- Apple is developing an ‘onscreen awareness’ feature that will allow Siri to understand and interact with the content currently on your screen
- Apple will also provide APIs to developers for integrating screen awareness into their third-party apps and is currently testing ChatGPT integration, which will allow Siri to answer questions based on screenshots
- Although not available in the iOS 18.2 beta, on-screen awareness could arrive in time for iOS 18.4 in 2025
Among digital assistants, Siri has done quite well (certainly compared to Cortana, rival Microsoft’s ill-fated assistant), and now Apple is working to make Siri even smarter by giving it a better idea of what you’re looking at. your screen and call it ‘onscreen awareness’.
Apple detailed the development of this feature on an Apple Developer Documentation page, which also notes that it will be included in several upcoming beta versions of the Apple operating system (OS) for testing.
Apple originally showed on-screen intelligence in June 2024 and this is a pretty solid indication that it’s still in development.
The core idea of screen awareness is quite simple: if you’re looking at items on your screen, say a document or a browser with a page open, and you have a question about something you’re looking at, you can ask Siri ( equipped with Apple Intelligence). Siri should then be able to respond to your query with relevant information or perform an action it is asked to do, such as sending content to a supported third-party app.
If it works as intended (and that’s a big ‘if’), it will result in a smarter Siri, where you won’t have to describe what you want it to do as extensively as you probably should at this point. For example, you can have a document open and ask for a summary without having to enter the contents of the document yourself.
Apple’s plans for Siri’s near future
MacRumors reports this that Apple has provided APIs for developers, allowing them to make the content of their apps available to Siri and Apple Intelligence. The idea is to provide developers with this API many months before release so that you can use the on-screen visibility with third-party apps when it’s officially rolled out.
Currently, we know that Apple is testing out ChatGPT in the latest iOS 18.2 beta (among other Apple operating systems), and the ChatGPT integration combined with Siri will allow you to ask questions about items (such as images, PDFs, and videos) on the screen. Siri then takes a screenshot and passes it on to ChatGPT to answer your question. This means that for now this functionality is limited to screenshots of your screen.
However, the on-screen consciousness is a bit different, as noted by MacRumors, because the idea is to integrate the on-screen consciousness in a more direct way.
Siri’s on-screen awareness should be some sort of ability to instantly analyze, interpret, and interact with the content on your screen. If someone sends you their number and you want to save it, you can tell Siri to create a new contact without having to add any additional instructions or require Siri to perform many intermediate steps.
Will Siri survive and even thrive in the AI era?
Apparently, on-screen awareness isn’t really in the iOS 18.2 developer beta yet, and MacRumors speculates that it’s one of many Siri features we won’t see for a while, but this news is still promising. One prediction is known on-screen and may be included in iOS 18.4, which is expected to release in the first half of 2025.
If successful, Siri could become a more helpful digital assistant, and with Apple’s sense of design, it could become the digital assistant of choice for many. This development reminds me of what Microsoft is aiming for with Copilot. That hasn’t been very well received based on what we’ve seen so far, leaving the target open for Apple.