Apple is currently working on improving Siri’s onscreen awareness, which will allow it to communicate with your screen


  • Apple is developing an ‘onscreen awareness’ feature that will allow Siri to understand and interact with the content currently on your screen
  • Apple will also provide APIs to developers for integrating screen awareness into their third-party apps and is currently testing ChatGPT integration, which will allow Siri to answer questions based on screenshots
  • Although not available in the iOS 18.2 beta, on-screen awareness could arrive in time for iOS 18.4 in 2025

Among digital assistants, Siri has done quite well (certainly compared to Cortana, rival Microsoft’s ill-fated assistant), and now Apple is working to make Siri even smarter by giving it a better idea of ​​what you’re looking at. your screen and call it ‘onscreen awareness’.

Apple detailed the development of this feature on an Apple Developer Documentation page, which also notes that it will be included in several upcoming beta versions of the Apple operating system (OS) for testing.