Microsoft Copilot Vision is almost ready to watch what you do
Microsoft’s Copilot AI assistant will start seeing what you’re looking at online, according to a tease post on the Copilot Labs is experimenting, but giving the AI the ability to see everything on your screen is now becoming an option for everyone.
After about a month of gathering feedback, Copilot Vision is ready for a general rollout, including integration into the Microsoft Edge browser. Once it is available, all you have to do is click on the screen-like icon in the browser to activate the tool. Copilot Vision essentially turns an AI eye on the same things you look at, from websites to documents. It can even read text, whether typed or handwritten, and displayed on the screen. The AI observes and reacts to what’s on the screen to answer any questions you have and provide ideas about what to do next.
Rather than researching a future trip from an options website, Copilot Vision just provides details and recommendations and answers your questions without the need for another search. If you see a recipe online but want to change it, Copilot Vision can tell you some substitution options or provide cooking tips without having to leave the page or open the chatbot.
If only your browser could see what I see… oh wait, Copilot Vision will be able to do that very soon 👀 pic.twitter.com/EtLGyMGPq3November 1, 2024
Vision and images
This approach makes Copilot Vision different from typical chatbots because you don’t have to explain what’s happening that caused you to ask for help or advice. It can see the same content as you. It’s part of Microsoft’s effort to make Copilot a useful tool that people can turn to when they need advice.
If you’re concerned about privacy, Microsoft has said that Copilot Vision data will not be transferred from previous sessions. Microsoft does not store or use your browsing data after the session is over. It also won’t work on every website, at least initially. Copilot Vision only looks at some popular websites that meet Microsoft security levels. With ChatGPT and Claude both entering the Copilot space with desktop apps, these types of features could be key for Microsoft to compete in the broader AI assistant arena.