ChatGPT adds eyes to its voice with a new screen and video sharing feature
- The sixth day of 12 Days of OpenAI brought visual skills to ChatGPT Advanced Voice Mode
- You can stream video from your camera or share your screen with the AI
- For now, only ChatGPT Plus and Pro subscribers have access
ChatGPT took a fresh look at day six of the 12 Days of OpenAI thanks to a new visual capability related to the Advanced Voice Mode that lets you share your screen and stream video live to the AI chatbot. Instead of uploading photos and screenshots to ask ChatGPT questions, you can now directly show what you’re looking at and ask for advice.
The feature is a bit like video chatting with a friend, even though he doesn’t share his own image. But you can hear the AI’s voice, making the conversation hands-free. If you want to try it out, you can tap the voice icon in the ChatGPT app and then tap the video icon to stream video from your camera. To share your screen, simply tap the three-dot menu and select ‘Share Screen’.
Imagine struggling to put together a new IKEA bookshelf (because who hasn’t?). Instead of staring at confusing instructions, you can point your camera at the half-built mess and ask ChatGPT, “What did I do wrong here?” The AI can view the parts and provide step-by-step guidance.
If you need to adjust some settings on your phone or computer and aren’t sure how to do it, you can share your screen with ChatGPT so the AI can guide you through the menus and buttons to get things in order. No more digging through endless tech forums or asking your friend who is good with computers.
The feature could even make ChatGPT a better sous chef. If your recipe says, “whisk until it thickens,” and you’re not sure if you whipped enough, just point the camera at your bowl and ask ChatGPT for confirmation or for a suggestion you might want to order.
All-seeing AI eyes
OpenAI’s CPO Kevin Weil and his team demonstrated how ChatGPT can help make pour-over coffee by pointing the camera at their brew setup to show off the new feature during the debut. The AI understood the coffee making equipment and guided them through the steps like a virtual barista.
The feature encourages treating ChatGPT more like a person than a computer, even more than just the voice interface. Being able to ‘see’ makes the AI seem more present in the real world and less like a chatbot in a vacuum.
OpenAI did acknowledge that sharing what your camera sees may make some wary of using the feature. It’s not on automatically, you have to activate it every time you use it so that a video recording isn’t accidentally made.
The new feature is now only available to ChatGPT Plus and Pro users. Enterprise and Education tier subscribers will get access next month, but OpenAI hasn’t said if or when the free tier will get access.
That makes sense because it will likely require a lot of computing power, and OpenAI doesn’t want ChatGPT to go down again after what happened on Wednesday.