Google is rolling out new AI features for Android phones, plus some that are only for Pixel
- Expressive Captions add feelings and (gasp) emotions to Android’s Live Captions
- Google Lookout gets new Gemini improvements
- Gemini on Pixel phones now remembers more information about you and your interests
Google has released a number of new AI features for most Android phones, plus a few that only work on the Pixel range of devices.
Some of the new features work in Google Gemini, Google’s AI app, but most work across the Android operating system, meaning there will be some big changes for Android users. Let’s break it down.
Expressive captions
Firstly, there are Expressive Captions, a way to bring more emotion into the rather boring and lifeless world of subtitles and captions. Google introduced Live Captions on Android in 2019, so if something is playing audio on your phone, it will also have captions. They work in much the same way as automatic subtitles on YouTube videos: your phone generates them live, based on what it interprets what the person is saying.
Live Captions, however, remain rather lifeless. With Expressive Captions, Google uses the power of AI to help you get more of the context of what’s happening. You’ll see bracketed comments such as (whispers) and (cheers and applause) appear between the captions, giving you a better understanding of the feelings and emotions being conveyed.
Intense vocalizations are written in CAPITAL LETTERS, while vocalizations such as (sigh), (grunt), and (pant) also appear. Ambient background noises are also tagged. Because Live Captions are part of the operating system, you now get Expressive Captions on most things you watch, so keep that in mind.
Twins in lookout
Google Lookout is an assisted vision app available on Android devices through the Play Store. It uses AI to help people with low vision or vision loss explore the world around them using their phone’s camera. You can use it to read text, scan food labels, identify currency, and even scan your surroundings to identify things like doors and windows and find out how far away they are.
Now Gemini AI is part of the Lookout app, which means it has much richer, more helpful descriptions. When you take a photo in the app, you now hear the description read aloud in a natural-sounding voice, and you can ask Loopout follow-up questions.
New Gemini Extensions
The more extensions Google Gemini has, the more apps it can add its AI magic to. Now Gemini has a new Spotify extension, which means you can play your favorite songs or discover playlists from within Gemini.
Gemini now also connects to your phone and messaging app, meaning you can call contacts and send messages from Gemini. Google says you’ll “soon” be able to use Gemini to control your smart home devices. Finally, Gemini now also connects to Maps, giving you more information about places.
More Android features
Also in the December update you’ll find new pizza-themed sticker combinations that you can share with friends via Emoji Kitchen in Gboard, Google’s keyboard app. You can also use QR codes in Quick Share, making it easy to share photos, videos, and documents without having to add people as contacts or authenticate devices. Now you just generate a QR code for people to scan. Documents you scan with your phone are now also improved, with better contrast and white balance.
Pixel-only functions
If you have a Google Pixel phone, you’ll get some extra features in the December Android update. There are lots of minor updates to things like the camera and security settings, but the most important feature is Gemini Saved Info. This means you can ask Gemini to remember more about you, what you’re interested in, and what you like, so it can tailor its answers to more of what you’re interested in.
Pixel users will also get some updates to the exclusive Pixel app Pixel Screenshots. Pixel Screenshots uses Gemini to analyze and retrieve the information in your screenshots. Now you can use Pixel Screenshots to save things with just one tap when you use Circle to Search.