Google is quietly building an omnipresent AI that will be linked to all your devices and apps – and ‘knows everything about your life’

Confidential documents presented at Google's recent internal summit outline the tech giant's plan to create an artificial intelligence (AI) designed to become a “life story teller” for its users.

But to do this, AI will require unprecedented access to every user's personal data.

It's not clear where this experimental AI, currently dubbed “Project Ellmann,” will fit among Google's apps and services, but the team behind it works at Google Photos — and their presentation suggested a dedicated AI chatbot.

“We can't answer tough questions or tell good stories without a comprehensive view of your life,” read one part of a Google product manager's presentation.

Taking off from ChatGPT competitor Gemini, Project Ellmann will use “large language models” (LLMs) to collect personal information from context that is said to include biographies of users and their loved ones, as well as stock photo “moments.”

But the new developments may worry those angry about Google's secret collection of millions of individuals' sensitive medical records, codenamed Project Nightingale in 2019 — or anyone eagerly collecting digital privacy tips.

“We go through your photos, looking at their tags and locations to pinpoint a meaningful moment,” according to another presentation slide, obtained by CNBC.

The slide continued: “When we step back and understand your entire life, your overarching story becomes clear.”

In short, the project hopes to create a personalized ChatGPT-style chatbot tailored to your interests and life history — as drawn from your internet search history, phone camera rolls, and other data, and will certainly include Google Wallet purchases and more.

According to a presentation by the project manager, the Google Photos team devoted months to confirming the MBA's ability to detect patterns — after ingesting search results, user photos, and other data to “answer previously impossible questions” about a person's life.

The ambition of the team's plan to build intimate, in-depth portraits of their users is in the project's name: a nod to literary critic and biographer Richard Ellman, who won the National Book Award for his biography of novelist James Joyce.

With each slide, the Google team hopes that Ellmann will eventually be able to describe a user's personal photos in more detail rather than “just pixels with labels and metadata.”

In one example, they discussed how Ellmann LLM could scan a user's photos, compiling them into, say, “memories” or “moments” from the birth of that user’s child, or a collage of photos taken at a high school class reunion. .

“It's been exactly 10 years since he graduated,” one of the photo analysis's presentation slides explained, “and it was full of faces that hadn't been seen in 10 years, so it was probably a reunion.”

As an example of the types of “previously impossible questions” that Elman’s project could help users answer, the presentation presented hypothetical requests for users to find out when their siblings had last visited them or which city they should move to.

Elman, based on his presentation, can answer both.

Elman also appears to be able to predict and recommend purchases, and even provided a summary of the user's eating habits.

“You seem to enjoy Italian food,” the Ellmann LLM project noted in one slide. “There are many pictures of pasta dishes, as well as a picture of pizza.”

Since the presentation came from a manager at Google Photos, CNBC speculated that the company may be planning to house the new AI product within the Google Photos app.

Google Photos has more than 1 billion users and stores 4 trillion photos and videos, according to Google Google Cloud blog post.

In a more direct reference to who the Google team sees as its competition, the team summed up “Ellmann Chat” to its colleagues by asking them to “imagine opening ChatGPT but it already knows everything about your life.”

Google feared the worst when rival OpenAI unleashed its own AI-powered chatbot, ChatGPT, a year ago.

A response to its AI-powered chatbot, Gemini — which is designed to power Google's Bard chatbot — outperforms ChatGPT's GPT-4 on the majority of cognitive tests, Google said.

Gemini can tell users when an omelet is cooked, suggest the best kite design, or help a soccer player improve his skills, judging by the latest and most popular trivia. Research paper.

But she is particularly skilled at mathematics and physics, fueling hopes that she might lead to scientific discoveries that improve people's lives.

Google claims that Gemini outperforms GPT-4 in 30 out of 32 performance metrics — including text generation, question answering, reasoning, image understanding, and “logical reasoning.”

In its Gemini paper, Google outlined the different capabilities of AI when it comes to images, including what it can knit from different colored threads.

While Gemini will only work in English for now, the company said the technology will have no problem diversifying into other languages.

In addition to the Ellmann and Bard projects, Google plans to publish Gemini within its main search engine as well.

The company appears to have been tripped up by a leaked presentation to CNBC, issuing clarifications on privacy issues via a spokesperson.

“Google Photos has always used AI to help people search for their photos and videos, and we're excited about the potential of LLM to open up more meaningful experiences,” the spokesperson said.

“This was early internal exploration,” the spokesperson stressed.

“If we decide to roll out new features, we will take the time to make sure they are useful for people, and designed to protect users' privacy and safety as our top priority.”