Nvidia's AI-powered NPCs are getting better, but they still sound creepy

At the consumer electronics fair (CES) on Monday, Nvidia showed off the latest developments in its generative AI-powered NPCs, showcasing automated conversations between a player and computer-generated characters that could change the way games are made. Nvidia's Avatar Cloud Engine (ACE) technology combines speech-to-text recognition and text-to-speech responses with generative AI facial animation and automated character personas to spit out computer-generated character interactions.

Nvidia's special talk at CES 2024 Here, ACE senior product manager Seth Schneider demonstrated the technologies working together. According to Schneider, the demo interprets a player's speech and converts it into text. That text is then fed into a cloud-based large language model to generate a response from an NPC. The response text is entered into Omniverse Audio2Face, which lip-syncs the generated spoken audio, which is then displayed in the game.

The demo is an update of an earlier version of the technology shown at Computex in 2023, in which a character spoke to the futuristic ramen shop owner, Jin. Nvidia's new demo expands on this by letting Jin and another NPC, Nova, have AI-generated conversations that can be unique on each playthrough.

The CES 2024 demo also shows off new technology from another company called Convai that will allow AI-powered NPCs to do more than just talk. These NPCs can also interact with objects in their environment. In the newer demo, Jin is shown pulling out a bottle of liquor when asked to “bring out the good stuff” by Schneider. According to a benefit shared by Convai, environmentally aware AI NPCs could interact with a number of objects, including bowls, bottles, lights and other props in the scene.

Nvidia says a number of game developers are already using its ACE production services, including the Audio2Face facial animation generative AI and Riva automatic speech recognition. Schneider liked to mention “top digital avatar developers.” Genshin impact publisher Mihoyo, NetEase Games, Tencent and Ubisoft as some of the companies creating AI-powered NPCs for their products.

It's not yet clear which games will feature these types of AI-generated NPCs, but Nvidia and Convai boast that the technology will integrate “seamlessly” with game engines like Unreal Engine and Unity. It's also unclear whether the outcome in the real world will be good, or, like Jin and Nova's conversation, unnervingly creepy. Both characters sound robotic and strange in their respective deliveries, despite the clever output of almost convincing conversations.

One thing is almost guaranteed to come out of Nvidia's new demo: the increased suspicion that the bad NPC interactions we'll experience in future games will be created by AI rather than an actual human.