Niantic, the creator of pokemon gorecently invited me to its London offices to see its new mobile pet sim peridot. I saw a presentation, tried out the game – which will be released on May 9 for iOS and Android – and interacted with some of the developers, including the computer scientist who heads the small research team developing the machine learning technology behind peridot‘s potentially groundbreaking new augmented reality technology.
But no demo can better illustrate its potential peridot than half an hour I spent with the game a few days later, when I got a test version to try at home. I have two young children (ages 4 and 6), who were instantly charmed when they saw the adorable creature of Niantic’s invention called a Peridot hatch on my phone screen and run around my house. We petted it, fed it, threw a ball at it that seemed to bounce off the walls in my living room, and watched it run around and behind the furniture, but a little erratically.
Image: Niantic
When I checked baby Dot’s desires, I saw that he wanted to eat a dandelion, which must be taken from lawns. Our backyard is paved, but there’s grass across the street, it’s a pleasant spring evening, and there’s still some time to kill before bathing—let’s go! We left and the kids screamed when they saw the Peridot run out of the driveway ahead of us. They were impressed that the game could tell the difference between grass, foliage, and paving stones, allowing the creature to forage on different items from each.
I noticed my Peridot wanted to forage from a Habitat, which this game calls the map-based local points of interest shared by all of Niantic’s AR games (in pokemon go, they are called PokéStops). I saw this was just a short walk away and I hadn’t visited it before (I’m not a pokemon go player) — why not? We went there. It turned out to be a Victorian stink pipe (I live in London, you know?), which looks like a lamppost without a lamp and is in fact some kind of giant straw designed to expel noxious gases from the sewers below, far above the heads of the subjects of the good queen. I had never heard of these things or noticed them before. We returned home after having a good time, getting some unplanned exercise, and learning a little about the local history (and leveling up our pet). Niantic couldn’t have written it better.
peridot is a pretty typical pet sim, in the style of something like Nintendogs, crossed with Niantic’s extensive map data source and a new generation of AR technology. You interact with your pet to keep him happy and earn growth points, helping him progress from baby to teen to adult. In the adult stage, the creature will want to be “released” into a habitat where it can breed with other Peridots (the animals are sexless) and father a new baby for you to care for. (Producer Ziah Fogel says the plan was for players to say goodbye to their adult Peridots for good at this point, but the testers got too upset, so you can hold them.) Niantic has built enough variables into the creatures’ DNA that each one is genetically unique and each pairing will create a new equally unique baby.
Image: Niantic
You can imagine the location-based features Niantic has built into this game genre; take your pet for walks, feed from different environments, participate in breeding with other local players as a sort of social endgame. You can also imagine the gentle but persistent trickle of objectives, progression, currencies and rewards built into a free-to-play live service game like this, as well as the customization features (yes, your pet can wear hats).
But Niantic has another mission with it peridot, which is to push the boundaries of AR and change the way people think about it. The idea is that your pet should, on some level, be able to understand and respond to the real world it (or rather your phone’s camera) is looking at. This is the work carried out by Niantic’s R&D team, housed over six floors of a narrow office building crammed into London’s Covent Garden area, and led by Niantic’s lead researcher Gabe Bostow.
The AI investigation begins by accurately mapping 3D space and finding paths around real-world objects, as well as clearing obstacles, meaning your pet will realistically disappear behind chairs and logs. (Bostow seems frustrated that the design team requested that a shadow showing the Peridot’s position behind the object be put back into the game, lest players get too stressed out about losing their charges.) It continues using “semantic segmentation,” as Bostow puts it, of your phone’s video feed of the environment, using machine learning to train the code to tell the difference between grass and water, to recognize a TV screen so your Peridot can chide you about screen time, and to identify classes of objects such as pets, people and plates of food. The neural networks required for this were not possible on mobile phones five years ago, says Niantic. peridot certainly puts a strain on your phone’s CPU and GPU, if the effect on battery life is anything to go by.
Image: Niantic
peridot started life as a tech demo, and while the pet sim genre is a natural fit for Niantic’s endeavors, you can tell. In some ways, it still feels more like the skunkworks project of a progress-hungry tech company than a labor of love from a game developer. Bostow’s team is sharing its findings with the academic community, while the code will be available for third parties to license through Niantic’s Lightship suite of development tools. You can imagine the benefits the technology could have for robotics or other assistive technologies; Bostow likes to envision a future where he can hold up his phone to see the location of pipes behind walls and under floors when he’s busy improving his home. There are probably more sinister uses for this combination of machine learning environments with Niantic’s industrial scale data farming, but don’t think about that – look how cute these little critters are!
In practice, the technology is far from perfect: it can’t recognize food as reliably, can see water outside better than inside, and has trouble with windows. But there’s an undeniable magic to seeing your Peridot running around your house, jumping on tables and noticing your cat. It feels a bit alive, and it will only feel more alive as the game releases and players start training Niantic’s neural networks en masse. As Fogel says, a game about growing tiny creatures and introducing them to the world is a good thematic match for technology that will itself improve understanding of the world and the realism of its behavior over time.
I think it’s impressive. But it’s not my response that matters, or tell you if peridot becomes a success. My kids loved it within seconds and my 4 year old was wandering around the house yelling for “pennydot” within minutes. Plus, we’re all familiar with Victorian stinkpipes now.