Google Research is showing off a new way to use AI to read handwriting, which could revolutionize the way machines convert what you put on paper into digital letters. The InkSight system transforms photos of handwritten words into digital text using AI without the need for intermediary devices.
The idea is to replace the sometimes fallible optical character recognition (OCR) with AI that can mimic how humans actually learn to read, specifically by rewriting existing text to learn what entire words look like and mean. For this, the researchers had to guide the AI in recognizing and imitating human handwriting.
“Digital note-taking is growing in popularity, providing a durable, editable and easily indexable way to store notes in the vectorized form known as digital ink. However, a significant gap remains between this method of note-taking and the traditional pen-and-write method of taking notes on paper, a practice still preferred by a large majority,” the researchers explain in their report. paper. “Our approach combines reading and writing priors, making it possible to train a model in the absence of large amounts of paired examples, which are difficult to obtain. To our knowledge, this is the first work to effectively represent handwritten text in random photos with various visual features and backgrounds.”
InkSight is more than just an alternative technology. It provides more accurate results in conditions that are not ideal. For example, if the photo was taken in low light, the text is partially obscured, or is on a confusing background when examined with OCR. The researchers found that people were able to read 87% of the text fragments created by InkSight. Two-thirds were so good that people couldn’t tell from the handwriting; you can see below what it looks like when InkSight is working.
Written by AI
If you like writing things by hand, InkSight has some potential benefits. Imagine writing by hand in a paper notebook and then showing the notes on your camera so they are instantly searchable and organized in the context of previous notes on physical pages. If you’re like me and have particularly messy handwriting, InkSight can help you turn your chicken scratch into typed text that’s still exactly what you scribble.
On a larger scale, this could be a crucial tool for deciphering and converting handwritten text through the ages into digital form. Even if the text is in a language without much digital presence, InkSight can help preserve handwriting and build training resources for those languages.
Google isn’t the only place developing AI tools for deciphering handwriting. Amazon’s new Kindle Scribe, for example, improves the e-reader’s ability to convert handwritten notes into readable text. There’s also Goodnotes, a digital note-taking app that can read handwriting, and recently debuted handwriting editing tools that use Goodnotes Smart Ink technology to convert handwriting into typed text. The added tools let you edit handwritten notes as if they were typed, including aligning notes, copying and pasting, and reflowing the text to make it more logical.