The massive amount of statistical calculations needed to do this are happening on the iPad itself, rather than at a data center. “It’s gotta be happening in real time, right now, on the device that you’re holding,” Federighi says. “Which means that the computational power of the device has to be such that it can do that level of processing locally.” Original: Popular Mechanics • The iPad’s Handwriting Recognition Shows How Apple Does Machine Learning via 9 to 5 Mac • Craig Federighi explains how the Apple Pencil Scribble feature in iPadOS 14 was developed A computer’s ability to read handwriting, then translate it into letters and numbers it can understand, has been a challenge going back decades. …the iPad can understand your scrawl and, with Scribble, convert it to typed text. It works like most machine learning—examples inform rules that help predict and interpret a totally new request—but taps into a smarter data set and greater computing power to do what had stumped generations of previous machines. While Alexa and Siri rely on a connection to faraway data centers to handle their processing, the iPad needs to be able to do all that work on the device itself to keep up with handwriting (and drawing—machine learning also helps the Notes app straighten out an imperfect doodle of a polygon, for example). That takes way more effort than you’d think. “When it comes to understanding [handwriting] strokes, we do data-gathering. We find people all over the world, and have them write things,” says Craig Federighi, senior vice president of software engineering at Apple. “We give them a Pencil, and we have them write fast, we have them write slow, write at a tilt. All of this variation.” That methodology is distinct from the comparatively simple approach of scanning and analyzing existing handwriting. Federighi says that for Apple’s tech, static examples weren’t enough. They needed to see the strokes that formed each letter. “If you understand the strokes and how the strokes went down, that can be used to disambiguate what was being written.” USE CASES You’re handwriting notes on your iPad ($329 and up) with the Pencil during a meeting, and you want to see a map of Zanzibar. You can now swipe to the Maps app and write “Zanzibar” into the search field, rather than pecking at the screen’s keyboard. Or, you want to email a few lines of those handwritten notes. You select that section, copy, then paste into an email, where it shows up as if you typed them. Or you write down a phone number, and you can tap to call it.