Apple has introduced a new feature in the camera system that automatically recognizes and transcribes text on your photos, from a phone number on a business card to a whiteboard full of notes. Direct text, as the function is called, does not need any questions or special work from the user – just tap the icon and you are ready.
Announced by Craig Federighi on the virtual stage of WWDC, Live Text comes to iPhones with iOS 15. He demonstrated it with a couple of photos, one from a blackboard after a meeting, and a couple of snapshots that included restaurant signs in the background.
By pressing the Live Text button at the bottom right, the text was detected a little underline, and then the swipe allowed it to be selected and copied. As for the board, it collected several sentences with notes, including bullet points, and with one of the restaurant characters grabbed the phone number, which could be called or stored.
Some types of text strings can also be recognized: a tracking code will be viewed as such, and a link to the tracking URL will be made immediately available. Translation can also be done quickly, to or from any language supported by Apple̵
The feature is reminiscent of many found in Google’s long-developed Lens app, and the Pixel 4 added a more robust scanning function in 2019. The difference is that the text is taken more or less passively on each image taken by an iPhone running the new system. – you do not need to enter scanner mode or start your own app.
This is a nice thing for everyone to have, but it can be especially helpful for people with visual impairments. A snapshot or two enables any text, otherwise difficult to read, to be dictated or saved.
The process takes place entirely on the phone, so do not worry about this information being sent to a data center somewhere. It also means it’s pretty fast, but until we test it ourselves we can not tell if it’s instantaneous or, like any other machine learning feature, something that happens within the next few seconds or minutes after you take a shot. However, your back directory of photos becomes live text in the phone’s inactive moments.