Amazon’s highly anticipated smartphone, the Fire Phone, comes with a new technology that combines image, text, video and audio recognition to make it easier than ever for users to identify (and hopefully purchase) some 70 million products in Amazon’s database. The Firefly technology can also scan QR codes, bar codes, and contact information. In terms of entertainment, the technology recognizes 245,000 movies and TV episodes, 160 live TV channels, and 35 million songs.
Firefly works by matching the captured image, video, or audio with products in its vast database. Users can scan a movie to learn more about the actors in it or even to get a link to start downloading it straight to the Fire Phone. Similarly, music recognition will give users the opportunity to download the song or learn more about the artist.
Firefly can also recognize some 70 million products, such as household items, food packages, books, DVDs and CDs. This capability, though, is still limited to mostly flat objects.
“While a smartphone may be able to recognize somewhat flat items like books, it’s still very difficult to discern objects like a purse or a stuffed animal,” reports MIT Technology Review. Image factors, such as angles, shadows, and lighting, make it particularly difficult to match complicated 3D objects to Amazon’s database.
Amazon is also offering app developers the chance to use Firefly in their own apps. StubHub and iHeartRadio created apps that could find concert tickets or make a new radio station based on an artist recognized by the Firefly technology. Also, MyFitnessPal, a health app, can display nutrition data of Firefly-detected food items. App developers can integrate their own image-recognition technology and databases of known objects with Firefly’s technology in their own apps.
The future of Firefly may have less to do with online shopping and more about learning about surroundings. Later this year, Amazon will also add artwork recognition and foreign language translation to Firefly. A wine label recognition powered by Vivino is also in the works.