How Amazon's AI-powered store might work its magic

Amazon Go is billed as a checkout-free store powered by AI and machine learning. Here's what might be going on behind the shelves

How Amazon's AI-powered store might work its magic

Amazon Go sounds like the ultimate retail experience: No checkout lines, no registers. Just walk in, grab what you want, and leave.

But what’s most eye-opening about the retail and cloud giant’s first foray into a brick-and-mortar presence is Amazon's claim that its “Just Walk Out” store is powered by “the same types of technologies used in self-driving cars: computer vision, sensor fusion, and deep learning.”

The details provided so far are sketchy, but based on what Amazon has revealed—and what’s already state-of-the-art for those technologies—it’s possible to make some educated guesses about what Amazon has put together and how others may benefit from it in time.

Attention shoppers

There is a promotional video for Amazon Go that depicts the process of shopping in the automated store, the first of which is set to open in Seattle in early 2017.

Customers enter the store and swipe through a turnstile using a smartphone app. Anything taken from the shelves is added to a virtual shopping cart. Put things back, Amazon states, and the items will be automatically removed from the cart. To make a purchase, just leave the store and one’s Amazon account is automatically debited for the items in the virtual shopping cart.

Amazon credits the way all this works to “computer vision, deep learning algorithms, and sensor fusion, much like you’d find in self-driving cars.” The first of those three provides the biggest hint as to how the system might work: Once users enter the store and log in, their presence could be captured on camera. Their general presence—the look of the person on camera, their gait, and so on—could be tracked as they move through the store by way of already-extant algorithms designed to recognize such things.

Likewise, when a shopper takes a product from the shelf it could be tracked using the same motion-detection and object-association algorithms. It seems easy enough to correlate the image of a product or its placement on the shelf with the item added to the cart, as object recognition algorithms are already commodity machine-learning components.

Machine learning isn’t the only secret sauce

What’s less clear is whether Amazon Go is powered entirely by the machine-learning/AI mix described above, or also uses more mundane technologies that have long been mentioned as ingredients for a checkout-less store.

For instance, it’s possible that each shelf is equipped with sensors to detect items being removed or added. Taking an item from a shelf could prompt the machine-learning back end to look for something resembling an act of removing an item, making it easier in the long run to detect such things by movement alone.

Another tech that is likely used is RFID technology, which is already widely used in retail to track goods through the supply chain. Smart self-checkout by way of RFID-tagged goods has been discussed for years, but it hasn’t taken off for consumer checkout for a variety of reasons, including the cost of tagging every item; how to deal with items that have to be weighed at checkout; the fact that RFID scans weren’t fast enough to catch all the items as they were walked out of the store; and so on.

Amazon’s reference to “sensor fusion” is the tipoff to possible RFID use. But instead of relying on RFID alone, Amazon may be using RFID tagging as one part of the checkout process—a way to provide more confirming information to the store’s AI rather than as the only source of data for what’s being bought.

A lot of what Amazon is claiming for Amazon Go is already possible for machine learning: recognizing people as they walk into a store, identifying the objects they take from the shelves, and tracking whether those objects make it out the door or not. Enough of that tech is out there, and in commodifiable forms (like open source) to boot.

The trick is to make all that work in concert, in real time, and with a high enough level of accuracy that it requires no human supervision. That’s what Amazon is likely to bring to this game—not the tech itself, but the way it all works together. But it isn’t likely to be long before others pick up the same pieces and follow suit.