Deep inside a Seattle building, behind a plain set of doors, lies what looks like an everyday convenience store. Except this one is different.
It’s a glimpse directly into the future, a place where engineers and researchers are hard at work solving one of retail's most complex challenges: how to create a seamless and highly accurate checkout-free shopping experience.
While the lab looks exactly like a store that uses Just Walk Out technology, it’s a mock store environment that serves as a testing ground for the upgrades Amazon makes to the system.
At Amazon, we remain steadfast in our commitment to Just Walk Out technology, and continue to invest in research and drive innovation as we believe frictionless, checkout-free shopping will be the future of physical retail. We offer Just Walk Out technology as a service to third-party retailers, and it is currently available in over 200 third-party locations at airports, stadiums, universities, hospitals, and more in the U.S., UK, Australia, and Canada. We will launch more Just Walk Out stores in 2024 than any year prior.
While the concept of Just Walk Out is simple—customers enter a store by tapping their credit card or mobile wallet, grab what they need, and simply leave without standing in a checkout line to pay, while the purchase is automatically charged to their payment method—the system behind it is anything but.
To solve the problem of “who took what,” Just Walk Out uses cameras, weight sensors, and a combination of advanced AI technologies to accurately determine the variety and quantity of items a customer selects, and ultimately leaves the store with.
To figure out the “who” part of the “who took what” equation, Just Walk Out associates a shopper with their payment method at the entry gate. Just Walk Out does not use biometrics-based technologies like facial recognition to identify shoppers—the system only tracks how their hand interacts with the products and fixtures (such as shelves or fridges), and correctly identifies the products and quantities they leave the store with. In fact, if a shopper comes to the store again with another credit card, the system would have no idea it was the same person.
“Figuring out the ‘what’ and how many items were taken by a shopper is a challenging AI problem to solve,” says Chris Broaddus, senior manager of Applied Science, Amazon Web Services (AWS). “For example, the system needs to accurately recognize how a shopper’s hands are interacting with the shelves—are they picking up a product, returning it, or just rummaging within the shelf?"
To address these complexities, Just Walk Out’s new AI system looks at multiple inputs—cameras, weight sensors, and other data simultaneously—and prioritizes what’s most important to accurately determine the variety and quantity of items selected. It also uses continuous self-learning and transformer technology, a type of neural network architecture that turns inputs (sensor data, in the case of Just Walk Out) into tokens to generate outputs (receipts for checkout-free shopping).
The lab allows researchers to quickly simulate a variety of other complex scenarios that might occur in a real store and adjust the system as needed. Multiple people grabbing products off the same shelf simultaneously? Check. A shopper changing their mind and putting an item back? No problem.
Customer behavior can also differ based on location and store type. For example, sports stadium stores can get densely packed with fans wearing the same team colors or jerseys during games. At airports, you’ll find people walking into stores with big bags that can block the cameras’ view of shelves.
Just Walk Out engineers and researchers test these scenarios in the lab first by mapping a store’s layout, usually with Light Detection and Ranging (LiDAR) technology, which uses laser light to create detailed 3D maps of a space. This helps researchers optimize the number of cameras needed, thereby driving down hardware costs, and their placement to ensure comprehensive coverage of the shopping area.
Along with cameras, weight sensors are another crucial component of the Just Walk Out system. They provide data inputs that are particularly helpful for products that are small (like candy bars and small packs of chewing gum, typically weighing just a few ounces) and potentially hard to “see”. Researchers use weights to validate the accuracy of weight sensor installations on these shelves.
The Just Walk Out research team also tested the new and advanced multi-modal AI model at this lab. Previously, the AI system analyzed shopper behavior sequentially, processing their movement and location in the store, then what they picked up, and then the quantity of each item. The new Just Walk Out AI model is a significant advancement in the evolution of checkout-free shopping. It uses the same transformer-based machine learning models underlying many generative AI applications, and applies them to physical stores. It analyzes all sensor data simultaneously, and supports even complex shopping scenarios with variables such as camera obstructions, lighting conditions, and the behavior of other shoppers, while allowing the team to simplify the system.
“The new multi-modal foundation model further enhances the Just Walk Out system’s capabilities by generalizing more effectively to new store formats, products, and customer behaviors, which is crucial for scaling up Just Walk Out technology,” says Broaddus.
For retailers, the new AI model makes Just Walk Out faster, easier to deploy, and more efficient. For shoppers, the new model means faster receipts and worry-free shopping at even more third-party checkout-free stores worldwide. As Just Walk Out technology scales, we will continue to invest in the future of checkout-free shopping by driving research in areas like cutting-edge AI, and delivering innovations that meet the needs of retailers across verticals.
