I’m excited to unveil the latest advancement in the technology behind Just Walk Out, our checkout-free service for retailers. A new multi-modal foundation model further increases its accuracy by using the same transformer-based machine learning models underlying many generative AI applications, and applies them to physical stores. We accomplish this by analyzing data from cameras and sensors throughout the store simultaneously, instead of looking at which items shoppers pick up and put back in a linear sequence. For retailers, the new AI system makes Just Walk Out faster, easier to deploy, and more efficient. For shoppers, this means worry-free shopping at even more third-party checkout-free stores worldwide.
A huge leap forward for checkout-free shopping
The new Just Walk Out multi-modal AI system helps figure out "who took what" in stores with higher accuracy.Just Walk Out uses cameras, weight sensors, and a combination of advanced AI technologies to enable shoppers in physical stores to buy food, beverages, merchandise, and more, without having to wait in a checkout line, or stop at a cashier. Just Walk Out technology, which launched in 2018, was built using generative AI and leading-edge machine learning available at the time to figure out “who took what.” Previously, the AI system analyzed shopper behavior sequentially—their movement and location in the store, what they picked up, and the quantity of each item—each action processed one after another. However, in unusual or novel shopping scenarios (such as if a camera view was obscured due to bad lighting or a nearby shopper), the sequential approach could take time to determine purchases with confidence, and sometimes required manual retraining of the model.
The new Just Walk Out multi-modal foundation model for physical stores is a significant advancement in the evolution of checkout-free shopping. It increases the accuracy of Just Walk Out technology even in complex shopping scenarios with variables such as camera obstructions, lighting conditions, and the behavior of other shoppers, while allowing us to simplify the system.
Just Walk Out is getting smarter and more accurate
The new Just Walk Out AI system is able to achieve higher levels of accuracy by analyzing all sensor data simultaneously, rather than sequentially. It looks at multiple inputs—cameras, weight sensors, and other data—and prioritizes what’s most important to accurately determine the variety and quantity of items selected. It also uses continuous self-learning and transformer technology, a type of neural network architecture that transforms inputs (sensor data, in the case of Just Walk Out) into outputs (receipts for checkout-free shopping).
For example, a shopper might pick up and put down multiple varieties of yogurt, in different combinations, and as they are doing so, another customer might reach for the same item, or the freezer door could fog up, obscuring the cameras’ view. In complex situations like these, the new model can quickly and accurately determine the actual items taken by each shopper. This is thanks to its ability to simultaneously process inputs from various sources (including, in this scenario, weight sensors on the fridge shelves), continuously learn from these inputs, and decide which are most important in order to accurately sort out who took what. This minimizes receipt delays and increases the ease of deployment of the technology for retailers.
The new Just Walk Out AI model reduces the rare need for model retraining in unfamiliar shopping scenarios, as the self-learning system continues to teach itself. To support Just Walk Out’s continuous learning, it is trained on a 3D map of the store to understand placement of fixtures (such as shelves, fridges, freezers, coffee machines, and more) and products, as well as an image catalog of the store’s merchandise so it can visually recognize every item accurately. This enables the new AI system to recognize shopper behavior, even if the store is remerchandised and items are rearranged or misplaced.
Delivering a seamless checkout-free shopping experience
The improvements to our AI system are so seamless that you will continue to enjoy the same contactless checkout-free shopping experience you’ve come to expect at Just Walk Out stores, all while protecting your privacy. Upon entering a Just Walk Out store you present a form of payment at the entry gate. The system immediately associates you with that payment method and begins mapping your journey around the store, adding or subtracting items from your virtual cart as you pick them up or put them down.
Just Walk Out does not collect any biometric information—the system only tracks how you interact with the products and fixtures (such as shelves or fridges), correctly identifying the products and quantities you leave the store with.
The new Just Walk Out multi-modal foundation model represents the latest innovation in checkout-free shopping. Just Walk Out is currently available in over 170 third-party locations at airports, stadiums, universities, hospitals, and more in the U.S., UK, Australia, and Canada. We will launch more Just Walk Out stores in 2024 than any year prior, more than doubling the number of third-party stores with the technology this year. As we scale, the system will continue to learn from everyday shopping scenarios and raise the bar for accuracy and convenience, delivering the benefits of AI to retailers and customers around the world.
