Four key physical retail technology takeaways from re:MARS 2022

Since we use computer vision algorithms across all our physical retail technologies, pushing the state of the art forward in this space has always been important to us. In the case of Just Walk Out technology, which allows shoppers to save time by skipping the checkout line in many Amazon stores, select Whole Foods Market stores, and several third-party retailer stores, we’ve made ongoing inventions at all layers, including sensors, optics, and machine vision algorithms. As a result, we’ve reduced the number of cameras required in Just Walk Out technology-enabled stores to make them more cost-effective, smaller, and capable of running deep networks locally. Our sensors and algorithms have evolved to detect a broad range of products and differences in shopping behavior in full-sized grocery stores, all while ensuring an effortless customer experience. We also increased the diversity of environments our algorithms have to account for as we deploy Just Walk Out technology to third-party retailers.

Similarly, when our team built the Amazon Dash Cart, a smart shopping cart that helps customers skip the checkout line in many of our U.S. Amazon Fresh stores, they developed a robust set of computer vision and sensor fusion algorithms to detect items while in motion, including accurately capturing weight and quantity. The machine vision algorithms also have strict latency budgets, as we keep track of a customer’s receipt in real time.

At Amazon Style, our physical apparel store that offers a personalized, convenient shopping experience for customers to find styles they love at great prices, we built new technologies and algorithms to make this experience effortless for customers. Our algorithms use the information a customer provides—whether that’s a detail they entered into our Style Survey or the items they scanned while shopping on the floor of the store—to build a diverse set of recommended items, balancing similarity to their current choices with a diverse set of options. The system also generates complementary selections, such as a shirt to match a pair of jeans to create a complete outfit. We went to great lengths to keep the fun in shopping while elevating the experience through machine learning algorithms.

When my team set out to reimagine the in-store shopping experience for customers, one challenge we faced was getting diverse training data for our AI models to ensure high accuracy. To address this challenge, our research teams built millions of sets of synthetic data—machine-generated photorealistic data—to help build and perfect our algorithms and provide a seamless customer experience. Combining real-world and synthetic data allowed us to build truly robust and smart algorithms that were ready to take on all types of scenarios in stores.

In the case of Just Walk Out technology, we built synthetic datasets to mimic many variations of realistic shopping scenarios. For example, we used synthetic data to create variations in lighting conditions to account for sunlight differences in stores, and we used it to create a crowd of synthetic characters to ensure that we could handle many customers in the store at one time. We also used synthetic data to generate virtual models of larger grocery stores for testing before we scaled to that size, so we knew when we launched Just Walk Out technology to the public, we were delivering a highly accurate experience for shoppers.

For Amazon One—a fast, convenient, contactless way for people to use their palm to make daily activities like paying at a store or entering a location more effortless—we needed data to train and test our AI algorithms across demographics, age groups, temperatures, and even variations like calluses and wrinkles that are unique to one’s palm, to enable the service to correctly determine whose palm was hovering over the device. As we started to build Amazon One, we soon realized the limited availability of public datasets consisting of palm and vein images to help train our algorithms. So, our research scientists further advanced already state of the art technologies to build huge volumes of diverse, realistic synthetic palm and vein images to train our AI models and prepare the service for a wide variety of users.

All our technology services are designed from the ground up with security top of mind and with a secure backend infrastructure, so all data follows Amazon’s stringent security approaches. Our services also have multi-layered security controls built into the hardware, software, and cloud infrastructure for end-to-end security.

For example, Just Walk Out technology cameras are custom-built devices that incorporate hardware-backed security capabilities and end-to-end encryption of data both locally and while being sent between our services. Amazon Dash Cart’s image data captured during item scanning is obfuscated and then encrypted to protect confidentiality. Meanwhile, for Amazon One, we built tamper and intrusion detection capabilities into the device hardware, allowing us to render the device unusable if we detect any attempts to compromise it. The Amazon One device processes all biometric data in hardware secure zones, and it’s built so it only connects to and sends data to our cloud services, which only accepts data that originates on our devices.

Since launching Just Walk Out technology in 2018, we’ve scaled rapidly and introduced new services that redefine in-store shopping experiences for customers. Today, Just Walk Out technology is available in more than 30 Amazon Fresh stores in the U.S. and UK, over 25 Amazon Go stores in the U.S., and two Whole Foods Market store locations. The Amazon Dash Cart is available in many Amazon Fresh stores, and Amazon One is available in select Amazon Go, Amazon Fresh, and Whole Foods Market stores across the U.S.

We’ve equipped more than a dozen U.S. and UK third-party locations with Just Walk Out technology, Amazon One, or a combination of both. Sports venues, like Climate Pledge Arena and TD Garden, and travel retailers, like Hudson and WHSmith, have enabled their stores with our technologies to make the shopping experience more enjoyable and efficient for their shoppers. In addition to our own Amazon Fresh stores in the UK with Just Walk Out technology, UK grocer Sainsbury’s has opened SmartShop Pick & Go, a Just Walk Out technology-enabled convenience store in London.

This kind of scaling comes with its own challenges. For example, over the past few years, we’ve increased the complexity of use cases where Just Walk Out technology is deployed—from 1,800 square foot Amazon Go stores to 40,000 square foot, full-size grocery stores—and had to account for differences in shopping behavior. For example, at an Amazon Fresh store, people tend to rummage for items like produce versus a grab-and-go salad or sandwich from Amazon Go. Similarly, expanding Just Walk Out technology to third-party retailers such as sports venues and travel retailers meant solving for unique challenges, such as being open 24/7, or hosting games and events where shoppers rush in at once at certain times. All these scenarios increase the complexity for our algorithms, and my team continues to innovate to meet the requirements of our customers and retailers.