Every week, marine biologist Lisa Steiner dials into a video call from her home on the remote island of Faial, in the Azores in the North Atlantic Ocean, to speak to a small team of data scientists and developers. They want to see, as she puts it, if she can break what they have been building for her.
The ‘what’ is a digital tool, designed by a team from global consulting firm Capgemini, using Amazon Web Services (AWS) machine learning technology, to help Steiner identify individual sperm whales faster and more accurately. Steiner tracks sperm whales, which can live for up to 70 years, by photographing their tail fins, or ‘flukes’, and storing the images in a database. Much like human fingerprints, flukes are unique to the individual. By cataloguing and matching the fluke images over time, Steiner can identify when the same animal has returned to the area.
A photograph of a sperm whale tail, taken by Steiner. To describe her research as a labor of love would be an understatement. When Steiner arrived on the Azores archipelago to begin studying sperm whales in 1988, the Berlin Wall was still standing, George Michael’s “Faith” was the best-selling album in the U.S., and it would be four years before the first-ever text message was sent to a cell phone. Perhaps unsurprisingly then, especially for someone who describes her own level of interest in technology “as probably a one,” Steiner’s research into the whales’ migratory patterns began—and continued—as very much a manual affair.
“We used to go out in a sailing boat from 9 a.m. until 10 p.m.,” she said. “I would photograph the whales using black and white film, but I wouldn’t know until I got it developed if I’d actually taken a clear shot.”
A digital camera, not to mention a faster catamaran, demand far less of Steiner’s patience today. Yet patience is a quality she seems to have in spades, as the daily reality of her research looks, to an outsider at least, something akin to the world’s longest-running game of "concentration."
Steiner has followed some whales for so long she can now recognize around 100 of them on sight. However, not even her vast knowledge and endless enthusiasm is enough to make seeking matches in an ever-growing database of more than 8,000 images anything less than painstaking.
Steiner has followed some whales for so long she can now recognize around 100 of them on sight. However, not even her vast knowledge and endless enthusiasm is enough to make seeking matches in an ever-growing database of more than 8,000 images anything less than painstaking.
"I did have a computer program," said Steiner, who added that she originally would simply lay the photos out on the floor. "But it’s from 2002, so nearly 20 years old, and quite cumbersome. Around 2010 to 2011, I started falling behind with the identifications."
Steiner's existing program required her to upload the images and manually input data describing the left and right contours of each individual whale's fluke into the system. The program was slow, and unable to pick up small changes over time. For example, if the contour of a whale's tail changed slightly due to wear and tear, the system couldn't recognize that it was still the same animal.
It was only when an employee from global consulting group Capgemini joined one of Steiner’s whale-watching tours and saw the potential for machine learning to assist her, that the idea for the Fluketracker was born. "I provided them with all my photographs, and they have trained an algorithm to know what a sperm whale tail looks like," Steiner said. "It has succeeded in finding matches that I couldn't, particularly those tails with very smooth contours that are difficult to distinguish from others."
The Fluketracker, which uses AWS machine learning technology to sort and match images at high speed, has already helped Steiner classify all her photos from 2019 to 2020—identifying more than 200 new whales. On her weekly call with the team, she provides feedback on how the tool is working, so they can improve the application itself, and the artificial intelligence, as well as adding new features.
"Now I just upload the images, wait a few minutes, and if there's a whale in the catalogue that’s already online, it will find the match," she said. "The end goal is really to make the platform publicly available, and open source, so anyone can submit photos."
"My research only covers the Azores, so it would be fantastic to gather photos from other regions," Steiner said. "The more we collaborate, the more information we can access, and the better we can understand where the whales go, and how they socialize. I really hope it will capture people's imaginations."
That is, of course, if she manages not to break it first.
