Numerous quarterback evaluation metrics exist, but they all drop the ball in one area—isolating the specific variables a quarterback must evaluate before a passing play and assessing whether the quarterback made the optimal decision after the play. To close this gap, the Next Gen Stats (NGS) team used Amazon Web Services (AWS) offerings to develop Passing Score, a new artificial intelligence (AI) quarterback measurement tool that evaluates passing performance.
Over the course of an NFL game, quarterbacks do far more than throw a few passes. They make hundreds of split-second decisions. As they approach the line of scrimmage, they assess factors like the defensive players’ lineup, time remaining in the game, and yards needed to make a first down. The variability of a play’s success significantly rises in complexity once the play starts and the quarterback must make a snap judgement on where to throw the ball. To compound the variabilities, a receiver could make an incredible play on an overthrown ball, or a defensive player might tip a perfect pass up in the air that triggers an interception. In the past, regardless of the play’s outcome, the statistical credit or blame was assigned to the quarterback.
Now, NFL fans have Passing Score. NGS spent nearly a year solving a series of complex technology problems to create the Passing Score tool. NGS had no existing playbook to follow, but they knew AWS is uniquely positioned to help tackle these challenges with its existing data-driven solutions for sports and deep technology partnership with the NFL.
Building on years of creating and deploying AWS-powered machine learning (ML) models, the NGS engineering and analytics team worked with the AWS Professional Services data science group to optimize and combine seven ML models, including a new model to predict the value of a pass before the ball is thrown. The combined ML models power the new NGS Passing Score. Each individual model provides compelling insights into what’s happening on the field, like predicted yards and completion probability. By combining the models, a score (a number between 50–99) better reflects the intuition used by fans, coaches, and players to assess a quarterback’s passing performance.
Ranking the 2021 QB performances
The 2021 NFL regular season delivered some memorable quarterback performances. We evaluated the regular season’s gameplay and combined that analysis with more than 16 million video frames of player data across the last three seasons of play to answer the question, “Who was the best 2021 regular-season quarterback?”
Building a winning combination
The NGS Passing Score is a simple representation of a complex equation. By design, it does not simply award all passing yards, touchdowns, and interceptions to the quarterback. Instead, it isolates the contribution of the passer on the outcome of a pass play, agnostic of the receiver. It also controls for the level of difficulty of every pass, to best encompass the traits of a passer that are most predictive of future performance. Finally, it rolls all this into a single metric.
Passing Score works by analyzing tracking data to predict how many yards a receiver will gain if targeted, combined with an enhancement to the completion probability model, which can now also estimate the probability of an interception. The new system created with the AWS team powers the “expectation” metrics used in the primary component of the score.
With this novel AWS-powered methodology now in place, NGS Passing Score can expand and factor in other dimensions like rushing, sacks, and pressures. To learn more about the new metric, check out this blog the NFL created to show you how it all plays out on the field.
