www.nbcsports.com
AI Comes to the NFL
I might have seen the future of football on TV Thursday night, with an assist from Artificial Intelligence, Andrew Luck’s former center at Stanford, and a team of Amazon techies based in Tel Aviv.
That is not a misprint.
One of the reasons the NFL was so eager to get a new and aggressive streaming partner in 2022 was on display in the Jacksonville-New Orleans game on Amazon Prime Video Thursday. Let me tell you what I saw on one of Amazon’s three streaming options for its games, Prime Vision with Next Gen Stats. On the Prime Vision feed, Amazon shows the all-22 camera angle, able to see the whole field; the tradeoff, of course, is that you don’t see the quarterback, large, in the center of the TV. You see everything, with no one bigger than anyone else—while hearing Al Michaels and Kirk Herbstreit call the game the same as on the regular streaming ‘cast.
With 5:26 left in the first quarter, the Saints had a third-and-seven at the Jaguars’ 32-yard line. Jacksonville cornerback Tre Herndon jogged to a spot two yards across from the left slot, in coverage on receiver Michael Thomas. On the all-22 view, Herndon leaned forward as quarterback Derek Carr began his cadence. Just then, a red circle was superimposed around Herndon—and on the other side of the formation, red also encircled linebacker Devin Lloyd—with black circles superimposed around the four Saints wide receivers plus running back Alvin Kamara.
The red circle was Amazon’s way of foreshadowing what AI told them from whipping through hundreds of factors—including anticipatory tics that could be gleaned from the two movement trackers in Herndon’s left and right shoulder pad—in split seconds: Prime Vision was predicting Herndon and Lloyd would blitz. Quite a leap of faith in the Herndon forecast. In the first six games of the season, Herndon, per Next Gen Stats, had blitzed only 12 times.
Carr took his time on the cadence. The red circle was around Herndon for two, three, four, five, six seconds, and the six-year vet corner showed nothing. Carr certainly could get no clue from the possible blitzer on his left. Finally, 8.31 seconds after the encircling of Herndon appeared on the all-22, Carr snapped the ball. Herndon streaked at Carr. Lloyd came, too, but was caught in traffic. No one touched the blitzing Herndon. Just as Carr was releasing the ball, Herndon, unseen, slammed into Carr and the football bounded harmlessly away. Incomplete.
Here’s the really amazing part of this: A soybean farmer in Iowa, were he a football nerd once his day job was done, could have been watching this Prime Vision view of the game just like me. And the soybean farmer would know more about the likelihood of Tre Herndon blitzing than Saints coach Dennis Allen or his offensive play-designer, Pete Carmichael, standing 20 yards away from him. Because the live feed you and I can see is banned on the sidelines and coaches’ box upstairs (more about the pitfalls of that later), people from the Everglades to the Cascades can see a blitz coming better than the teams on the field can.
It’s sort of revolutionary. Viewers should love this. The NFL must love the fun of it today. But Artificial Intelligence strikes fear into those who think it might go too far. The competitive guardrails on this, for the NFL, had better be sturdy.
---
Amazon had the idea when it got the NFL contract in 2021 of an alternate telecast heavy on analytics. Some of the elements, like tagging skill players pre-snap, started last year. But the biggest element, predicting blitzing, debuted 11 days ago in the Denver-Kansas City game. I heard about it in a smart story in The Athletic by Ted Nguyen. The brainchild behind the idea, Sam Schwartzstein, told me he began working on it in “late April or May” with Amazon AI experts, a machine-learning team based in Tel Aviv.
Keep in mind Amazon’s not CBS or ESPN, with virtually all the idea people solely based in the U.S. Amazon’s based in Seattle, but has campuses in more than 50 countries, including Israel. Schwartzstein has been a leader at Amazon in getting some people unfamiliar with American football, very familiar in a short time. Just who is Schwartzstein? He started 13 games at center for Stanford in Andrew Luck’s last season, 2011, and they became fast friends and smart forecasters of defensive tendencies. There is so much that’s ironic about Schwartzstein’s role in introducing new technology to the Thursday night games, but how about this nugget: He has assembled a crew of smart former players as advisers to Amazon’s Prime Vision with Next Gen Stats—including Luck. Schwartzstein meets with him on Monday nights to talk Amazon football business.
“We are in a unique spot as a sports broadcaster that is a tech company first,” Schwartzstein told me Friday night. We spoke for 30 minutes; the conversation will air in The Peter King Podcast dropping late Tuesday.
“So,” Schwartzstein said, “we looked at all these different ideas of what we could do. The first one came to mind is how do we identify the players who are going to blitz … take you into the mindset of what I used to do when I played center in college … There weren’t a lot of things that we could do to help people to watch defense in a unique way. But then talking with our science team, they said, I think we can do this with machine learning and AI. We went through the process to be able to identify using machine learning where we don’t have a readout of the rules or the specific reasons why someone’s being highlighted as a potential blitzer. But we know that’s it’s being ingested from thousands of plays that are then creating that identification tag of ‘this player is likely going to blitz.’ You can never be 100 percent right; we’re just giving you an idea of looking at the defense the same way the quarterback is.”
Two Amazon coordinating producers for the Thursday games, Alex Strand and Betsy Riley, went to Tel Aviv last spring to meet with the AI team, and to begin explaining football to the non-fans there. They ended up building the software and the model that ID’s which players on every play were likely to blitz, using pieces of physical, statistic and analytical information. Schwartzstein lives in the Bay Area. Tel Aviv is 10 hours ahead. So if he’d wake up at 7 a.m. in California, on some days he’d be tutoring the Tel Aviv team in the late afternoon and evening on Football 101.
“We probably had 15 different ideas,” Schwartzstein said. “I can’t give you the exact number that have gone to production, but a lot have gone to the wayside that we’ve tried to accomplish and pushed off for later times. We have the ability to continue ideating with them and talking with them about different ways we can help expose new things to our fan bases. What I really like is we’re not afraid of the big hairy audacious goals. We are looking to try and do things that people said that you can’t do.”
The goal this year was predictive blitzing. Amazon trusted the red-circling so much that Schwartzstein and the game producers of Prime Vision just let it go when the game starts. “I can’t turn it on and off,” he said.
The factors. That’s what I wonder about. Think of the scores of known football factors as a quarterback comes to the line, and then add the minutiae of what Next Gen Stats knows, and then add what can be read from the movement trackers in every shoulder pad. The amount of information that can be processed and interpreted by AI in seconds is, of course, mind-boggling. “It knows the alignment of every player on the field, offense and defense,” Schwartzstein said. “And then there’s expectations of all the plays where players have blitzed from. It’s taking that bevy of information to make a prediction. I can tell you … that it’s seen so many different plays and so many different scenarios that it’s intelligently highlighting unique players.”
Like Herndon, with 5:26 left in the first quarter Thursday night. AI figured he was blitzing. The Saints either didn’t or blew an assignment, and let him rush, and it cost them a third-down conversion in opposing territory in a game they lost by a touchdown. Sort of a big deal.
---
“This is the tip of the iceberg,” said Schwartzstein, and he’s right. What I saw Thursday night will make me come back for more, to see the game in a different way than I ever have. The potential for more cool innovations for home viewers is there. It’s fun. It’s smart. It’s great.
But the potential for mayhem is too, because AI may know more than an offensive coordinator about what a defender like Tre Herndon is likely to do on a given play.
We can all think of the dangers for this totally new tool. When I asked Amazon about the delay from live game to being able to see the Prime Vision feed, a spokesperson emailed: “The vast majority of Prime Video’s TNF streams travel from the stadium to the screen in 10 seconds or less. This delay matches and is often less than what viewers receive from live games on broadcast and cable. Prime Vision’s technology adds a minimal amount on top of that, usually three seconds or less.”
The exact time of delay depends whether you’re watching the stream on home internet, Xbox, Apple TV, your phone, or other devices. Understand two things here. Encircling a defender in red isn’t a guarantee that he will blitz; it’s simple saying the AI program suggests he’s likely to blitz. But in this case, Herndon was circled for 8.31 seconds, which is an eternity before the ball is snapped. (To be fair, most red circles are evident for less time.) My concern is, Amazon’s delay has to be enough time so that some person in some stadium won’t be able somehow to alert a team with information that could be an advantage competitively. It appears to be long enough, but that must be policed.
“No one’s using it for nefarious reasons,” Schwartzstein said.
The NFL’s got to be sure it has multiple layers in place to ensure no one does.
www.tmz.com
