After today, we've probably had enough about statistical modeling, FPI, SP+, all of that. (If you've already stopped reading, my projection of the interest in this topic was off. So it goes with projections.) But there is one more piece of this that I want to look at before we all move on to the next thing.
Since 2014, I've been keeping track of FPI's preseason ratings compared to the final ratings. With the final 2019 ratings in, that's 774 individual rankings. It's a decent sample size at this point.
How good of a job does preseason FPI do at projecting what the final ratings will look like? It's important to note that's not really FPI's job in the preseason. The job is to run the model and determine a pure power ranking––Team 1 is better than Team 2 is better than Team 3 and so on. That's all the 2020 rankings released so far are meant to do.
As Brad Edwards mentioned on Hail Varsity Radio yesterday, the next step in the FPI process is using those power rankings to run hundreds of thousands of simulations based on team's actual schedules. From that, you'll get projected win totals which will be a better real-world representation of what these rankings mean.
ESPN CFB Insider Brad Edwards breaks down why Nebraska received a Top 25 FPI ranking https://t.co/AcQ35O5wOa
— ESPN LINCOLN KLMS (@ESPNLincoln) February 19, 2020
All that said, I don't think most people view spring projections that way. If Nebraska shows up at No. 22, many will expect Nebraska to be around No. 22 at the end of the year. It's a limited view of the model, but probably the predominant view. Kentucky fans don't care a whole lot about how close FPI got to getting Tennessee right, they care about Kentucky.
But if you do look at all 774 rankings and ratings and compare preseason to end-of-season, it looks like this. The standard error for a wins projection between 2014 and 2019 was 2.2 wins. A team projected to win seven games, then, would fall between five and nine wins about two-thirds of the time. The range here spans from 2019 Navy on the high end (+7.1)––FPI projected 3.9 wins for Navy last season and the Midshipmen won 11––to 2015 UCF on the low end (-6.4), a projection of 6.4 wins and a season with zero. (I expect Nebraska's projected win total to be around seven when those numbers come out from FPI.)
On the rankings side, you're looking at a standard error of 18.55 spots. Last year's Navy squad is the most underrated team of the span, but 2018 Louisville takes the "honor" of being the most overrated. On the ratings side, it's a standard error of 5.86 meaning teams are within a touchdown, plus or minus, of their rating about two-thirds of the time.
If you want to apply that to Nebraka's No. 22 rating to start 2020, the past performance of the model gives you an idea of just how wide the range here might be. Two-thirds of the time, Nebraska should finish the season ranked between No. 4 and No. 40. If you applied that range to last season, the Huskers' could've been expected to finish between No. 13 and No. 49 after starting at No. 31. They actually finished at No. 47, towards the bottom of the expected range but still within it.
Here's how this has worked for Nebraska over every year I have these calculations for at this point.
2014: Nebraska is projected at No. 41 in the preseason with a projected regular-season win total of 7.7. The Huskers win nine games in the regular season, Bo Peini gets fired anyway, and Nebraska is No. 28 in the final rankings after a bowl-game loss to USC. It is the biggest positive gain in the rankings (+13 spots), preseason to end-of-season, of the span.
2015: The Huskers aren't viewed much differently in the preseason than the previous season's team––No. 42 ranking, 7.7 projected wins. Nebraska wins five in the regular season, backdoors into a bowl game and makes it six in the bowl in Mike Riley's first season. Despite the losing record, this is FPI's most accurate ranking for Nebraska. The Huskers finished No. 39, a gain of three spots.
2016: In terms of wins, FPI nailed it with a preseason prediction of 8.9 against a regular-season total of nine. The ranking, however, was off by more than one standard deviation. No. 25 in the preseason, Nebraska finishes No. 48. After winning nine games. This is part of the value I find in models such as this. Results matter, yes, but not as much as how well a team is actually playing. The 2015 team lost a bunch of close ones and played better as whole compared to the 2016 team that won four more, but didn't do so impressively. Some might view that as a flaw. I think it's a strength.
2017: This was the much-discussed 5.5-win projection, which felt a little crazy if you just looked at it from the point of view of the Huskers having just won nine the previous season. Nebraska won four games, Riley was fired, and the Huskers ended the year at No. 70 after starting at No. 58. It was bad, but FPI kinda said it would be.
2018: Here's another one where things were pretty accurate. Scott Frost's first team was No. 51 at the start with 5.6 wins as the projection. Twelve games later it was four wins with a No. 57 ranking. That 2018 season was a wild ride with the 0-6 start and 4-2 finish, but by the end of it all things had mostly smoothed out to about what FPI's model predicted.
2019: As mentioned above, we're towards the outlier range here. While FPI wasn't as high rankings-wise as the opinion polls, putting Nebraska at No. 31, it did apply that rating to a favorable schedule and project 8.1 wins. Nebraska falling 3.1 wins short is the biggest discrepancy, positive or negative, of the six seasons presented here. From that view, it's not a huge surprise people are a little skeptical of FPI right now. Its most recent result, from a Nebraska point of view, was the one that was off the most.
But if you weren't doing the wins math above, you have: 1) The 2019 season, which was the biggest gap between projected and actual wins and the 2015 season whish was off by 2.7, 2) One season (2016) that was dead on (+0.1) from a wins perspective, and 3) three other seasons where Nebraska's win total was 1.6 or fewer wins away from the projected total.
Do with that what you will.
The Grab Bag
- Greg Smith released his first big-board ranking for the 2021 recruiting cycle. He also gave us some honorable mentions.
- Nebraska v. Michigan State. Hoiberg v. Hoiberg. Preview.
- Jacob Padilla on the prospect of revamping transfer rules.
Today’s Song of Today
Brandon is the Managing Editor for Hail Varsity and has covered Nebraska athletics for the magazine and web since 2012, Hail Varsity’s first season on the scene. His sports writing has also been featured by Fox Sports, The Guardian and CBS Sports.
