Football Outsiders and their flawed argument about NFL place kickers

(Update: October 13, 2013 - See the full analysis here)

I was reading the Football Outsiders basics, and I got to this part about field goal kickers:

Field-goal percentage is almost entirely random from season to season, while kickoff distance is one of the most consistent statistics in football.

This theory, which originally appeared in the New York Times in October 2006, is one of our most controversial, but it is hard to argue against the evidence. Measuring every kicker from 1999 to 2006 who had at least ten field goal attempts in each of two consecutive years, the year-to-year correlation coefficient for field-goal percentage was an insignificant .05. Mike Vanderjagt didn’t miss a single field goal in 2003, but his percentage was a below-average 74 percent the year before and 80 percent the year after. Adam Vinatieri has long been considered the best kicker in the game. But even he had never enjoyed two straight seasons with accuracy better than the NFL average of 85 percent until 2011, when he followed up his 26-for-28 2010 campaign by going 23-for-27 (85.2 percent).

On the other hand, the year-to-year correlation coefficient for kickoff distance, over the same period as our measurement of field-goal percentage and with the same minimum of ten kicks per year, is .61. The same players consistently lead the league in kickoff distance, particularly Billy Cundiff, Olindo Mare, and Stephen Gostkowski.

In the New York Times article that they cite, “NFL Kickers Are Judgers on the Wrong Criteria”, they say:

There is effectively no correlation between a kicker’s field-goal percentage one season and his field-goal percentage the next. But average kickoff distance shows more consistency from season to season than almost any other individual statistic in the N.F.L.

So, let’s sum this up.  Football Outsiders is saying that they have discovered that field goal percentage from year to year is incredibly inconsistent, that this “theory” is “controversial”, and that kick-off distance “shows more consistency from season to season than almost any other individual statistic in the N.F.L.”

I wouldn’t describe this as either a “theory” or “controversial”.  If you’ve taken any statistics class beyond Intro Stat, this should be totally expected.  It would be strange if this wasn’t the case, and the explanation is incredibly simple.  Field goals are attempted from different distances at different angles with 11 300+ pound men trying to kill you.  There are many variables.  Kick-offs are are taken from the same spot every time and no one is trying to kill you.  Less variables.

In general, you shouldn’t compare rates like this when you aren’t controlling for other factors that may affect the rates.  I’ve written about this before in response to former Football Outsiders writer Bill Barnwell’s “study” of mortality rates of football players versus baseball players.   He shows, correctly, that baseball players have a higher mortality rate than football players, however, he failed to control for age.  The cohort of baseball players that he considered was on average older than the football players.  So, essentially what Barnwell (who is still blocking me on Twitter) demonstrated is that older people die more often than younger people.  (And of course this isn’t even considering the fact that he should have been using survival analysis techniques and looking at survival times rather than comparing mortality rates, which is usually not advisable.)

Football Outsiders seems to be making the same mistake with their field goal “theory” (and since Barnwell used to write for FO, I suppose it could just be Barnwell making the same mistake.)  Let’s take a look at the variability of field goal attempts for kickers from year to year.  To do this, I’ve created a Shiny app displaying boxplots of field goals attempts for NFL kickers with green dots for distances of made field goals and Red dots for misses.  Take a look at Mike Vanderjagt below:

Screen Shot 2013-09-23 at 11.46.09 PM

Recall, that this is what FO says about Vanderjagt in support of their argument that field goal percentage is inconsistent from year to year.

Mike Vanderjagt didn’t miss a single field goal in 2003, but his percentage was a below-average 74 percent the year before and 80 percent the year after.

You’ll notice in the graph that in 2003, Vanderjagt’s median distance of a field goal attempt was 31 yards the year that he made all of his kicks.  In the previous year, 2002, his median kick distance was 9 yards longer and in 2004 his median kick distance was 34 yards.  By directly comparing rates, without considering distance of the kicks, the comparison is completely meaningless.

Football outsiders says, in their FAQ:

Q: What are we talking about here?

Football Outsiders brings you a series of brand new, in-depth statistics you can’t find anywhere else.

I don’t quite know what is meant by “in-depth statistics”, but between this kicking “theory” and former FO writer Barnwell’s mortality “study” at Grantland, I don’t really have much faith in Football Outsiders’ ability to correctly apply introductory/intermediate statistical concepts to football data.  Now, everyone makes mistakes, but these seem like unbelievable simply ones to make for a website whose major selling point is applying statistical concepts to football.

Cheers.

P.S.

All this being said, I do read Football Outsiders, and I enjoy their writing. But I am highly skeptical of all of their statistical analysis.

About these ads

7 thoughts on “Football Outsiders and their flawed argument about NFL place kickers

  1. […] the Redskins food truck … tonight is the big PBS Frontline special on NFL concussions … Stats in the Wild vs. Football Outsiders … “Is Pedialyte’s Big New Market Just A Bunch Of Drunks?” … great read […]

  2. […] of all, I stand by everything I said in my previous post about Football Outsiders, but I guess there is more to add.  Turns out that there is a student here who used to intern for […]

  3. […] visited the Redskins food truck … tonight is the big PBS Frontline special on NFL concussions … Stats in the Wild vs. Football Outsiders … “Is Pedialyte’s Big New Market Just A Bunch Of Drunks?” … great read about online […]

  4. […] recently wrote an article about Football Outsiders flawed argument about NFL place kickers. The Football Outsider’s people responded, and I wrote a rebuttal here.  The original piece […]

  5. […] However, such hot streaks likely tend to revert toward the mean (even though the subject is hotly debated). But it’s understandable that McCoy might have real confidence in Novak, even if it is […]

  6. […] (tried to) point out some their mistakes in a non-smug manner.  For instance, I wrote about their flawed “findings” about place kickers.  I thought I was being polite, and I got this, what I would consider, smug response from […]

  7. […] I’ve written about this before, in that we shouldn’t expect there to be consistency from year to year for field goal percentage within a kicker.  It’s because field goals are taken from different distances. You can see just how much variability there is in the distances of place kickers from year to year with this sweet shiny app that I made. […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 1,970 other followers

%d bloggers like this: