(Update: October 13, 2013 – See the full analysis here)

I was reading the Football Outsiders basics, and I got to this part about field goal kickers:

Field-goal percentage is almost entirely random from season to season, while kickoff distance is one of the most consistent statistics in football.

This theory, which originally appeared in the New York Times in October 2006, is one of our most controversial, but it is hard to argue against the evidence. Measuring every kicker from 1999 to 2006 who had at least ten field goal attempts in each of two consecutive years, the year-to-year correlation coefficient for field-goal percentage was an insignificant .05. Mike Vanderjagt didn’t miss a single field goal in 2003, but his percentage was a below-average 74 percent the year before and 80 percent the year after. Adam Vinatieri has long been considered the best kicker in the game. But even he had never enjoyed two straight seasons with accuracy better than the NFL average of 85 percent until 2011, when he followed up his 26-for-28 2010 campaign by going 23-for-27 (85.2 percent).

On the other hand, the year-to-year correlation coefficient for kickoff distance, over the same period as our measurement of field-goal percentage and with the same minimum of ten kicks per year, is .61. The same players consistently lead the league in kickoff distance, particularly Billy Cundiff, Olindo Mare, and Stephen Gostkowski.

• “NFL Kickers Are Judged on the Wrong Criteria,” New York Times, November 12, 2006

• Pro Football Prospectus 2007, Arizona chapter

In the New York Times article that they cite, “NFL Kickers Are Judgers on the Wrong Criteria”, they say:

There is effectively no correlation between a kicker’s field-goal percentage one season and his field-goal percentage the next. But average kickoff distance shows more consistency from season to season than almost any other individual statistic in the N.F.L.

So, let’s sum this up.  Football Outsiders is saying that they have discovered that field goal percentage from year to year is incredibly inconsistent, that this “theory” is “controversial”, and that kick-off distance “shows more consistency from season to season than almost any other individual statistic in the N.F.L.”

I wouldn’t describe this as either a “theory” or “controversial”.  If you’ve taken any statistics class beyond Intro Stat, this should be totally expected.  It would be strange if this wasn’t the case, and the explanation is incredibly simple.  Field goals are attempted from different distances at different angles with 11 300+ pound men trying to kill you.  There are many variables.  Kick-offs are are taken from the same spot every time and no one is trying to kill you.  Less variables.

In general, you shouldn’t compare rates like this when you aren’t controlling for other factors that may affect the rates.  I’ve written about this before in response to former Football Outsiders writer Bill Barnwell’s “study” of mortality rates of football players versus baseball players.   He shows, correctly, that baseball players have a higher mortality rate than football players, however, he failed to control for age.  The cohort of baseball players that he considered was on average older than the football players.  So, essentially what Barnwell (who is still blocking me on Twitter) demonstrated is that older people die more often than younger people.  (And of course this isn’t even considering the fact that he should have been using survival analysis techniques and looking at survival times rather than comparing mortality rates, which is usually not advisable.)

Football Outsiders seems to be making the same mistake with their field goal “theory” (and since Barnwell used to write for FO, I suppose it could just be Barnwell making the same mistake.)  Let’s take a look at the variability of field goal attempts for kickers from year to year.  To do this, I’ve created a Shiny app displaying boxplots of field goals attempts for NFL kickers with green dots for distances of made field goals and Red dots for misses.  Take a look at Mike Vanderjagt below:

Recall, that this is what FO says about Vanderjagt in support of their argument that field goal percentage is inconsistent from year to year.

Mike Vanderjagt didn’t miss a single field goal in 2003, but his percentage was a below-average 74 percent the year before and 80 percent the year after.

You’ll notice in the graph that in 2003, Vanderjagt’s median distance of a field goal attempt was 31 yards the year that he made all of his kicks.  In the previous year, 2002, his median kick distance was 9 yards longer and in 2004 his median kick distance was 34 yards.  By directly comparing rates, without considering distance of the kicks, the comparison is completely meaningless.

Football outsiders says, in their FAQ:

Q: What are we talking about here?

Football Outsiders brings you a series of brand new, in-depth statistics you can’t find anywhere else.

I don’t quite know what is meant by “in-depth statistics”, but between this kicking “theory” and former FO writer Barnwell’s mortality “study” at Grantland, I don’t really have much faith in Football Outsiders’ ability to correctly apply introductory/intermediate statistical concepts to football data.  Now, everyone makes mistakes, but these seem like unbelievable simply ones to make for a website whose major selling point is applying statistical concepts to football.

Cheers.

P.S.

All this being said, I do read Football Outsiders, and I enjoy their writing. But I am highly skeptical of all of their statistical analysis.