Full Rankings

Cheers.

Introduction

I recently wrote an article about Football Outsiders flawed argument about NFL place kickers. The Football Outsider’s people responded, and I wrote a rebuttal here.  The original piece from Football Outsiders that sparked my interest in this topic is below:

Field-goal percentage is almost entirely random from season to season, while kickoff distance is one of the most consistent statistics in football.

This theory, which originally appeared in the New York Times in October 2006, is one of our most controversial, but it is hard to argue against the evidence. Measuring every kicker from 1999 to 2006 who had at least ten field goal attempts in each of two consecutive years, the year-to-year correlation coefficient for field-goal percentage was an insignificant .05. Mike Vanderjagt didn’t miss a single field goal in 2003, but his percentage was a below-average 74 percent the year before and 80 percent the year after. Adam Vinatieri has long been considered the best kicker in the game.But even he had never enjoyed two straight seasons with accuracy better than the NFL average of 85 percent until 2011, when he followed up his 26-for-28 2010 campaign by going 23-for-27 (85.2 percent).

On the other hand, the year-to-year correlation coefficient for kickoff distance, over the same period as our measurement of field-goal percentage and with the same minimum of ten kicks per year, is .61. The same players consistently lead the league in kickoff distance, particularly Billy Cundiff, Olindo Mare, and Stephen Gostkowski.

“NFL Kickers Are Judged on the Wrong Criteria,” New York Times, November 12, 2006

Pro Football Prospectus 2007, Arizona chapter

The basic gist of the NY Times article is that kickers are inconsistent from year to year and the argument is based solely on field goal percentage made.  There are two major flaws that I perceive in this argument.  The first is that they don’t control for any covariates that would affect variability such as the most obvious predictor, distance of the field goal attempt.  (I see that Football Outsiders is controlling for length of kick when they are calculating DVOA, but I’m still not convinced they controlled for distance when making the statement that “Field-goal percentage is almost entirely random from season to season”.)  The second is that I don’t think FO is making a distinction between the actual percent of field goals made by a given kicker and the probability of a given field goal kicker making a field goal.  For example, let’s say a given kicker kicks 40 field goals per year all from the same distance and the probability of success on every field goal is the same at 0.8.  Here is what ten simulated seasons look like under these conditions:

rbinom(10,40,0.8)/40

[1] 0.925 0.825 0.875 0.775 0.800 0.825 0.750 0.875 0.700 0.875

You can see that the kicking percentage for this kicker has quite a bit of variability. In their first season they made 92.5% of their kicks and then in their ninth year they made only 70% of their kicks. But their ability over these ten years was exactly the same. The differences in the field goal kicking percentages is completely due to random chance even though their ability was constant. So, we aren’t so interested in answering the question of whether or not kickers field goal percentages change from year you year; That’s essentially a meaningless question. We’re interested in whether or not the probability that a kicker will make a field goal, their ability, changes from year to year.

So I agree completely with the statement that “Field-goal percentage is almost entirely random from season to season”. But again that doesn’t really mean anything. The question that I think we’re really interested in here is “Does the ability of a place kicker change significantly from year you year?”  It seems like FO was trying to answer this question by comparing field-goal percentages, but they really should have been asking whether the probability of making a kick for a given kicker varies from year to year.

I’m going to explore this using mixed effects logistic regression modeling to look for evidence of variability in kickers abilities from year to year.

Analysis

To begin this analysis I needed to collect some data. I collected data from www.pro-football-reference.com in their box scores.  My full scraping code can be seen at the end of this post in the appendix or viewed on github here.

summary(kick.dat$yards)
#Min. 1st Qu. Median Mean 3rd Qu. Max.
#18.00 28.00 37.00 36.37 45.00 76.00
table(kick.dat$good)
#0 1
#1610 7273
table(kick.dat$year)
##2011 2010 2009 2008 2007 2006 2005 2004 2003
##1053 997 968 1038 993 990 986 892 966
tapply(kick.dat$good,kick.dat$year,mean)
##2011 2010 2009 2008 2007 2006 2005 2004 2003
##0.8309592 0.8224674 0.8057851 0.8448940 0.8257805 0.8202020 0.8103448 0.8082960 0.7960663

I collected data from 2003-2004 through 2011-2012 season. The median length of field goal attempt over this period of time was 37 yards. 8,883 total field goals were attempted (987 per year on average) of which 81.88% of attempted field goals over this period of time were converted. It also appears field goal success percentage is trending upwards over the course of these 9 years. This could easily be explained if the mean length of field goals was getting smaller over the course of these nine years, but based on the box plot below, it does not appear that there is a trend of field goal distance getting shorter on average.

Models

I built a logistic regression model with the outcome 0/1 for missed/made field goals. I initially included only a fixed for distance of the field goal (actually square root of distance), but after some exploration, I also needed to include a dummy variable for the the year. This means that as a whole for all kickers, the baseline probability of making a field goal is significantly different from season to season. Next, I added a random effect for kicker and another random effect for kicker by year. The first random effect assesses the different abilities of kickers in the NFL and the second random effect assesses the difference in abilities between years for a given kicker. If this second random effect variance estimate is large, that indicates evidence that kickers abilities are actually different from year to year. The output from this model appears below.

Generalized linear mixed model fit by maximum likelihood ['glmerMod']
Family: binomial ( logit )
Formula: good ~ I(yards^0.5) + year + (1 | name) + (1 | name:year)
Data: kick.dat

AIC       BIC    logLik  deviance
7237.243  7322.346 -3606.622  7213.243

Random effects:
Groups    Name        Variance  Std.Dev.
name:year (Intercept) 8.690e-11 9.322e-06
name      (Intercept) 4.317e-02 2.078e-01
Number of obs: 8883, groups: name:year, 352; name, 78

Fixed effects:
Estimate Std. Error z value Pr(>|z|)
(Intercept)   9.91805    0.30610   32.40  < 2e-16 ***
I(yards^0.5) -1.32140    0.04445  -29.73  < 2e-16 ***
year2010     -0.13349    0.12694   -1.05  0.29297
year2009     -0.28794    0.12606   -2.28  0.02236 *
year2008      0.02418    0.12908    0.19  0.85141
year2007     -0.17933    0.12834   -1.40  0.16232
year2006     -0.25091    0.12796   -1.96  0.04991 *
year2005     -0.30528    0.12727   -2.40  0.01646 *
year2004     -0.31881    0.13116   -2.43  0.01507 *
year2003     -0.35993    0.12817   -2.81  0.00498 **
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Results

Below is a plot of the probability of making a kick by distance of the kick. Each line represents a different season with the blue curves representing the more recent years 2007-2011 and the red curves are for the seasons 2003-2006. So it looks like, on the whole, kickers in the NFL are getting better (This Sloan paper reaches the same conclusion).

Next let’s look at how much variability there is within kickers between years.  The plot below displays the probability that a fixed kicker will make a kick based on the distance of the field goal when a fixed effect for year is NOT included in the model.    The red and blue lines represent the 5th and 95th percentile of the distribution of the probability for a kicker between years.  This means that there IS variability in the ability of a kicker from year to year when not including a fixed effect of year, but this variability is relatively small. However……

If a fixed effect of year is included in the model the standard deviation estimate for the random effect of year by kicker is 9.322e-06.  The plot below is the same as the one above, but a fixed effect of year is included here.  The red and blue lines represent the 5th and 95th percentile of the distribution of the probability for a kicker between years.  From this plot, you can see that there is almost no difference within a kicker between years.  This variability is almost entirely explained by the group improvement in all of the kickers in the NFL.

Conclusions

• There is evidence there is considerable variability between the abilities of NFL kickers.
  
• The variability that kickers seem to exhibit from year to year does not seem to be within kickers, it seems to be that kickers are getting better from year to year.  When you control for this group improvement there is ALMOST NO detectable variability in the ability of kickers from year to year.
• Kickers as a whole seem to be getting better over time.  
• 2008 was an especially good year for kickers.  Almost 84.5% of field goals were made.

Appendix

If you are interested in my scraping code you can see it below or on github here.

library(XML)
#Define function for getting kicking data
get.x<-function(date,team){
	url<-paste0("http://www.pro-football-reference.com/boxscores/",date,"0",team,".htm")

x<-readHTMLTable(url,header=FALSE)$pbp_data

x<-x[x$V1!="",]

x<-as.character(x$V6)

x<-x[!is.na(x)]

out<-x[unlist(gregexpr("field goal",x))>0]
out}

#get.x('20120930',"atl")

#Getting the data
month.hash<-list()
#month.hash[['08']]<-31
month.hash[['09']]<-30
month.hash[['10']]<-31
month.hash[['11']]<-30
month.hash[['12']]<-31
month.hash[['01']]<-31
month.hash[['02']]<-29

#kick.list<-list()
year<-"2012"
t.vec<-c("gnb","phi","htx","nor","crd","den","cle","min","tam","chi","nyj","oti","rav","rai","nyg","ram","was","buf","kan","clt","car","sea","mia","sfo","sdg","dal","pit","nwe","jax","atl","det","cin")
	for (yearmonth in c(paste0(year,c("09","10","11","12")),paste0(as.character((as.numeric(year)+1)),c("01","02")))){
			#for (yearmonth in c("201109","201110","201111","201112","201201","201202")){
	month<-substring(yearmonth,5,6)
for (day in c(paste0("0",c(1:9)),as.character(c(10:month.hash[[month]])))){
	for (t in t.vec){
		d<-paste0(yearmonth,day)
			print(c(year,yearmonth,day,t))
			tmp<-try(get.x(d,t))
			print(tmp)
			if (class(tmp)!="try-error"){kick.list[[year]][[paste0(d,t)]]<-tmp}
				}
				}}

make.kick.df<-function(year){
kick.vec<-c(unlist(kick.list[[year]]))
#remove '(no play)'
kick.vec<-kick.vec[unlist(gregexpr('(no play)',kick.vec))<0]

#pull out yardage
yards.index<-unlist(lapply(gregexpr('[0-9]',kick.vec),min))
yards<-substring(kick.vec,yards.index,yards.index+1)

#Pull out kicker name
name<-gsub(" ","",substring(kick.vec,1,yards.index-2))

#Did they make it?
good<-yards
good<-0
good<-(unlist(gregexpr("field goal good",kick.vec))>0)+0

out<-data.frame(name,yards,good,year=year)
out
}
kick.df<-list()
kick.df[['2012']]<-make.kick.df('2012')
kick.df[['2011']]<-make.kick.df('2011')
kick.df[['2010']]<-make.kick.df('2010')
kick.df[['2009']]<-make.kick.df('2009')
kick.df[['2008']]<-make.kick.df('2008')
kick.df[['2007']]<-make.kick.df('2007')
kick.df[['2006']]<-make.kick.df('2006')
kick.df[['2005']]<-make.kick.df('2005')
kick.df[['2004']]<-make.kick.df('2004')
kick.df[['2003']]<-make.kick.df('2003')
kick.df[['2002']]<-make.kick.df('2002')
kick.df[['2001']]<-make.kick.df('2001')
kick.df[['2000']]<-make.kick.df('2000')
#save.image("Kick_Database.RData")

kick.dat<-do.call(rbind,kick.df)
kick.dat$yards<-as.numeric(as.character(kick.dat$yards))
#write.csv(kick.dat,"kick_dat.csv")

First of all, I stand by everything I said in my previous post about Football Outsiders, but I guess there is more to add.  Turns out that there is a student here who used to intern for Football Outsiders, and he pointed me to the methods section on their website where they say this:

Field goal kicking is measured differently. Measuring kickers by field goal percentage is a bit absurd, as it assumes that all field goals are of equal difficulty. In our metric, each field goal is compared to the average number of points scored on all field goal attempts from that distance over the past 15 years. The value of a field goal increases as distance from the goal line increases.

I agree completely with this.  But you need to mention this fact or at least link to it even when you are describing the basics.  Schatz tweeted this, among other things, to me (thanks for the mention):

That’s not it at all.  There is a difference between simplifying your explanation of a complex statistical concept to a non-expert lay audience and simplifying your analysis to a point that renders your conclusions meaningless.  Leaving out the fact that you are controlling for the distance of the field goal shouldn’t disappear when they are trying to write a simple summary.  It’s an essential piece of the analysis, without which, renders all of your conclusions about place kicking meaningless.  And in reading just the FO Basics it’s not clear at all that they are actually controlling for distance or anything else.

Maybe this is my fault for not reading more about the methods, but I tend to think that the onus is on Football Outsiders to make it clear, even in their simple summary, that they are controlling for field goal distance.  But there is still the issue of why this wasn’t mentioned in the NY Times article either.

Schatz argues that they left the fact that they were controlling for field goal distance out of the NY times article for the purpose of simplicity.  Or in his tweeted words:

Sure, I understand that things have to be simplified for a larger audience, but this is the NY Times.  They published this article, for example, in 2007 about statistics that are misleading when you don’t properly control for explanatory variables.  So, I think it would have been alright to explain in the NY Times that kicking distance is controlled for because without controlling for kicking distance, the conclusions are meaningless.  We’re also talking about the same NY Times that hosted Nate Silver’s blog until recently!

Finally, even when controlling for field goal distance (and possibly other factors), I’m still not convinced that the ability of a place kicker varies randomly from year to year, though I don’t have any hard analysis (right now) to back this up.  Though, Mike Lopez pointed me to this article from Sloan analyzing field goal kicking, and if you look at Table 5 there is some evidence that kickers’ abilities, at least some kickers’ abilities, are consistent from year you year.  If kicker ability was changing dramatically from year you year we wouldn’t expect to see Janikowski twice in the top five best seasons or see Kris Brown twice in the bottom five seasons.  This is certainly not proof that kicker’s performance from year to year is highly variable, but it also doesn’t support that argument either.

I would be very interested to see the full analysis that Football Outsiders performed to reach this conclusion, as I think it would be interesting to try and reproduce this.

Cheers.

P.S. I will once again state for the record, that I enjoy Football Outsiders and read it regularly.

(Update: October 13, 2013 – See the full analysis here)

I was reading the Football Outsiders basics, and I got to this part about field goal kickers:

Field-goal percentage is almost entirely random from season to season, while kickoff distance is one of the most consistent statistics in football.

This theory, which originally appeared in the New York Times in October 2006, is one of our most controversial, but it is hard to argue against the evidence. Measuring every kicker from 1999 to 2006 who had at least ten field goal attempts in each of two consecutive years, the year-to-year correlation coefficient for field-goal percentage was an insignificant .05. Mike Vanderjagt didn’t miss a single field goal in 2003, but his percentage was a below-average 74 percent the year before and 80 percent the year after. Adam Vinatieri has long been considered the best kicker in the game. But even he had never enjoyed two straight seasons with accuracy better than the NFL average of 85 percent until 2011, when he followed up his 26-for-28 2010 campaign by going 23-for-27 (85.2 percent).

On the other hand, the year-to-year correlation coefficient for kickoff distance, over the same period as our measurement of field-goal percentage and with the same minimum of ten kicks per year, is .61. The same players consistently lead the league in kickoff distance, particularly Billy Cundiff, Olindo Mare, and Stephen Gostkowski.

• “NFL Kickers Are Judged on the Wrong Criteria,” New York Times, November 12, 2006

• Pro Football Prospectus 2007, Arizona chapter

In the New York Times article that they cite, “NFL Kickers Are Judgers on the Wrong Criteria”, they say:

There is effectively no correlation between a kicker’s field-goal percentage one season and his field-goal percentage the next. But average kickoff distance shows more consistency from season to season than almost any other individual statistic in the N.F.L.

So, let’s sum this up.  Football Outsiders is saying that they have discovered that field goal percentage from year to year is incredibly inconsistent, that this “theory” is “controversial”, and that kick-off distance “shows more consistency from season to season than almost any other individual statistic in the N.F.L.”

I wouldn’t describe this as either a “theory” or “controversial”.  If you’ve taken any statistics class beyond Intro Stat, this should be totally expected.  It would be strange if this wasn’t the case, and the explanation is incredibly simple.  Field goals are attempted from different distances at different angles with 11 300+ pound men trying to kill you.  There are many variables.  Kick-offs are are taken from the same spot every time and no one is trying to kill you.  Less variables.

In general, you shouldn’t compare rates like this when you aren’t controlling for other factors that may affect the rates.  I’ve written about this before in response to former Football Outsiders writer Bill Barnwell’s “study” of mortality rates of football players versus baseball players.   He shows, correctly, that baseball players have a higher mortality rate than football players, however, he failed to control for age.  The cohort of baseball players that he considered was on average older than the football players.  So, essentially what Barnwell (who is still blocking me on Twitter) demonstrated is that older people die more often than younger people.  (And of course this isn’t even considering the fact that he should have been using survival analysis techniques and looking at survival times rather than comparing mortality rates, which is usually not advisable.)

Football Outsiders seems to be making the same mistake with their field goal “theory” (and since Barnwell used to write for FO, I suppose it could just be Barnwell making the same mistake.)  Let’s take a look at the variability of field goal attempts for kickers from year to year.  To do this, I’ve created a Shiny app displaying boxplots of field goals attempts for NFL kickers with green dots for distances of made field goals and Red dots for misses.  Take a look at Mike Vanderjagt below:

Recall, that this is what FO says about Vanderjagt in support of their argument that field goal percentage is inconsistent from year to year.

Mike Vanderjagt didn’t miss a single field goal in 2003, but his percentage was a below-average 74 percent the year before and 80 percent the year after.

You’ll notice in the graph that in 2003, Vanderjagt’s median distance of a field goal attempt was 31 yards the year that he made all of his kicks.  In the previous year, 2002, his median kick distance was 9 yards longer and in 2004 his median kick distance was 34 yards.  By directly comparing rates, without considering distance of the kicks, the comparison is completely meaningless.

Football outsiders says, in their FAQ:

Q: What are we talking about here?

Football Outsiders brings you a series of brand new, in-depth statistics you can’t find anywhere else.

I don’t quite know what is meant by “in-depth statistics”, but between this kicking “theory” and former FO writer Barnwell’s mortality “study” at Grantland, I don’t really have much faith in Football Outsiders’ ability to correctly apply introductory/intermediate statistical concepts to football data.  Now, everyone makes mistakes, but these seem like unbelievable simply ones to make for a website whose major selling point is applying statistical concepts to football.

Cheers.

P.S.

All this being said, I do read Football Outsiders, and I enjoy their writing. But I am highly skeptical of all of their statistical analysis.

I’ve recently converted to R Studio for R package development and I love it.  As someone who doesn’t really understand the details of git, it makes git incredibly easy to use (And git is DEFINITELY worth using!)  However, when you check a package in R for upload to CRAN, you need to use the –as-cran option, which R studio does not use by default.  Though you can set it to be your default.  The screen shots of how to do this are below.  Cheers.

Cheers.

Deadspin announced today on their website that they are starting something called “Regressing“.  They describe it as “Deadspin in a lab coat.”  This sounds awesome.

Cheers.