Stats in the Wild

First of all, I stand by everything I said in my previous post about Football Outsiders, but I guess there is more to add.  Turns out that there is a student here who used to intern for Football Outsiders, and he pointed me to the methods section on their website where they say this:

Field goal kicking is measured differently. Measuring kickers by field goal percentage is a bit absurd, as it assumes that all field goals are of equal difficulty. In our metric, each field goal is compared to the average number of points scored on all field goal attempts from that distance over the past 15 years. The value of a field goal increases as distance from the goal line increases.

I agree completely with this.  But you need to mention this fact or at least link to it even when you are describing the basics.  Schatz tweeted this, among other things, to me (thanks for the mention):

That’s not it at all.  There is a difference between simplifying your explanation of a complex statistical concept to a non-expert lay audience and simplifying your analysis to a point that renders your conclusions meaningless.  Leaving out the fact that you are controlling for the distance of the field goal shouldn’t disappear when they are trying to write a simple summary.  It’s an essential piece of the analysis, without which, renders all of your conclusions about place kicking meaningless.  And in reading just the FO Basics it’s not clear at all that they are actually controlling for distance or anything else.

Maybe this is my fault for not reading more about the methods, but I tend to think that the onus is on Football Outsiders to make it clear, even in their simple summary, that they are controlling for field goal distance.  But there is still the issue of why this wasn’t mentioned in the NY Times article either.

Schatz argues that they left the fact that they were controlling for field goal distance out of the NY times article for the purpose of simplicity.  Or in his tweeted words:

Sure, I understand that things have to be simplified for a larger audience, but this is the NY Times.  They published this article, for example, in 2007 about statistics that are misleading when you don’t properly control for explanatory variables.  So, I think it would have been alright to explain in the NY Times that kicking distance is controlled for because without controlling for kicking distance, the conclusions are meaningless.  We’re also talking about the same NY Times that hosted Nate Silver’s blog until recently!

Finally, even when controlling for field goal distance (and possibly other factors), I’m still not convinced that the ability of a place kicker varies randomly from year to year, though I don’t have any hard analysis (right now) to back this up.  Though, Mike Lopez pointed me to this article from Sloan analyzing field goal kicking, and if you look at Table 5 there is some evidence that kickers’ abilities, at least some kickers’ abilities, are consistent from year you year.  If kicker ability was changing dramatically from year you year we wouldn’t expect to see Janikowski twice in the top five best seasons or see Kris Brown twice in the bottom five seasons.  This is certainly not proof that kicker’s performance from year to year is highly variable, but it also doesn’t support that argument either.

I would be very interested to see the full analysis that Football Outsiders performed to reach this conclusion, as I think it would be interesting to try and reproduce this.

Cheers.

P.S. I will once again state for the record, that I enjoy Football Outsiders and read it regularly.