Stats in the Wild

First of all, I stand by everything I said in my previous post about Football Outsiders, but I guess there is more to add.  Turns out that there is a student here who used to intern for Football Outsiders, and he pointed me to the methods section on their website where they say this:

Field goal kicking is measured differently. Measuring kickers by field goal percentage is a bit absurd, as it assumes that all field goals are of equal difficulty. In our metric, each field goal is compared to the average number of points scored on all field goal attempts from that distance over the past 15 years. The value of a field goal increases as distance from the goal line increases.

I agree completely with this.  But you need to mention this fact or at least link to it even when you are describing the basics.  Schatz tweeted this, among other things, to me (thanks for the mention):

That’s not it at all.  There is a difference between simplifying your explanation of a complex statistical concept to a non-expert lay audience and simplifying your analysis to a point that renders your conclusions meaningless.  Leaving out the fact that you are controlling for the distance of the field goal shouldn’t disappear when they are trying to write a simple summary.  It’s an essential piece of the analysis, without which, renders all of your conclusions about place kicking meaningless.  And in reading just the FO Basics it’s not clear at all that they are actually controlling for distance or anything else.

Maybe this is my fault for not reading more about the methods, but I tend to think that the onus is on Football Outsiders to make it clear, even in their simple summary, that they are controlling for field goal distance.  But there is still the issue of why this wasn’t mentioned in the NY Times article either.

Schatz argues that they left the fact that they were controlling for field goal distance out of the NY times article for the purpose of simplicity.  Or in his tweeted words:

Sure, I understand that things have to be simplified for a larger audience, but this is the NY Times.  They published this article, for example, in 2007 about statistics that are misleading when you don’t properly control for explanatory variables.  So, I think it would have been alright to explain in the NY Times that kicking distance is controlled for because without controlling for kicking distance, the conclusions are meaningless.  We’re also talking about the same NY Times that hosted Nate Silver’s blog until recently!

Finally, even when controlling for field goal distance (and possibly other factors), I’m still not convinced that the ability of a place kicker varies randomly from year to year, though I don’t have any hard analysis (right now) to back this up.  Though, Mike Lopez pointed me to this article from Sloan analyzing field goal kicking, and if you look at Table 5 there is some evidence that kickers’ abilities, at least some kickers’ abilities, are consistent from year you year.  If kicker ability was changing dramatically from year you year we wouldn’t expect to see Janikowski twice in the top five best seasons or see Kris Brown twice in the bottom five seasons.  This is certainly not proof that kicker’s performance from year to year is highly variable, but it also doesn’t support that argument either.

I would be very interested to see the full analysis that Football Outsiders performed to reach this conclusion, as I think it would be interesting to try and reproduce this.

Cheers.

P.S. I will once again state for the record, that I enjoy Football Outsiders and read it regularly.

(Update: October 13, 2013 – See the full analysis here)

I was reading the Football Outsiders basics, and I got to this part about field goal kickers:

Field-goal percentage is almost entirely random from season to season, while kickoff distance is one of the most consistent statistics in football.

This theory, which originally appeared in the New York Times in October 2006, is one of our most controversial, but it is hard to argue against the evidence. Measuring every kicker from 1999 to 2006 who had at least ten field goal attempts in each of two consecutive years, the year-to-year correlation coefficient for field-goal percentage was an insignificant .05. Mike Vanderjagt didn’t miss a single field goal in 2003, but his percentage was a below-average 74 percent the year before and 80 percent the year after. Adam Vinatieri has long been considered the best kicker in the game. But even he had never enjoyed two straight seasons with accuracy better than the NFL average of 85 percent until 2011, when he followed up his 26-for-28 2010 campaign by going 23-for-27 (85.2 percent).

On the other hand, the year-to-year correlation coefficient for kickoff distance, over the same period as our measurement of field-goal percentage and with the same minimum of ten kicks per year, is .61. The same players consistently lead the league in kickoff distance, particularly Billy Cundiff, Olindo Mare, and Stephen Gostkowski.

• “NFL Kickers Are Judged on the Wrong Criteria,” New York Times, November 12, 2006

• Pro Football Prospectus 2007, Arizona chapter

In the New York Times article that they cite, “NFL Kickers Are Judgers on the Wrong Criteria”, they say:

There is effectively no correlation between a kicker’s field-goal percentage one season and his field-goal percentage the next. But average kickoff distance shows more consistency from season to season than almost any other individual statistic in the N.F.L.

So, let’s sum this up.  Football Outsiders is saying that they have discovered that field goal percentage from year to year is incredibly inconsistent, that this “theory” is “controversial”, and that kick-off distance “shows more consistency from season to season than almost any other individual statistic in the N.F.L.”

I wouldn’t describe this as either a “theory” or “controversial”.  If you’ve taken any statistics class beyond Intro Stat, this should be totally expected.  It would be strange if this wasn’t the case, and the explanation is incredibly simple.  Field goals are attempted from different distances at different angles with 11 300+ pound men trying to kill you.  There are many variables.  Kick-offs are are taken from the same spot every time and no one is trying to kill you.  Less variables.

In general, you shouldn’t compare rates like this when you aren’t controlling for other factors that may affect the rates.  I’ve written about this before in response to former Football Outsiders writer Bill Barnwell’s “study” of mortality rates of football players versus baseball players.   He shows, correctly, that baseball players have a higher mortality rate than football players, however, he failed to control for age.  The cohort of baseball players that he considered was on average older than the football players.  So, essentially what Barnwell (who is still blocking me on Twitter) demonstrated is that older people die more often than younger people.  (And of course this isn’t even considering the fact that he should have been using survival analysis techniques and looking at survival times rather than comparing mortality rates, which is usually not advisable.)

Football Outsiders seems to be making the same mistake with their field goal “theory” (and since Barnwell used to write for FO, I suppose it could just be Barnwell making the same mistake.)  Let’s take a look at the variability of field goal attempts for kickers from year to year.  To do this, I’ve created a Shiny app displaying boxplots of field goals attempts for NFL kickers with green dots for distances of made field goals and Red dots for misses.  Take a look at Mike Vanderjagt below:

Recall, that this is what FO says about Vanderjagt in support of their argument that field goal percentage is inconsistent from year to year.

Mike Vanderjagt didn’t miss a single field goal in 2003, but his percentage was a below-average 74 percent the year before and 80 percent the year after.

You’ll notice in the graph that in 2003, Vanderjagt’s median distance of a field goal attempt was 31 yards the year that he made all of his kicks.  In the previous year, 2002, his median kick distance was 9 yards longer and in 2004 his median kick distance was 34 yards.  By directly comparing rates, without considering distance of the kicks, the comparison is completely meaningless.

Football outsiders says, in their FAQ:

Q: What are we talking about here?

Football Outsiders brings you a series of brand new, in-depth statistics you can’t find anywhere else.

I don’t quite know what is meant by “in-depth statistics”, but between this kicking “theory” and former FO writer Barnwell’s mortality “study” at Grantland, I don’t really have much faith in Football Outsiders’ ability to correctly apply introductory/intermediate statistical concepts to football data.  Now, everyone makes mistakes, but these seem like unbelievable simply ones to make for a website whose major selling point is applying statistical concepts to football.

Cheers.

P.S.

All this being said, I do read Football Outsiders, and I enjoy their writing. But I am highly skeptical of all of their statistical analysis.

I’ve recently converted to R Studio for R package development and I love it.  As someone who doesn’t really understand the details of git, it makes git incredibly easy to use (And git is DEFINITELY worth using!)  However, when you check a package in R for upload to CRAN, you need to use the –as-cran option, which R studio does not use by default.  Though you can set it to be your default.  The screen shots of how to do this are below.  Cheers.

Cheers.

Deadspin announced today on their website that they are starting something called “Regressing“.  They describe it as “Deadspin in a lab coat.”  This sounds awesome.

Cheers.

One of my students tweeted this at me today.  I think it is awesome.  But I suppose this also means they aren’t paying attention to me.  But still, this is awesome.  So I guess it’s ok that they aren’t paying attention if they are creating brilliant images like this.

Argggghhhhhhh and Cheers.

Check out this graphic from the New York Times.  Eli Manning has now started 150 games in a row for the Giants.  That’s pretty hard to grasp.

Cheers.

I attended the New England Symposium of Statistics in Sports (NESSIS) last Saturday at Harvard Science Center (See the sweet logo below) where I presented a poster.  The conference was organized by Mark Glickman and Scott Evans Scott Evans

My poster (see below) was about openWAR, which is  a project I am working on with Ben Baumer and Shane Jensen.  Our goal is to create a completely open source version of wins above replacement (WAR) based entirely on publicly available data.  We’ve implemented openWAR in R and the package is currently available on github here: openWAR.  When we think it’s ready for primetime, we’ll be putting in on CRAN.

I missed the first featured session because it was at 9:30am, and that’s not how I roll on Satudays.  During the parallel sessions at 11:30am, I decided to attend the non-NBA series of talks.  The first talk was by Robert Carver and he talked about R.A. Dickey and the curveball.  He was followed by Stephanie Kovalchik who gave an interesting talk about trends in tennis intensity.  She had a lot of really interesting data visualizations of tennis trends over the past few decades, but I can’t seem to find them online.  If anyone knows where I can find there, please point me in the right direction.  After her, Dennis Lock gave a talk about using random forests to estimate win probability.  At the end of the day I was trying to explain random forests to someone from ESPN (how awesome is that sentence), and I knew that random forests were essentially regression trees based on bootstrapped samples.  When I went to look this up to make sure I wasn’t lying about random forests, I found out that at each step the set of predictors in the regression tree is randomly chosen.  I did not realize this, but makes total sense.  Otherwise, the trees in the forest would all be very similar. So I learned something, and isn’t that the whole point of these conferences?

The final talk in this session was by Michael Pane who was attempting to cluster pitches based on pitch F/X data and improve classification of MLB pitches.  They call their procedure CLUMPD and they made a sweet interactive shiny app.  But I didn’t write down the URL,  and I can’t seem to find it by googling it.  Hopefully when they post the slides, the link will be in there.

Following the session I ate lunch with Ben Baumer, Mike Lopez, and one of Mike’s friends from UMass on the rocks outside of the Harvard science center.  After lunch I mean to go the the afternoon featured speaker, but I ended up talking to two San Francisco fans about my poster.  I asked them if they were presenting at the conference, and they told me that they didn’t even know the conference was going to be there.  They were just baseball fans in town to see a few Red Sox games and they apparently just stumbled across NESSIS and my poster.  After talking to the two guys from San Francisco, I talked to one of the members of the Tuft’s SABR club about openWAR for the rest of the time allotted for the featured speakers.  After we finished talking the actual poster session started at 3:30.  I met and spoke with a ton of interesting people.

Here’s a list of some of the interesting people that I talked to while at my poster:

• Vince Gennaro – Author of Diamond Dollars: The Economics of Winning in Baseball, President of SABR, consultant to MLB teams, all around baseball fanatic
• Eric Van – Former consultant for the Boston Red Sox
• Michael Humphries – Author of “Wizardry: Baseball’s All-Time Greatest Fielders Revealed”
• James O’ Malley – Professor at Dartmouth
• Andy Andres – Teacher SABR 101 at Tufts
• Doug Noe –  Professor at Miami (OH) (This was my favorite meeting because I had never met him before, but he told me that he really liked my blog and that I had actually written about him before.)

Right at the end of the poster session, Eric Van came over to my openWAR poster and criticized our definition of replacement player.  The way that we have defined it, about half of the players we have defined as being in the replacement group are below the average replacement player.  While I’m not sure that this isn’t ok technically, it’s a huge success for our larger idea.  By making openWAR completely transparent people are free to criticize, critique, and complement every single piece of our procedure (and we definitely welcome constructive criticism), rather than gues at what’s going on inside the black boxes of baseball reference and fan graphs WAR.

NESSIS then closed with a panel discussion.  The panel consisted of Ben Baumer, Eric Van, and Vince Gennaro.  The picture below is the panel, with Carl Morris (you know he’s a big deal cause he’s got a Wikipedia page) saying some words before the discussion began.  The panel was ultimately moderated by Andy Andres.

One of the interesting points the panel made was that in the beginning of SABRmetrics, a lot of the most interesting work was being done by fans and not necessarily the teams themselves.  This has entirely changed today due to the fact that baseball teams have access to mountains and mountains of data that are simply not available to the public or the public can’t afford.

Van also pointed out that the numbers don’t tell you everything.  You can’t just view numbers and ignore the personality of players.  For instance, if the numbers say that a guy should hit 6th instead of 2nd, you have to weigh the improvement your team will gain against the psychology of moving a guy from 2nd to 6th in the line-up.  In his words:

The numbers are just sign posts. You have to actually watch the game to see if you’re onto something. -Eric Van

The whole discussion was fantastic, and it was really interesting to hear the perspective of three people who have actually worked in baseball as statistical analysts.

Fantastic overall conference.  See you in 2015!

Cheers.