THE BOOK--Playing The Percentages In Baseball

Thursday, December 01, 2011

Is there a goalie skill?

I used Gabe’s data found here to select all goalies with at least 3000 shots faced on 5 on 5, over the 2007/08 through 2010/2011 seasons.
http://behindthenet.ca/goalie_shot_quality.php?sort=12&mingp=3000

This gave me 42 goalies that were the most used.  I calculated the league average goals/shot rate (5.62%), as well as how many standard deviations each goalie was from the mean (given the number of shots he faced), or his z-score.

If all goalies were the same, and all the shots they faced were of the same quality in front of the same defense, we’d expect the standard deviation of all the z-scores will be 1.00.

Of course, we won’t find that because we expect at least one of those three things to be different:

1. There’s no reason to expect that all goalies have an identical skill.

2. Nor would be expect each goalie to have faced identical quality of shots, even if their total shots faced is from 3000 to 8000.  Though the hope is that it will cancel out to a large degree.

3. In addition, given that goalies are married to a limited number of teams, and there is an obvious difference in defense talent and defense setups, we don’t expect that to work itself out in the wash.  Indeed, this would be an example of a systematic bias, and so, would actually be somewhat persistent.

So, no longer do we have 1.00 as the standard baseline to use to presume a random context for each goalie.

Anyway, so the standard deviation of all the z-scores is 1.32, which shows a substantial amount of ... well, something non-random.

To the extent that there’s SOME goalie talent, the one goalie that stands head-and-shoulders above them all is Tim Thomas.  His z-score is 3.30, while the next three are at 2.29, 2.19, and 2.11 (Bryzgalov, Luongo, Hiller, respectively).  After that, it’s a steep drop.

Bringing up the rear, by far is Brian Elliott (-3.13), and then a cluster of goalies.

We have to add about 3000 shots faced at league average to do a regression toward the mean to remove the noise (presuming that the goalie is the ONLY variable in play here).  On the other hand, if we believe that the other two variables have at least as much impact as the goalie, we need to add 7500 shots at league average for regression toward the mean.

What is the impact here?  Let’s look at Tim Thomas.  He faced 6395 shots and allowed (or rather, was in the net when there were scored) 298 goals.  He is, unadjusted, +69 goals better than average over the 4 year period.  Regression toward the mean however would suggest regressing that total at least 32% and perhaps even 54%.  That would then given us a true talent estimate for Tim Thomas of +47 goals, or perhaps even +32 goals.  Divide by 4, and that’s a value of +12 goals (or perhaps +8 goals).  This is limited to the 5 on 5 time, which would roughly correspond to 75% of all shots faced.  Increasing the above totals by four-thirds, and we get +16 goals (or +11 goals) per season.

***

It could even be much lower than that.  Take for example the goals per shots allowed of the 42 goalies with the most shots faced (hence, the goalies that the coaches trusted the most) and the rest of the league.  Now, we have a bias because a coach will be guided by observed outcomes as to who he’s going to put in nets.  Take for example the possibility that all goalies are equal in talent (but the coach thinks otherwise).  If one goalie gives up 2 goals a game, and the other gives up 3 goals a game, which goalie will continue to see ice time? 

So, we expect to see some difference between the two groups of goalie, based purely on the selection bias issue.  How much?  Well, I don’t know!

The regular goalies allows goals at a .056 rate per shot, while the backup goalies are at .061.  Is it possible that selection bias is at play here, such that the better goalies have an actual talent of .058 goals per shot (instead of .056) and the backup goalies are at .059 goals per shot (instead of .061)?  Sure, I’m not excluding anything here.