THE BOOK--Playing The Percentages In Baseball

Monday, October 10, 2011

Goodbye to you, WPA for starting pitchers

In a game for the ages, Roy Halladay and Chris Carpenter did what they did best.

In a poll on my site, asking for which pitcher threw the better game, nearly 50% called it a draw.  20% went with Carpenter and 30% went with Doc.  Basically, it was 52/48 for Doc, with plenty of uncertainty.  For all intents and purposes, it looked like a draw to the readers.

I introduced four Game Score metrics modelled on the Bill James original, and it gave a slight advantage to Carp over Doc.  If you had to give it odds, it was probably 55/45 in favor of Carpenter.  (Game Score like innings pitched, and Carpenter went the whole nine.)

In terms of trying to have an overall evaluation of their performances, you are left to conclude that it was pretty close.

Except for WPA. 

WPA called it +0.77 wins for Carpenter and +0.24 wins for Halladay if you were to follow Fangraphs, and Baseball Reference says +0.81 for Carpenter and +0.28 for Halladay.  Either way, the gap is 0.53 wins.  A gap of 0.53 wins is equivalent to 5.3 runs in a typical game!  (Of course the higher the leverage, the lower the number of runs it takes to turn into a win.)

So, what happened?  Why did WPA simply blow it here?

The first thing is we have to understand what is WPA is doing.  As it pertains to pitchers, and as Fangraphs and Baseball-Reference calculate it, the pitcher gets 100% of the credit, and the fielders are ignored.  This is not because these two sites believe this as a philosophy.  It is simply a data limitation: if no one is telling them how much impact a fielder actually had on each play, then we simply ignore the fielders altogether.  It’s not the best solution, but it is the same solution as any other metric used.  If you look at a pitcher’s OBP allowed or SLG allowed, the impact of the fielders are ignored, and the pitcher is credited with all of it.

Basically, WPA is simply based on a pitcher’s runs allowed, and his innings pitched.  (We’ll ignore discussion of partial innings for this article.)

Ok, so if WPA is based on a pitcher’s runs allowed and innings pitched, then how does one guy giving up 1 run in 8 innings and another guy giving up 0 runs in 9 innings cause a 0.53 difference in wins?

WPA has a second component: timing.  Let’s take a look at how the game progressed, which started with Halladay giving up an early run, and then, it was zeroes the rest of the way:

What we see right off the bat is that the Phillies chances of winning goes down immediately from the starting point of 50/50.  And as the game progresses to its eventual conclusion of 100% win for the Cardinals, then each out that Carpenter was getting had a bit more win value than Halladay’s outs.

Basically, Carpenter’s outs were feeding off each other, and he’s rolling down the hill toward a win.  Halladay’s outs were simply trying to stem the tide as much as possible.  It’s a case of Halladay setting Carpenter one step back, and then Carpenter getting two steps forward.  Time was running out on Halladay.  And it’s all because of that initial run allowed.

Another way to consider it is this way: imagine that each pitcher was getting a shutout through 8 innings.  Each one step forward was matched by the other guy, such that at the top of each inning, the chance of winning was always 50/50.  For every +0.05 inning for one pitcher, it was exactly matched back with his opponent’s +0.05.

And then you get into the 9th inning.  You are in the 9th inning, it’s still a 50/50 game, both pitchers have earned say +0.52 wins to that point.  Doc gives up a run in the top of the 9th, sending the Phillies chance of winning down to 20%.  That single run costs Doc 0.30 wins, taking him from +0.52 heading into the 8th, down to +0.22 heading into the bottom of the 9th.  Carpenter, with an 80% chance of winning shuts down the side, and he gains +0.20 wins for the 9th inning.  That shutout inning takes him from +0.52 at the start of the inning to +0.72 to end the game.

So, the final tally, as far as WPA is concerned, becomes:
+0.72 Carpenter
+0.22 Halladay

That difference is 0.50 wins.

In this respect, it’s exactly like a pitcher’s Won/Loss record: Carpenter gets the win, and Halladay gets the loss:
1-0 Carpenter
0-1 Halladay

The difference here is 1 win exactly, but that’s only because we don’t consider the rest of the team at all.  If we give Carp and Doc half-credit for their W/L record, then the gap between the two, using a pitcher’s Won-Loss record is 0.50 wins.  And that’s just like WPA.

WPA for starting pitchers simply hides in alot of number crunching what a pitcher’s Won-Loss record already does cleanly and obviously.

Now, does this mean that WPA for starting pitchers is redundant?  Not totally.  While the gap between the above two pitchers was 0.50 wins using WPA, you can see that they both ended up with a positive number.  Had the pitchers given up 1 run per inning, the gap between the two pitchers would STILL have remained at 0.50 wins, except now, each pitcher would have posted negative WPA.  In the case of the starting pitchers each allowing 8 runs through 8, they would have been standing at -0.85 wins each.  Even the winning pitcher would end up in the negative.

So, WPA for a starting pitcher is more like a combination of W/L record AND ERA.  The losing pitcher of a 1-0 game will end up with a higher WPA than the winning pitcher of a 9-8 game.  Which, one would think, sounds right.

The main problem is that it treats both metrics, the W/L record and the ERA, with similar value.  One metric shows the gap at a full win.  The other metric shows the gap as extremely close (1 run is 0.10 wins).  WPA takes the middle position, and gives the gap between the pitchers at 0.50 wins.

Nonetheless, it’s not very appealing to see a 1 run game resulting in a 0.50 win difference. But, that’s at the very essence of any game, that you have a winner and a loser, that you end up with a binary outcome.  It’s not like the game ended with 0.52 wins for Philly and 0.48 for the Cards, or that it ended 0.55 wins for the Cards and 0.45 wins for the Phillies.

WPA is a story stat, as studes calls it.  But, I don’t think WPA for a starting pitcher really adds much to the story.  I think it describes the Cards/Philly as a game perfectly.  But in terms of including the identity of the starting pitcher in that story?  I think WPA fails to tell that story.

Since WPA for a starting pitcher is very reliant on the context of whether the team happens to be winning or losing, it will overshadow the performance of the pitcher to some degree.  While it’s not as bad an indicator as a pitcher’s W/L record, neither is it better than a pitcher’s ERA.

And so, I say, goodbye to you, WPA for starting pitchers.

(Note that WPA for a relief pitcher is another matter entirely.  As untelling WPA for a starting pitcher is as a metric, it’s quite the opposite for a relief pitcher.)

Game Score however tells a much better picture. I said I had 4 versions, each one looking at a particular subset of a pitcher’s performance:
Game Score 1: Runs Allowed
40 + (6.4 * IP) – (10 * R)

81 Doc
98 Carp

Game Score 2: K/BB
40 + (0.4 * IP) + 3 * (SO – BB)

61 Doc
53 Carp

Game Score 3: FIP
40 + (2.5 * IP) + FIPcore

71 Doc
69 Carp

Game Score 4: Linear Weights
40 + (8.4 * IP) + LWTScore

74 Doc
101 Carp

So, depending how you look at their performance, we can see different stories being told here.  The runs allowed story favors Carpenter, but when you exclude balls in play, the story slightly favors Halladay.  Given the lack of consensus by the fans who watched the game, that kind of story seems much more accurate than anything WPA can tell us.

And I say, Hello Game Scores!