THE BOOK--Playing The Percentages In Baseball

A good case study for looking a an appropriate threshold would be Francisco Rodriguez’s 2008 season relative to the top closers over the last few years.  Have to account for number of appearances as well.

This is where I feel helpless as I don’t know how to pull this out of a database.  Are there any good primers on this sort of stuff, namely for baseball?

K-Rod’s 2009 game logs, sorted by WPA:

http://www.baseball-reference.com/players/gl.cgi?n1=rodrifr03&t=p&year=2009#pitching_gamelogs::42

And this one is also sortable:

http://www.fangraphs.com/statsd.aspx?playerid=1642&position=P&season=2009

B-R.com is preferable, because Sean shows you the game situation when the pitcher entered.

His 2009 is not a good test case, as he had no appearances between the two threshholds.  You should focus more on middle relievers.

His 2008 is more interesting.  He had one where he had a runner on 2b, 1 out, ahead by 3, let that runner score, and finished the game.  That was +.056 according to both Sean and David.  LI entering the game was 1.3.

Is that a Shutdown, or not?  It’s close.  Hence, anything between .050 and .070 is the gray area of discussion.

My suggestion for the threshold:  whatever value will give us approximately 1,202 shutdowns last season.

In 2009, there were 1202 saves in MLB.  Somebody (essentially) has decided that there were 1,202 relief pitching performances that were worthy of special notice last season.

I suggest that we take them at their word, and find the 1,202 best relief pitching performances by WPA.  If nothing else, it would give us numbers that we can directly compare to save totals.  For example, did Fernando Rodney really have 37 of the 1,202 best relief performances of 2009?  Is he over-rated by the save, or under-rated?

I suppose if that cutoff gives us a number like .0594 WPA, it would be reasonable to round it off and set the threshold for a “shutdown” at .06 instead. 

This method might set the threshold far too low, but I think it would be a good starting point.

That’s seems like good enough suggestion, though we should amend that to saves+holds. 

Using Crash’s results (using .05 as the threshhold) and we had 143 shutdowns for the 09 Phillies, and 174 for the 08 Phillies.

How many saves+holds are there?  100 or something?  So, we may need to go all the way up to .070.  Like I said, you CANNOT go above .080, as you will ignore valuable shutdowns.

Maybe Crash can help us out here with some more granular data.

Good point.  The ‘09 Phillies had 44 saves and 81 holds--125 total.  So it looks like the 143 “shutdowns” is close, and raising the threshold a little bit, maybe to .06, might improve it.

There were 1,202 + 2,155 = 3,357 saves plus holds in MLB in 2009.

Good stuff.  Using Sean’s Play Index, these are the number of positive, and negative games at each threshhold:

.050 3952 - 2395
.060 3549 - 2182
.070 3273 - 2010

At .066, there are 3365 games.
At .067, there are 3343 games.

So, to match saves+holds, the threshhold would be between .066 and .067.

Great suggestion, thanks!

***

There’s an extra added issue that Sean and David park adjust the WPA numbers.  That would need to get resolved, or neutralized, as you can’t start saying it’s a shutdown in one park and not another.  Otherwise, someone is going to come in and talk about the identity of the teams next.

There were 276 games with WPA of between .060 and .069.  These are the guys with the most number of games in that range:

n Player
8 Heath Bell
6 Mike Adams
6 Jonathan Broxton
5 Luke Gregerson
5 Eric O’Flaherty
4 Trever Miller
4 Sean Burnett
4 Matt Guerrier
4 Eddie Guardado

So, I would say that if we look at Heath Bell’s 2009, we can get a bit of insight.  I’m going to guess it’s because of PETCO, and so, he got a lower run environment that affected his WPA.

Let’s see… yup, that’s what it was.  He came in the 9th, ahead by 2, and all 8 games at home. 

I think that settles it then.... threshhold is .060 WPA.

Excellent.  Shutdowns and Meltdowns it is.  .06 WPA.

I like this stat!

This discussion reminds me of a husband and wife deciding on a name for their baby.

http://www.fangraphs.com/blogs/index.php/shutdowns-meltdowns/

And just like that, FAngraphs has implemented Shutdowns and Meltdowns:

http://www.fangraphs.com/leaders.aspx?pos=all&stats=rel&lg=all&qual=y&type=3&season=2009&month=0

Last year, David Aardsman had 46 shutdowns and 6 meltdowns.  In 1983, Bob Stanley was 39-13.  In 1977, Goose Gossage was 40-15.

DAvid said he’ll get it on the player pages this weekend.  For now, you have to go year-by-year.

Enjoy!

So something like Shutdown % would be a decent quick-and-dirty look, right?

Top 5 teams in SD%:

Tigers, 72.4%
Padres, 71.4%
Mets, 71.1%
Rays, 71.0%
Nationals, 70.6%

Only teams this year above 70%.

or Shutdown to Meltdown ratio?  Not every meltdown is a blown shutdown, right?
Great stuff!  Great work!!  Hope it catches on ms.
vr, Xei

Is there some reason these stats need to be symmetric?  Seems like a lot of thought went into the SD threshold, but a lot less into the meltdown threshold.  Which then leads to the result in #13, where you really want SD / SDchances.  But no one has yet calculated, or really, even defined SDchances. (enter with WP < 0.94 doesn’t really work here)

Reading the fangraphs thread, I see I’m likely to get hit for not bringing a suggestion along with my critique.  I will note a need to do some more research to figure out what situations fall close to the treshold, unfortunately I don’t have time right now.  My apologies.  Hopefully tonight.

Also, every single releiver had at least one MD in 2009, for instance.  That may be desirable, since almost all closers rack up at least 1 blown save, which seems like the motivation here.  Lidge and Gagne being the very notable recent exceptions on this score.  Tango notes that the SD/MD ratio is 1.6, which does seem low to me.  I’d expect more like 2/1, which would be similar to out% for pitchers.  OK, that’s not a great motivation, but it’s something.

Has there been any thought about making a similar stat for starting pitchers to replace wins/losses/quality starts/etc.?
The cutoffs would be very different, maybe something like < 0 = "hurt the team", > .15 = “helped the team” or something like that?

Adam, I’ve been thinking the same thing. I may be easiest (tho not best) to redefine a quality start with WPA. Then we don’t have to worry about assigning a loss and a win in every game.

@ Adam F

i feel like it would be a lot harder to do because of the length of time a starter is usually in the game.  it also have to deal with that old bit about pitchers “pitching to the score”, ie performing better in tight games and letting up a bit in blowouts. for example, look at the following lines…

cliff lee - http://tinyurl.com/38tasld
7ip, 0r, 3h, 0bb, 8k, .420wpa

colby lewis - http://tinyurl.com/38tasld
9ip, 0r, 3h, 1bb, 10k, .645wpa

roy halladay - http://tinyurl.com/32qo2ad
9ip, 0r, 3h, 1bb, 6k, .246wpa

clay bucholz - http://tinyurl.com/35vt5hm
5.2ip, 4r, 8h, 3bb, 2k, .038wpa

the first three are very similar.  colby lewis and lee both get no decisions, but both went out and had great games.  they both had high wpa because the score was so close.

conversely, look at halladay’s line.  almost identical to the line lewis had, but because his team piled on the runs he gets a bit above a third of the wpa. 

and then there is the line for bulchoz.  not a good outing, but because his team scored some early runs he walks away with a positive wpa, and during his rough innings his team still had a 70% shot at winning.

wpa would be dependent on your team’s offense and when they scored the runs. it would probably play out similarly to the win loss record, really.

sorry, i probably should have thrown up a “counter” case for the bulchoz game too.

chad billingsley - http://tinyurl.com/2wyg8ab
6ip, 4r, 7 h, 2bb, 3k, -.162wpa

billingsley’s line is almost identical to the bulchoz line, but because he allowed the runs in the first (and before his team could plate a few) his wpa is exactly 20% lower. 

@JD Sussman

calculating a pitcher’s game score is preferable to the quality start notion i think. it is not perfect, but it tells you about how good the game was quickly and is independent of your own team’s offense.

Right, there was no thought for the meltdown.  We can change that of course.  It all depends what kind of ratio we are looking for.  As it is, the bad relievers end up with a 1:1 ratio, so if you do shutdown minus meltdown, you get “0” for a replacement-level reliever, which works, sort of.

I’m open to hear more.

As for starters, because of the starter/relief issue, and the quantity issue, you’d have to change the baseline.

Say, a quality start might be something like WPA minus IP*(-.01) is the “newWPA”, and then set the threshhold as +/- .05 or something around newWPA.  Something like that.

I had some ideas for using WPA, LI and IP for awarding wins and losses a while back, but tango hated it.

http://www.fangraphs.com/forums/topic.php?id=493#post-966

I’d really like to see a NetSD column.  It’s easy enough to calculate after I snag the Excel, but I like that it really shows who stinks (it confirms my belief that Jesse Crain is one of the worst relievers in the league).

If anyone’s interested in looking these up, which level of save-style situation do closers get a shutdown for, assuming they start the inning clean and don’t give up the lead?

1-run?
2-run?
3-run?
4-run?

Does home/away matter at all?

Are there any situations where a setup guy could pitch one inning, allow a run, and still earn a shutdown, like maybe up 3 in the 8th?  (Or avoid a Meltdown?)

ceph: re-reading it, I can see why I didn’t like it.  My comments there stand as accurate.

The idea I have here for starters, of doing WPA + .01*IP is good and logical.

I looked this up and giving up a run while getting 3 outs is a meltdown only if you start between up 1 run (i.e. you allow the tying run) and down 1 run (you go from being able to tie with one swing to needing more).  Also, it can be meltdown in some early innings under other score situations, where the three outs aren’t so meaningful.  Contrary to my earlier worries, this seems pretty reasonable.

I’m pretty sure only getting the last out lets you give up a run and earn a shutdown.

tango: i still disagree but that’s fine. im not trying to re-argue it, i was just hoping someone would take the idea and go with it, using WPA to assign win/loss/save - because i think that’s a good thing to use for it.

shutdown/meltdowns are a good save/hold/blown save replacement, i think.

now if someone could fix Wins, Losses and Quality Starts...i’d be happy. I hate all of them.

The number of shutdowns and meltdowns and games for the average team:

118, 73, 475

That sets the % of shutdowns and meltdowns as 25% and 15%, with 60% of the relief games being neither.

The bad reliever is going to have as many shutdowns as meltdowns, meaning that if you take the difference, a difference of zero marks the replacement level.

Are we happy with that, or do we want the number of meltdowns to be smaller, and truly represent a bigger drop than just a .06 win difference?

ceph: the argument I had with you was when you included LI, since you’d be double-counting it.  I already have the idea to use WPA for starters to convert to “good” and “bad” starts, as I posted twice above.  Do you have an issue with that, just doing WPA + .01*IP ?

***

“I’m pretty sure only getting the last out lets you give up a run and earn a shutdown. “

Yes, getting the last out of the game without giving up the lead let’s you do ALOT of bad things in an inning and still earn a shutdown.  For example, you can enter the 9th up by 2, walk 4 batters in a row, and then get three outs, and you earn a shutdown.  That’s the same thing as giving up a leadoff HR and getting three outs.  Or just getting three outs.  In all cases, you came up with a non-zero chance of losing the game (say about .07) and finished the game with no chance of losing the game.

i think that any wpa win/loss concept will need to find a way to minimize the effect of the score on your performance.  referring to my example in post 19, the line for halladay and lewis differed by only 4 k’s, and as i understand it wpa shouldn’t change much because of that difference.

that leaves a 40% change in win probability which is largely attributable to the score of the game. pitching well in a close game should hold an extra value, but 40% is ridiculous. 

tango, i think that your idea is a decent start, as it would help polarize good performances from bad.  if i may, i would propose a change in the formula though.  instead of a linear change by ip, how about something that gives you a bonus based on the league average for ip.  say the average is 5ip...you would get +/- .01 for going 6/4 innings, +/- .022 for going 7/3 innings, +/- .035 for going 8/2 and +/-.05 for going 9/1.  the numbers would obviously need fine tuning, and maybe an offset to start with.

additionally, i would propose that instead of calculating that pitcher’s wpa to determine a w/l based on the game he is in, base it against a neutralized standard. make that standard based off of what a league average run distribution through a game might be.  i am making up some numbers again, but maybe make it that the pitchers team scored 1 in the first, then 0,1,0,1,0,1,0,1 runs in the subsequent innings.  this could potentially eliminate the massive swings i showed in posts 19/20.

Tango:

My writing was poor.  I meant to indicate the ONLY way to get a shutdown while giving up a run is to also record the last out (I’m not 100% sure of this, I haven’t found a WE table with all the out states and score states, the links on the Book Wiki don’t include that).  I didn’t mean to imply that there was something problematic about being able to record just the last out, give up a run and get a shutdown, because, the last out is pretty d*mn important. (though defenders of the save might say the same about the last inning)

dutchbc:

Why would you bother?  Just use FIP or xFIP or WAR and convert to a record if you must.  You’re trying to split the difference between context-neutral and context-driven, it seems kind of pointless to me.

Larry: check Amazon’s Look Inside for The Book.

Or check this: http://tangotiger.net/wins.html

@ larryinla

xFIP/FIP/WAR are all based on just walks, k’s, and hr.  i think this is excellent for measuring a pitcher over the course of a season, but not as useful on a game by game basis.  i want to minimize the effect of the pitcher’s offense from the w/l, not remove all in game context.

i still think game score is a decent way to judge wins/losses.  just set upper/lower limits so that all starters end up with similar totals of wins and losses.  i was initially against using wpa for starters, but because a bunch of people have brought it up i am trying to think of ways to make it work.

TT:I already have the idea to use WPA for starters to convert to “good” and “bad” starts, as I posted twice above.  Do you have an issue with that, just doing WPA + .01*IP ?

What’s the reason for .01? Did I miss that?

A lot comes down to philosophy behind this all. I’m serious - MD/SD is meant to replace Sv/BS. But what I’d prefer is a stat that evaluates the game as a whole - all pitchers get the same look. So you award a Win to whoever gave the most value to the game. A Save to whomever recorded the highest leverage performance. etc.

Not every starter deserves a win, or a loss. sometimes all they did was eat innings. That’s valuable in the long run, but not necessarily worth a W, L or Sv…

That’s sorta the thing, WPA + .01*IP would replace QS...but that’s only half of what people want. They want to know which pitcher got the W - who pitched the best and added the most value.

That’s what I was trying to get at when I came up with the formula I used. All pitchers got a look, everyone had a chance for a W or an L, and a Sv if a reliever did an exceptional job (basically top Shutdown, philosophically).

But really, i’d take anything to replace W and L. And QS. Just thoughts I have, not an actual argument for/against anything.

A replacement-level pitcher will have a -.09 WPA per 9 IP.  So, to get that pitcher to a zero, you add .01 wins per IP.

A little late to the game here, but I’m seeing a flaw with the Shutdown/Meltdown logic, which I also posted in a comment at Fangraphs. 

Shouldn’t the Shutdown/Meltdown be based only on the team’s change in WPA while the team is on defense? In other words, wouldn’t it make more sense to disregard changes to WPA while the player’s team bats? He’s not influencing that by his pitching, and it shouldn’t show up in the Shutdown/Meltdown as if he did. 

Consider the following fictitious scenario.  A reliever enters the game in the sixth with a 57% win expectancy.  He give up two runs, leading to a 45% win expectancy.  The offense scores three, boosting the win expectancy to 69%.  He comes back out and walks the lead-off man, dropping the win expectancy to 64% and prompting a pitching change.  He’s been completely ineffective, but managed to get a shutdown by virtue of the run support of his offense.  If you looked only at what happened while he was on the mound, he had a WPA of -0.12 in the sixth and a WPA of -0.05 in the seventh - clearly a Meltdown.  Right?

You misunderstand WPA.  The player ONLY gets tracked when he’s involved in the play.  When his team is batting, the pitcher is not getting any WPA.

i am pretty sure that the stat is only based on what that individual pitcher’s contribution to wpa would be. 

the only effect that the team’s offense would have is if they score some runs and make his next inning a higher/lower leverage appearance.  either way, that pitcer would still have the previous inning’s performance as part of his wpa contribution.

or maybe i am just misunderstanding your question.