THE BOOK--Playing The Percentages In Baseball

Another variation on the question is how much a truly outstanding game would tell us about a pitcher.  For example, a young pitcher goes 9 IP against an average team with a line of 0 R, 14 Ks, 0 BB, and 3 H.  How likely is it that he’s an above-average pitcher (cat 4 or 5)?  I’d guess the probability is surprisingly high (though of course that depends on what one thinks is ‘surprising’.

Interesting research.  Is there any difference in the quality of teams that the different categories of pitchers have brilliant games against?  In other words are one run above average pitchers much more likely to have a brilliant game against a superior opponent than one run below average pitchers?

If most major league starters are capable of brilliant games against even the best of opponents, then what factors determine whether a pitcher pitches a brilliant game (i.e. the top range of what he is capable of) or a lessor performance?

Guy, Joe Sheehan wrote about this awhile ago:

http://www.baseballprospectus.com/article.php?articleid=5903

Thanks, David—interesting article.  To me, those results suggest that extreme performance, especially if demonstrated two or more times, can actually be a pretty significant indicator of talent.  It would be interesting to analyze a metric like James’ game score, which does a decent job of measuring pitching dominance with a single number, and see if there’s a benchmark at which we can say a pitcher who exceeds this level—perhaps twice—is almost certain to be a very good pitcher.

Another way to look at the stats is that a given pitcher has a 32% probability of being above average, but that rises to 43% if he pitches a brilliant game.  That seems like a pretty substantial increase to me.

I was really trying to emphasize (and quantify) the point that a pitcher’s performance in any given game is merely a normal curve centered around his mean with a SD of whatever the SD is of run scoring in a game.

I just read the BP article and it does not tell us anything at all as Joe just scratched the surface.  But I don’t think there is anything to tell us that we don’t already know.

I think that the BP article is misleading, suggesting that a great performance (or bad, or whatever) is indicative of some level of talent on that particular day.  I don’t like that characterization.  A pitcher’s performance in any one start, regardless of how that is defined and measured (ERA, RA, K/BB, BA against, etc.) is going to have random fluctuations around that pitcher’s true talent on that day, and there is no end to the magnitude of that fluctuation, of course.

If I pitched enough games, I would eventually have a 14/0 day!

The notion that Joe cited from the rotowire guy, that, “If a pitcher demonstrates a certain performance level in one game...” is a misleading one and flat out wrong.

A brilliant performance by a pitcher in any one game is MOSTLY plain old dumb luck.  Sure that pitcher is likely to be better than average since a better than average pitcher’s luck is better than a below average pitcher’s luck.  (Although the fact that we have more average and below average pitchers in MLB, actually mitigates that statement, as you can see with my little study.) But to characterize that brilliant performance as a “demonstration of some level of ability” is a mischaracterization of what is going on.  Sure, to some extent the “random fluctuations” of a pitcher’s performance include making good pitches (and good deicsions, etc.) more often than what is typical, but the underlying talent is still the same.

If I pitched enough games, I would eventually have a 14/0 day!

But that’s just the point:  you never would.  (Actually, I know nothing about your pitching ability.  Let’s say I never would.] Even one successful game in MLB would require being above a certain talent level, and a 14/0 game is very likely an indication of being good.  Sheehan’s point isn’t that an extreme performance means the pitcher was actually that talented (14/0) on that day, but rather that some performances are so great that only a pitcher of true talent >= X has a non-trivial chance of being that good even once. 

Let’s compare pitcher A, who’s true talent is 6 K/9 and 4 BB/9, and pitcher B, who’s true talent is 10/9 and 2.5/9.  You can estimate this better than I, but I’d guess that pitcher’s B chances of throwing a 14/0 games are at least 50x better than pitcher A’s.  Just looking at Ks, 14 Ks is +3.5 SDs for a true 6 K/9 pitcher, but just +1.8 SDs for a 10 K/9 pitcher. Basically, pitcher A—like me—will never accomplish 14/0.  Remember, we’re calculating Prob (14k) * prob (0 K), and pitcher B will be considerably higher on both.  [that assumes Ks and BBs are independent, which probably isn’t quite true, but whatever....].

There isn’t any argument here unless someone does not understand how one game is a sample around a pitcher’s means with SD X (for any one measure) and that the SD X is basically computed using the binomial probabilities.  I know you know that.

My point was that I WILL eventually throw a 14/0 game, given enough starts and so will everyone else.  Of course, my chances are less than a major league pitchers, etc.

The question is how much better or worse and then you have to compare that to the percent of each class of pitchers in the population of pitchers, which was one of the salient points of my original post - that the fact that there are many more average and below-average pitchers means that even though an exceptional pitcher is much more likely to throw a great game, when a great game is thrown, the chances that it is NOT ny a great pitcher is much more than it would seem from the performance itself.  Imagine a population of pitchers where only 1% are great and the rest are average or worse.  If a random pitcher throws a fantastic game, it is overwhemlimgly likely that that pitcher was NOT one of our great pitchers, unless the game was ridiculously great.

The point I was trying to make about the old BP article was that the great game is NOT an example of some “high level of performance that a pitcher is capable of,” although we are getting into semantics.  It is merely a sample of a pitcher’s performance, most of it being a “lucky sample” and an equivalent (X SD from the mean) lucky sample for a good pitcher is necessarily going to be better than one from a poorer pitcher, but there is NOT LEVEL of performance that ANY pitcher is not capable of, including me and you.  There is no magic number of SD that makes something likely, unlikely, impossible, etc.

This is just a Bayesian problem. We’re thinking about it incorrectly: It needs to be solved backwards. The question we want to ask is, given that a pitcher has a game with x Ks and y BBs, what is the probability that his true talent ERA is z? I guess the hypothesis of people searching for significance in these results is that the probability that such a pitcher is not very good is relatively low.

Mickey, your threshold is way too easy to surpass. Joe suggests two different performance levels: One game of 13 Ks, 0 NIBB, or three starts of 10 Ks, 0 NIBB. Mickey, since you’re more familiar with Bayesian analysis than me, do you mind calculating the probability that a pitcher who meets either of those thresholds would have a true talent ERA of 6.00, 5.00, 4.00, 3.00, or 2.00?

Remember, ERA = 5.40 - 12*(K-BB)/BFP.

Guy makes the same argument Bill James did.

The Sheehan BP article mislocated the original James discussion on “Signature Significance”; it occurred in his 1985 Baseball Abstract in the team essay on the Detroit Tigers, who had started the 1984 season 35-5. Most of the detailed discussion involved team performance, but James did lead off with the following (p.29):
“...in certain relatively rare cases of extreme performance,significant separations in data can occur in surprisingly small samples, including one game. A perfect example would be the game in which Roger Clemens struck out 15 batters without walking anyone. That game, in and of itself, presents credible, or “significant,” evidence that Clemens is a pitcher of some quality. Why? Because a poor pitcher never (almost literally never) has such a game. (...) We are in the habit of looking for direct significance; one game is never directly significant. No one game makes a man a proven pitcher. What small data samples can occasionally provide is indicative significance - the significance of the signature they bear. Another example would be a player hitting three home runs in a game; the occasion of a player who does not have at least 12-15 range home run power doing that, as Freddie Patek did, is extremely rare.”

[Note - when James wrote this, Clemens had a career 16-9 record spread over 2 seasons.]

Part of the difference between MGL’s sim and James’s proposition is that James is talking about really really great games as bearing significance.

Of course that is easily done, David, but I am not comfortable using true talent in terms of ERA to look at K and BB threshholds.  There is still a wide array of true talent K and BB within each ERA group.  I have to find out the average K and BB rates in each ERA class. I’ll do that by using Pecota’s K/9 and BB/9.

OK, here are the average BB/9 and K/9 for each class of pitchers, according to Pecota, where the average pitcher is defined as 3.0 and 6.0, and the average league ERA is 4.50:

True ERA True BB/9 True K/9

3.00 2.30 8.40
3.50 2.32 7.40
4.00 2.68 6.99
4.50 2.76 5.83
5.00 3.25 5.53
5.50 3.71 5.34
6.00 4.53 4.99

I’ll do the Bayesian calcs later.

James is correct and it is nothing earth shattering.  It is merely a (Bayesian) probability issue.

People do get hung up on sample size sometimes.  As I like to explain to some people who don’t have a good grasp of P&S a sample of 10 can be more enlightening than a sample of 100 or even 1000, depending on what you are looking for.  Getting 10 heads in a row is more indicative of a biased coin than 85 heads in 100 flips.  It is the same principal with “signature peformances.”

What you have to be careful of is using the Bayesian model, and incorporating the distribution of talent in the population.  If we know there are no great pitchers in our population, then no matter how great the performance, there is zero chance that the pitcher is great.  As I said, earlier, if there are only 1% great pitchers in our population, then depending on how great the performance, a performance anything worse than boggling, was probably NOT by one of those great pitchers.

B-R allows you to search for game scores.  Below are the 32 pitchers (49 games) who posted a game score of 97 or higher in a 9-inning game.  It’s definitely a good signifier of talent (Bobby Witt notwithstanding). But it may be true that something like 2 90+ games would be a more effective screen. 

Player, Date
Warren Spahn, 9/16/1960
Tom Seaver, 5/15/1970
Steve Carlton, 4/25/1972
Sandy Koufax, 9/9/1965
Sandy Koufax, 6/4/1964
Roger Clemens, 8/25/1998
Roger Clemens, 9/7/1997
Roger Clemens, 9/18/1996
Roger Clemens, 4/29/1986
Randy Johnson, 5/18/2004
Randy Johnson, 4/21/2002
Randy Johnson, 5/8/2001
Randy Johnson, 9/16/1992
Ramon Martinez, 6/4/1990
Pedro Martinez, 8/29/2000
Pedro Martinez, 5/12/2000
Pedro Martinez, 9/10/1999
Nolan Ryan, 5/1/1991
Nolan Ryan, 7/15/1973
Nolan Ryan, 7/9/1972
Nolan Ryan, 6/11/1990
Nolan Ryan, 4/26/1990
Mike Witt, 9/30/1984
Mike Scott, 9/25/1986
Mike Mussina, 9/2/2001
Mike Mussina, 8/1/2000
Len Barker, 5/15/1981
Kevin Millwood, 4/14/1998
Kerry Wood, 5/25/2001
Kerry Wood, 5/6/1998
Jim Bunning, 1964-06-21(1)
Jim Bunning, 1958-07-20(1)
Jason Schmidt, 5/18/2004
Hideo Nomo, 5/25/2001
Gary Peters, 7/15/1963
Ernie Broglio, 7/15/1960
Eric Milton, 9/11/1999
Don Wilson, 6/18/1967
Dennis Eckersley, 5/30/1977
David Wells, 5/17/1998
David Cone, 10/6/1991
David Cone, 7/18/1999
Curt Schilling, 4/7/2002
Catfish Hunter, 5/8/1968
Bobby Witt, 6/23/1994
Bill Stoneman, 6/16/1971
Bill Singer, 7/20/1970
Bartolo Colon, 9/18/2000
Andy Benes, 7/3/1994

That’s 1957 to date.

Two other thoughts:

1) A great game will be a somewhat stronger signifier of very good or great talent than the straight Bayesian calculation in the circumstances we care about, i.e. a young pitcher with limited ML track record.  First, because he’s at an age where talent level ‘X’ is even less likely than for all pitchers.  Second, because pitching a complete game—and maintaining excellent performance over 9 consecutive IP—is an additional signifier.

2) I wonder whether excellent game performances could be a useful indicator of talent for young pitchers, even beyond what we can infer from standard probability distributions.  If some young pitchers are inconsistent, beyond the random fluctuation we expect, then their peak performance—say, their best 25% of starts—could provide useful information.  It might suggest they have the potential for a higher ‘ceiling,’ a true talent level greater than indicated by their mean performance (i.e. they are in a sense pitching at two or more different true talent levels, depending on the day).

Has anyone ever looked at this?  Do young pitchers (in minors or MLB) show any more variance in performance than veteran pitchers?

I don’t like MGL’s list in #12.

When David said:
“Remember, ERA = 5.40 - 12*(K-BB)/BFP. “
he was quoting some research that I posted on Fanhome that was inspired by Guy.

In short, that is a best-fit equation using only K, BB, and BFP.

My best fit shows that when the K and BB numbers match, you get an ERA of 5.4.  MGL’s list shows the K numbers still exceeding the BB numbers, and the pitcher has an ERA of 6.00.

There were 207 pitchers:
- since 1994
- had at least 150 BFP
- their K/(BB-IBB) was at most 1.00

The results:
- average K/BB ratio of 0.91 (i.e., 10% MORE walks than K)
- ERA of 5.97

In MGL’s list, he has the true talent of pitchers with an ERA of 6.00, when the pitcher’s walk rate is 10% LOWER than his K.

On the flip side, there were 157 pitchers:
- since 1995
- had at least 150 BFP
- at least 75% of their games were starts
- their (K-NIBB)/BFP>=.15 (the elite… RJ, Clemens, Maddux, Pedro, Brown, Mussina, Schilling, etc)

Their actual ERA was 3.26, and my equation estimated 3.16.

Finally, if I remove the “starting pitcher” qualifier, I get 620 pitchers.  Their actual ERA was 3.11, and my equation estimated 3.12.

The equation works.

MGL is getting that list from Pecota, so it includes major and minor league pitchers.  If you’re looking at only major league pitchers who walk as many as they strike out, then you are selecting pitchers who probably do something else right, like getting groundballs.  Think of Kirk Saarloos or Kirk Rueter.

It wouldn’t surprise me if you take all professional pitchers who walk 4.5 and strike out 5 per game, you get something like a 6.00 ERA.

Tom, I was using the Pecota projections for all starting pitchers.  I simply looked at all SP with an equivalent ERA between 5.75 and 6.25 (and 5.25 to 5.75, etc.) and their average eqBB and eqSO.  Maybe my scale for BB and K were wrong.  eqBB is scaled to 3 per 9 innings and eqK is scaled to 6 per 9 innings.  If that is not the MLB average then I have to redo the scale.

My two favorite examples among the small sample sizitis are:

Bill Gullickson (2nd draft pick overall), 18K, 2 BB, at the age of 21:
http://www.baseball-reference.com/boxes/MON/MON198009100.shtml
Seven months later, he threw 6 WP in one game. 
Career ERA of 3.93, in a league of 3.83.

Roger Clemens, A ball, around 105 batters faced, 36K, ZERO walks, at the age of 21:
http://www.thebaseballcube.com/players/C/Roger-Clemens.shtml

MGL: can you spit out the full-line for your sample (BB,K,H,HR, etc) at both the high-end and low-end?

Re: Gullickson, I don’t think he is really much of a counter-example (if that’s what you were suggesting).  A league-average ERA from a starter is, of course, quite valuable, and at least through age 27 he was clearly an above-average starter (2.3 K:BB). 

But this kind of example does raise an interesting question.  In his first two seasons, during which he threw that game, Gullickson was a 7.1 K/9 pitcher.  But for his career, just 4.5 K/9.  Was the 18 K game a reflection of his real talent at that time, which subsequently changed (perhaps due to injury)?  Or was the K rate those first 2 seasons a fluke?  With pitchers, it will always be tricky to distinguish between flawed predictions and substantial changes in true talent.

He was the #2 pick, and he did make the pros at age 20.

If you are looking for stat lines:
- since 1995
- had at least 150 BFP
- at least 75% of their games were starts

Here they are, per 9 IP:

KWclass         n        H       ER      HR       NIBB       SO       szERA


a: .15-max    157     7.78      3.17      0.83      2.20      9.11      3.14 


b: .10-.15    469     8.80      3.97      1.03      2.56      7.15      3.95 


c: average    878     9.47      4.50      1.11      2.96      5.87      4.50 


d: .00-.05    532     9.97      5.12      1.21      3.61      4.87      5.02 


f: under 0    82     10.46      5.98      1.31      4.61      3.99      5.58

The BABIP are all very close, with a range of +/- 4 points from top to bottom.

By the way, “szERA” is strikeZone ERA, and it’s based on the equation in post 17.

As well, it looks like it breaks down at the high end, but that’s because of it’s linearity. 

I can do the following:
1. Start with K and BB per BFP
2. Use league averages for BABIP and HR/H to reconstruct the pitching line
3. Apply Markov or RC

I will end up with something close to the actual ERA.

15+ K, 0 BB, 0 HR:

Roger Clemens x 4
Pedro Martinez x 3
Randy Johnson x 2
Dwight Gooden x 2
Kerry Wood
Luis Tiant
Frank Tanana
Vida Blue
Curt Schilling
Sam McDowell
Mike Mussina
Nolan Ryan

So when Clemens did it in his 18th career start, it was rather more significant than the 4.92 ERA he came into the game with.  Ditto for Kerry Wood in his 5th career start (probably the greatest game ever pitched) and his 5.89 ERA entering the game.  (You could argue from Wood’s mere 116 career ERA+ that the only reason we think he fits comfortably on the above list is that he’s famously on it, but I don’t think anyone believes the career record is a measure of his original talent.)

To make the list better, I wouldn’t go with 15 K.  After all, a guy who allows alot of hits has more chances to get more K.  Remember, we are always going after 27 outs (in a 9-inning game), and if you have bad fielders who can’t get those outs, that increases your chances of getting a K.

So, it’s a technicality, but I’d say you have to do:
over 50% of the PA ended on a K
0% ended in a walk
0% ended in a HR
Minimum 24 PA

As well, when K are more abundant today than in yesteryear, your list is skewed to non-golden (bronze?) age of baseball. 

A further adjustment could be that instead of making it 50%, you make it 3x the league average. 

(If you want to get real picky, make it that the Odds are 5x the league odds.  So, if the league rate is 16.667, or odds of 0.2:1 of getting a K, then your threshhold is 5x that, or 1:1, or 50%.)

No pitcher on the list who debuted before the 60’s.  Had that never happened before, or are you just looking at games in the retrosheet era?

Tango:  First, 50% of PA is way too high.  A typical 15-K, 0 BB game wouldn’t even qualify (assuming a .300 BABIP).  And this is much more complicated than we need.  Suppose two pitchers have BABIP/DER of .270 and .310—a huge spread.  That’s only a difference of about 1 PA in one of these games, so maybe .4 Ks.  Not worth worrying about. 

Historical era is a potential problem, though I’m not sure how big a factor it is.  Some of the recent surge in K-rate is result of using more short relievers, and complete games are down dramatically. 

I still like the idea of using Game Scores, since it sums everything into a single score and B-R makes it accessible.  Unless we think it really does a poor job of capturing the kind of pitching dominance we’re looking for.

There is no doubt that once you get into the realm of ridiculously great games, like 15/0/0, that primarily only great pitchers will populate that list.

Using the Bayesian model, what happens is that the probability of each class of pitchers throwing the game in question overwhelms the distribution of pitchers of each class in the population for extremely good and bad performances.

For example, if a certain good performance is twice as likely by a certain good class of pitchers than a certain worse class of pitchers but the worse class has twice the presence in the population as the better class, than the chances of that performance coming from one class or the other is the same (50/50).  However once the chances of the better class exhibiting that performance is 10 or 100 times greater.

So the “signature performance” dynamic that James talks about has a very different “conclusion” depending upon the threshhold of that performance.

Even if 50% is too high a threshhold (6 of these pitchers would drop off the list), then choose whatever appropriate threshhold you want.  The key is to use per PA.

Among the exactly 15/0/0 crowd, Mussina faced 27 batters and Ryan and Pedro faced 34.

***

Fun facts about our gang of 19.  This is the list that Eric is reporting (since 1957):

http://www.bb-ref.com/pi/shareit/PwVt

They allowed 96 hits on 281 BIP (.342), with only 15% extrabase hits.  Batters may have decided to chop their swing in the face of such dominance.

My favorite fact about those 19 games is that Clemens did it twice for the Sox, 12 years apart, and Gooden did it twice for the Mets—in consecutive starts.  Talk about careers in a nutshell.

It would be great to get a list of comparable games from before 1957, with an appropriate lower K threshold.

So, bad fielders who can’t get outs increase the chances of a high-K game?  So, if you wanted to smash the all-time K record, you’d put someone like Don Baylor at 1B and have him drop an easy foul popup?

It’s funny—I agree with both Tom and Guy about using BFP as the denominator.  I always calculate K and BB rates that way when I have the BFP data, and I’m often chagrined to see how rarely that adjustment matters (more so in small sample sizes where BABIP can be crazy).

Here’s an interesting variation (I think) on MGL’s original question: 
Marcel (4.98) and PECOTA (5.12) both say that Jason Marquis has a true talent ERA of about 5.00.  So far, he’s 1.70 in 7 starts, 47.2 IP.  With this additional information, what is the chances that Marquis is really a 5.00 ERA pitcher (in that park, in 2007), vs. the chance that the systems underestimated him?

The question could be rephrased as:
“What is the chance that a true .350 wOBA pitcher will pitch as .220 with 200 PA?”

That performance is 3.6 SD from the true .350.  In order to get him down to 2.0 SD, we have to have a wOBA below .300, which would make him one of the best pitchers in baseball.

However........ his BABIP is .193.  And, as we know, that would get heavily regressed.

So, Marquis .220 wOBA or 1.70 ERA would typically presume a typical profile.  Better than avg K/BB, better than avg HR, better than avg BABIP.  That is hardly the case here.

If we look at it component-wise, I would bet that Marquis forecast would likely change to “average” pitcher.