THE BOOK--Playing The Percentages In Baseball

Is this method “mathematically correct,” or is it good because it’s supported by the empirical data? My impression is the latter.

It is “mathematically correct”, when the league mean is .500, and the distribution of variables involved follow a normal distribution, and not “too tight”.

I did a test once on Fanhome (long-since lost, but perhaps existing somewhere) that showed that using more real-life data, that this model would be off by around .002 OBP points.

That is, I played Pedro Martinez against a group of MLB players, using a reasonable approximation of what the talent distribution was for those players.  I then did a head-to-head of Pedro against each one.  I should have come back with exactly Pedro’s starting rate, and I was off by .002.

It was from these tests that made me find that the Odds Ratio method only works on single confrontations, and not “group level” results.  So, a .600 team facing a .400 team will NOT necessarily follow the Odds Ratio (log5) process.

To the same extent then, Pedro does not have a “true rate” of .280 or whathaveyou.  That number is really a compilation of his tools in competition with other players.  If you look at things more granular (pitch by pitch), you will likely end up with something different.

Also in that Fanhome thread, someome proved something to that effect by using runners with a speed of 10m/s against a runner of 12m/s, and how the odds ratio method may guess the result might be a win of .900 for the 10m/s, you can easily contruct a model where the true answer is 1.000, simply because of the distibution of running times.

This is a very fascinating topic.

Tango,

Could you explain why the result would work for players but not teams?  Because for players, the result is really a “group” result on pitches.  There’s no real “atom” of performance, so I don’t understand why it would work for players but not teams.

It works better for players than teams, since all players are close to the mean, relative to teams.

Player’s OBP are in the .280 to .430 range, while teams are in the .350 to .650 range.

Simply put, the true OBP SD is around .030, while the true team SD is around .060.

As well, the “atom” of performance (great word!) is closer to the player than to the team.  That is, the atom of the performance would be a single pitch.  One to 8 pitches gives you a PA.  For teams, you are looking at W/L records made up of around 300 pitches total per game.

***

If you remember when I talked about the true talent level of sports league and the home-side advantage, I talked about “confrontations”.  You will find, if you work it out (and maybe I’ll do just that), that the atom confrontations won’t necessarily match the Odds Ratio method.

***

Ok, maybe I will do it.  Let’s make the confrontation level the PA, and that you have a current team that wins .499 of them and loses .501, each time.  And there are 80 such confrontations.  Such a team will win .421 times.

You also have a team that wins .501 and loses .499.  They’ll win .579.

Now, the Odds Ratio method would say that these two such teams, facing each other, should win .655 times.

However, the Odds Ratio method would make their single-PA confrontation a .502/.498 affair.  80 such confrontations would mean a .723 win percentage!

Quite a difference, wouldn’t you say?

I think the secret in unraveling this lies with tennis, where the single confrontation is much better handled, as well as having an extremely wide level of talents.  Where is the Retrosheet of Tennis?

I believe that the mathematically correct method of determining the outcome of an event with two adversarial forces is the z-score method. This method assumes you know the rate of outcome for each opposing force and the average rate and standard deviation of all outcomes and that the rates are normally distributed.  For Tango’s example above of a batter with a .400 OBA batter and a .250 OBA pitcher with a league average OBA of .333 and a standard deviation of .030.  The batter’s z- score is 2.233, the pitcher’s is equal to -2.777.  The two z-scores are added together for a result of -.544.  The expected outcome would then be .333 + (-.544 * .030) or about .317.  The Odds Ratio Method was developed for a completely different statistical problem and although it is not a bad estimator for the result of adversarial confrontations it has no mathematical basis.

Okay, I think what’s happening is that the odds ratio happens (by coincidence?) to work for some cases, but it doesn’t have to.

Imagine a basketball league of 11 teams.  These teams play games of infinite length, so the better team *always* wins.  So the 11 teams have winning percentages of 1.000, .900, .800, and so on.

Obviously, the odds method doesn’t work here—the actual head-to-head probability is either 1.000 or .000, but the odds method predicts something in between.

Tango, what are the assumptions that make the odds method work? 

Maybe it only works well where the standard deviation of a team’s performance is large compared to the difference in performance between opposing teams?  That would explain why the more atoms there are, the worse it works.

Peter,

That .030 is the standcard deviation of what?

In post #4 Tango gave .030 as an example of the SD of “true” OBP.  I used that for the example for both pitcher and batter OBP distributions.  You could use separate SD’s for the batter OBP and pitcher OBP to calculate each z- score if each distribution is different.

Maybe I’m remembering this wrong, but I thought the method used in The Book was to take player A + adversay B - mean.  Gets you the same result as Peter’s method:  .400 + .250 - .333 = .317.  But a little easier.

In the case where the batter and the pitcher are taken from the same distribution, the z-score method simplifies to Guy’s calculation. In other words, it’s not just coincidence that they give exactly the same answer.

The difference between .308 and .317 is not much. I wonder what would happen if we looked at more extreme cases—that would clarify the differences between the approaches.

What comes to mind for me is the park factor for HR in Coors. In 2002 Juan Pierre had 592 AB for the Rockies, and hit 1 HR. Using a linear park factor we may end up with a “true” HR rate that is negative. On the other hand a purely multiplicative park factor is not correct either.

Tango said on another thread that the odds ratio method is useful here. That at least will always give a positive answer. How would the z-score method work in this context? Is it even appropriate? Because it deals with normal distributions, there’s always the possibility of a negative answer, but in practice that may or may not be a major concern. I think Pierre is a good test case.

Here’s an alternate formula which gives the same results and is more succinct:
(SF-SFA)/(SF+A-SA-FA)

S[uccess]= probability of outcome for player or team
F[ailure]= probability of opponent allowing that outcome
A[verage]= average probability of outcome.

Ultimately this formula requires about the same number of operations as Tango’s, if you stick to the case with a common league average. (To handle Tango’s last case in which eg batter and pitcher are from 2 different leagues, my formula would have to be applied iteratively and so involve more computation.)

There are no underlying assumptions about normal distributions; the amount of dispersion around the mean doesn’t affect the accuracy. Mathematically this is just an algebraic derivation of multiplying joint probabilities [where probabilities S,F and their complements are expressed as offsets from average: A+s,A+f,1-(A+s),1-(A+f) ]

Apart from not exactly knowing the true probabilities, how does this formula break down when applied to baseball?
1) it depends on binary valued outcomes.
2) it assumes the probabilities are not emergent.

If I wanted to model the outcome of Pedro Martinez ‘99 vs Mark McGwire ‘99, I wouldn’t want to stop with predicting OBP. I’d want to construct a whole batting line with probabilities of home runs and strikeouts and walks. Notice the infinite number of ways you can slice the outcome pie - getting on base/not getting on base; hitting a double/not hitting a double; striking out/not striking out; hitting a ground ball/ not hitting a ground ball and so on. You need not get consistent results by applying the odds ratio method to all these different probabilities and then rolling them up.

I think I have seen a couple of general studies which deal with the sample size problem by defining groups of pitchers and groups of hitters; these studies suggested that the odds ratio method “works” when it comes to predicting eg batting average, but this doesn’t go very far in proving how well the method works for individuals.

Joe, in your formula, letters that are next to each other are multiplied, right?  So SF equals Success Rate times Failure Rate?

Suppose you’re doing free throws.  A hits with probability .650.  B hits with .700.  If you have a sudden death tournament, where the winner is the first one to hit while his opponent misses, then the chance of winning is exactly predicted by log5.  Right?

Now, suppose you make it best of 5.  Now, B has a higher chance of winning than before.  Make it best of 55.  B now has a very, very high chance of winning, probably more than 99%.  Make it best of 5555, and B will win pretty much 100% of the time.

So log5 works for best-of-1, but not best-of-anything-else.

Doesn’t this prove that no one formula can work for every sport in every situation?  That is, you have to figure out a log5-type formula empirically, based on what works.

I didn’t do the intermediate probabilities in #13, but I’m assuming from Tango’s comments in #4 that log5 won’t work for the best-of-more-than-one case ...

If they each take 1 shot (and continue until someone wins), the .70 guy will win .557 according to the binomial or log5.

If they each take 10 shots, the .70 guy will win .603 according to the binomial.

What does log5 say?  Well, the .70 guy in 1-shot will win .842 in 10 shots, if facing a .500 opponent.  The .65 guy in 1-shot will win .771 in 10 shots if facing a .500 opponent.  So, what happens, according to log5 (Odds Ratio Method) if the .842 facing the .771?  That’s a .613 result, which is different from the “true” result of .603.

I should note that when I say the .70 guy will win .842 in 10 shots against a .500 opponent, I mean exactly .500, and not a distribution of opponents such that their mean is .500.

If a .70 faced a .400, .500, and .600 opponent equally, and they each took 10 shots, then his record would have been .824, not .842.  Similarly, a .65 facing the same three opponents equally would have performed at .754, not .771.

And the result of .824 against .754, according to the Odds Ratio method?  0.604, which compares rather favorably to .603.

I think if I chose a normal distribution of opponents, and if that distribution was “wide enough”, that I would have gotten a match between the binomial and the Odds Ratio method.

Cool, thanks, Tango.

So are we any closer to having a summary of when the Odds Ratio method works, and when it doesn’t?

It works:

-- for free throw contests, even best-of-10s against players of different calibers
-- for players better than teams

But it doesn’t work

-- for infinite-length games where the better team always wins
-- for teams as well as players.

Is that right so far?

Extending the uniform distibution of opponents to .300, .400, .500, .600, .700, I get a result of .590 using the Odds Ratio method, of a .70 against a .65 player.

If I weight the opponents as 1,2,4,2,1 (making it look more “normal"), the Odds Ratio method says .598.

If I weight the opponents as 1,3,8,3,1 (making the opponents more tighter around the .500 mean), the Odds Ratio method says .602.

As you can see, I’m sure there’s some mathematical basis upon which the Odds Ratio method will work exactly.  The standard deviation of that last group of opponents is .097.

Using just my guts and what I’ve posted here, I’d say that it “works” the best when you are closest to:
- mean being .500
- the difference between opponents is “close” (say their difference is the same as the SD of the league they are in)

***

Here’s another test.  What happens when a .60 faces a .40?

The binomial says a .60 v .50 with 10 throws will win .688.  Obviously, a .40 v .50 will win .312.  Odds Ratio says a .688 v .312 will win: .829

A .60 v .40 will win .836 according to the binomial.

***

The binomial says:
.6 v .6 = .500
.6 v .50 = .688
.6 v .40 = .826

An even distribution of such opponents means .6 v average opp = .671

And of course, .4 v average opp = .329.

Odds Ratio of .671 v .329?  0.806.

I don’t see how the “odds ratio” applies to Phil’s hypothetical game. These shot probabilities are probabilities of independent events - the separate shots - they are not probabilities of the outcome of a jointly determined event.

It appears to me that Tango and Phil have gotten confused here. The outcome of Phil’s “10-shot” game between a 70% shooter and a 65% shooter has nothing to do with the mean percentage of the shooter population; their outcome is the same whether that population mean is 5, 50 or 95%. A 50% shooter is not a .500 opponent.

Tango’s binomial calculation of win probability in this game looks correct; it won’t make any reference to a 3rd value (a mean). Note that there is an inflection point after 10 shots; Tango’s binomial calculation reflects the fact that ties after 10 shots are resolved by switching to the 1-shot sudden death rules to resolve the percentage of games which are tied; the relative chances for winning change at that point on, for the reason Phil noted before. Also, the chances of ties needing sudden death resolution vary greatly depending on how closely the opponents are matched. in this game, the 70% shooter will be tied with the 50% shooter after 10 shots about 11% of the time; the 65% shooter will be tied about 14% of the time.

The odds ratio doesn’t apply to this game, which is why it isn’t working.

****
Studes: yes

Maybe I should bump this out to 80 shot games, to reduce the ties as much as possible.

***

Going back to Joe’s post:

No, we’re not confused.  Let me start from the beginning.  We are starting with the “atom”, and simply creating a distribution of successes, given the success rate, using the binomial.  So, a .70 shooter will be successful a certain number of times 0, 1, 2… 10 times.  A .65 shooter has his own scoring distribution.

Given these two scoring distributions, we can then use basic probability to see how often each win will.  This is our head-to-head win probability.

The important part is this: we’ll know everyone’s head-to-head win probability, but we won’t know their “atom” rates.

We can run our .70 shooter against our .65 shooter, against our .60 shooters, our abundance of .50 shooters, our .40 shooters, our .35, 30, and .70 shooters.  Against each head-to-head, we come up with head-to-head win probabilities.

The weighted average of our head-to-head win probabilities gives each shooter his own “game winning percentage” (as opposed to his atom shooting percentage).

So, given that we have the game winning percentage for each of our players, the question is: “can we use the Odds Ratio method to predict the game winning percentage for any two given shooters?”

That is, will we get the same result if
- we use the Odds Ratio method, which knows only about each shooters “game winning percentage”
- we use the binomial method, which knows each shooters “atom shooting percentage”

My contention is that, yes, we can, under certain conditions.  Those conditions depend on
- the closeness of the metric to .500,
- the game winning percentages following a normal distribution
- that distribution being “Wide enough”
- each team being “close enough” to each other

Now, it’s simply a matter of testing to quantify each of these variables.

Tango,

here’s what I think is confused (from entry 15):

“What does log5 say?  Well, the .70 guy in 1-shot will win .842 in 10 shots, if facing a .500 opponent.  The .65 guy in 1-shot will win .771 in 10 shots if facing a .500 opponent.  So, what happens, according to log5 (Odds Ratio Method) if the .842 facing the .771?  That’s a .613 result, which is different from the “true” result of .603.”

The .70 guy is a 70% shooter. By binomial, he’ll win .842 against a 50% shooter. This is where you misleadingly say .500 opponent. Similarly the 65% shooter will win .771 by binomial against 50% shooter [not .500 opponent].

Then you use the odds ratio method as if those percentages of .842 and .771 were achieved in the framework of a .500 league, ie as if it just so happened to be the case that a 50% shooter also would be a .500 player in this league. But those absolute winning percentages of .842 and .771 apply just the same in a league in which the average shooter happens to be a 30% shooter or a 70% shooter.

The odds ratio method assumes that .842 and .771 are achieved relative to a particular league average context. That assumption is violated here. It assumes that the probabilities you are manipulating are simple probabilities applying to a jointly determined result. With the sudden death rule to break ties, that assumption is violated too, because a separate probability of winning applies after shot 10. (Imagine the tie breaking rule instead was to play another whole 10-shot game...; just to reiterate, your binomial solution knows not only the “atom shooting percentage” but also the tie-breaking rule.]

You pose the question “can we use the Odds Ratio method to predict the game winning percentage for any two given shooters?” The simple answer is no; your answer appears to be “yes, if I make about 5 more assumptions about context, I can get a very good approximation from it.” I can’t disagree with that answer, but I don’t see it leading to a useful result.

I took a more real-life scenario, where the shooting percentage was not an average of .50.

Let’s say the league mean is .340 (like OBP), and you have a .350 OBP team facing a .330 OBP team, and the winning team is the one that gets on base the most, given 40 PA each.

Under that scenario, the binomial says the .350 team will beat the .330 team .528 times, lose .379, and tie .092.  The tie-breaker goes to the .350 team 52.2% of the time.  Final win% is .5777.

Now, what does the binomial say about a .350 OBP team against a .340 OBP team?  .5384.  And a .330 OBP against a .340 OBP? .4613. 

What does the Odds Ratio say about a .5384 facing a .4613 team?  Final win% of .5766.

***

How about something wider, like a .370 OBP team against a .310 OBP team?  The binomial win% is .719.

.370 v .340 gives binomial win% of .6130.
.310 v .340 gives binomial win% of .3846.
.6130 v .3846 Odds Ratio gives win% of .717.

Now, we won’t necessarily have both teams facing an exact single common opponent of .340, but rather, some average that is .340.  Let’s say we have the following head-to-head, with the resulting binomial win%:
0.37 0.31 0.719
0.37 0.32 0.685
0.37 0.33 0.650
0.37 0.34 0.613
0.37 0.35 0.576
0.37 0.36 0.538
0.37 0.37 0.500

0.31 0.31 0.500
0.31 0.32 0.461
0.31 0.33 0.422
0.31 0.34 0.385
0.31 0.35 0.348
0.31 0.36 0.314
0.31 0.37 0.281

Giving a “normalish” distribution of teams, the weighted average of the .370 OBP team is .612 win%, and the .310 OBP team is .386.

That is, all we know about team 1 and team 2 is that their winning percentage against typical competition is .612, .386, respectively.

The Odds Ratio method would give their matchup, absent any other information, as .715.  As we noted, the binomial win%, the “true” win% said .719.

Instead of the typical normalish distribution of opponents, let’s assume that there is a uniform distribution in opponents (i.e., the .310 opponent is as prevalent as the .340 as the .370).  In that case, the win% of our teams is .611, .387, making for an Odds Ratio method win% of .714. 

As it is, I can never get a distribution of teams, such that the Odds Ratio method to give me a result of .719, even though that’s the true win%, according to the binomial.

The Odds Ratio is close, but that’s as far as I can take it.

Tango,

Is this the thread you were referring to?

http://mb9.scout.com/fbaseballfrm8.showMessageRange?topicID=1100.topic&start=1&stop=20

Yup, that’s the one!  So many memories.  That’s the thread that made me think of my “confrontations” theory.

In rluzinski’s link, change the last two characters from “20” to “99”.

If I’m doing my math correctly, the Odds Ratio works just about perfectly in this post:

http://www.baseballmusings.com/archives/018795.php

Studes, you’re right on.  I made a note in that thread.

I don’t mean to get too basic, but can someone explain to me why/how the odds ratio works?  Why is it that, in this case, the result is so different than the simpler P+H-L?

To simplify slightly, the assumption of the odds ratio method is that if hitter’s likelihood of event is 10% better than average (1.1 times average) and pitcher Y’s likelihood of allowing that event is also 10% worse than average (1.1 times average), their joint likelihood is 1.1 * 1.1 average = 1.21 times average or 21% above average.

In the strikeout example, the hitters strike out at roughly .46 times the average rate, while the pitchers strike out opponents at roughly 1.46 times average rate; their net outcome should be about .67 times average.

1) I went ahead and verified that the odds ratio is algebraically equivalent to the formula I posted in #11 above.

2) remember that the batting line is a larger set of outcomes; if you did the same calculations with each element of the batting line (BB,HR, etc) as was done here with strikeouts, and summed them, the total predicted probabilities would not sum to 100% exactly. The “odds ratio” calculation of 11.83% probability of strikeout is still just an approximation, and would need to be adjusted further by rescaling so that the set of individual outcome probabilities does sum to 100%.

Awesome, Joe.  Thank you.

Joe:
Am I right that your formula is:
(S*F-S*F*A)/(S*F+A-S*A-F*A)?

Also, how good an approximation is the even more simple formula suggested by your analysis above:
S*F/A?  It breaks down at extremes (e.g. mean of .9), but seems like it might be a pretty good shortcut for stats like BA, OBP, SLG.  In the example over at baseballmusings, the result would be 11.2% vs. the correct 11.7%—close enough to tell us that hitters were likely not determining the result any more than pitchers.

The shortcut is a very good approximation when S,F approach A. I’ll be brief because i’m not at home ...

The formula can be refactored as
S*F*(1-A)/(A+S*F-S*A-F*A)
in the denominator, if S and F approach A, it approaches A+ A*A -A*A -A*A [which is = A-A*A or A*(1-A)]. Cancelling 1-A from numerator and denominator, you’re left with S*F/A.

The short cut is S/A * F/A =S*F/A*A.  This is a ratio to apply to the average, so multiply again by A to get it back to a rate, and the shortcut is S*F/A

This is a “stupid post,” but I can’t get something right here.  When I follow your Odds Ratio formula, assume league average of .500, one team is .600 and the other is .400, I get a resulting win percentage of .500.

The “odds” formula I get is 1.5 (the .600 team) times .67 (the .400 team) divided by 1 (the league average.  Comes out to .500.

Can anyone tell me what am I missing?

In the case of wins, you’d have to do:

1.5/0.6*1.0 = 2.25

When you look at “times on base per out”, it is what it is.  For the batter it’s times on base earned, and for the pitcher is times on base allowed.

For wins and losses, the W for one is an L for the other.  So, you need to flip it.

Thanks, Tango.  One last stupid question, if you don’t mind.  What if you have a .500 team playing a .600 team?  The overall league average is still .500, of course.  Do you still use the “wins” formula, even though the two teams don’t average .500 themselves?

Many apologies for not intuitively understanding this better.

1 divided by 1.5 times 1 = 0.667

So, the .500 team will play .400

Great stuff...I think I’ve gotten everything working right in my spreadsheet. Here’s a question that I can’t seem to figure out. A team (let’s call them the Spurs) has a 70% chance of winning a game. What are the chances of that team (ignoring home field advantage) sweeping the series in 4 games? Or winning 4 games to 1? 4 games to 2?

Is there a formula in Excel that can tell me the chances?

You do something like this:

p^4*q^(n-4)* (n-1)C3

For n in {4..7}

(n-1)C3 is n minus 1 choose 3

When n=4, that is 1
When n=5, that is 4
When n=6, that is 10
When n=7, that is 20

Maybe someone can check my numbers…

I did that calculation for a thread at Phil’s Blog

http://sabermetricresearch.blogspot.com/2007/06/winning-world-series-in-x-games.html#links

Here is a link to the spreadsheet with the math:

http://tinyurl.com/2lqmrw