THE BOOK--Playing The Percentages In Baseball

Running my quick Marcel, I get him to be a .346 hitter if I consider his .420 batting average to have occurred in 500 at bats, not 219 at bats.  So, I have that big issue with this as well.  No way we can say he’s a true .348 hitter right now.

As well, looking at his chart, it looks like he made his probability distribution as a normal distribution with .348 as the mid-point.  This would also be false, and you would need to have a left-skewed distribution, since we know he comes from a population where the true talent is mostly concentrated in the .220 to .320 levels.

Since he’s looking at whether he’s going to cross some extreme threshhold, all of this has a big impact on the results.

I think you also need to account for possibility of a skewed sample in terms of quality of opposition.  Nearly 30% of Chipper’s ABs have come against FL (5.00 ERA) or Pit (5.22).  Only 15% of his ABs have come against the 6 lowest-ERA staffs in the NL (AR, ATL, Chi, LA, Phi, StL).  I don’t know the weighted BA-allowed for his opposing pitchers, but wouldn’t surprise me if it’s well above average.

Tango, I don’t get your point here:

“Here’s my problem: if the chance that a true .319 hitter will hit .400 or better in 219 tries is one in 4 thousand, how is it that a true .348 hitter will hit .450 (in 331 at bats no less) just one in 1 thousand?  It’s not.  That number is actually 20 thousand to 1.”

Just because it happened once in a thousand simulations doesn’t mean that the odds of it happening are 1 in 1000.  Isn’t this just a sample size issue caused by the fact that he “only” ran 1000 trials?

I really don’t see how Nate gets to his conclusion.

First, if Pecota had him as .316 pre-season, how can his performance in 219 AB possibly change our estimate of his true talent to .348?  (Not to mention his age.)

Second, the range of possible true talents around .348 can’t possibly be symmetrical (.370 is far less likely than .330).

Third, even if he IS now a .348 hitter, hitting .385 the rest of the way is about 1.5 SD above that.  Throw in the risk of injury, and I still don’t see a 12-13% chance of getting to .400 w/ necessary PAs.

Tango, to try to figure out what’s going on with respect to your last point, I’d turn your analysis around a bit: since we know that the .420 sample average is based on 219 at-bats, what kind of prior distribution would we have to assume in order to get a posterior distribution centered around .348?

Well, first, eyeballing the first graph, it looks like it’s centered closer to .315 than to .310, so for simplicity let’s assume a normal distribution around .315.

Since the 219 AB sample has a built in standard deviation of around .031 points of BA, the kind of quick and dirty regression you use in the book suggests that the prior distribution would have to have a standard deviation of around .021 points.

And indeed, if you look again at that first graph, that looks just about like what he has. And here’s how he explains where that prior comes from: “I have generated a normal distribution based on the performance of Chipper’s comparables, after regressing the comparables’ batting averages to the mean.” That’s a bit unclear, but i take it to mean that his comparables’ actual performance was a .315 average with a standard deviation of (something like) .028, which regression would tell us breaks down into .019 due to sample size (600 AB) and .021 due to variation in true talent.

If I have that right, it seems reasonable to me, though whether it’s “right” depends on how much faith you put in PECOTA’s methodology compared to more traditional Marcel-type analysis. I’m an agnostic on that point.

I actually had tries to link to this article but the Book server was busy.  Here is what I wrote:

I have said this before, Bayesian probability is the correct and rigorous way to do projections.  Marcels, our formulas, like PA/(X+PA), for regressing toward the mean, etc., are all approximations of the rigorous Bayesian method.

That being said, the rigorous Bayesian method is quite simple.  As long as we know the exact distribution of true talent in the population that a player comes from, we can combine that with basic binomial (or multi-nomial for things other then BA or OBP) expectations, using basic Bayesian probability methods to come up with a distribution or a mean of a player’s likely true talent.  That is exactly what Nate did.  He also threw in some nice simulation things (the pitchers, the walks, etc.) which were not really necessary, but made his results “nicer” (they look like they were modeled after “real life") and a little more accurate.

There are, however, two problems I can see with his methodology.  What he used as a proxy for the likely distribution of true talent in the population of players from which Chipper belongs, is Pecota’s pre-season estimate of the distribution of Chipper’s likely true BA.  In other words, he skipped a step in the Bayesian model for computing Chipper’s current likely true BA (the “step” is included in the Pecota formulas).  While this is fine from a mathematical perspective, the “problem” is that we don’t know if that distribution is even close to being correct, since we don’t know how Pecota came up with them .

The second problem is that the basic Bayesian model that he uses assumes that a player, like Chipper, does not and can not change his true talent at any given time.  It seems like it does, but it doesn’t.  It assumes that we don’t know what his true talent is (we can estimate it at any time), but that it is static at all times.  It estimates that true talent from two things (which is what a Bayesian model does - it uses two probabilities - one, the prior, or a priori one, which is that Pecota distribution, and two, the current one, which is his current 2008 performance, and the binomial uncertainty surrounding that).

Basically, it (his Bayesian model) says that, given what we think (in this case, he is assuming that it is absolutely true - the Pecota pre-season distribution of Chipper’s likely true talent BA, that is) his chances are of being a .300, .250, .320, 350 (etc, etc. - in fact every possible BA) before the season started and given that he has hit .420 in 219 AB so far, what are the chances that he is a true .300, .320, .250, .410 (etc, etc. again).  Then it uses that new “likely true BA” distribution in the simulation.

Again, that is technically the correct and completely rigorous way to do it, but, as I said, it assumes that the pre-season Pecota “likely true BA” distribution is accurate, and perhaps more importantly, the model assumes that those are the only two probabilities that go into the Bayesian methodology - one, the likely pre-season true BA distribution, and two, his BA in 219 AB so far this year (and of course the binomial distributions surrounding that BA).  What it does NOT include, which is a possibly important parameter, is the distribution of probabilities that a player’s true BA can change during his career, for whatever reasons.  This can drastically change the numbers.  I think it can only change the numbers (his chances of hitting .400 or greater this year) upward in this case.

As I said, it seems like his Bayesian model includes that chance, but it doesn’t.  All of the numbers in the resultant distribution, the ones which go into the simulation, are actually answering the question, “What if we were wrong in or pre-season mean estimate of his true BA?” That is different from, “What are the chances that his true BA has changed?” And will yield different results in both the resultant distribution and in his simulation.  Of course if you include the “chances of a true BA changing” parameter, you are really combining the questions in your final distribution (the final distribution represents the chances that a player’s true BA has changed and the chances that you were “wrong” in your initial distribution).

Let me give you an example.  Let’s say that hitters’ true BA do not change.  So when we use Silver’s Bayesian model and it produces a distribution whose mean is .350 (IOW, that is our mean estimate of his true BA now that he had done so well in 2008), that automatically means that we were “wrong” in our pre-season estimate (which was a distribution with a mean of .310).

But, what if there were a great chance that batters’ true BA can change significantly from year to year, both up and down?  Well, if we estimate now that Chipper is likely a true .350 hitter, there is a significant chance (at least more than zero, which was the chance in the example above) that we were “right” with our pre-season .310 estimate, but that it has changed and he is now a true .350 hitter. 

Basically, if there is a significant chance that batters’ true BA can change a lot from year to year, it makes our estimate of Chipper’s mean true BA now a lot higher than if batters’ BA do NOT change much from year to year.

So, as I said, those set of probabilities are an important part of the Bayesian problem here.  (A Bayesian problem can use more than one set of a priori probabilities).  Yet, Nate does not include that at all.  He assumes that a batter’s true BA never changes, only our estimate of that true BA.

Also, off the top of me head, his “revised” estimate of Chipper’s true mean current BA of .348, seems a little high, given that it started out at .310 before the season started.  Since he used a pure Bayesian model to get to that .348 (and he has no parameter for a true BA actually changing - only our estimate of it), his distribution before the season started must have either been quite wide or skewed towards a higher BA.  Or else, I am just miscalculating in my head what a Marcel would put his current true BA at.

Sam/3: My basic point stands that the 4000:1 was calculated based on a static true talent level, while the sim he generated was based on a dynamic true talent level.  So, they are not comparable.

***

Guy/4: exactly what I’m saying.  The age thing too should also not be ignored.

It’s almost certain that the way he did his comparables is that he treated his current 2008 as a full-season.  This may even point to an issue I have with PECOTA in general, in that I’m not even sure if Nate distinguishes between rate stats of a guy with 300 PA and 600 PA.  In any case, it’s almost certain that he’s treating his .420 BA in 2008 as if it came from someone with 500-600 AB.

Here’s my problem: if the chance that a true .319 hitter will hit .400 or better in 219 tries is one in 4 thousand, how is it that a true .348 hitter will hit .450 (in 331 at bats no less) just one in 1 thousand?  It’s not.  That number is actually 20 thousand to 1.

First of all, he does not do it “that” way.  He does not assume that Chipper is a .348 hitter in his sim.  He takes the distribution centered on .348 and has his sim randomly (according to the expected frequencies of course) select points in that distribution.  One time it will be .320. Another time .370.  etc.  Plus he has different pitchers, etc.

I assume that the results of that kind of sim will be close to the result of a binomial distribution around a mean of .348, but it should generate some more extremes if nothing else because he uses different quality pitchers.  That will create more variance of course. 

And more importantly, as Sam said, just because something happened in a sim one time in a 1000 times run does not tell us what the odds of that event occurring are.  It could easily be 200 to 1 or 10,000 to 1.  Now if he ran off 100,000 seasons in the sim and something came up 100 times, then I think we could assume that the odds of that event occurring are near 1 in a 1000.

Anyway, I thought in my original post that the .348 sounded high given .310 going in and only 200 some odd AB so far this year.  But if he did the Bayesian model correctly then the .348 must be correct. But, as I also said in my original post, even if that .348 is correct, that assumes that his original distribution around the .310 (the famous Pecota “confidence intervals") are accurate.  Whether they are or are not is nother story. If Nate says that that is the approximate distribution of similar players after regressing their sample BA toward a mean, then I guess I have to believe him.  The problem of course, it who are you using as “comps.” If you are using players with similar stats, that is circular reasoning and wrong!  You really have to use as your comps players who are similar in any feature you want, but you are not allowed to look at their stats.  Not at all!  You can’t look at power hitters (high HR), etc.  As soon as you look at any other players’ stats and use them as comps, you are using as comps players who also got lucky or unlucky (depending on whether the player you are analyzing is above or below average in a certain stat or stats).  Now, if you are regressing the stats of the players who are comps, then what is the point of using those players as comps?  You might as well just regress the player you are analyzing.  Now, if you want to use similar players with similar stats and then use those same players future (or past) stats as a proxy for your player’s true talent, that is fine.  THat is actually what I assumed that Pecota does, so I am not sure why he talks about “regressing” the stats of the similar players.  Either you use “out of sample” stats for those comps and don’t regress, or if you are using “in sample” stats for similar players, it makes no sense to even use those players since you are just using them as a proxy for the player you are analyzing - you might as well just regress his stats.

"By the way, I ran a quick Marcel.  I’ve got him as a true .330 hitter.  I don’t see how .348 is possible, unless Nate is treating his 219 at bats as if it was 500-600 at bats.”

Before the season, PECOTA had him at .316 and MARCEL had him at .307.  I did this same exercise using CHONE, PECOTA, ZiPS, and MARCEL in the post I link to in my name.  If you weight in this year’s PAs using .8, .8^2, etc. to his PECOTA, you get .337.  .348 really does seem outrageously high, and that has a huge effect on the result.  When I did it I got 3.6%.

I have a separate question though, although I’m not quite sure how to explain my thought process here.  It arises because we’re dealing with such an extreme mark (.400).  Take two scenarios, one in which he hits .200 over the next two months, another in which he continues to hit .420.  These simulations are not factoring in any changes to his true BA over the rest of the season.  The addition of this would increase his chances of hitting .400, since the .420 example (where he still has a shot at hitting .400) has a larger effect than the .200 example (where he does not).

MGL/6, last paragraph: I’d bet anything that my last paragraph in post 7 explains that.

To go from a .310 estimate to a .348 estimate (38 point jump) because he is currently a .420 hitter (110 point jump) makes the regression of our sample as 65% toward the forecasted mean.

There’s no way that 219 at bats will let you regress only 65% toward the preseason forecast, if that preseason forecast was based on the amount of PA that goes into Chipper’s recent career.  It’s got to be at least 80%.  It’d be 80% for OBP.  For BA, it would be even more regression.

Like I said, I’d bet that Nate treated 2008 as if it was 500-600 at bats.

The sim is assuming two things exactly and then uses Bayesian probability to come up with a distribution of true BA for the season.

One, it has a distribution of true BA before the season starts.  For example, it says that there was a 10% chance he is around a .290 hitter, 15% around a .300 hitters, etc.

Two, it has him hitting .420 for the 219 AB (or whatever it is) so far this season.

So we have a player who had x1 probability of being a y1 hitter before the season started and yet hit .420 in 219 AB, a player who had an x2 probability of being a y2 hitter before the season started and hit .420 in 219 AB.  From all of these, we can compute the probability that now he is an x1 hitter, x2 hitter, x3 hitter, etc.

That is a simple Bayesian problem.

From the distribution that results he simply has the simulator pick a BA randomly using the chance that Chipper is a true whatever that batting average is, for each AB.

That is the correct way to do it.  Actually he should run each season separately using one BA for that whole season and weight the results by the chances that Chipper presently has that particular true BA.

IOW, if there is a 10% chance now that he is a .330 hitter, he should run a 1000 season sim using .330 as Chipper’s true BA.  If there is a 15% chance that he is a true .340 hitter, he should run another 1000 season sim with Chipper as a .340 hitter.  Then he should average all the results (he can’t choose every possible BA, so he will have to choose maybe 10 or 20 possible ones), weighting each result by the chance that Chipper has that BA as his true BA.

The way he does it, by doing just one 1000 season sim and randomly choosing a BA for each AB (I think that is the way he does it) should yield the same results, only it will be for a much smaller sample size.

Anyway, he is NOT, as I already said, including in the model or any of the calculations the chance that Chipper’s true BA actually changes over time - only his (the model’s actually) estimate of that true BA).  If he included the chance that his true BA actually changes, he would have to specify in the model the chance that his true BA changes and by how much and how often.  That would be difficult.  I’m not sure how you would do that.  If it is true that player BA changes, then it is more likely that his true BA is higher now than either his model or a regular Marcel thinks it is.

I don’t think anyone is disputing you mgl… sounds like you are arguing with yourself!

The two main points still stand: a .348 forecast seems much too high.  VegasWatch probably has the best estimate for PECOTA, not Nate.  When VegasWatch did the Marcel forecast as of today, he ended up with .330, which was the same one I got real quick.  So, I’m presuming he did PECOTA right when he says it should be .337.  And that is a far cry from .348.

The second point is that it doesn’t look like Nate used the population distribution of MLB, since we’d expect Chipper’s distribution to have a left-skew using Bayes.

The second point I’m not too concerned with.  The first however, is a huge problem.

2005 PA- 432 * (0.8^3) = 221
2006 PA- 477 * (0.8^2) = 305
2007 PA- 600 * (0.8^1) = 480

221+305+480=1006

This year he has a .420 BA in 260 PA.

So, .316*(1006/1266) + .420*(260/1266) = .337

That’s all I did.  He would need to hit .420 for 188 more ABs (448 total) to get to .348.

.348 was what Chipper was hitting from the start of 2006 thru yesterday. 2004 & 2005 were injury years, 2004 had a career low BA although still with power, so it might be reasonable to say that .348 is Chipper’ “true” level at this moment, over the last 2.4 seasons.

He’s got a .423 BABIP, which would be unprecedented to keep up. His past two seasons were .339 and .348, which are the 2nd & 3rd highest of his career. .330 in 2001 and .349 in 2002 were his only other seasons above .320, although he’s only been below .314 four times. Since the beginning of 2006, it’s .360. That level may be sustainable - Jeter is career .361, Ichiro .357, Miguel Cabrera .354.

This year Chipper also has a career low SO% of .092, tying 2000. Nine times he’s been between .118 and .131. That’s about 8 more balls in play so far this season.

Adding 400 more ABs at .348 to what he’s done already in the 1st 200 AB gives him a projected .374 at the end of the season.

MGL:
There are two related problems.  One is the normal distribution Nate uses for his talent estimate.  It assumes that it’s just as likely Chipper has been an unlucky .330 career hitter as a lucky .290 hitter, which can’t be true.  That leads to another normal curve for the new talent estimate, which says a new true talent of .370 is just as likely as .330—you can’t possibly agree with that. 

Or look at it this way:  Based on 6898 AB, Pecota said Chipper was a .316 hitter.  He’s hit 100 points higher for 219 AB, and that raised the talent estimate by 32 points.  That’s what you’d get if you just weighted his 2008 partial season at 1x, and the entire rest of his career at 2x! 

I think Nate’s preseason estimate signficantly overestimated the likelihood Chipper was a true .330+ hitter, and the new estimate hugely overestimates the probability he is a true .360+ hitter.....

Marcel only looked at starting with 2005:

2005: .296/.412/.556 (109 games)
2006: .324/.409/.596 (110 games)
2007: .337/.425/.604 (134 games)

2005 does not stand out as a season that needs to be discarded due to injury.

Why start with 2006?  That’s cherry-picking.  He missed 52 games, which I will presume is because he was injured in 2006.  So, I see no reason to start Chipper’s clock at 2006 as opposed to 2005.

Even if you somehow justify taking his 1100 AB since 2006, those still need to be regressed.  I’ll guess the AB regression required x/(x+AB), where x=300.  So, you need a 20% regression.  And if he was a .348 since 2006, that puts him as a true .332 hitter.

As far as I’m concerned, he’s around a true .330 hitter, give or take.  0.348 is not justifiable.

No, I’m not arguing with anyone. Just thinking out loud.

And yes, Brian’s “analysis” is not right.

Discarding seasons because you think a player was injured is not right.  You might tweak them but you better be careful.  I don’t like doing that at all.  Fist of all, players play injured all the time and that should be included in a projection (the chance that they are injured and still play in the future).  Second of all, if you discard or discount any seasons where you think a player was playing hurt AND his stats were depressed, you are creating a biased sample of historical stats.  You have to also discount or discard seasons where a player is playing hurt and had GOOD stats (he got lucky!).  No one does that! 

Plus some people seem to think that if a player has a high sustained BA or whatever stats you are looking at, that somehow you don’t have to regress it.  Of course you do.  If Chipper or any other batter were hitting at a consistent .348 level for 3 years, you would still have to estimate his true BA at something like .320 or .330 (I don’t know how much to regress off the top of my head).

The question with the .348 that Nate comes up with is two fold:  One, is his distribution of likely BA before the season started, the one which centered on .316, correct?

I don’t know. I’m not sure if it can be perfectly normal, which it seems to be from Nate’s graph.  It is certainly symmetrical around the .316.  We know that the distribution of BA talent (not weighted by playing time) in baseball is heavily left-skewed, but this is not that kind of distribution.  Whether this distribution can be symmetrical, again, I don’t know.  I’d have to think about it.

The second part of the question is whether Nate did the Bayesian calculations correctly.  If the pre-season Chipper “likely true BA” is correct, then we can come up with the distribution of likely current true BA quite easily, given his current .419 BA in 222 AB.  I can do it one BA at a time, using a computer.  There is probably a cleaner way to do it using the whole distribution at once (assuming that it is in fact normal), but I don’t know how to do that.  In any case, if he did the Bayesian math correctly than his .348 estimate of the current true BA mean is correct.  If it is wrong, then his pre-season distribution must be wrong.

I doubt that he made a mistake in the Bayesian calculation.  So most of you who are disputing the .348 must be disputing the validity of the pre-season distribution.  What do you think it should look like and how would that affect the resultant distribution (of his likely current possible true BA’s) after doing the correct Bayesian calculations?

Since Chipper’s average for any given plate appearance will be slightly higher or lower than his true average based on park, opposing pitcher, game state, all estimates made using a binomial distribution will be a little optimistic.  Does anyone know if this would make a significant difference?  If it’s unclear what I mean, if a player’s true average is .990 at home and .010 on the road, but the coach starts him every game anyway, he will bat around .500 over the course of 200 at bats almost inevitably, while a player whose true average is .500 home and away, he will have much more variance for 200 at bat average.

MGL, I take your advice, although before that I was going to agree with Tango in #16.

Before I got into the conversation, I did want to take a look at Chipper’s year by year stats. The first thing that jumped out at me is where Nate got his .348. I agree that starting at 2006 is cherry-picking, but both you guys have written before about how as you get further back in time the stats can be discounted. I wrote that it “might be reasonable” to do it that way.

I know the math formulas tell you how much to regress, but has anyone here (I know, I haven’t finsihed The Book yet) done an empirical study? When I was working on park factors, it seemed I wasn’t regressing enough. I lnew what the upper and lower bounds should be, but even after regression the outliers were dominated by the one and two year samples. I remember The Book says to regress not to a population mean, but to the player’s mean, which can be hard to establish. And if a player’s mean can change over time, how can we be sure that Chipper’s a .330 hitter (give or take) instead of mid .340’s? intuitively, 2.4 seasons does seem fairly solid, although I do prefer 1500-1800 PAs (3 full seasons).

This is a very similar argument to that on BP’s Playoff Odds Report. We have long and short term performance data, including what’s in the books so far for this eason, and then we attempt to predict what will occur the remainder of the season, and then add it to the performance so far to get the end of season predicted total.

I don’t want to do “arbitrary” cherry picking, but in examining a player’s career, can a trained eye detect certain patterns or influences? Personally, I look at the BA/OB/SA, then also BABIP, ISO, BB%, SO%.

In Chipper’s case, 2004 & 2005 (and before) will be discounted already because of the amount of time that has passed since now to then. Chipper has been consistent over his career in most categories, but how important are the old seasons compared to the last two in predicting the rest of the season, or next season? Projections put Chipper’s extablished BA at .316, which is lower than each of his last two seasons. Now that 2008 so far is considerably above that, I think it’s good evidence that .316 is too low and maybe there is a new higher level. A .423 BABIP has been shown historically to not be sustainable, but given his recent performance, and what others have shown is possible, .350ish is likely.

One of the things I noticed in Chipper’s 2008 stats were that his SO% dropped while his BABIP went up (very much). I have seen this happen to other players (Steve Pearce 2007 & Nate McLouth 2004 in the minors quickly come to mind) but it doesn’t carry over. It might not last for a full season. Looking at a player’s career, I would see this as an outlier.

Tying the last two points together, Jay Bruce this year has also cut his SO% while boosting his BABIP. Looking at his last three seasons combined he projects a high SO% with a high BABIP. The only player in ML history to have a SO% as high or higher with a BABIP as high or higher is Ryan Howard. That combination of skills is possible, but very unlikely. Bruce’s BA line translates to around .295, but all his comps in BA, BABIP, ISO, BB% & SO% all hit in the 265-275 range, so I don’t think he can sustain the BA unless he cuts his SO%, or is a very special talent.

Should we not discount injuries? In Chipper’s case they were several years ago, just giving further reason in addition to elapsed time to discount those years. Jason Bay had a very consistent performance for three seasons. Before 2007, he had knee surgery, and performed in line with his previous record for two months, then tanked the last four months of the seasons. There’s evidence that he wasn’t running well and wasn’t driving the ball. He had many questions coming in to 2008. Is the knee healed? Assuming it is, will he return to the pre-injury level, or should we expect something in between? I felt it was an either/or - if he was healthy, he should perform like he did previously (which he has done so far). If the injury lingers, then he should perform worse.

MGL:  If you look at Nate’s graph for the pre-season estimate, it looks like the probability of Chipper being a true .340+ hitter was something like 20% (assuming each line = 5%, as on his last chart).  I don’t know the correct number, but that seems very high for a guy who has never hit .340 in a season over 13 seasons, and has a .310 average.  I assume Pecota throws out most of that prior data, and it may not improve the mean forcast to use it, but it should impact the size of the error bands.  And as Tango notes, the range should have a left skew. So I’d guess the pre-2008 likelihood Chipper was a true .340 is closer to 2% than 20%.

Unless.... we think what Chipper hit in 1998 is now irrelevant.  If players can make large changes in true talent late in their career, then arguably we don’t care what Chipper hit more than 3 or 4 years ago.  But such changes seem pretty rare to me, so I wouldn’t disregard 13 years of data.

Guy, I don’t think that’s right. Over 480 AB, the random variance for a player with a true .316 batting average is around .021, meaning that his chances of hitting .340 this season (which is really what PECOTA is telling you) were around 15%. And that’s if we have his true talent exactly pegged. If we add some variance due to our imperfect knowledge of Chipper’s true talent (though with his sample size and consistency, I do think we have a pretty good idea of just what kind of hitter he is), 20% does not seem out of the realm of possibility.

The problem as far as I can tell is that PECOTA is not giving us a range of his true talent, but simply of his outcomes. So even if we *knew* that Chipper was a true .316 hitter, Nate’s exercise would arrive at a different answer, probably not far from .348.

Actually DAvid, I think you are wrong and Guy is right.

That first chart is Nate estimating what Chipper’s true rate is, not his sample.  The chart is showing Chipper’s true rate with a mean of around .315 and 1 SD = .020 (eyeballing it).  So, Nate is saying that there’s a 16% chance that Chipper’s true rate, as of Apr 1, 2008, was at least .335.

That seems rather high to me.

It seems to me that what Nate did was not start with the correct population distribution of BA, which I will guess is around 1 SD = .025.  He probably used something much higher (i.e., based on sample BA, rather than regressing the sample BA).

I don’t know why Nate “spared” us from all the details.  This is a webpage, and so, we are not constrained to space.  Furthermore, simply a link to a page that showed his details would have spared us from all this second-guessing and reverse-engineering.

***

Here is Pinto’s chart:
http://www.baseballmusings.com/archives/027174.php

Tango/Guy, you’re right. I should have re-read the article before posting. In that case, I simply can’t believe that PECOTA is giving us the right distribution of outcomes for Jones. No way that we couldn’t predict Chipper’s true batting average with a standard error of more than 20 points...I think.

Bayes and Regression

I selected all players with at least 200 at bats in any season since 1993.  That gives me 4407 hitters.

For each hitter, I figured out the standard deviation that the binomial would expect given his number of AB.  Then, I figured out how many standard deviations his actual batting average was from the mean (of .267): his z-score.  I then took the standard deviation of the z-score.  It was 1.39.

If batting average was purely random, we’d expect 1.00.  As it stands, it was definitely not random.  While parks and uneven competition can contribute to a larger than 1.00 z-score, most of it is tied in to the true talent distribution of players.

r = 1-(1/1.39)^2 = .482

So, if you get a z-score of 1.39 from a group of players, that would imply that a year-to-year correlation of these players would give you r=.48.

The average number of AB of this group of players was 419 (harmonic mean of 376).

So:
r = AB / (AB+419) = .48, making AB = 449.

If we use the harmonic mean (376), then AB = 403

Our correlation equation becomes one of:
r = AB / (AB+449)
r = AB / (AB+403)

Similarly, we could have taken the unweighted standard deviation of our 4407 batting averages to get: one SD = .0303

The average of the binomial SD for batting average is one SD = .0225

Remembering that:
variance observed = variance true + variance binomial
.0303^2 = variance true + .0225^2

variance true implies one SD = .0203

Our correlation for this group of players is therefore:
r=(.0203/.0303)^2= .45

Fairly close to what we got.  Note that in our case here, we did not do any weighting, and therefore, we expected a somewhat lower number.

Anyway, going back to treating this as our equation going forward:
r = AB / (AB+400)

Then we know that at AB=400, correlation = .50, which implies that the standard deviation from the binomial equals the true standard deviation of our population of batting averages.

Since the binomial at AB=400 is one SD = .022, then we know that the true spread of our population of MLB players has a true talent of 1 SD = .022.

If the mean is .267, then this implies that we have a true distribution of players of .267 +/- one SD = .022.  95% of players will be roughly .267 +/- .044.

Now, the Bayes question to ask is: given that we have this known distribution of .267 +/- oneSD=.022, then what is the likelihood that someone will hit .420 in 220 AB was really the guy at .220?  At .225?  At .26493022?  At .3221?  At .38764?

You figure out the probability of each of those answers, multiply by the frequency of each of those talent levels existing and multiply by the batting average in question, and you get Chipper’s Bayes-generated true talent batting average.

Now, we also have further information.  The guy who hit .420 in 220 AB is not some random player, but he also happened to come in with a career .310 (or whatever) BA, or if we treat this as some Marcel weighted number, say we also know that he got 320 hits in his previous 1000 (weighted) at bats.  You add that (say 410 hits in 1220 at bats), and you ask the question again.

I’m going to lunch, and someone else can pick it up from here.  But, I can pretty much guarantee that the answer will not be .348.

For it to be .348, I’m guessing the true spread in BA would have to be .030, or even higher, or that his past performance is weighted less than I am doing here.  And I think this is where we have the disconnect.

I actually do understand all the math, but I think the problem is we are aiming at a moving target.

Players do change. I’m browsing the Lahman batting database, with my query adding lots of calculated columns (Age, BABIP, ISO, BB%, SO%, etc). With the luxury of hindsight, I can see where players talent changed from one level to another, for whatever reason (got fat, back gave out, snorted coke, shot up, finally got healthy).

The problem we have is identify when the change occurs. If a player has been consistent (within a SD) for five years, then does much better or much worse, is this a one year blip, or are we at a new level? Is the change fluid, evolutionary, or does it come in spurts, up or down (from one plateau to another).

If Chipper is projected to be a .315 hitter, but for 2.4 seasons he’s been hitting .348, and for 200+ ABs he’s over .400, then maybe he’s not really a .315 hitter and his “true” BA skill today is not the same as what it was one or two or three years ago.

Have we asked how high must his true talent level be to produce the results we are seeing recently?

FWIW, I’ve got the split-half reliability for 250 PA’s at .328 and for 300 at .351.  (Chipper’s had 264 as of right now… call it .330?)

The NL is hitting .259 as a league.

.33 * .419 + .67 * .259 = .312, which is around CJ’s career BA.  However, he’s also had seasons in the .330 range the last two years, suggesting that he’s probably closer to a .330 hitter.  Also, Chipper’s batted ball profile (something that stablizes much quicker than BA) is different this year, with more line drives.  So, it’s likely that he’s a different hitter this year.

I looked at this subject in the StatSpeak roundtable last week.  Even given the most optimistic projection (he’s a true .350 hitter), I have his odds at 1 in 26 of finishing the season above .400.

Brian/26:

We have no idea what his true BA is.  All we can do is estimate what it might be.  And, entering 2008, we thought it was .310.  We now think it’s .330.

IMPORTANT: we don’t even know if his true talent itself has changed.  It could simply have always been .330 for the last 2.5 years, but we didn’t have enough information to proclaim that.  OR, he could actually have improved his talent this year, so much, that while we were rock-solid sure he was .310 entering 2008, we are now rock-solid sure he’s .330 today.

But, all of that is irrelevant.  All we have to go on, god notwithstanding, is our best estimate of what his true talent is today, and our uncertainty level around that estimate.

And, I would like to see the evidence that he is now a true .348 hitter.  I don’t want to be “spared” that detail.  I don’t want to give Nate, who does otherwise great work, any benefit of the doubt on this one.  It’s not up to me to prove that he is not a .348 hitter, but up to Nate to prove that he is. 

And, of all the things he wrote in the article, the two things that would make him go from a 12% chance of hitting .400 to a 1% chance of hitting .400 had nothing to do with all the extra niceties that he did, but rather:
a) his estimate of .348 being his true talent,
b) the uncertainty level of this .348 being so incredibly high, and so symmetrical around that .348, that he could possibly hit .438 one time in 1000 and hit .420 or better what looks to me like 1% of the time.  And this had nothing to do with running the sim “only” 1000 times.  Ten times out of 1000, he’s hitting at least .420.

I will remain in a state of disbelief, as I await evidence.

FWIW, I’ve got the split-half reliability for 250 PA’s at .328 and for 300 at .351.  (Chipper’s had 264 as of right now… call it .330?)

I *plead* with you Pizza to, in addition, report the average number of PA.  As it is, since 250 PA in half a season is so high, I will assume that the average is around 300 PA, and the one for 300 PA is an average of 325.

Proceeding on that basis:
r = x/(x+300)=.328
implies x = 615

r = x/(x+325)=.351
implies x = 601

So, that’s fairly consistent, that you would get:
r = .50, when number of PA = 600 (at bats = 540 or so).

Note, I get similar numbers when I use a threshhold of 400 AB (z-Score = 1.41, implying r=.50 when average AB = 520).  That is, I get virtually identical results as Pizza.

However, I think here we see a huge selective sampling issue.  By discarding so much of the players from our sample, it makes it look like there’s little differentiation among players.  And, it’s true!  At the very highest level of playing time, there’s little variance among guys in batting average.

But, do we want to say that Chipper is necessarily drawn from this group of hitters?  We could.  But, then we force him to not allow him to be drawn from the possible .330 talent level.

On the other hand, by lowering our threshhold to include as many “reasonable” MLB players that we can, our spread in talent level increases (while the mean decreases).

(Pizza: in this case, you would have to report what the mean PA is of your population.  I am guessing it’ll be around .280-.285.)

So, either Chipper is drawn from a population with a mean of .267 with 1 SD = .022, or he’s draw from a population of .282 with 1 SD = ... I dunno… say .014.

I sent Nate an email asking him to engage us here.

Did no one read my post (#5)? I think what Nate did was pretty clear.

Don’t have much time, but just to clarify a couple of things:

The .348 average does not really come from PECOTA.  The only thing that PECOTA was used for was to give a rough outline of the true talent distribution around Chipper’s batting average forecast.  But the performance of the individual comparables was regressed to the mean in order to estimate their true level of talent, and the distribution was normalized, which is not something that we do for normal PECOTAs.

There are a lot of different ways we could have estimated the true talent distribution without any reference to the PECOTAs.  Maybe some of you can experiment with this.  My guess is that the finding will actually be fairly robust, as no matter what the shape of the bell curve, the Bayesian mathematics are really going to accentuate the tail of the distribution.  On the other hand, if the distribution is asymmetrical, that might change the findings significantly (I don’t really know).

Andeux/31:

I think what he tried to do was clear.  I think it’s also clear in what I am doing that I’m not going to get a 13% chance to break .400.

Regardless, he further compounds the talent level issue by giving such a wide uncertainty level around that .348.  He gives him a decent chance that he’s at least a .370 hitter.  On that basis, of course he’ll get a 13% chance of breaking .400.

I don’t think that Nate started with the league population mean of .267 +/- .022 (actually, probably more like .262 this year).  It seems pretty clear to me that he didn’t.  There’s simply no way he could get that .348 mean with the distribution that he’s showing (his second to last chart) if that were true.  He’s claiming that Jones is nearly 4 standard deviations from the mean, with a decent chance that he is 5 standard deviations from the population mean.  That’s a little hard to accept, isn’t it?

Nate, thanks for stopping by.  Perhaps for his sake, if Nate has a chance to come back, you guys can precede your post by “NATE:”, if it’s a specific question that he can answer.

NATE: your uncertainty level around the .348 true mean seems to be around 1 SD = .020, and fairly symmetrical.  Can you confirm what the SD actually is?

Using Bayes, with the following assumptions:
.267 = pop mean
.022 = standard dev of pop
219 = trials
92 = successes

I get a mean forecast of .321 for the batting average.

That is, if we know nothing at all about a hitter, other than he is 92 for 219, Bayes says he’s .321, with an uncertainty level of 1 SD = .017.

What if we add in 900 at bats of .310 hitting?  Bayes says .320.  Furthermore, the uncertainty level around this .320 is 1 SD = .012.

***

How do we get him to be a true .330 hitter?  The population spread has to be 1 SD = .025.  If that is the case, then Bayes says our uncertainty is 1 SD = .019.

If this hitter also happens to add in 900 at bats at .310, then Bayes brings him all the way down to .324 as a true hitter, with an uncertainty of that estimate of 1 SD = .012.

***

How do we make him a true .348 hitter?  Our population spread must be 1 SD = .031.  That is impossible, since we know that the *observed* population spread is 1 SD = .030.  Our uncertainty level in this case is 1 SD = .024.

***

I will have to retract one thing, and that is that the spread of our true talent uncertainty does follow very closely a symmetrical distribution.  So, I’ll have to take back that objection.

The rest of what I’ve said stands.

By the way, if we add in a .267 league batting average of 400 at bats to 92 for 219, we get 199 hits in 619 at bats, for a batting average of .321.

Therefore, our regression toward the mean equation works!  You add in 400 at bats of average batting average (.267) to your players total sample, and you get .321.  The more precise Bayes says that a 92-219 player probably is a true .321 hitter.

Sweet, right?

More by the way: the reason that Marcel said that he was a true .330 hitter, as opposed to the true .320 hitter, is that I’ve been using a standard regression model of adding league average performance of 240 PA.  While this works out good overall, it doesn’t work for individual components.  For example, for Ks and walks, you should use a much smaller number.  For OBP a bit smaller.  For batting average, as we’ve found, you need to add 400 AB.

So, if you’ve got Chipper with 92 for 219 today, and you credit his 2006/07 performance of 306 for 924 (.331) at say 70% for 214 for 647, and you add in .267 on 400 AB, you get:
AB, H
219, 92
647, 214
400, 107
---- ---
1266, 413 = .326

So, that I think becomes our best guess, that Chipper, today, is a true .326 hitter.

We can do further breakdowns by looking at strikeouts, linedrive rates, etc.  We’d need to do the same exercise, in terms of knowing the true spread among the population in each subcomponent.

Regardless, I doubt you’ll get him up to anything other than a bit over .330, if even.

Tango, when I quote those split-half reliability coefficients, they are generated by articificially restricting the sample for each player to X number of plate appearances.  So, when I say 250 plate appearances, I’m taking the first 500 PA’s for each player, assuming he has that many over the two year stretch that I’m looking at, splitting them in half by evens and odds, and running a correlation between the two numbers.  So, when you see 250, I’m comparing a sample of exactly 250 PA to exactly 250 PA. 

The selectivity of that sample is driven down by the fact that in order to make it into my sample for 250, a player only had to amass 500 or 600 PA over a two year period.  Still a selective sample, but at least a little better methodologically.  I’m using data from 2001-2006 as my source.

I’m actually working to generate coefficients down to individual levels of PA (so what’s the split half at 228 PA or 357), but those charts take forever for my computer to generate.

513, 555 therefore are my new numbers for Pizza Cutter for PA, respectively.

But the performance of the individual comparables was regressed to the mean in order to estimate their true level of talent, and the distribution was normalized, which is not something that we do for normal PECOTAs.

There are a lot of different ways we could have estimated the true talent distribution without any reference to the PECOTAs.

I don’t know exactly what he means in the first paragraph above, but all you really have to do to generate these true talent distribution curves is to take all players in history who have a similar profile at a similar age, and other similar characteristics (for example, in this case, switch hitters), and then see how each one does over some subsequent time period.  I have always assumed that is what Pecota does anyway, more or less.

If you take the average of all those “subsequent” performances, you will get your player’s mean true talent of course.  If you take those individual performances of all the similar players and regress those toward the mean, you will get your distribution.

For example, let’s say that there were 130 players similar to Chipper as of the end of last year, and all those players had at least one more year of playing time left.  If you take all those players and look at their year after you identify them as similar players, the mean of those players’ performance will be exactly equal to Chipper’s expected performance in 2008, essentially his pre-season projection.

If you look at each of those 130 players individually and regress their one-year stats to get their “true stats” for that one year, then you can come up with a nice distribution of Chipper’s likely true talent.  For example, if 20 of those 130 similar players, or 15%, hit .320 in that next year, and you estimate them to be true .312 hitters in that year, then Chipper has a 15% chance of being a true .312 hitter.  Etc.

The problem with that methodology, and it is a big problem, I think, is multi-fold.  One, you are unlikely to have much of a sample size of similar players.  And the more similar you want to make them, the smaller the sample.  And with a small sample, you will have lots of noise in creating that distribution and even the mean.  And if you want to have a large sample, you will probably end up including players who are not all that similar to your player in question, which makes your conclusions (the mean and distribution of your true talent estimate) pretty unreliable.  Two, all kinds of things change over the years, not the least of which is player fitness and aging.  Do you really want to use comps from the 1950’s for today’s players?  I think that Pecota uses a fairly recent time period for their players comps, but again, that creates a small sample size.  Three, for your distribution (the mean is no problem), how do you go about estimating each player’s true performance in that one year?  You would have to do the same thing for those players as you are trying to do for your player in question!  That would be a mess.  If you are just going to do a Marcel on them then you might as well just do a Marcel on your player in question!

Tom #37 - Why regress with 400 PA of league avg BA when we have 13 prior years of Jones own performance. I do remember from The Book you saying about being able to regress to the individual players’s prior established record (presumably of enough length to not require regression of it’s own).

Replacing .267 league BA with Chipper’s .303 from 1993-2005
AB, H
219, 92
647, 214
400, 121
---- ---
1266, 427 = .337

I am intuitively quite comfortable with that.

Well, your intuition is wrong, unfortunately.  You take all players with 5000 or 10000 career PA, and you will find that players with 3000 PA will forecast just as well.  That is, all those extra PAs really provide almost no additional information.

Read here for one such study:
http://tangotiger.net/banner.html

Basically, you need regression exactly as I have laid out.

***

In any case, I have provided the Bayesian solution. (Regression is really just a shortcut for Bayes.) And in the Bayesian solution, a guy who is hitting .420 after 219 at bats will receive almost no additional information from us to determine how good a hitter he is.  That is, regardless of how bad a hitter he has been prior to Apr 1, 2008, the performance of .420 is so extraordinary that we don’t need any more information!  The Bayes result, based strictly on 219 AB, gave us a true talent estimate of .321.  And when I tried different amount of AB of .310 hitter added to that, it always stayed at .321. 

So, regardless of intuition or comfort, you can’t escape the fact that Chipper is a .321 hitter.

(Note that since the BA across the league is down around 5 points, all of our original estimates need to go down 5 points to make the fair comparison.  This is an additional reason to make the .348 estimate even more unlikely.)

It seems to me there’s an implicit assumption in the Bayesian calculation, that Chipper’s 92/219 was an unbiased sample.  But I think it’s likely that it came against substantially below-avg pitching.  If so, that makes it less likely he has make a big jump in talent. 

(BTW, he went 1-for-5 against Zambrano tonight, and dropped his average by 5 points.  This is really hard.  I’d gladly take the “no” bet with 8:1 odds....)

Guy, according to the Baseball Prospectus stats page, Chipper has faced very close to average pitchers this year.  That is assuming I’m interpreting their page correctly, which I’m not sure about.

http://www.baseballprospectus.com/statistics/sortable/index.php?cid=350487

Guy/45: I basically agree but it is a lot of work to do an accurate accounting of quality of opposition. BP has very simple reports, and they show Chipper’s opposition yielding .246/.336/.386, which would mean his pitchers faced have been a bit better than average. Then again, that’s an incomplete way to do it and it wouldn’t surprise me if you were more correct than that number indicated.

(Cliff Lee’s opponents: .250/.320/.381. Just throwing that out there.)

5th:  Looks like that isn’t a factor for Chipper.  (Though I’m a bit skeptical of the BP numbers:  I checked a few teams, and several—including Braves—appear to have faced good pitchers.  I wonder if they’re weighting pitchers properly.)

Still, I think in general the possibility of skewed samples after 60 games (home/away, quality of pitchers, platoon edge) does somewhat increase the probability of extreme performances in absence of large change in true talent.

The BA is the only “quality” we are looking at here.  And, I don’t know how the opp BA is calculated.  I presume it is weighted based on the number of AB (good) and I presume they are using to-date (2008) performance (bad).  You really want to know the pitchers’ true talent level in BA.

Guy’s basic point stands that we can have a bias here.

Mike/46 was marked for moderation and has been unqueued.

***

Mike, that does not look very average to me…

PED comments moved here:
http://www.insidethebook.com/ee/index.php/site/comments/does_driving_out_peds_affect_hitters_disproportionately/

In post 35, I said this:

Using Bayes, with the following assumptions:
.267 = pop mean
.022 = standard dev of pop
219 = trials
92 = successes

I get a mean forecast of .321 for the batting average.

That is, if we know nothing at all about a hitter, other than he is 92 for 219, Bayes says he’s .321, with an uncertainty level of 1 SD = .017.

What if we add in 900 at bats of .310 hitting?  Bayes says .320.  Furthermore, the uncertainty level around this .320 is 1 SD = .012.

Repeating this exercise for Chipper now (98/249, .394), and Bayes tells me that he’s a true .317 hitter.

This is what should happen.  Chipper is still Chipper.  Our evaluation of him should not change drastically, if he goes on a hot or cold streak.  Bayes had him as a true .321 hitter when he was hitting .420, and Bayes has him as a true .317 hitter after going on a 6/30 (.200) run.

A regression toward the mean model would give you similar results.  I suspect that Nate’s process has him dropping substantially from the .348 true talent level to probably something like .335.

One good way to evaluate these models is to see how “tight” the true rate remains.  We don’t want to see his true rate bouncing up and down all over the place, only to settle at .320 at the end of the year, if some other model can keep it tight around .320 the whole time.

You can similarly evaluate post-season odds this way, as you don’t want the Rays to have a 90% chance of making the playoffs followed by 30%, then 80%, then 10%, then 100%.  Well, you could if the case warrants it.  But, if you have a competing system that has less variability and leads to the same conclusion, then the competing system is probably better.

I ran an updated CHONE projection last week for Larry and got .318, pretty close.

Can you explain the calculation process for the Bayesian method used above?  If I change those 4 inputs around, how would I calculate the BA estimate?

Never mind, I see post #25.  It might take awhile to understand it though.

In Excel, you’ll want:
NORMDIST

I find the frequency of some mean with some standard deviation happening between a .1795 and .1805 batting average (midpoint of .180), and go up in steps of .001 all the way up to .450.

If you multiply the frequency by the sample rate and add them all up, you get the mean you started with.

BINOMDIST

You use this function to figure out, for each batting average from .180 to .450 could have produced a 98/249 or better average.

Multiply this probability by the frequency calculated earlier to give you your distribution of true rates.

And from there, you do a weighted average, and you get the Bayes batting average.

Play around with it, and if you get stuck, let me know.

Since you are assuming normal distributions for both probabilities, isn’t there a short cut to doing the Bayes calculations, rather than taking every possible BA (in increments, as you do it)?  Using calculus?

Chipper is now 108/285 (.379).  Bayes now has him as a true .315 hitter.

If you add 900 AB of .310 hitter, regression to the mean has him a .317 hitter.

Like I said, a good way to test a system is for the talent level of a majority of the players to remain fairly flat season-to-season.

The highest we got Bayes on Chipper was .321 and the lowest was .315, even as his BA went from .424 to .379. 

As I’ve shown at the start of all this, Nate’s estimate of .348 was unsupportable, and so was his forecast as to the chances of him breaking .400.

For what it is worth, I have watched Chipper quite a bit in the last couple of weeks, and he “looks” like a .240 hitter, taking some awful swings and having some awful AB’s.

The reason I say that is because everyone was talking about how “locked in” Chipper was and how he clearly had changed his talent level, etc.

I can’t emphasize enough how much nonsense and crap I think that kind of “analysis” is. Sometimes a batter looks great and sometimes he looks horrible.  You cannot, and I mean CANNOT, tell whether a batter is a .280 or a .320 hitter from looking at him (I am not including a professional scout in that).  And even more importantly, in virtually any stretch of PA, any batter can look like a .400 hitter or a .200 hitter.

Of course, the proponents of batters “pressing” or being “locked in” will say that he WAS “locked in” before but that he is “pressing” NOW.  That is after the fact.  I challenge any of these guys to tell me when a batter is locked in or pressing and we’ll track his next X number of PA.  If that batter beats his projection when “locked in” by more than 10 points, or is more than 10 points less than his projection if he is “pressing,” then I will give that person X dollars.  If not, I win X dollars.  Again, open challenge.