THE BOOK--Playing The Percentages In Baseball

I have no idea where that “one in 1,650” figure is coming from. If PECOTA’s average error on a pitcher forecast is roughly 1 run (and that’s what Tango’s testing has shown), I don’t see how that’s even remotely possible.

Okay, I do have an idea. If forced to guess, I’d say he used a t-test. I don’t think that’s the right approach, though.

First of all, I assume he meant, “AT LEAST a .35 difference,” and not EXACTLY a .35 difference.

One more minor quibble with wording:

A better approach is to contrast the aggregate performance of players on Classic rosters with an estimate of what they would have produced had they not signed up for the tournament.

I don’t think that Pecota’s estimates are “what they would have done had they not played in the WBC.” That is like saying that “Pecota is a projection system that estimates a player’s performance assuming that they don’t die, get severely hurt, or eat pizza.”

Pecota has nothing to do with whether they played in the WBC or not.  That was an awkward way of wording what he was comparing.

More importantly, since most readers of the WSJ have no idea about Pecota and forecasting systems in general (which is fine), Dan should have showed what Pecota predicted for all pitchers who did NOT play in the WBC and how they did (as a control).  For all the readers know, Pecota underestimated ALL pitchers by .35 runs.

Plus, Dan refereed to Nate’s own study (I think) that compared the WBC pitchers with their April performance.  Dan and I talked about that in an email.  I suggested that he break up the performance by month or perhaps first half/second half.  I would want to know, if there were an effect, whether it was spread out over the whole year or just at the beginning of the year.  Plus, if he found a large effect in April and then diminishing effects or no effect after that, that would be stronger evidence that there was a cause-effect relationship.

Dan emailed me and asked me the best way to do a statistical test on the significance of the data.  I am no statistical expert by any means, but I suggested one of two ways:  convert the pitcher performance into wOBA against, which Tango suggested to him, also in an email I think.  Or, figure out the standard error in runs, which is what Colin is suggesting I think.

As far as whether a T-test is right for this kind of analysis, I have no idea.  I don’t think it really matters as long as he is in the ballpark with the “odds.” But, I think he could have just said that the chances of this occurring by chance is extremely small, if that is true of course.

I do applaud him for even talking about that though in a mainstream publication.  Most authors would not.  Most authors show an anomaly, make a conclusion or statement, and never explain how, with sample data, there is a finite chance that the conclusion or statement is completely wrong, and that the results occurred by sheer chance, and that calculating what the finite chance is is important in how certain we are of the conclusion or statement.

This discussion is better than many of the injury risk associated with the WBC. The standard is discussion just notes that WBC pitchers have an ERA worse than their previous season and conclude the WBC pitching hurt them. This is so obviously a candidate for a regression effect that I don’t make anything of it.

It’s more interesting to see whether they’re exceeding their projections from systems that regress performance in some way. I worry greatly that two WBCs is too small a sample to know. Nevertheless, we begin to get statistically significant data when we compare players with their projections from good systems.

“the odds were one in 1,650 that they would post an E.R.A. 0.35 higher than what had been projected.” I’m not sure what this means, but I would like to point out that most of these pitchers belong to an extreme performance group and that considerably improving on their ERAs is going to be difficult.

#3, the projections he is talking about (Pecota) already include regression toward the mean.  If you took all of the elite pitcher projections from any other year (non-WBC), you would find that the projections and actual performance were the same (more or less).  No regression. Dan probably should have done that too as a control.  Showed the same pitchers, or same basic type of pitchers (elite pitchers, as you say), and their projections and actual performance for the year before and/or the year after.

So he (and Nate) is definitely finding a significant effect.  There is nothing wrong, methodologically, with his study, although he may have screwed up the standard error calculation, or used the wrong test, as Guy suggests.  He definitely used the wrong words, as I already pointed out (should be AT LEAST .035).

I missed this thread.  Yes, Tango told me to convert the ERA to wOBA and then just use the binomial distribution.  It should indeed have said at least 0.35, although I highly doubt anyone actually managed to misinterpret it.

Dan, do you know what happens when you are (extremely) late to a party?

BTW, for your study, did you look at all pitchers who were on the WBC rosters, or pitched a certain minimum of innings?  If the latter, did you pro-rate or weight anything by the number of innings they pitches in the WBC?

In other words, if were to accept at face value that a pitcher in the WBC will see his ERA be .35 runs worse in the reg season, is there any reason to think that would depend on whether he actually pitched in the WBC and/or how many IP he threw?