THE BOOK--Playing The Percentages In Baseball

It was an excellent post by Dan on the James article and James in general.  I wrote this on the blog:

He (James) seems almost obsessed with pointing out that analysts’ characterization of many things as “random,” may not be.  What he does not seem to understand is that analysts (at least the good ones) merely look to see whether there is statistical evidence that something exists as a skill to a significant extent or at least to some extent, X.

If there isn’t, they “conclude” that no such skill exists, to a signficant extent or to X extent.  That does NOT mean that they are “right” of course.  Since these analysts are working with sample data, they all realize (at least they should) that there is a finite chance that they made a Type I (or Type II) error and that they are “wrong” about their conclusions (of course “wrong” can mean many things and the chance that they are “wrong” varies with the size of the effect that you are “wrong” about).

James sort of dangles that in front of their faces and says, “How can you say that ‘no skill exists’ yada, yada, yada,” without understanding the “limitations” that these analysts are fully aware of with respect to analyzing their sample data.

He also seems to continually come up with different methods of defining “clutch” which is fine (I have no problem with that) and then cherry picks players who have or have not exhibited clutch play in certain periods or throughout their careers, but NEVER does the proper statistical work to see if there is any evidence that these differences are not merely noise!

I genuinely don’t think that he fully understands the difference between finding large spreads in clutch performance in the past and assessing the chance that these differences are due to skill or luck.  That of course is the key to these kinds of analyses.  You wil ALWAYS find great differences in any performance splits among players (the smaller the sample, the more and greater differences you will find).  So what?  They key thing is to then analyze the variance you find in the samples and see if they are likely due to chance or to skill, and then to quantify “how much” they are due to skill (IOW, how much to regress those differences for a given sample size).

As I said, I don’t think James understands this concept.  All he ever does is point out large differences in performance splits and say something like, “Come on guys. How can these large differences be due only to chance?” Which is silly of course.  They may be (due almost completely to chance) or they may not be - but we can surely investigate that.  Bill apparently cannot.

Finally, and I have mentioned this several times already in various venues, when we work with sample data trying to find the existence of “effects” like a skill component to clutch (or platoon splits or whatever), if we come up with “no evidence of an effect” there is ALWAYS a significant chance that there is a small effect that we missed.  That is the nature of working with sample data where there is ALWAYS sample error and there is nothing the researcher can do about it.

Now, how small or large this “small effect” may be (that we missed) is proportional to the size of the sample error in our calculations, which is essentially proportional to the size of our samples.

Even though we (as researchers) don’t always say it (I try to), when we say something like, “There is no evidence of a clutch skill in the sample of data we analyzed given our definition of clutch” what we really mean is that, “There is a 99% (or whatever) chance that no skill exists beyond a level of Y” or something like that, and you can replace the 99% and the Y with any numbers you want.

That, BTW, as Dan mentioned in his blog, is what we (Andy) did in The Book.  Rather than say that clutch exists or does not (which is a silly dichotomy, as I keep saying, when dealing with sample data), we say that, “There is likely an X amount of variance in clutch skill in the population, as compared to, say wOBA skill, such that in Y amount of PA we can regress a player’s clutch splits Z percent in order to estimate his clutch skill.”

And even then, we explain that those numbers have an uncertainty level as well!

So, from now on, let’s appease James, when we do a study looking for a particular effect, by never concluding with the “yes/no” dichotomy and merely reporting how much skill there likely is (from zero to whatever in terms of regressing for any particular sample size) AND we’ll include our uncertainty with respect to that as well. 

Maybe that will make James happy.  I don’t think so, because I don’t believe he understands what I am talking about.  He is a brilliant guy but he is not nearly a statistician and I think that he has some “civilian biases” (e.g. that if you find some large “split” it certainly must be due to SOME skill) with repsect to these kinds of issues.

I’m not a James apologist, but I don’t see what the big deal is.  A lot of people (not those on this site) clearly have gone overboard on the clutch hitting thing, saying that there is clearly no such thing as clutch hitting in the majors.  Try saying the opposite at BBTF and see how quickly you get shouted down.

I agree it would have been nice if James had put more of a “study” in the Annual, and it would also be nice if he had stayed involved with the sabermetric mainstream.  I definitely think he can and should be faulted for the latter.  It’s going to be very interesting to see how he handles the debates that are likely to occur on his new site.