THE BOOK--Playing The Percentages In Baseball

Monday, April 19, 2010

The null hypothsis does not always have to be zero, does it?

Phil:

Doesn’t that seem like a reasonable adjustment for HFA? I think it does. I’m not sure what an average batter hits for ... say, 35 runs? That means if the player hits for 25 runs at home, he’ll be cut 10% of the time. If he hits for 25 runs on the road, he’ll be cut 4% of the time. What’s wrong with that?

What the authors would say is wrong with that is that the signficance levels for the last two terms were too low, so we have to drop them. To which I say, nonsense! They look almost exactly as you’d think they would based on your prior expectation of managers not being dumb. If they’re not significant, it’s because you don’t have enough data!

Looking at it in a different way: the authors chose the null hypothesis that the managers’ adjustment of HFA is zero. They then fail to reject the hypothesis.

But, what if they chose a contradictory null hypothesis—that managers’ HFA *irrationality* was zero? That is, what if the null hypothesis was that managers fully understood what HFA meant and adjusted their expectations accordingly? The authors would have included a “managers are dumb” dummy variable. The equations would have still come up with 4% for a road player and 10% for a home player—and it would turn out that the significance of the “managers are dumb” variable would not be significant.

Two different and contradictory null hypotheses, neither of which would be rejected by the data. The authors chose to test one, but not the other. Basically, the test the authors chose is not powerful enough to distinguish the two hypotheses (manager dumb, manager not dumb) with statistical significance.

But if you look at the actual equation, which shows that home players are twice as likely to be dropped than road players for equal levels of undperformance—it certainly looks like “not dumb” is a lot more likely than “dumb”.

***

It’s like when I looked at the gap in wOBA between starter and relievers, and the gap was some 27 or 32 points, using only 4 years of data, and limited number of players.  You’d think that would have some statistical significance, but not much.  Then I looked at 50 years of data and found the same difference.  That’s because I didn’t include my prior expectation that pitchers thrown harder in relief than in start.  Typically speaking, when you get an extreme result with a limited data set, and you have no prior, then looking at a larger data set should get you a less extreme result.  But, if you DO have a prior, then that changes everything.

Like flipping a coin.  You flip 100 times, and you get 54 heads.  Yawn.  You keep doing it, or someone keeps doing it,and finally someone reports “hey!  I got 87 heads” Now, if you have some prior belief that the coin is weighted, then this may be significant.  If you have no prior, then it’s a mild yawn.  If you have a prior that it is NOT weighted, it’s a total yawn.

Priors are crucial.