from Amazon
Thursday, March 18, 2010
Statistical Significance, or the reason that mathematician Ron Fisher is on MGL’s “On Notice” Board
Glove-slap Chuck:
Fisher’s P value eventually became the ultimate arbiter of credibility for science results of all sorts — whether testing the health effects of pollutants, the curative powers of new drugs or the effect of genes on behavior. In various forms, testing for statistical significance pervades most of scientific and medical research to this day. But in fact, there’s no logical basis for using a P value from a single study to draw any conclusion.
This article is so well-written and well-researched, but I think this is misleading:
Correctly phrased, experimental data yielding a P value of .05 means that there is only a 5 percent chance of obtaining the observed (or more extreme) result if no real effect exists (that is, if the no-difference hypothesis is correct).
There’s a gap between the main part of the sentence and the parenthetical part. The parenthetical part is correct and what we care about. I would say, he should have said:
Correctly phrased, experimental data yielding a P value of .05 means that there is only a 5 percent chance that the observed result occurred if no real effect exists
That is, it’s not that what we observed is an indication that this particular observation is real. It means that what we observed is an indication that something is going on TO SOME DEGREE, from non-zero up to what we actually observed.
If someone wants to take a stab at how to better phrase, please do so.
The main point is that if the p value of a .330 OBP hitter who gets a .530 OBP in the clutch is .05, this ONLY means that there’s 5% chance that he got an OBP that high by luck, and so we have to conclude that we’re 95% sure that his true OBP is somewhere above .330, though the information provided here tells us nothing about our best guess as to his true OBP in the clutch.
It does not mean that we’re 95% sure that his clutch OBP is .530. And this last part is pretty much how I see conclusions being made.