THE BOOK--Playing The Percentages In Baseball

In searching for genes that raise your risk of developing some disease, for instance, geneticists applied a traditional test: is a link to a gene less than 5 percent likely to be a coincidence? Unfortunately, when you’re testing thousands of genes, that lets too many false positives through.

Everyone should familiarize themselves with this concept.  It is sometimes called curve-fitting.

Exasperating this is “publishing bias” which is the tendency of researchers to only publish studies where the null hypothesis is not supported by the data, especially in medical research financed by pharm. companies.  Imagine that there are 100 studies performed on a certain drug.  The drug has no benefit but, by definition, 2 or 3 of them will find a false positive effect at the 2 sigma level.  What if only those 2 studies were published?  Or even 10 were published including those 2?  Unfortunately that is what happens in science to one degree or another, and that has to be kept in mind when reading these studies.  Heck, when I am casually looking at data myself or even when I am doing/writing a formal study, I usually do not publish it, or continue with the research, unless I get an “interesting” result.

I guess one of the best ways to combat this bias is for someone to always duplicate a study that someone has reported as positive and make sure that the results, either way, are published.  This does happen of course.