THE BOOK--Playing The Percentages In Baseball

I’ve been looking for Murray Chass, have you seen him?

Well put, Tom. These days I do little more than scan my local newspaper online because I get more information and better analyses from blogs and aggregator sites. This not only applies to sports but to other news as well.

People often maintain that this is a generational thing, and perhaps it is, in the main, but not necessarily. I’m in my sixties. Of course, I may be the exception in that I worked in software engineering for the majority of my working career.

I think the peer review worked very well in this case.  I am not sure how well it works in the case of statistical studies.  Ideally, there would be enough information in the description of a statistical sudy to be able to replicate the study, but the complexity of some of the studies makes that practically impossible.  Throw in the multiple sources of data and multiple ways of manipulating the data that require use of non free software and you pretty much have to take somebody at his word about his methodology.  This is somewhat true in the academic world as well, but I think it is worse on line.

"you pretty much have to take somebody at his word about his methodology.  This is somewhat true in the academic world as well, but I think it is worse on line.”

I’m not so sure the academic world is any better:
http://www.msnbc.msn.com/id/29255336/

Guy: fantastic article.

McKitrick, economics professor at the University of Guelph, and co-author Bruce D. McCullough, professor of decision sciences at Drexel University in Philadelphia, summarize replication efforts of more than 1,000 economics articles published since the 1980s. Most authors did not release their data when asked, and of those who did, only a small number of results were reproducible.

Great article Guy!

The first paragraph from the Wikipedia article about the organization that is making the critique:

The Fraser Institute is conservative and libertarian think tank based in Canada that espouses free market principles. Its stated mandate is to advocate for freedom and competitive markets. It generally opposes public policy solutions based on government spending, taxes, deficits, and regulation. Some of the public policy stands taken by the Institute include: greater free trade throughout the world, privatization of various government services, the freedom to own and acquire firearms without controls, marijuana legalization, competition in primary schooling, and greater private sector involvement in the delivery of healthcare insurance and services. In terms of present hot topics in public policy, the Institute opposes government regulatory action as a possible solution to global warming. They argue that regulations have “the potential to impose high costs on Canadian citizens and drastically increase the regulatory state, while providing little or no environmental benefit.”

#7 - despite if you think that Fraser might have an agenda, if someone is going to do a study and publish the results, I wouldn’t pay any attention if the results can’t be replicated, and especially if they obstruct efforts at replication. I was taught that this was basic scientific method. When I and other here do studies we identify the data source and describe the methods. I don’t freely offer the exact source code, but if someone requested it for this purpose I would share.

Silver King offers a little “peer review” of our comments.

I’m not sure that what he’s pointing out is really to the point. Lots of work in the study of science has shown that having a vested interest in outcomes produces good scientific work. (If anyone wants a short bibliography on this, I’m happy to recommend some stuff.)

Indeed, the scientific response to the Frasier Institute Study is not to point out what interests they have in their conclusions, but to assess the methods by which they have produced those conclusions; the public attitude of science (which is sometimes violated in private) is that a scientists personal biases are irrelevant to the assessment of their work and a *scientific* response assess the merits of the evidence, experiments and reasons that the work itself presents.

(It’s worth mentioning that there’s something scientifically unorthodox about the Frasier Institute study: it’s not published in a peer reviewed journal. They’re clearly at least partially a policy institute. Nevertheless, a good criticism will look at their reasons and evidence, not their personal, ancillary motivations.)

To a certain degree, every scientist has some personal bias simply because he or she receives grants, tenure track jobs, continued citation and other rewards for having his or her work being seen to be credible.

“This is somewhat true in the academic world as well, but I think it is worse on line.”

It depends a lot on the academic stuff in question. High energy physics experiments are so expensive to run that there’s a cost prohibition on replicating them. That’s an extreme case. (Physicists don’t typically own or operate the accelerators that test their theories; rather they apply to a lab like FERMI lab with their studies and FERMI lab selects and performs the ones they deem most efficient and meritorious.) But even simpler studies can be difficult to reproduce in practice, like one that recently concluded that the evolutionary origin of spiciness in peppers is an anti-fungal property of capsasin would be very expensive and difficult to reproduce because you’d have to spend a few weeks in Peruvian jungles to do the tests that they did.

Most re-checking in academia actually probably comes from those scientists that need to rely on another scientists for their personal research.  In such cases, the cost of relying on faulty work is high; hence, scientists will do careful rechecking. A major explanatory factor of the reliability of science is that significant scientific work is used by other scientists in the process of further discovery. If that significant work is faulty, the faults are unearthed in subsequent use; this produces a kind of incidental re-checking procedure for the most significant scientific claims.

This is a nice site for visualizing those relationships (although not just limited to blogs): http://www.touchgraph.com/TGGoogleBrowser.html

It would be interesting to quantify those weights. Artfacts (http://www.artfacts.net/marketing_new/?Services,Artist_Ranking) doesn’t divulge their precise system, but I believe that a higher ranked artist confers greater weight on a gallery, which in turn increases the weight on other artists exhibiting at that gallery. The same could be applied to blog posts (although maybe in a roundabout way Google already does this).

That is so f-ing cool.... you guys are great!

Studes must be in love with dackle at this point…

I think Silver King raises a valid point.  I wasn’t aware of the institute’s ideological agenda.  As philosofool says, such an agenda doesn’t mean we should reject the study out of hand.  But it does entitle us to bring additional skepticism and scrutiny as readers, especially when the findings are consistent with the group’s agenda.  Surely it isn’t a coincidence that their press release highlights three examples of prominent flawed studies (they claim), and that all three were the basis for forms of government intervention that the institute disapproves of.  Did they find no flawed studies conducted by libertarians, or just not mention that (either way, it concerns me as a consumer of the research). 

Their ideology could also impact the findings directly.  When they say that researchers wouldn’t surrender their data, do they mean “wouldn’t give it to the Fraser Institute?” I can certainly imagine why researchers might not respond to a group they think has motive/intent to discredit their work, rather than conduct a fair evaluation.  So I think the sponsorship does matter....

I like Rob Neyer but I think he came off as a little harsh in this case.  Rob has editors at ESPN, but it is a luxury most bloggers cannot afford.  So of course they are going to make more mistakes.  For instance I did a 45 Worst Contracts list, and once people took their hacks at it some needed revisions became clear.  Of course by then it was too late, I’d already published it.

Strangely enough, I was looking up “Linear Weights” this morning on Google, and was pleasantly surprised to see that Tangotiger.net was the #1 result.