THE BOOK--Playing The Percentages In Baseball

Good article. I like this quote:

“Many authors are nowadays determined to achieve publication for publication’s sake, in an effort to secure an academic position and are not particularly swayed by the argument that it is in their own interests not to publish an incorrect article.”

I am also shocked to find gambling at Rick’s.

The market idea for publishing is neat.

In my experience, which is limited to economics, the referee process works very well at top journals and then degrades rapidly with journal quality.

It’s a shame that the article does not discuss testing these claims with data.  We have lots of measures of paper success and journal prestige. The fact that top journals consistently publish the highest quality papers (I would take a form of this bet at 99:1 or more) is fairly good evidence that the peer review system is working correctly, at least at the top end.

Along these lines, check out Bradbury’s latest post: http://www.sabernomics.com/sabernomics/index.php/2010/09/on-the-jqas-study-of-attendance-and-winning/. It looks like people just keep shopping papers til someone will publish.

mcnside, I interpret this the other way: JC is worked up over something relatively minor. His complaints have nothing to do with the methodology, questions, or content of the paper. Instead it is about miscitation of a referee’s own work. Yes we want to fix those. But papers should be evaluated on their contributions. Harping on this stuff is silly.

I’m sure that journals had their day, but they seem hopelessly antiquated now. I am really only familiar with nutrition and exercise publications, and they are, for the most part, embarrassing. It’s more cronyism than any attempt to produce excellent (or even decent) work.

I liked the marketplace idea, but I think we already have it most of the way. When I publish an article on HT it gets commented on. If people think I’m all wet then I will hear about it. Other people will see those comments as well. For me, the gold standard is getting mention on The Book Blog.

If these journals were similar published in an open format like HT or most of the other online magazines then anyone would be free to comment, discuss, laud, pan, what have you. The cream would rise to the top on it’s own. I have a feeling than many in the academic world would not appreciate this level of scrutiny - although the good one’s would.

I suppose this sort of breaks their economic and social model. Probably time, in my opinion.

My standard is simply: if you were to start a clean slate, would [enter whatever object, task, process] be done this way?

In the age of internet, with information flowing as fast as possible, what would happen if someone had this novel idea:

“Hey! I have a great idea!  We’re going to have a print document!  Not only that, it’s going to take 6 months from submission to printing!  On top of that, we’ll get other like-minded folks with similar talents as us to review things!  I know, I know, we have 10,000 readers a week looking at our site, but if we can limit that to the 5 best referees, wouldn’t that be great?!?!”

Uh, no, I don’t think that system would ever come to fruition.  It’s ridiculous frankly.

Eric S. Raymond is known for stating that, in open source software, “all bugs are shallow with enough eyes on them.”

Eric S. Raymond is not known, for example, for his work in missile trajectory calculations or temperature sensor data analysis.

I’m simply pointing out that, in some (many?) places, enough eyes is sufficient to be confident that the presented data and conclusions are correct. In others, not so much.

Andy,

You can take JC however you want. Personally, I thought he was pretty harsh. That said, he’s presenting clear evidence that the authors simply shopped their paper til they found someone to publish it. Kinda blunts the whole theory behind peer review if you can just keep shopping til you find peers who don’t hate your paper.

I’m with Tango on this one. Throw it on the internet and ask for feedback. I think it’s stupid to “publish” research in journals that few people have access to unless they’re willing to shell out hundreds of dollars. Information wants to be free. It’s stupid to hide it away for only the initiates.

mcsnide,

This isn’t as terrible as it sounds.  There’s definitely a hierarchy of journals so, really, you work your way down.  If you don’t make necessary changes (or, maybe, it’s just not possible), then your paper ends up in a less-respected journal and it’s given less weight.

Depot is right: “shopping your paper” just means moving down the hierarchy.

As to the point of refereed journals: I agree that new findings are often “refereed” more quickly and effectively through the internet/blogosphere. Although before this, it was not typically the referees who validated work, but people presenting at seminars and conferences and getting feedback. That happens faster now. The media’s focus on the ‘refereeing process’ always kind of baffled me.

One large value of having journals and a hierarchy of quality is that it makes entry into a field much easier. If there have been 10,000 papers published in the field you are interested in making advances, it is impossible to read them all and quickly figure out where the cutting edge is and what are the important outstanding issues. Quality control and journals help solve this problem by helping researchers focus on papers known to be very important.  Journals are records of important discoveries.