THE BOOK--Playing The Percentages In Baseball

The “pseudonymous” jab is a red herring.  If you started making crazy assertions and stopped using evidence to back up your claims, you would be criticized and it would stick just as surely as criticism sticks to Bradbury.  It’s not as though you could start posting under a new pseudonym - everybody would know it was you.  So your credibility is on the line just as surely as his is.

A couple things:

1) This really all goes back to Steven Levitt’s ridiculous attempt at sabermetrics and JC’s attempts to defend it b/c he couldn’t stand the idea that someone with credentials might know less about a subject than someone without.

2) I’m not sure I agree with Tango’s conclusion that the equation in that paper is ridiculous.

What the equation suggests is that *any* veteran player who was given 600 PA in a given year is likely to have been paid more that year than a veteran who was given 400 PA and hit .330/.400 (or whatever it was).  It’s not really saying that he had more value that year but rather that he was anticipated to have more value (got paid more).  That doesn’t seem ridiculous.

Am I misunderstanding your objection?

J/2: Right, that is the suggestion.  And, unless you can show me the empirical data upon which this is made, the status quo has to be that it’s a ridiculous claim.

Furthermore, the way each parameter is used in the equation makes no sense.  Who would mix a counting stat with a rate stat the way it was done?  Not to mention the FA, arb, slave status.  The whole idea that they can throw in the paramters in the way they did to come out with something meaningful, when we have a better system of combining various performance stats is preposterous.

I can go through each of their equations that they published and show how it has no logical underpinnings.  That is, correlation is not causation.  That they then decide to take each annual equation and try to come up with a trend is simply unacceptable.

You know, I was going to say what I thought about JC, but I couldn’t think of how to say it in a way that is polite enough for this blog.

Blackadder-

My thoughts exactly.

J: I should correct you that the performance data was from the prior year.  So, 600 PA in year X-1, of whatever performance level, are being paid more in year X than the .330/.400 guy with 400 PA in year X-1.

Here’s another example of the Sakes/Hauer model, if you use their equation from Table 2, years 2001 (which you can see in post 29 above).

Make someone a free agent, with 600 PA, a .330 OBP, .400 SLG outfielder.  His salary is 3.2MM.  Make him a SS or C, and it’s 3.4MM.

Does that make sense to anyone?

What if we give him 500 PA instead, with a .500 SLG and .400 OBP.  Pretty attractive, right?  Well, it’s the same 3.2MM$ salary for the OF, and the same 3.4MM$ salary for the SS.

Now, if they are suggesting their empirical model, based on actual performance data of 2000 that says that an OF with 600 PA, a .330 OBP and .400 SLG is making in free agency as much as the SS with 500 PA, with a .400 OBP and .500 SLG, then should I accept that?  Do I need to go further to disprove them?  Even if they can somehow justify it, isn’t it more plausible to think that this is some sort of anomoly whereby the uncertainty in the data is so large as to make such a conclusion insignificant?

You have to have SOME logical underpinnings, don’t you?  You can’t just throw everything into the regression grinder, and think that you will get a filet mignon, or even a kielbasa?  Sometimes, you get dog food.

Why is it to me to disprove such an unreasonable claim, and not for the authors of the paper to prove it?

***

Primer thread:
http://www.baseballthinkfactory.org/files/newsstand/discussion/chop_n_change_interview_with_jc_bradbury_economist_and_sabernomics_blogger/

*******You can’t just throw everything into the regression grinder******

Dude, people have written books about regression analysis.  People even lecture about regression analysis at some of the finest universities in the world.

All kidding aside, fuck you JC for suggesting that Sabermetrics is all about rhetoric and not (serious) research. The problem with academics is that they often aren’t students of the game and they wrongly assume authority when stepping outside their field to describe the poetry that occurs on the field....

I don’t get it....a virologist wouldn’t spontaneously decide to go to a conference on neurosurgery and be confident in the assumption that he knows all he needs to understand in order to apply his specialty across disciplines....

BTW...JC, puts the piled higher and deeper in PhD with rants like the most recent one he embarrassingly and so casually injected into the marketplace…

Other than that, he cool.

Can someone please point me to the original article or Tango’s comments to it?  I am curious as to exactly what was the original claim.

Thanks.

http://www.insidethebook.com/ee/index.php/site/comments/why_i_hate_regression_techniques/

Tango-

Thanks.  I’ll read it this evening though I’m pretty sure I know how this story will end already.

-Eric

JC left a comment on the blog post.  A couple of parts that rubbed me the wrong way:

And finally, if you think there is a better way, then do a superior study. Science doesn’t work by pointing out imperfections, it progresses by replacing the inferior with the superior. Seize the opportunity and do the study. Should it yield different conclusions, I assure you that Hakes and Sauer would agree that you have surpassed their work.

Is not pointing out potential flaws in the initial study a prerequisite to building upon it?  And how would the vaunted peer review process work if reviewers were required to produce a competing study before commenting on the initial?

Instead, you defer to a guy who is a prolific writer, but doesn’t even use his real name. The only people reviewing his work is another group of people who may or may not be qualified to evaluate the argument.

I say this with all due respect. 

What a prick.

I don’t think JC is due much respect, David.  He has yet to earn any as far as I can tell.

The part that rubs me wrong is this:

During my hiatus, I spent most of my time working on a book project. Although it was not my original intention, the book is completely about valuing players. In The Baseball Economist, I used a simple system that I had developed to value players. It was actually developed as a side project, but I thought it might be of interest to readers so I included it in the book at the last minute. I’m glad I did, because when I used it to value players on Sabernomics, I received a lot of feedback, especially questions. So, I decided to lay it all out there; and when I was done, I had a 200-page manuscript instead of the single chapter I had planned on writing. I completely tore down and rebuilt my model, which allowed me to improve the model and correct past mistakes. The book is at the editing stage, and I’m just beginning to circulate drafts to publishers right now.

Talk about rewriting history in his favor!  As if Colin didn’t take (venerated but anonymous sabermetric icon) Tango’s WAR model and knock such a huge hole in the ship that was JC’s model that it sank to the bottom of the ocean.  And as if it wasn’t Colin’s thorough thrashing of him on his own blog that precipitated his hiatus.

To admit that he had to completely tear down and rebuild his model because Internet sabermetricians tore it to shreds would be too much to expect, I guess.

I’m not seeing any obvious negative emotion in my response. If you are interpreting this, I want to make it clear that offense wasn’t a response I felt.

No, no, of course not.

I can’t tell my dean, Yoda27 thinks my study is nifty, nor does he care that FatBoi34 thinks my work is hogwash.

Right.  I see.  Mmm-hmm.

Mike, I can’t wait till JC rips Pitch f/x.  I’d love to see your response

Also, I love how Colin’s 3 1/2 excellently researched, peer edited, and non-pseudonymous articles on THT - which is definitely not a yoda27 type blog (not like there is anything wrong with that) - don’t count as legitimate academic criticism.

JC should just go away.

"Instead, you defer to a guy who is a prolific writer, but doesn’t even use his real name. The only people reviewing his work is another group of people who may or may not be qualified to evaluate the argument.”

Apparently ad hominem rebuttals are acceptable in academic circles?  It’s not the quality and logic of the argument being made but the accepted credentials of the person making the argument.  How enlightened.

But I guess when you can’t defend your position on its merits, you don’t have many other options.

What’s ironic, is that his point presupposes that the only relevant credentials are the statistical kind.  An understanding of the underlying system being analyzed is irrelevant...?  And if it’s not, what’s the system for identifying someone’s bonafides in that area?

http://www.hardballtimes.com/main/article/the-root-part-1/
http://www.hardballtimes.com/main/article/the-root-part-2/
http://www.hardballtimes.com/main/article/the-root-part-3/

Seriously, those would easily be excepted by whoever JC is appealing to.  He used his full name, cited sources and had it edited by the excellent editors at THT.  There is no possible way that you could dismiss those articles simply because it was on a blog - if THT even counts as a blog.

So JC thinks The Baseball Economist is scholarship and “The Book” is rhetoric.....

Again, fuck you prick. And by the way, hubris and scholarship aren’t compatible.

I have no idea why he is making me the centerpiece of his bashing. 

In any case, I’m open to discussing any article, research, book, or whatever that I have produced that JC is criticizing.  Because, if I were to strip away all of his griping, and focus on the merits of his arguments regarding my work, I see nothing at all of mine that he is actually challenging.

All we get is his summary opinion.  And opinion without evidence is, as I’ve said in the past, the very definition of b.s.

I am catching up. 

I read the Hakes and Sauer article ‘An Economic Evaluation of the Moneyball Hypothesis’ and am a bit confused on how they reached their conclusion that an inefficiency existed in the pricing of offensive performance for the period 2000-2003 but that it was largely corrected for 2004. 

Specifically, I am curious how they came up with a pricing model that:

1. Includes players not yet eligible fo arbitration.  I am curious how salary and performance correlates for that group.

2. Includes multiyear contracts.  I am curious how previous year performance correlates with current year salary for a contract signed before the end of the preceding season.

3. Includes Plate Appearances.  Because PAs can be earned for non-offensive reasons they seem to be adding noise to a study that purports to value offensive performance.  To a lesser extent, they may be collected more often by high OBP players at the top of the line-up thus deflating OBP coefficients (or, perhaps it works the opposite way, deflating the SLG because of the lack of high SLG guys at the bottom of the line-up).

I think there are also problems lumping in arbitration eligible players (whose salaries aren’t solely market dependent, ignoring replacement level and using OPB and SLG as the offensive metrics (It’s been a long time since I read Moneyball, and Lewis may have couched many of Beane’s strategies in those terms for the telling; but it seems likely better metrics were in place in Oakland and available to Hakes and Sauer).  I suspect readers here may be able to suggest a few more improvements to the original paper’s process.

I read Tangotiger’s ‘Why I hate regression “techniques”’ and thought the examples there adequately show that the model used by Hakes and Sauer (and, by implication, any conclusions drawn from using it) should be considered suspect.

The other interesting thing about this is the assumption that works published in peer-reviewed journals have undergone a more rigorous review process than internet blog posts. The peer review process for a journal article is a crapshoot. You can get friendly/unfriendly reviews based solely on who is picked to review the article. And while anyone is free to write a rebuttal, the rebuttal itself needs to get past the same process to appear in print. So it’s entirely possible that a mistaken article could get friendly reviews and a correct rebuttal unfriendly ones. Then, you’re stuck with a wrong, uncorrected article in press.

OTOH, posts on this site and others may be replied to by anyone… including the same people that could be picked to review it by a journal! Furthermore, the back-and-forth dialog allows for a more efficient form of communicating and correcting errors.

I can’t say for certain that one is better than the other, but there are certainly inherent advantages to this medium and if I would guess that the “classic” peer-review method is the worse option for finding and correcting mistakes. The assumption that the other is better isn’t just haughty. It’s also possibly (and in my estimation, probably) incorrect.

I think the Kovash/Levitt paper is a good example.  Kovash was ALREADY aware of the two major problems we had with the paper (he told me himself).  But the paper was out there WITHOUT that notation.

So, as Shawn/23 is noting, it is a very inefficient form of communication.

... and to provide an example of how this works in the “untrusted” world of blogs, take a look at the last post I happened to comment on here:

http://www.insidethebook.com/ee/index.php/site/comments/some_actual_data_and_research_on_hot_and_cold_pitchers_for_one_game_rather_/

I thought there was a mistake, brought my concern up in the comments, and mgl quickly admitted his error. And that all happened within about a day of the original post… and the correction within a couple hours of the issue being raised. Boy do I wish the peer-review process were that open, efficient, and apolitical. My “real world life” would be full of a lot less stress.

I have nothing to add ... you guys are nailing it.  I’m just posting this to get on the subscription list for the thread.

In the comments, JC says this:  “I disagree with many economists on various issues, sometimes siding with non-economists over economists....If I thought the criticism was valid (and I do have significant experience doing and teaching econometrics) I would agree that the findings were wrong. But I think the Hakes and Sauer paper’s findings are right.” JC makes this claim every time he defends the work of a fellow economist.  But here’s the thing:  he always defends their work.  He and Berri often praise each others’ work, and even defend one another when one is criticized by non-academics.  But I never see these guys (and the others at the Sports Economist) offer serious criticism of each others’ work.  Note the total silence about the Levitt/Kovash paper.  Whether they’re incapable of seeing these errors, or choose not to say anything, I can’t say.  But the result is a mutual admiration society.  And they demonstrate the solidarity of a religious sect if outsiders offer criticism.  So when JC says that “capital S” sabermetrics has become a “club,” he would better apply that description to sports economics.  There is more mutual criticism on this blog in one day (or on BTF in one hour), than the sports economists muster in a good year.

(The one exception is that Bradbury criticized Zimbalist.  But that was only because Zimbalist savaged JC’s book.  Zimbalist acts a bit like the dominant bull elephant trying to protect his status.)

But lets take JC at his word here, and assume he thinks Hakes and Sauer are right.  What does that say about his judgement?  The central claim of the paper is that in one year, 2004, OBP went from being severly under-valued to properly valued.  And yet, the vast majority of the players in their sample were in a multi-year FA contract, or were still subject to arbitration decisions by arbitrators whose decisions are governed entirely by PRIOR years’ contracts.  It is simply impossible for player salaries to adjust like this in a single year.  Putting aside the details of whether Hakes and Sauer’s model was or was not properly specified, the simple fact is that they cannot possibly be right—it’s an economic and mathematical impossibility. Doesn’t that matter?  To stand by this conclusion, even after later research reveals that this “correction” was gone by 2006, is unfathomable.

Similarly, how can anyone run a model that says the worst hitter in baseball is worth $12M, and not say “oops, my model clearly has an error?” I don’t fault JC for having a problem in his model—could happen to anyone.  But where is his judgement and basic common sense?  This tells me he is using these econometric tools without the necessary constraints of good judgement and common sense.

I’ll ditto Guy/27.

***

I just posted a new thread:
http://www.insidethebook.com/ee/index.php/site/comments/the_sauer_hakes_moneyball_spreadsheet/

Hopefully that spreadsheet is editable.  This way, you guys can plug in the numbers yourself and see what you get.

I don’t recall f-bombs ever being dropped on this blog, certainly not directed at someone.  Well done, JC.

Of the many reasons Bradbury is absolutely infuriating to me, one of them is the horrible name he is giving to academics.  For the opposite extreme, check out, say, Terry Tao’s blog:

http://terrytao.wordpress.com/

Ok, a lot of the posts there are pretty technical, obviously.  But there are plenty of posts designed for pretty general audiences as well, which elicit plenty of responses from people who clearly are not mathematicians.  Tao is never anything less than a complete gentleman with commenters, spending lots of time explaining things and readily admitting when he is wrong.  See, for instance, this thread:

http://terrytao.wordpress.com/2008/02/05/the-blue-eyed-islanders-puzzle/

Tao, of course, is a million times smarter and more successful than Bradbury (or pretty much anyone else).  He’s one of the best mathematicians in the world.  If anyone could be an arrogant prick and brush of criticism, it’s him.  My point is not that JC should be as smart as Tao, but that it’s possible, even when you are dealing with subjects MUCH harder than those Bradbury is dealing with, to engage with the broader public in a reasonable and responsive manner.

So I sat down and read the paper. Other than all the obviously wrong things already pointed out - there are massive problems with assumptions of causality here that are, quite frankly, contrary to the nature of human existance.

When Bradbury says:

The coefficient on OBP is negative—higher OBP lowers your salary. You don’t need to plug in any values to see that this is counter-intuitive. Part of the reason why the salaries remain so stable when Tangotiger adjusts the inputs is that the higher value for OBP cuts into the impact of SLG. As Hakes and Sauer acknowledge in the text, the coefficients on OBP are not even statistically significant—the market appeared to be ignoring the relevance of OBP at the time. That’s their argument.

He is completely ignoring, um, the temporal order aspect of causality. To wit - OBP in that season cannot affect a player’s salary, because salary is set prior to the season! OBP in year X (potentially) affects salary in year X+1 and later, but it cannot change salary in year X. (If this isn’t something that comes up frequently in economics reserach, perhaps they should involve a physicist or a Time Lord or some other “linear nature of time” subject-matter expert in their peer-review process.)

Now of course they compound the problem by including players on multiyear contracts, whose salaries are (roughly) the same in year X and year X-1 - in other words, the market made the exact same decision on those players in both years. Those players in fact should have outnumbered the players who were free agents and were “reevaluated” in the market! In other words, if a player signs a multiyear deal in 2002, how he performs in 2004 says nothing about how teams view his OBP or SLG - no matter how he performs, his salary remains fixed. If his OBP jumps from .338 to .408, it doesn’t mean teams value OBP more highly; it in fact cannot mean that.

Future research may in fact validate or invalidate the conclusions of the study - Bradbury on his part is urging people who disagree with the study’s methods to prove the conclusions wrong. To be blunt about things:

YOU FAIL THE SCIENTIFIC METHOD.

A badly designed study that reaches the correct conclusion is still a badly designed study. And Bradbury’s constant invocation of peer review in defense of the study does far more to destroy the credibilty of peer review in economics than it does to revive the credibility of the paper in quesiton.

Colin:
That’s not a fair criticism.  Hakes-Sauer use the prior year’s offensive performance as their independent variables. 

However, the point about multi-year contracts is right, and also applies to the Arb players whose compensation will be based entirely on similar contracts signed prior to the year in question.  (There’s also a question as to whether a dummy variable for Arb player fully captures the downward impact on salary of not being a FA).

Guy is correct:

“Since salary is determined prior to performance and is based on expected productivity, we use the prior year’s performance as an indicator of expectations.”

So, I missed that. My appologies.

That said, yes, the only players that matter for such a study are those players whose salaries are determined after the prior year. (And you’re right about the arb players - their salaries are determined by what the ARBITRATORS consider important, and thus are a confounding input if you’re trying to isolate what TEAMS value.)

Well, arbitrators base their decisions on prior settlements, and probably loosely follows the FA market.  So it’s not the arbitrators’ opinion, but precedent that largely rules.  But precedent, by definition, can’t reflect some new valuation by GMs. 

If you look at a large pool of seasons, I don’t know that it’s a huge problem to include multi-year contracts.  If you studied the relationship of salary in year Y to avg performance in years Y-1, y-2 and y-3, I think you could discern what teams were paying for, even though some of those contracts were signed in year y-2 or y-3.  The key is looking at a LOT of years together.  The single-season models are nonsense, as H-S themselves reveal in the 2nd paper (though they seem unaware of this).

Guy, yes, fantastic point about how arb salaries are set.  They are based on “comps”, and so, it would be impossible for a market to correct itself so quickly.

Don’t take this to mean I’m not on your side here, Tom (I am), but before the 2008 season there were 8 cases decided by an arbitrator.  All the rest settled, presumably based on contemporary valuations rather than the way arbitration works.  So, the point about how arb. works is valid, but not terribly significant…

This link lists the 2008 cases that went all the way to an arbitrator’s decision…

http://stlcardinals.scout.com/2/732598.html

Here’s a link listing the 7 cases that didn’t settle prior to the 2004 season:

http://sports-law.blogspot.com/2004/02/baseball-arbitration-results-via.html

Most players who file for arbitration settle before going before the panel, but their salaries are still determined largely by the restraints of the arbitration process.  The player and club each submit a salary figure that they think the panel will feel is fair, and the panel decides between the two figures (not just that the panel sets a value-it has to be one or the other).  So both sides have to submit figures based not on what they value, but based on what the panel will value the player at based on precedents.  When the two sides settle, it is at some point between the two figures submitted within the restriction of the arbitration panels ideas of player valuation.  In most cases, if the panel were to assign a value and not just pick one side’s submission or the other’s, it would probably be close to what the players and clubs settle at, and if either the club or player didn’t think the panel’s valuation would be between the two submitted values, they wouldn’t settle.

If a club values a player at $5 million, but the arbitration process sets the value at roughly $3 million based on precedents, they won’t submit $5 million to the panel.  They might submit $2-2.5 million.  They’re not going to settle for that much higher than the panel values the player either.  If the submissions to the panel are $2.5 and $3.5 million because both the team and the player know that precedents set the value at around $3 million, they are never going to settle at $5 million just because that is what the team values the player at.  The team would just laugh at that idea and go before the panel until the player agrees to settle somewhere between their submissions.  Everything is still restricted by the panel’s method of valuation.

Once again, ditto Kincaid.  It is set by precedent, and based on how both sides think the panel will decide.  This does not change over the course of one off-season.

I have nothing to add but am just commenting to subscribe so I can read future responses.

“ ... relative to other academic disciplines, which have the strongest tendency to celebrate difficult work but ignore sound-yet-unimpressive work?  My hunch is that economics, along with most other social sciences, would be the worst offenders, while the fields closer to engineering will be on the other end of the spectrum.  Engineers should be more concerned with truth since whatever they build has to, you know, work.  What say you?  More importantly, anyone have any evidence?”

From here.

Hakes and Sauer provide, inadvertently, a really nice example of confirmation bias.  In the first paper, they happen to find a large coefficient for OBP in 2004 and decide it’s a trend because it’s what they hoped/expected to see.  In the second paper, http://business.clemson.edu/Economic/storage/papers/Hakes-Sauer-Moneyball-IJSF-Sept07-rev3.pdf, they again find that OBP was undervalued in comparison to SLG.  But their data also pose a bit of a problem, because there appears to be a general upward trend in the price of BBs and OBP, starting 10 years before Moneyball.  See table 3 which shows the coefficient for OBP rising:
1986-93 0.49
1994-97 1.14
1998-03 1.39
2004-06 3.36
That shouldn’t happen, economists say—the correction should be sudden.  So how to Hakes and Sauer deal with this?  They explain it away, by breaking down the data into one-year models (table 5) and then abusing the meaning of statistical significance: 

“Looking across the 21 sets of annual labor market regressions in Table 5, it is
clear that hitting for power has consistently been rewarded in salaries, and at more or less
the same “price.” Even hitting for average (Batting) has traditionally influenced salaries, allowing for a couple of tiny hiccups. Plate discipline (Eye), however, and on-base percentage as a whole were typically ignored prior to 1995. Then, after a hesitant flirtation in the late 1990s, returns to on-base and Eye dropped sharply in 2001 and remained statistically insignificant until the sudden two-year burst in 2004-2005. The magnitudes of the coefficients are still reasonably high in 2006, but they are not statistically significant.”

It’s true that in most years before 2004, the coefficient for OBP is not significant.  But they’ve already revealed that when you pool the data into multi-year segments, the relationship IS significant and there’s a clear upward trend.  So they just make it go away by creating smaller samples in which the coefficient is no longer signficant!  I also like how the 2003 coefficient of 2.12 is dismissed as “insignficant” while the 2006 coefficient of 2.14 is “still reasonably high.”

The secular increase in OBP coefficients may mean the price of OBP was in fact rising pre-Moneyball.  Or this may simply reflect the fact that OBP was becoming more highly correlated with SLG (something the authors seem unaware of), or just be a function of the errors in their model. I don’t know.  But I do know that you can’t make it go away by slicing your sample into smaller pieces. 

Table 5 also shows the separate salary coefficients for BB rate, BA, and power (and OBP and SLG) for each year 1986 to 2006.  What it clearly reveals is the enormous instability of these annual models.  For example, in 1996 the price for BA dropped by 65%, then tripled the next year.  Any fair-minded person looking at this table would say you can’t possibly draw any meaningful conclusions from dramatic 1- or 2-year swings in these coefficients.  Yet they stick to their story about a 2004 “correction” in the price for OBP. In for a dime, in for a dollar appears to be the intellectual standard in sports economics today.

Guy - That is a perfect example of why so many people mistrust statisticians.  That’s data manipulation at it’s very best.

I don’t know that people mistrust statisticians (certainly, they shouldn’t). 

To say that people mistrust statistics may even be inaccurate.

People often do mistrust the source of statistics and have some awareness that statistics out of context can be misleading (understatement has been my tendency lately).

And that is our theme here.  The authors have stats from a model that does not share the context (has no relationship) to the rest of their paper.

Robin Hanson:

Academics change their beliefs only when a sufficiently impressive work appears to earn that respect, even if that work provides little info.  ...

So a paper suggesting academics change their opinion on a very important subject will be held to a higher standard of impressiveness.  It must use more impressive math, data, analysis, or have a more prestigious author.  ...

These academic blindsides in principle offer an opportunity for bloggers to contribute to intellectual progress via thoughtful posts that add info but are not impressive enough by academic standards, and via drawing reasonable conclusions from these and other unimpressive sources to which academics refuse to listen.

Whole thing here.

Robin Hanson is awesome.