from Amazon
Friday, October 23, 2009
Should we hold BPro accountable for unannounced changes?
Colin points out:
Baseball Prospectus has a metric called Wins Above Replacement Player, which (ideally) should be a pretty straightforward proposition. But let’s look at a few things. I have a collection of sortable WARP1 reports from October 9th, covering 1996-2009. How does it match up what’s currently on the site?
[snipped examples]
So what’s going on here? Aggregate WARP1 per season has declined by roughly 150 over the past few weeks. That’s right - nearly 150 wins per season have simply disappeared from Baseball Prospectus’ website. Why? It seems that BPro has changed how they convert runs to wins. Before, they were using a scale where 8.2 runs = roughly 1 win. Now it’s a lot closer to 9.2 runs per win. (For a fuller explanation, check out this discussion of the subject.)
This is, by all accounts, good (it has the nice side effect of making individual player WARP sum up to team WARP, assuming a replacement level of .220 - we can have that conversation another time). Just one question - why hasn’t BPro announced this change?
How accountable is a site in making adjustments to their metrics? Suppose Sean Forman decides to change his park factors to do it a different way? Does he need to announce this? What if he decides to change ERA+ to do it as Guy suggests (2- ERA/lgERA), instead of lgERA/ERA? Does he need to tell us? What if Fangraphs updates their WPA numbers to reflect the 2009 park factors, rather than using the 2008 PF?
It seems to me that making adjustments to factors that still keeps the mean and standard deviation roughly equal, but moves individuals up-and-down is one thing, and you can make the argument that a website should not necessarily make an announcement (though announcements are ALWAYS appreciated). But, in the case of WARP, the mean point has changed. I think that makes it announcement-necessary. Especially since Clay wrote an article in the BPro annual of his epiphany, and made the announcement that the mean point of WARP will be changed. I think status updates are appreciated. Clay is a straight arrow, so I’d say it’s more of him making the announcement a low priority, rather than consciously dismissing the users.
Regardless, I feel for any of the researchers out there who download data, and then find that the data has been changed.
Recent comments
Older comments
Page 3 of 344 pages « First < 1 2 3 4 5 > Last »Complete Archive – By Category
Complete Archive – By Date