THE BOOK--Playing The Percentages In Baseball

I disagree with VW’s conclusion about PECOTA.  Before last year, they were the best system 3 out of 4 years.  Why should we overreact to a bad season so much?  Shouldn’t we weight and regress this data like anything else? 

Still, he’s right that CHONE is probably the best.  They have been the best in 3 out of the last 4 years, and they have been the best in their “career” as well.

While I think they are all about the same, to be honest, other than a lot of adjustments I do, mine are basically a Marcel. There isn’t any secret sauce.  And I don’t even look at them. The computer spits everything out.

What does RMSE stand for? Which projection system is best for players?

I don’t see any reason to weight the results (if you mean giving recent years more weight), unless someone has changed their methodology for the better or something like that.  And regress?  regress what?  What for?

Standard error maybe, so the comparative results can be put in perspective.  IOW, if for the last 3 or 4 years, everyone is withing 1 or 2 standard errors of one another, they are all basically the same…

Unless the O/U (I assume it is the Vegas over/unders taken from one or more sports books) he uses are prior to when all the other public systems come out, the reason they do so well is because they converge towards the public systems.  Basically the publication of Pecota, Chone, ZIPS, and all the other public projections have ruined any betting opportunities!  Without them, the Vegas over/unders would be terrible.

Remember what Vegas had in 2008 for the Rays (60 something?) and what most of the sabermetric forecasters had (80 something?)?  And then the Rays’ Vegas number moved probably 5 or 10 wins.  After that fiasco, the Vegas numbers pretty much reflect the public forecasts, at least the reputable (sabermetric) ones…

And there definitely is no reason to criticize Pecota just for having a bad 2009.  As you can see, for 05-09 they were in the middle of the pack of the 5 best of the best and pretty much indistinguishable from the top 2.

twac00, RMSE = “Root Mean Square Error,” which is just a fancy term for taking the square root of the sum of all the squared errors for each team.

If I have 70 wins for Team A and they win 68, and
80 wins for Team B and they win 85, and
90 wins for team C and they win 86 games,

the RMSE is:

the square root of:

70-68 squared plus 80-85 squared plus 90-86 squared, or

sqr(2^2 + -5^2 + 4^2) or sqr(45) or around 6.71.

You don’t think that projection systems have sampling and random error, just like most things in this world?  Serious question.  Especially given that we are only dealing with 30 observations a year, if you are going to be projecting the projectors, it seems very necessary to weight and regress.

Like MGL says, unless there is a known change in the projection engine, why would you weight the years differently?  It’s more or less the same system running year after year, and some years it will be better or worse due to sampling or random error, but whether it got lucky or unlucky last year as opposed to 2 years ago doesn’t really make any difference.  As long as it’s the same projection system, it’s safer to assume that its forecasting ability hasn’t changed than to assume that because the RSME changed, the game somehow changed to better fit the system already in place.  So we should act under the assumption that the true forecasting ability of the system hasn’t changed and take all our sample at full value rather than diminish older results.

The point of weighting years for projecting players is that players’ talent levels change.  This year tells us more than last year because maybe the player didn’t just get lucky or unlucky, maybe he also got better or worse.  There’s no reason to assume that for a projection system (unless you know there were significant changes), so there’s no reason to weight years.

Didnt Nate Silver leave BP?

Could that be the reason for PECOTA performing worse this season?  Maybe the methodology HAS changed.

It is definitely true for Marcel that it’s one system run the same way every year.  It might be for MGL, as it seems his system is Marcel with MLE/Park/League/quality of opponent adjustments (and I might have left something out.

In my case, the only thing that really stays the same is the name CHONE and the fact that I run it.  I’m always changing things here and there, and will again this year once I get all my data lined up.

From what I’ve read about PECOTA, it’s a far clunkier, time consuming system than eve I run.  I thought it was mentioned that it was being moved from excel to a database environment.  That’s a process I’ve worked with as well, my mid season spreadsheet update was produced by Access instead of excel.

There are a few thousands of things that can go wrong in such a conversion process.  If Nate Silver is no longer involved, I’d treat the system as a blank slate, an untested rookie, and not a veteran who just had a slump.  If Nate is still working on it, I would expect him to learn from some of the mistakes of 2009 - Wieters got the most publicity but the Cristian Guzman projection was out there as well - and come back strong.

We have reason to believe that the ‘09 PECOTA was not the same as the prior years:

http://baseballprospectus.com/article.php?articleid=8653

That said, until we look at the individual player forecasts in detail we won’t know for sure what happened. (ID mapping is a pain in the rear, so I’ve been putting off a detailed projection system analysis for a while now.)

Nick, presumably the outputs of each forecasting system already includes the “regressing” although I am not clear what you want to be regressed and why…

I think he means something like if CHONE got a 5.87 RMSE for 2007, then I was probably a bit lucky in getting that, and if you want to project what error my system will have for 2010, you’d take my data points and add in something representing the average RMSE for all forecasters.

But I don’t know why you’d bother with that.  We’re just looking at past performance here.