from Amazon
Friday, January 27, 2012
Who’s evaluating the 2011 forecasts this year?
Anyone going to step up? Anyone? The hard part is collecting all the data, and matching all the players. If someone ELSE does all that hard work, I can step in and do the rest.
The test is pretty simple.
1. Calculate wOBA for every forecast, and for the actual. I’ll do something simple like
numerator = 0.7*BB + 0.9*1B + 1.3*(2B+3B) + 2*HR
denominator = BB+AB
It really doesn’t matter much what you do here. You just need something that focuses on the important stats, and make sure everyone forecasted those stats.
2. Calculate each population mean, by weighting by actual PA (AB+BB). For missing players, either give them the population mean (you HAVE to do this for Marcel, since by definition, Marcel has no missing players), or set the wOBA at 20 points below the rest of the population mean.
3. Recalculate new population mean (where applicable).
4. Baseline each player to a common mean (set to .330, but it doesn’t really matter what you set it to). So, if the pop mean in #3 is .327, and you have a player forecasted for .377, his adjusted forecast is .380.
5. Calculate the difference for every player.
6. Present the average absolute difference, and the RMSE, and in both cases, weighted by the actual PA.
That’s it, that’s the basis.
Then you can do fun stuff, like splitting by career performances. Guys with 1500 or more PA in the last 3 years, guys with fewer than 250 PA, guys who had a .380+ wOBA in the last three years, shortstops, etc, etc. Look for whatever attribute of a player you want. And compare the systems, and look for bias.
Bueller?