from Amazon
Friday, February 12, 2010
The Marcels takes on the field
Jared Cross and his students provided me with the raw data of their forecast evaluations. It contains a list of 438 players who had a foreacst for all the evaluation systems, with at least 50 PA in 2009.
Jared and his team also normalized all the foreacsted OPS so that the simple group average matched the actual simple group average of 2009. For example, Aaron Hill is showing with a Marcel forecast of .747, compared to the simple group average of .760, or -.013 relative to the group average. Hill’s actual OPS was .829 against the simple group average of .733, or +.096. Marcel was therefore off by .109 in OPS.
Repeating this step for all the systems, and this is how far off each system was on Aaron Hill:
0.058 Fantistics
0.090 Chone
0.109 Marcel
0.110 ZiPS
0.110 Steamer
0.112 Sporting News
0.135 PECOTA
0.146 2008
Relative to Marcel, we see that Fantistics did much better, and that PECOTA (and just taking Hill’s 2008 performance as the forecast) did much worse. The other systems were all within .020 OPS(*) of Marcel deviation, and therefore were basically similar. I give a win to Fantistics, a loss to PECOTA and 2008, and a tie to the other systems, relative to Marcel.
* Twenty OPS points represents about 5 runs in a full season. I reason that if someone forecasts 87 RBI and someone else forecasts 98, and the actual is 90, then that’s the point where you go from a tie to a win/loss.
I repeated this step for all 438 players. Marcel v Chone gives this tally:
Marcel v Chone
W: 93
L: 107
T: 238
Players: 438
So, of the 438 players in the dataset, Marcel made the clearly better prediction than Chone 93 times. That’s 93 wins for Marcel. Marcel lost out to Chone on 107 players. That’s 107 losses for Marcel. And in 238 players, it was too close to call, as the forecast-differentials were all within 20 OPS points of each other. That is, 54% of the time, it doesn’t matter which forecasting system you choose between Chone and Marcel. With Chone being right 107 times and being wrong 93 times, that means that Chone is just 14 players ahead of Marcel. That is, in 3% of the players does it add real value.
Giving 2 points for a win, 1 points for a tie, and 0 points for a loss, and the average win percentage for Chone v Marcel is .516. That is, we can say that Chone will have the better forecast 51.6% of the time. This is actually the BEST forecasting system against Marcel. Here are the totals for all the systems, based on the data provided to me:
Win% v Marcel
0.516 Chone
0.500 Marcel
0.500 ZiPS
0.487 Sporting News
0.487 Fantistics
0.482 Steamer
0.455 PECOTA
0.377 2008
As you can see, Marcel had an outstanding year.
What if I change the “20” to a “10” OPS poinst in terms of establishing the tie? Things change slightly:
0.516 Chone
0.509 Fantistics
0.500 Marcel
0.495 ZiPS
0.491 Steamer
0.478 Sporting News
0.460 PECOTA
0.364 2008
Fantistics takes a step up, while the others pretty much stay in their spots.
If you have to go with one forecasting system for 2009 (for hitters), Chone would be the one.
I did a similar process in 2008:
http://www.insidethebook.com/ee/index.php/site/comments/evaluating_the_2008_forecasting_systems/#14
In that case, I showed these results for hitters:
win% system W L T
0.530 Chone 234 174 579
0.514 ZiPS 235 207 542
0.482 PECOTA 212 247 528
0.473 THT 204 257 526
In terms of the big 3 (Chone, ZiPS, PECOTA), it was the same order as in 2009.
And in terms of overall (hitters and pitchers), I said this in the same thread for the 2008 systems:
Adding up the hitters and the pitchers, and we get:
win% system W L T
0.522 Chone 418 339 1076
0.502 PECOTA 413 405 1015
0.501 ZiPS 433 431 969
0.476 THT 374 463 996Chone, among these 4, is the clear leader. I hope no one tries to tout their system as the best, other than maybe Chone.
Matt also studied the forecasting systems for the 2007/08 seasons:
http://statspeak.net/2009/04/testing-the-projection-systems-strengths-and-weaknesses.html
(link broken) and said this:
--CHONE was the best at projecting most things.
Therefore, once a system is able to beat Marcel, Chone is the next system up the ladder that a forecasting system needs to challenge. Chone is the heavyweight.
Good stuff. Yes we will run the same thing for pitchers.
My bold prediction: 2010 Steamer will beat Marcel.