from Amazon
Wednesday, October 15, 2008
Challenging Nate Silver (and all other forecasters)
Nate issued a challenge to a pollster a few months ago. That pollster, too “big” for Nate, didn’t take him up on it. Nate, wisely said, regarding PECOTA:
“Sometimes being more accurate means you’re getting things right 52 percent of the time instead of 50,” says Silver. “PECOTA is the most accurate projection system in baseball, but it’s the most accurate by half a percent.”
Which is of course what I’ve been saying. PECOTA is not necessarily the most accurate (Chone might be), but the basic idea is that same: the best one is an 84-win team (.519), the next best ones are 83-win, and your typical baseball fan is an 81-win team. So, calling one forecasting system “the best” would be as dubious as calling an 84-win team better than an 83-win team, especially since we are not even sure if PECOTA is the 84 or the 83 win team.
Anyway: Nate Silver, MGL, Dan Syzmborski, Rally Monkey, David Gassko, Chris Constancio, Bill James, Pete Palmer, Ron Shandler, Tom Tippett, and anyone else. And I’ll put in Marcel of course, “the people’s choice”. Step right up. Time for a challenge and put all the forecasting systems out there to fight it out. I have a simple enough idea, which I’d be open to modify. Each one of you provides forecasts for as many players as you wish. I’ll tell you exactly what metric I’ll be using, and what position each player can play. You provide a ranking based on that metric. I then put you in one huge league. I’ll draft the players for you in snake-style draft. That’s the team you get for the whole season, no trading or moves.
Then, I’ll do it again, with a new order. And then again. And again. And again, and again, and.... and again. I’ll do one hundred (maybe one thousand) drafts, all done programmatically. Then, we’ll see how everyone does at the end of the year. The winner gets to say “Best Forecasting System of 2009”. No one gets to put on their book “deadly accurate” unless it’s true.
In the spirit of fun and highlighting how close all these forecasting systems truly are, I encourage all you readers out there to push for your favorites to join. There’s no reason for any of them to be all snooty like American Research Group was to Nate.
I will update this post, as challengers accept.
For those who don’t know me, I created the Marcel The Monkey Forecasting System (affectionately called The Marcels) as a way to provide the minimum competence level all forecasters should beat. I want and prefer that the Marcels not win. And I have even provided the exact model so that you can create your own. So, there is no conflict of interest here.