THE BOOK--Playing The Percentages In Baseball

Tuesday, November 29, 2011

Forecaster’s Challenge: Final 2011 Results

I ran four competitions, three unofficial, and one official. I’ll run them all down.  I’m going to list the results of all the pro forecasters who finished ahead of Marcel.  For those that finished below Marcel, I will list them in alphabetical order. 

FAN_ID: a unique number for internal purposes

POINTS_CT: the average points for each team.  The league leader in Fantasy points has around the same number of points as the league leader in RBIs does.  That’s the kind of scale you should think of. 

WINS_CT: the number of leagues won.

VALUE_CT: the number of draft points won in the league.  You get 11 points for a win, 5 for a 2nd place finish, 3, 2, 1 for 3rd through 5th.  C

First, the unofficial competitions.

Setup 1: All Pros
All 22 Pros in the same league, 100 leagues.

We see that the winner is the Consensus pick, which is the consensus of the other 21 forecasters.  Rotoworld was a close second (and Rotoworld also finished in 2nd place in the inaugural competition two years ago).  Steamer, a forecasting system created by students of Jared Cross did well. 

Other highlights are the Fangraphs Community being average, and Marcel not doing well at all.

Baseball Info Solutions
Future of Fantasy
Geoff Buchan
PECOTA
Statspeakblog

Setup 2: Head-to-Head Pros
Two-person league, each Pro facing off against one other Pro, 42 leagues each Pro

Rotoworld won 41 of its 42 leagues, while the Consensis was close behind.  Marcel has a slightly above-average showing, winning 24 of 42.  The Fangraphs Community was right behind Marcel.

Ask Rotoman
Baseball Info Solutions
BigScoreSports
Bloomberg Sports
Fangraphs Community
Future of Fantasy
Geoff Buchan
John Eric Hanson
PECOTA
Pat Senechal
Rotochamp
Statspeakblog

Setup 3: 1 Pros v 21 Random Joess
Each Pro faces off against Random Joes, whereby Random Joes created based off the Consensus of Pros (Value for each player for each Random Joe is within +/-5$ from the Consensus for 95% of the players, with the other 5% of players effectively removed from the pool for the Joes); 22 leagues

Marcel does pretty well here.  KFFL takes the top honors.  So far, we’ve seen KFFL, CAIRO, FEIN, and Mike Podhorzer all finish ahead of Marcel in all three unofficial competitions.

Ask Rotoman
Baseball Info Solutions
BigScoreSports
Bloomberg Sports
Consensus
Fangraphs Community
Fantistics
Future of Fantasy
Geoff Buchan
John Eric Hanson
PECOTA
Razzball
RotoWorld
Rotochamp
Statspeakblog
Steamer

***

Finally, the official competition.

Setup 4: 1 Pros v 21 Random Fangraphs Readers
Large collection of Fangraphs readers were pooled in various fashions to create 21 overall draft lists for each league; 22 leagues

In this competition, each Pro forecaster is put in his own league against 21 average forecasters (Joe).  In order to make the apples-to-apples comparison, the Pro faced off against the same 21 average forecasters, from the same draft spot.  So, when Marcel drafts first against Joe1 through Joe21 in the Marcel Draft1 league, then Rotoworld also drafted first against Joe1 through Joe21 in the Rotoworld Draft1 league.  Each of the 22 Pro forecasters followed the same pattern.  In addition to that, Marcel then drafted second against 21 new Joes (Joe22 through Joe42).  And so on, and so on, and so on.

This is I think the one setup that most mimics reality. This model, I believe, should represent a reasonable group of average Joes that you would find in a Fantasy League.

And the official winner is Ask Rotoman.  Congratulations Ask Rotoman!  It barely edged out Consensus by a single point and a single win.  Steamer and Rotoworld also had a strong showing.  The Fangraphs Community did about average, and Marcel did not do well.

Baseball Info Solutions
BigScoreSports
Future of Fantasy
Geoff Buchan
PECOTA
Statspeakblog

Since there are 22 points allocated per league, and there are 22 participants per league, the average numbe of points is 1 per team.  And with 22 leagues for each Pro, that means the average number of points for a Pro is 22.  As you can see, most Pros were above the league average.  That’s because the league average included all those Joes, and those Joes did not do well.

This is actually a huge selling point for the Pros: almost *any* forecasting system is better than *no* forecasting system.

Conclusion

Unless you are very confident in your system, you would be foolish to stick to a single system.  Hanson won two years in a row, but finished middle of the pack this year.  Marcel did great last year as well, but not so well this year. But, everyone believes their system is an exception.  So, save yourself the trouble: let everyone else worry about their system, then just take the consensus of those.  You’ll do great.

Post-script:

I also turned everyone’s scores in each of the four competitions into a z-score (number of standard deviations from the mean).  I averaged the scores out, and here are the results.

We see that even though Ask Rotoman won the official competition, it was Consensus that was able to do the best across all four setups.  Given that it barely lost to Ask Rotoman in the official competition, it’s heartening to see Consensus (essentially the S&P500 index) do as well as it did. 

Marcel finished in the middle of the pack.  Think about that.  There are 20 other individual pro forecasters (excluding the Consensus), and we know they put more thought into their picks than Marcel did.  And yet Marcel was beaten by 10 of them and lost to 10 others.  This is the kind of result you’d expect if there was no such thing as talent at being a forecaster.

Baseball Info Solutions
BigScoreSports
Bloomberg Sports
Future of Fantasy
Geoff Buchan
John Eric Hanson
PECOTA
Rotochamp
Statspeakblog