from Amazon
Thursday, July 22, 2010
Forecasters Challenge 2010 - Mid-season results
After a series of posts where I highlighted the results of various fun competitions, now we get into the official competition.
In this competition, each Pro forecaster is put in his own league against 21 average forecasters (Joe). In order to make the apples-to-apples comparison, the Pro faced off against the same 21 average forecasters, from the same draft spot. So, when Marcel drafts first against Joe1 through Joe21 in the Marcel Draft1 league, then Chone also drafted first against Joe1 through Joe21 in the Chone Draft1 league. Each of the 22 Pro forecasters followed the same pattern. In addition to that, Marcel then drafted second against 21 new Joes (Joe22 through Joe42). And so on, and so on, and so on.
In the end, each Pro ended up in 22 leagues, each facing identical competition.
Where did these Joes come from? That was the fun part. Fangraphs asked their readers in the offseason for their forecasts. So, what I did was this: for each team, I took the 21 fans that provided the most number of forecasted players. For the Yankees, I’d have Reader1 through Reader21. For the Redsox, I’d have Reader22 through Reader42, and so on. I then randomly selected one fan for each of the 30 teams, and called that collection of readers Joe1. Of the remaining readers, I repeated those steps, until I ended up with 21 Joes, from Joe1 to Joe21. I then put them into the competition as noted in the earlier paragraph.
I then did the same thing for Joe22 to Joe42: randomly selecting one fan from each team. In this way, I am able to mix-and-match 630 (21x30) fans’ picks into 21x22 collections (462 Joes).
This model, I believe, should represent a reasonable group of average Joes that you would find in a Fantasy League.
We now come to the results, as of the All-Star Game break:
fan_id points_ct wins_ct value_ct fan_tx
108 968 20 227 Chone
115 960 19 224 John Eric Hanson
105 921 18 218 Brad Null
118 967 17 203 Steamer
116 929 15 198 KFFL
299 917 15 192 Consensus
112 904 14 189 FantasyPros911
113 897 14 184 FeinSports.com
126 890 13 184 Bloomberg Sports
129 858 12 163 Wells Oliver
132 898 12 163 Fantistics
120 895 11 155 Razzball
127 859 9 143 Future of Fantasy
131 832 8 129 Fangraphs Community
217 841 6 115 Marcel
109 828 6 109 Chris Gehringer
102 801 6 100 Ask Rotoman
125 797 4 75 BigScoreSports
111 757 1 35 Fantasy Scope
106 751 0 32 CAIRO
130 689 2 29 Baseball Info Solutions
The current leader is Chone, winning 20 of his 22 leagues. It’s actually a very tight race. Here’s what those columns mean:
FAN_ID: a unique number for internal purposes
POINTS_CT: the average points so far for each team. The league leader in Fantasy points has around the same number of points as the league leader in RBIs does. That’s the kind of scale you should think of. So, you can see how Chone and Hanson and Steamer all have similar quality teams.
WINS_CT: the number of leagues won.
VALUE_CT: the number of draft points won in the league. You get 11 points for a win, 5 for a 2nd place finish, 3, 2, 1 for 3rd through 5th. Chone earned 227 points out of a maximum of 11x22 (242).
Since there are 22 points allocated per league, and there are 22 participants per league, the average numbe of points is 1 per team. And with 22 leagues for each Pro, that means the average number of points for a Pro is 22. As you can see, all Pros were above the league average. That’s because the league average included all those Joes, and those Joes did not do well.
This is actually a huge selling point for the Pros: *any* forecasting system is better than *no* forecasting system.
***
FangraphsCommunity v The Joes
This one is an interesting competition. FangraphsCommunity is the concensus of all the Fangraph readers. The 462 Joes in the competition is a random collection of Fangraphs readers. The difference is this: if one Fangraphs reader makes an outlandish pick, he WILL see it in among the 462 Joes draft list. However, he will not see it in the FangraphsCommunity, because all the other Fangraphs readers will average him away to the bottom of the list. This is basically a fight between a community pick that the majority can agree with and a set of picks where everyone’s voice can be heard. The Fangraphs Community won 8 of his 22 leagues, while the other 21 Joes won 14 of their leagues (an average of 0.67 wins per league entered per Joe). 8 v 0.67. The lesson here: don’t listen to yourself, if you don’t have a system. Add your voice to the community, and then go with the community pick.
FangraphsCommunity v Concensus
In this case, it’s the concensus of all the Fangraphs readers against the concensus of the 21 Pros. The Pros won 15 of their leagues, with the Fangraphs Community won 8 of theirs. The lesson here: there’s a limit to how much the community can tell us. While they are about as good as a random Pro, they can’t match the muscle of the concensus of Pros.
Just The Joes
As it turned out, one Joe won all 22 leagues he was entered in. There were also 392 Joes that didn’t win at all. There were 242 Joes that didn’t finish in the top 5 in any of their leagues. Basically, 242 of the 462 Joes (52%) were completely useless… as individuals.
Just 10 of the individual Fangraphs Joes beat out the Fangraphs Community. The other 452 ended up worse. That’s a 2% success rate. That is an enormous amount of knowledge pooling that was effective in removing the outlandish picks. Basically, just 2% of individuals can beat the concensus. This is a far lower figure of the Pros, where their individual picks beat out their concensus picks 25% of the time.
Conclusion
Unless you are very confident in your system, you would be foolish to stick to a single system. Now, Hanson, having won last year, and is in 2nd place this year, perhaps he would be an exception. Chone, who has always done well in past year’s of testing, maybe he is an exception to. But, everyone believes their system is an exception. So, save yourself the trouble: let everyone else worry about their system, then just take the consensus of those. You’ll do great.