from Amazon
Monday, October 26, 2009
Forecasters Challenge 2009 - Who was similar to who
In the head-to-head drafts, I pitted Marcel against each of the other 21 forecasters twice (Marcel selected first in the first change, and second in the second draft), for a total of 42 head-to-head contests. In 40 of those 42 drafts, Marcel ended up with Endy Chavez. In all 42 of those drafts, ZiPS ended up with Endy Chavez. We can say, therefore, that Marcel and ZiPS were “similar” in their evaluation of Endy Chavez. And, naturally, these two were very different with each of the other 20 forecasters when it came to Endy.
For every players, I marked a player as “very different” if the number of times a forecaster ended up with him was at least 27 more than the number of times another forecaster got that player in his 42 head-to-head drafts. Endy, for example, would count as “very different” when looking at the Marcel/MGL head-to-head, Marcel/PECOTA head-to-head, but “very similar” when looking at the Marcel/ZiPS head-to-head. The “very similar” was noted as anyone that both forecasters selected within 5 times of each other. Since the gap between Marcel and ZiPS on Endy was 2, that counts as “very similar”
So, which forecaster was most similar to Marcel? Well, seeing that MGL and ZiPS both used the playing time forecasts of the Community, it’s no surprise that those two matched up highly with Marcel.
Marcel/ZiPS were very similar on 213 picks, while very different on 51 picks. Marcel/MGL were similar on 211 and different on 54. Among those that didn’t make any claims to having used the Community Playing Time forecasts, the biggest similarity was Fan #109 (checking to see who that is… Gehringer), with 114 very similar matches and 51 very different.
But, I’m more interested in who was most dissimilar to Marcel+Community. I have to figure this forecaster’s system is the most novel. Chone comes in with 65 very similars and 190 very differents.
When I look at all the pairs (22 x 21 = 462 pairs of head-to-head), the average number of big and small differences is about 100 players each. When I look only at Marcel, the average number of big and small differences with the other 21 forecasters is about 110 each. This tells me that the other 21 forecasters are more similar with themselves than with Marcel.
How about the most famous system in our group, that being Baseball Prospectus’ PECOTA?
They are actually fairly similar to all the other systems, with an average of 100 players each in the very different and very similar groups. Which system is the most similar to them? Would you believe Steamer (Fan 218), which is a group of high schools students? They have 88 big differences, and 93 small ones.
Marcel is the most different from PECOTA, which is fairly shocking to me.
Finally, which pair of forecasters are the most similar? That would be our top two finalists: Rotoworld and Hanson. They had only 14 players that were very different, and 201 players that were very similar. Hanson did rely somewhat on Rotoworld’s published playing time forecasts. The next two pair of forecasters that were close were MGL and ZiPS, with 26 very similar and 205 very different. Those two used the identical playing time forecasts. After that, it was Marcel/ZiPS followed by Marcel/MGL. All three systems used the same playing time forecasts.
The most dissimilar in the non-Community division was FeinSports and Cairo, with 148 big differences and 71 small ones.
Here are the overall “similarity scores”, which shows the average number of big and small differences, along with the total absolute differences for each forecaster compared to the other 21 forecasters. Marcel is Fan ID 217. You can find the rest of the matching key on the main website.
FAN_ID big small tot
218 78 100 7,468
110 74 101 7,546
109 77 100 7,701
122 71 104 7,715
101 77 100 7,892
116 73 104 7,927
115 93 94 7,986
121 94 94 8,130
112 85 104 8,148
119 100 96 8,400
105 103 98 8,413
108 111 98 8,436
120 92 107 8,587
204 100 106 8,613
207 109 95 8,682
113 100 107 8,688
203 102 108 8,748
217 106 111 8,834
111 110 95 8,902
106 128 80 9,073
102 114 101 9,146
214 123 94 9,394
Interesting. We’ve started to look at correlations in OPS and ZiPS and CHONE are looking like similar systems while Marcel and Steamer projections look similar to each other (although Marcel looks like it had more success). Sporting News and Pecota don’t seem that similar to any of the other systems we looked at (and didn’t do very well this year).
Maybe distinctions in playing time turns out to be more important in determining fantasy value. We’ll check that out.
We’ll have much more on this coming up.