THE BOOK--Playing The Percentages In Baseball

Monday, October 26, 2009

Forecasters Challenge 2009 - Who was similar to who

In the head-to-head drafts, I pitted Marcel against each of the other 21 forecasters twice (Marcel selected first in the first change, and second in the second draft), for a total of 42 head-to-head contests.  In 40 of those 42 drafts, Marcel ended up with Endy Chavez.  In all 42 of those drafts, ZiPS ended up with Endy Chavez.  We can say, therefore, that Marcel and ZiPS were “similar” in their evaluation of Endy Chavez.  And, naturally, these two were very different with each of the other 20 forecasters when it came to Endy.

For every players, I marked a player as “very different” if the number of times a forecaster ended up with him was at least 27 more than the number of times another forecaster got that player in his 42 head-to-head drafts.  Endy, for example, would count as “very different” when looking at the Marcel/MGL head-to-head, Marcel/PECOTA head-to-head, but “very similar” when looking at the Marcel/ZiPS head-to-head.  The “very similar” was noted as anyone that both forecasters selected within 5 times of each other.  Since the gap between Marcel and ZiPS on Endy was 2, that counts as “very similar”

So, which forecaster was most similar to Marcel?  Well, seeing that MGL and ZiPS both used the playing time forecasts of the Community, it’s no surprise that those two matched up highly with Marcel.

Marcel/ZiPS were very similar on 213 picks, while very different on 51 picks.  Marcel/MGL were similar on 211 and different on 54.  Among those that didn’t make any claims to having used the Community Playing Time forecasts, the biggest similarity was Fan #109 (checking to see who that is… Gehringer), with 114 very similar matches and 51 very different.

But, I’m more interested in who was most dissimilar to Marcel+Community.  I have to figure this forecaster’s system is the most novel.  Chone comes in with 65 very similars and 190 very differents.

When I look at all the pairs (22 x 21 = 462 pairs of head-to-head), the average number of big and small differences is about 100 players each.  When I look only at Marcel, the average number of big and small differences with the other 21 forecasters is about 110 each.  This tells me that the other 21 forecasters are more similar with themselves than with Marcel. 

How about the most famous system in our group, that being Baseball Prospectus’ PECOTA? 

They are actually fairly similar to all the other systems, with an average of 100 players each in the very different and very similar groups. Which system is the most similar to them?  Would you believe Steamer (Fan 218), which is a group of high schools students?  They have 88 big differences, and 93 small ones. 

Marcel is the most different from PECOTA, which is fairly shocking to me.

Finally, which pair of forecasters are the most similar?  That would be our top two finalists: Rotoworld and Hanson.  They had only 14 players that were very different, and 201 players that were very similar.  Hanson did rely somewhat on Rotoworld’s published playing time forecasts.  The next two pair of forecasters that were close were MGL and ZiPS, with 26 very similar and 205 very different.  Those two used the identical playing time forecasts.  After that, it was Marcel/ZiPS followed by Marcel/MGL.  All three systems used the same playing time forecasts.

The most dissimilar in the non-Community division was FeinSports and Cairo, with 148 big differences and 71 small ones.

Here are the overall “similarity scores”, which shows the average number of big and small differences, along with the total absolute differences for each forecaster compared to the other 21 forecasters.  Marcel is Fan ID 217.  You can find the rest of the matching key on the main website.