THE BOOK--Playing The Percentages In Baseball

Tuesday, August 31, 2010

Forecasters Challenge 2010 - Latest Update

By Tangotiger, 10:17 AM

There are four different setups, one is official, and the other three unofficial. Depending on how you see things, feel free to choose the setup that makes the most sense to you.

John Eric Hanson is the only forecaster to finish in the top 5 in all four setups. Hanson also won the official competition last year, and I think finished fairly high up in the other unofficial competitions I ran. So, yeah, he done good.

The baseline comparison point should be Marcel. Marcel finished middle of the pack in 3 of the competitions, and IS LEADING in one of the competitions (the head-to-head one… the setup that was the official competition last year).

The results. First the three unofficial setups:

Setup 1: All Pros: All 22 Pros in the same league, 1000 leagues.

fan_id points_ct wins_ct value_ct fan_tx
217 1220 382 5687 Marcel
108 1213 263 5195 Chone
115 1185 197 4034 John Eric Hanson
112 1113 21 1310 FantasyPros911
113 1107 24 1122 FeinSports.com
118 1060 39 1071 Steamer
299 1044 22 800 Consensus
116 1057 12 662 KFFL
105 1062 12 578 Brad Null
131 1016 8 369 Fangraphs Community
120 1008 9 366 Razzball
132 1038 4 305 Fantistics
126 997 3 140 Bloomberg Sports
109 939 0 45 Chris Gehringer
102 918 0 23 Ask Rotoman
106 927 0 17 CAIRO
129 928 0 15 Wells Oliver
111 713 0 0 Fantasy Scope
125 897 0 0 BigScoreSports
130 672 0 0 Baseball Info Solutions
127 872 0 0 Future of Fantasy

Setup2: Head-to-Head Pros: Two-person league, each Pro facing off against one other Pro, 42 leagues each Pro

fan_id points_ct wins_ct fan_tx
113 11563 42 FeinSports.com
108 11617 37 Chone
299 11823 37 Consensus
105 11653 35 Brad Null
115 11502 33 John Eric Hanson
118 11459 29 Steamer
132 11334 27 Fantistics
116 11270 23 KFFL
120 11314 23 Razzball
112 11165 21 FantasyPros911
217 11237 21 Marcel
131 11122 20 Fangraphs Community
102 11059 18 Ask Rotoman
127 11073 18 Future of Fantasy
126 10854 12 Bloomberg Sports
129 10438 11 Wells Oliver
109 10721 6 Chris Gehringer
130 10366 6 Baseball Info Solutions
106 10670 5 CAIRO
111 10100 2 Fantasy Scope
125 10416 1 BigScoreSports

Setup 3: 1 Pros v 21 Random Joes: Each Pro faces off against Random Joes, whereby Random Joes created based off the Consensus of Pros (Value for each player for each Random Joe is within +/-5$ from the Consensus for 95% of the players, with the other 5% of players effectively removed from the pool for the Joes); 22 leagues

fan_id points_ct wins_ct value_ct fan_tx
115 1875 21 236 John Eric Hanson
366 1747 19 222
132 1858 18 218 Fantistics
629 1911 18 218
113 1830 17 212 FeinSports.com
299 1780 16 203 Consensus
522 1803 14 194
547 1802 14 194
131 1765 14 187 Fangraphs Community
465 1708 13 186
497 1759 13 186
108 1818 13 184 Chone
120 1768 15 184 Razzball
116 1746 12 170 KFFL
653 1753 11 170
217 1720 12 169 Marcel
686 1816 10 168
105 1758 11 163 Brad Null
473 1790 10 162
743 1676 10 162
112 1721 10 157 FantasyPros911
432 1795 8 153
102 1706 9 148 Ask Rotoman
129 1714 10 148 Wells Oliver
564 1660 6 144
596 1677 8 144
126 1713 6 137 Bloomberg Sports
720 1625 7 136
760 1624 7 134
416 1571 7 132
358 1775 3 126
580 1678 4 124
399 1719 3 120
670 1656 7 120
489 1728 5 120
736 1695 4 119
415 1555 6 118
637 1699 5 118
111 1667 6 117 Fantasy Scope
106 1657 6 115 CAIRO
342 1623 6 115
127 1652 6 110 Future of Fantasy
317 1624 3 105
588 1629 4 103
109 1589 5 100 Chris Gehringer
735 1686 3 99
118 1607 4 93 Steamer
448 1577 3 93
407 1519 3 90
662 1638 3 84
514 1644 2 80
538 1565 1 80
678 1612 3 79
613 1571 2 77
711 1564 1 77
301 1580 1 76
663 1625 0 76
334 1600 1 75
620 1676 0 75
694 1602 1 75
130 1571 3 74 Baseball Info Solutions
391 1621 0 74
350 1579 1 72
498 1593 0 71
441 1662 0 70
325 1599 2 69
605 1557 0 64
702 1554 1 63
523 1601 0 62
309 1549 0 61
450 1535 0 60
571 1524 0 60
728 1585 0 57
125 1572 0 55 BigScoreSports
.. another 409 Random Joes below this line

Finally, the official competition

Setup 4: 1 Pros v 21 Random Fangraphs Readers: Large collection of Fangraphs readers were pooled in various fashions to create 21 overall draft lists for each league; 22 leagues

fan_id points_ct wins_ct value_ct fan_tx
115 1416 20 230 John Eric Hanson
660 1396 19 222
116 1343 18 218 KFFL
725 1349 18 216
108 1309 17 200 Chone
299 1282 16 197 Consensus
118 1367 16 194 Steamer
616 1393 15 189
105 1279 13 184 Brad Null
717 1283 14 181
690 1242 13 177
112 1277 12 172 FantasyPros911
113 1277 12 170 FeinSports.com
132 1301 12 169 Fantistics
126 1236 12 166 Bloomberg Sports
120 1305 12 162 Razzball
129 1222 12 160 Wells Oliver
127 1230 12 155 Future of Fantasy
131 1217 11 151 Fangraphs Community
217 1244 10 151 Marcel
109 1240 11 149 Chris Gehringer
576 1250 9 148
541 1275 9 138
549 1270 8 133
614 1227 6 131
573 1179 4 119
327 1187 5 118
445 1167 5 116
475 1192 5 111
751 1192 4 108
390 1167 4 101
467 1189 4 101
421 1226 3 100
302 1234 2 97
515 1167 2 95
701 1225 2 95
667 1265 2 93
558 1163 3 90
460 1161 4 88
344 1154 4 87
756 1146 4 85
579 1210 2 83
530 1145 2 82
333 1152 2 81
482 1175 3 79
371 1164 3 76
696 1165 2 76
102 1142 5 75 Ask Rotoman
617 1176 2 75
548 1238 1 75
664 1251 0 74
417 1194 2 73
638 1188 2 73
384 1130 3 70
386 1141 2 70
313 1204 1 66
345 1137 1 66
485 1152 1 66
606 1164 1 66
759 1141 1 64
646 1173 1 60
358 1115 2 58
408 1177 1 58
360 1143 0 58
301 1153 2 56
644 1160 3 56
499 1102 2 56
125 1099 4 55 BigScoreSports
111 1122 3 54 Fantasy Scope
366 1129 1 54
503 1101 0 54
432 1095 1 53
655 1175 0 52
706 1172 2 51
721 1139 0 50
451 1118 0 48
580 1157 1 48
598 1142 1 47
613 1135 2 47
680 1129 1 47
587 1141 2 45
649 1160 1 45
374 1124 0 44
130 1070 3 42 Baseball Info Solutions
628 1142 0 42
446 1096 0 41
737 1111 0 41
594 1142 1 41
636 1133 0 40
335 1107 0 39
500 1080 1 39
729 1125 0 39
569 1106 0 39
416 1143 2 39
394 1108 0 36
414 1136 1 34
476 1118 1 33
693 1088 1 33
535 1169 1 33
514 1059 1 33
314 1131 0 29
501 1065 0 29
315 1113 1 28
411 1102 1 28
709 1128 0 28
538 1131 1 28
494 1076 0 28
400 1086 1 27
427 1073 1 27
517 1093 0 27
447 1084 0 27
622 1053 1 26
403 1100 0 25
743 1113 0 25
452 1065 0 24
749 1115 0 24
376 1095 0 23
458 1082 0 23
340 1076 0 22
336 1049 0 21
401 1048 1 21
363 1086 0 20
490 1062 0 20
685 1069 0 19
658 1111 0 17
106 1066 0 16 CAIRO
... another 357 Fangraphs Readers below this line

• Sabermetrics • Fantasy

#1 Ken (see all posts) 2010/08/31 (Tue) @ 11:03

who is John Erik Hanson? are his forecasts publically available?

#2 Tangotiger (see all posts) 2010/08/31 (Tue) @ 11:21

John has posted here a few times. Here’s the history of his posts:

http://www.insidethebook.com/ee/index.php/site/member_posts/?name=JEH

http://www.insidethebook.com/ee/index.php/site/member_posts/?name=Eric%20Hanson

#3 (see all posts) 2010/08/31 (Tue) @ 12:48

CHONE’s looking good as usual. Second in two formats, sixth in one, and third in another.

#4 Tangotiger (see all posts) 2010/08/31 (Tue) @ 13:09

I think what Chone did this year, but not last year, was rely on the Community for the playing time forecasts. That I think is where the real value lies. Easy enough to check…

#5 (see all posts) 2010/08/31 (Tue) @ 13:18

I’m disappointed that the “deadly accurate” forecasts of PECOTA weren’t included. Also, I would have liked to see how ZiPS performed.

#6 Rally (see all posts) 2010/08/31 (Tue) @ 13:39

Tango #4 is exactly correct. And I second Bobby’s wishes. Colin, Dan, what’s up with that? No guts, no glory.

#7 Colin Wyers (see all posts) 2010/08/31 (Tue) @ 13:44

The short version?

I had indicated to Tom that I was going to participate, and sent him an entry, but I made some rather severe mistakes with the ID maps and so the submission wasn’t usable. He gave me the chance to fix the ID maps (assuming that I didn’t change the ranking any, just fix the ID attribution) but I was rather tied up at the time (free advice - don’t get a divorce if you can help it) and so I wasn’t able to get it to him.

#8 J. Cross (see all posts) 2010/08/31 (Tue) @ 13:51

Steamer is looking to have a big September and pull this one out.

I also think we might need to find out more about how Hanson makes his projections.

#9 Tangotiger (see all posts) 2010/08/31 (Tue) @ 13:53

I tried to get Dan, MGL, PECOTA, and Oliver in this thing, and it didn’t work out.

With Dan and MGL, they didn’t provide all the necessary categories, and I didn’t want to manually intervene like last year.

For PECOTA, I wasn’t able to get anyone at BPro to agree to put them in, and I wasn’t able to get someone to act as trustee. (The data I got was riddled with errors.)

For Oliver, it was a similar situation as PECOTA: THT wasn’t interested, and I couldn’t get a trustee in time.

#10 Tangotiger (see all posts) 2010/08/31 (Tue) @ 13:58

Since Colin posted that, and to reconcile our two posts: it wasn’t clear that Colin was going to act as official rep or whether he was doing me a solid, so that’s why I posted what I said in post 9.

#11 JEH (see all posts) 2010/08/31 (Tue) @ 13:59

@1 - No one has ever asked for my projections before, but if anyone wants them next season they are welcome to them.

#12 Tangotiger (see all posts) 2010/08/31 (Tue) @ 14:09

I converted the scores from the four competitions into standard deviations, and averaged it out. This is how it looks:

ALL      Pros      H2H      Joes      Readers     fan_id    fan_tx
1.45    1.8    1.0    1.7    1.4    115    John Eric Hanson
1.32    2.5    1.3    0.6    0.9    108    Chone
0.83    0.1    1.7    1.2    0.4    113    FeinSports.com
0.79    2.8    0.0    0.3    0.1    217    Marcel
0.75    -0.1    1.3    1.0    0.8    299    Consensus
0.57    -0.4    1.1    1.1    0.4    128    Sport Fanatics
0.43    -0.4    0.5    1.3    0.4    132    Fantistics
0.42    -0.3    1.1    0.2    0.6    105    Brad Null
0.37    -0.2    0.2    0.3    1.2    116    KFFL
0.16    0.2    0.0    0.1    0.4    112    FantasyPros911
0.16    -0.4    0.2    0.6    0.2    120    Razzball
0.07    -0.4    -0.1    0.7    0.1    131    Fangraphs Community
0.06    0.0    0.6    -1.2    0.8    118    Steamer
-0.32    -0.5    -0.7    -0.3    0.3    126    Bloomberg Sports
-0.33    -0.6    -0.8    -0.1    0.2    129    Wells Oliver
-0.40    -0.6    -0.2    -0.9    0.1    127    Future of Fantasy
-0.54    -0.6    -0.2    -0.1    -1.2    102    Ask Rotoman
-0.71    -0.6    -1.2    -1.1    0.0    109    Chris Gehringer
-1.11    -0.6    -1.5    -0.8    -1.6    111    Fantasy Scope
-1.22    -0.6    -1.3    -0.8    -2.2    106    CAIRO
-1.30    -0.6    -1.2    -1.6    -1.8    130    Baseball Info Solutions
-1.44    -0.6    -1.6    -2.0    -1.6    125    BigScoreSports

Whether you think one of the competitions should be worth more or less, the data is all there for you to reweight.

For example, Marcel might, after the fact, say that the Pros competition should count for alot more and the H2H count the least, and so, he’d want different weights.

***

I also want to call your attention to these participants: Marcel, Consensus, Fangraphs Community.

Marcel and Fangraphs Community would intersect, to some degree, on the playing time forecasts. Marcel actually used the PT forecasts from Tangotiger.net surveys, while the Fangraphs Community got their results from the Fangraphs.com survey. There should be little difference between the two. The big difference would be in the rate stats, and, as you can see, Marcel held its own. They were close in 3 of the 4 categories, while Marcel handily beat the Community in the 4th.

The other one is the consensus, which is the average of the 21 picks. Really, if I didn’t know about Eric and Rally, I would just go with that one. It’s going to be consistently good every year. And, given that a regular fantasyer just uses it as a guideline anyway for his draft list, it should do the sanity-check job.

***

The Official Competition is the one under “Readers”.

#13 Tangotiger (see all posts) 2010/08/31 (Tue) @ 14:12

No one has ever asked for my projections before, but if anyone wants them next season they are welcome to them.

Dude, you are going to be quite popular. I would also suggest that you get them posted at Fangraphs.com . David posts several of them, and I’m sure he’d go for yours as well, considering your track record.

#14 Ken (see all posts) 2010/08/31 (Tue) @ 14:31

yes, you should send them to Fangraphs, as well as Mays @ Last Player Picked.

but first you need a catchy nickname, something like the Hanson Brothers, or Projections From The John

#15 Rally (see all posts) 2010/08/31 (Tue) @ 14:44

Nobody ever asked me for projections before I started posting them. Make them available, and if they are good you’ll get a following, and it looks like yours are good.

#16 Tangotiger (see all posts) 2010/08/31 (Tue) @ 14:55

The Hanson Brothers! Ah-hahaha… call yours… Slapshot.

#17 Mike Rogers (see all posts) 2010/08/31 (Tue) @ 15:14

Hanson Brother’s or Slapshot HAVE to be your projection names, for sure.

#18 JEH (see all posts) 2010/08/31 (Tue) @ 16:06

Clearly I have under estimated the importance of a nickname.

Step 1. Get a nickname
Step 2. Build a draft list that does not include Alex Gonzalez
Step 3. Send to Fangraphs

Sounds easy enough.

#19 Clemente (see all posts) 2010/08/31 (Tue) @ 17:08

More important than the list, JEH--can we have some insight into your projection methodology?

#20 dkappelman (see all posts) 2010/08/31 (Tue) @ 18:56

JEH, just send me an e-mail (or fill out a fangraphs contact form) and we can get yours included this year.

#21 Brian Cartwright (see all posts) 2010/08/31 (Tue) @ 20:05

The biggest reason why I bailed on including Oliver was that this is geared towards fantasy and I didn’t project several of the categories that Tango wanted, and at the time I was right in the middle of launching with THT and didn’t have the time. I hope to have those things taken care of over this winter.

#22 JEH (see all posts) 2010/09/01 (Wed) @ 07:39

@19 I have alluded to it before here, but the only thing I am aware of doing differently here is the “over-weighting” a player’s peak performance instead of recent performance.

The hypothesis is that (as Shandler sometimes writes) once you demonstrate a skill set you own it. There are many reasons a player may fail to match a peak performance season subsequent to achieving it, but most of those reasons (e.g. injury (serious or nagging), fatigue, situation, luck, etc.) are coincidental.

In other words, there are lots of things that can go wrong, but very few that we can expect to go wrong again. Basically, if I can’t explain away the peak performance and can’t anticipate a recurrence of the limiting factors then I (apparently) give the player a better chance than most to realize those numbers again.

#23 Rally (see all posts) 2010/09/01 (Wed) @ 09:56

Interesting.

So you would like this guy:
2007 .320 38 HR
2008 .270 20 HR
2009 .220 14 HR

More than this guy:
.270 24 hr each year

Marcel would prefer the second player, as would most systems. If the first player had reversed his pattern, had the 38 homer year last, you and Marcel might see him the same.

I guess you have to have some adjustment to bring everyone down to average? Otherwise looking at the peak performance for everyone will result in forecasting an above average league.

How recent must a skill be for a player to be considered as owning it? Bobby Abreu once owned well above average power, he clearly does not now.

#24 JEH (see all posts) 2010/09/01 (Wed) @ 10:48

@23

In your example my guess is that I would like Player B over Player A in all but the most extreme circumstances (circumstances under which the decline could be largely explained by causes not likely to be present in 2010 . . . and with such a drastic drop-off I have a hard time coming up with a reasonable scenario for that.)

I would say, however, that I would probably like Player A more than most other systems.

As far as adjustments go, I do “over project” to around, I would guess, the 80-85% of the outcome curve.* For this exercise it doesn’t matter (a ranked ordering was all that was necessary), and for a regular fantasy baseball league it is essential (because [very short version] the terrible performances, mostly injury related, that pull down EV won’t as drastically affect a fantasy team that simply replaces the player).

*I would guess the projection systems in use here projects closer to the 30-35 percentile. This would be easy to test.

#25 Ken (see all posts) 2010/09/01 (Wed) @ 17:01

JEH: how did you project Konerko this year? He is a guy I have fought about with various people. The argument for not being so shocked is basically “he owned these skills, he had been injured before, etc” which is similar to what you are saying above. So I’m curious to see if even if your system, which seems to buy into these factors more than CHONE/ZIPS/MARCEL/PECOTA, saw anything different for Konerko this year

#26 Ken (see all posts) 2010/09/01 (Wed) @ 17:03

though to be fair, I should also ask how you projected guys like Arod, Berkman, Carlos Lee, etc

#27 JEH (see all posts) 2010/09/01 (Wed) @ 19:14

Ken-

The players you mentioned did not make my team in any of the 100 drafts Tom used and posted the summary of, likely due to their age (most players don’t get younger and if I associate a large part of the decline with age I usually count on it recurring the following season). The only aged 32+ bounce back players strongly represented on my rosters were Vlad Guerrero who got a projection boost from the move to Texas and Vincente Padilla (who was 272nd on my draft list and, apparently, lower on everyone else’s).

Younger players who had a down 2009 and that did make my team at exceptionally high rates include: Nick Markakis, Josh Hamilton, Francisco Liriano, Adrian Beltre, Conor Jackson, Kevin Slowey and Nick Johnson. Some hits, some misses.

Page 1 of 1 pages

<< Back to main

THE BOOK--Playing The Percentages In Baseball

Tuesday, August 31, 2010

Forecasters Challenge 2010 - Latest Update

Latest...