THE BOOK--Playing The Percentages In Baseball

Tango - Thanks for running the competition! It’s interesting, but perhaps not surprising, that consensus would do so well.

One tidbit: the five systems that didn’t beat Marcel in the All Pros setup (mine included) didn’t beat Marcel in *any* of the setups.

So some of us have more work to do if we want to be better than the monkey!

Once again, thanks for doing this, especially since this year it was a little more fun for me.

I do think you’re underselling Marcel’s competence rather strongly there at the end. It isn’t as if Marcel is thoughtless. There is a fair amount of intelligence built into him, intelligence that is the likely starting place for all projection systems.

Peter, that’s my point.  Marcel should be the starting point for all systems.  That is it’s raison d’etre.  It’s the minimum level of competence expected.

That’s why it’s disappointing how often systems are not built on top of Marcel.

I’ve long wondered if certain projections were more accurate for certain types of players or for certain stats.  Maybe Marcel nails the OBP of middle infielders but you want Roto World for the HRs corner outfielders are going to hit.  I’ve been trying to put together a database of last year’s predictions to try and look into this, but organizing the multitude of projections consistently is a huge burden.  I can’t even find a copy of last year’s ZiPs.  A proper study would obviously include multiple years of each projection system, but that’s completely beyond me.

I’ve taken a look at OBP for the half-dozen systems I’ve gotten into Access, and the results are disappointing.  The R-values (not squared) don’t reach .6 for all players, and are even worse for fantasy relevant players.

Two points:

Marcel is our current state of the art baseline, a system derived by multiple regression from the known and relevant data. To use similar methods, as I did 16 years ago, and not improve on Marcel would be very disappointing. But it is also possible that the only way to improve the current state of performance forecasting is to do something entirely new, using the new data we have now. Figuring out how to use them to forecast might take some time, and begin with some big failures that don’t mean improvement won’t eventually come.

The question about r-values and forecasting models is what the top score might potentially be. Each stat category has its own degree of difficulty, and in one season all are influenced fairly randomly by pure stochasm and injury. Maybe some new model will figure out a way to model that, but for now a “perfect” projection might score .66 (or whatever) in OBP. The balance would be unknowable noise. Getting almost there (and not very far from the starting point of Marcel or simply last year’s stats) is frustrating, but based on what we know now it shouldn’t be disappointing.

I second Peter/4’s first point: while I agree that Marcel is a great baseline for comparison of systems, I think it’s healthier to have systems that try very different methods.

Even if most of those methods aren’t as good (mine certainly didn’t perform better this year), it seems as likely that significant improvement in forecasting may come from a different approach as compared to refining a Marcel-based system.

Ideally people continue to do both.

If it’s not disappointing, then what is it that forecasters are selling the public? 

“New and improved” Coke?  (I hope you are old enough to get the reference.)

***

Byron: don’t waste your time looking for a bias at that level.  Even if it did exist, you’ll never find it.

The focus should be on what Marcel does not do: parks and MLEs.  And, I’ve demonstrated in the past that even with something like MLEs, where Marcel does NOTHING at all, it still is just about as accurate as the other systems.  (Which may simply be a selection bias issue.) And for parks, and there SHOULD be a bias, and I’m sure there IS a bias, you can’t find that bias, because of the inherent noise.

So, if you want to devote your time, devote it there.  You would be doing the community a service.

Good stuff.  Thanks, Tango.  I hope someone looks at the RMSE of a bunch of these systems this year (b/c we felt good about our system for Steamer this year).  I know that it’s lazy for me to hope someone else does it but we just had kid #2 so my available sabermetrics time is approaching zero.

Another very interesting observation is that the consensus of all these systems is basically the best one across all four competitions.

Consensus didn’t do quite as well in 2010, but it was still 2nd in one competition, 3rd in another, and above the median in all:
http://tangotiger.net/forecast/results2010.html

It’s early to be definitive, but it could be that the best projection system overall is simply the average of the forecasts.

Congratulations J Cross.  Having kids will certainly do that.  It sure gets fun when they start paying attention to the games.  My older daughter, 3 1/2 now, learned to root for Mike Napoli during the 2010 season.  Part of it being how much I like the guy, but the name Napoli is an easy one for kids to like.

This summer, while hearing his name when the Rangers were on TV, she said “Mike Napoli is my favorite player.  Is he your favorite player too daddy?”

She doesn’t understand trades yet, so I don’t think she figured out why daddy was crying.

I love reading this every year. Fascinating stuff.

I have a question about the consensus pick. Is that simply an average of all of the projection systems? If not, how is it calculated?

Right - I guess the net question is “how do I take the consensus”?

is it a straight average of every prediction?

is it playing time weighted (normalized) at all?

It’s a straight average of all the picks.

I turn each pick ranked #1 to #550 into a dollar value first.  If he’s not ranked there, he gets 0 dollars.

Then I just add up all the dollar values.

Pretty straightforward.

I’ve posted a few thoughts on my first year in the competition here:
http://blog.rotovalue.com/?p=181

I was happy to enter, because I was curious how well my model might stack up against others. And while the sample size is small, so far the answer is, not too well. But that’s incentive to improve it for next year!

Tango/7: I’ve never been one to give much value to projections, and said as much with my first published set in 1996 in How to Win at Rotisserie Baseball. Many years later Ron Shandler quoted things I’d written a couple times in his essay about how inaccurate projections are that supported his thesis before he wrote it up.

But comparing Marcel to other projection systems across the board misses at least part of the point. Any projection system that conservatively regresses its results to the mean, and gives all minor leaguers and missing players the same fill-in numbers (200 league average AB, for instance), is going to score decently against systems that try to project more extreme (in both directions) performances.

In my experience the value added for users is that active projectors create a dataset that better resembles what a finished baseball season dataset looks like, and the user can derive information from it about which players are expected to outperform their regressed stats, which rookies are expected to break out, and which players the projector thinks will underperform. If done right, these sorts of projections should score more meaningful hits in good seasons, while also missing more in bad seasons.

The projectors who do best in any given year beat Marcel and sometimes even the Consensus, but over time the more conservative regressions (and the Consensus is really just a Super Marcel) are going to win because of the Law of Regression, which will help you win a forecasting challenge but won’t win many competitive fantasy leagues.

That’s why I think this exercise is so interesting. Your drafts give us “real world” results for the different systems, and we get a chance to see how our work tests out in different situations against a broad competition. But I think Marcel’s and my goals are different, and so these results aren’t the only meaningful ones.

You are suggesting that it’s better to take riskier bets because the payoff is for win-place-show, and being average or being in last place is the same payoff.

That’s a theory to test.

I will remind people, not that it directly relates here, that my best system in a football pool, placing winners from 14 down to 1 (15 to 1 these days I guess), was to simply go with the odds.

Now, I will say that if the season lasted 8 or 10 weeks, I would not have won.  It was just by the 14th game or so when I finally emerged in the top 3.

So, yes, it depends at what point the signal exceeds the noise, such that risky bets end up really having negative value.

I’m saying that in attempting to model the upcoming baseball season’s stats in my projections, I am obliged to give some hitters 650+ AB and have some hit 40+ homers, etc. These sorts of decisions make the overall dataset less accurate, for one they remove injury mitigation from the numbers, but they make many individual players more accurate. (And others less so.)

And, it should be said, they make users happier because it looks like they’re looking at real baseball seasons, not stunted ones. I am always upfront about the trade-off here, and I’m also forthright saying that you shouldn’t derive your fantasy bid prices from the projections, because projections are always misleading. So the choice is either the dull and more accurate set or the sharper but riskier set. My projections are actually in between and fairly conservative, but as soon as you move away from the Marcel foundation you’re getting away from the info that’s in the math.

I don’t know that it is better to take riskier bets because the payoff is limited to the top few spots, though it looks to me that successful fantasy players do make riskier bets (though not necessarily the ones you’d expect) and attempt to exploit variance.

Well, all that is just theoretical and anecdotal.  Not to say you are wrong mind you or Shandler is wrong.

It’s just that this can be tested easily enough, if we have access to data, or even create challenges as I have, but targetting thousands of poolsters.  And, who knows, maybe this has been done.

I know if I were to ever run some fantasy site, I’d be running simulations and challenges out of my a$$ until I knew what really worked.

By the way, no criticism intended on anyone.

Thanks, Rally.  Definitely looking forward to when my daughter is old enough to enjoy the game.

Fantasy baseball sites really do have a wealth of data about how people draft and what works and what doesn’t work.  I haven’t really seen a site take advantage of this though I think ESPN’s player rater does to some extent in that it determine how many more points teams have, on average, with a player than they would with an average player.  The result isn’t any different that what you get by looking at how many standard deviations above average each player is in each category and summing across categories.

Tango, do you know if all the other participants used forecasting “systems”, as in mathematical models developed to produce the projections automatically? Am I the only one who does each projection manually by hand?

Geoff Buchan/6: You said it’s healthier to try systems with very different methods. I would think my projections are as different a method as it gets if I am the only one who does them by hand! But maybe I’m not.

Mike/21: I’ve always used a formula to create a baseline, Marcel-like set of projections, and then work by hand to tweak them. I don’t do much tweaking to the guys who have a few stable years, but I work hard by hand on minor leaguers and fact/fluke types using all available evidence to figure out what’s next for them.

Mike: I don’t know what they do.  Presumably there must be some manual intervention in the playing time portion of the forecasts.

Tango - Thanks again for running this!

One thing to consider in terms of Marcel vs. projections is the timing of this contest.  The majority of drafts occur 1-3 weeks before the season.  If you made the ‘due date’ March 15th and had to go with earlier ‘FG Community’ figures, Marcel might face more challenges.  That said, I’ve pored through most available PTEs throughout the preseason and they all need manual tweaking (I got f***ed in one draft when I didn’t realize Mark DeRosa was set at 500+ PAs)

Mike - I use automated projections based on various inputs for playing time and rate statistics.  But I do play around with PTEs that I feel are too high/low.

Rudy:

All? 
http://www.tangotiger.net/survey/index5.php

I don’t remember having to do any manual tweaking of these playing time estimates (but I can’t say that’s 100% correct; if there was some late pre-season injury that knocks a player out for the year, I would have changed that to 0):

Can you do the following for me:
1. Compare your estimates to those in the link, and show me the 10 that are most different, for hitters and pitchers

2. Show us how many PA and IP they actually had in 2011

Tango -
Will run that comparison.  Having some computer issues now.

In case it was unclear, I update my playing time estimates prior to submitting to this contest so it isn’t as if Marcel has an advantage (at least on me).

I’m just saying that by having the deadline prior to opening day, we’re removing one of the key variables for fantasy baseball success - estimating regular season playing time before rosters are set.

Here are some examples off the top of my head (with some fact-checking):

- Blake DeWitt was scheduled to be the Cubs starting 2B against RHPs through the preseason.  Darwin Barney named the starter on March 31st. 

- Zach Britton was supposed to be a Super 2 until the Orioles surprised everyone by putting him on the Opening Day roster on April 1st (after Matusz goes on DL).

- Roger Bernadina is thought to have the inside track at the Nat’s CF job throughout preseason - sent down March 28th (had to look that up) as he loses job to Rick Ankiel.

I know I benefitted in an AL-only league by taking a flier on Britton and, in NL-only leagues, that I dodged bullets on DeWitt (discounted his ABs) and Bernadina (got lucky).

It’s possible that moving up the contest deadline by 2 weeks only adds more luck vs. skill.  But it would be a bit more realistic.

Open to others’ thoughts on this....

Thanks again for running this + congrats to the half that beat me and condolences to the half that didn’t....

Just to choose one example: Bernandina was given 364 PA by the fans. Voting started March 24.  So, at least half the votes, if not more, was from before March 28.

He actually had 337 PA.

So, first, I reject your argument about the “inside track”.  Secondly, what a great pick by the fans.

In general, my playing time estimates (which were a mix of other sources + manual changes) were higher than the Community estimates.  I focused solely on players projected at 450+ PA by either system.

Top 10 Plate Appearances Projected Higher By Me vs. FG Community (mine/FG/Actual PAs):

1) Xavier Nady 487 / 280 / 223
2) Michael Young 481 / 653 / 689
3) Brandon Belt 325 / 495 / 209
4) Chris Getz 517 / 351 / 429
5) Carlon Beltran 542 / 380 / 598
6) Jason Heyward 712 / 563 / 454
7) Melvin Mora 565 / 421 / 135
8) Adam Dunn 724 / 583 / 496
9) Franklin Gutierrez 632 / 494 / 344
10) Brian Roberts 605 / 470 / 178

5 of the 10 didn’t clear 350 PAs.  Only pluses for me are Michael Young and Beltran.

Top 10 Plate Appearances Projected Higher By FG Community vs. Me (mine/FG/Actual PAs):

1) Alexi Casilla 464 / 520 / 365
2) Brent Morel 462 / 509 / 444
3) JJ Hardy 480 / 517 / 567
4) Carlos Ruiz 464 / 499 / 472
5) Danny Valencia 551 / 582 / 608
6) Mark Ellis 511 / 536 / 519
7) Peter Bourjos 529 / 553 / 552
8) Brendan Ryan 506 / 522 / 494
9) David Freese 453 / 466 / 363
10) Reid Brignac 529 / 540 / 264

All except Brignac cleared 350 PAs.  Hardy was really the only plus on the FG side.

Hard to say whether the fans were prescient or not.  I knocked him down to 270 PA after the demotion.  Had him at 420 PA prior to demotion (he had risk).  So average them together and you’re at 345 PAs.

I’m on board with crowdsourcing PA projections.  I’d take the crowdsourced #s over any of the other sources I’ve used throughout the years.

I just need them earlier b/c getting estimates at the end of the preseason for fantasy baseball is after the fact for most drafts.

Well, Fangraphs runs them pretty early (like, now!), so you can use them whenever you want.

I run mine with a week to go to the season.

Tango, I had a theory that some ‘perts project better for players who are projected for full AB while others better at projecting newer/or injury players. First did a points application that showed Marcel under-valued players (a full -12%) from those that hit 100% (razzball) on players that ended up with over 525 AB.  Went back and created two sets, based on projected consensus AB greater/fewer than 525.  The two sets are about equal in numbers of players.  The results are included in the following spreadsheet.  Would appreciate your review.  My conclusion:  One will get much better projections using consensus JAMES/ZIPS/RAZZZBALL for players projected at >525AB.

Use consensus MARCEL/FANGRAPHS/ROTOCHAMPS for players projected at <525 AB.

JAMES had an average RSQ for the 5 standard cats of >525 AB of .42.  Marcel for the same group, .34.

I tried, but couldn’t attach the spreadsheet here.  If you email me: I will forward it.

In 2009, I did a very extensive study of 9 forecasting systems, and I looked for bias like you are talking about.  I did it by how many career PA they had, how good a hitter they were, etc.

Marcel did not stand out with any bias.

Feel free to email me at
tom~tangotiger~net

Not surprising at all that you had looked at something similar already. Sounds as if my findings (from one year small sample) just the year-to-year randomness you addressed already.  Thank you for the quick response.  I will let this dog die.

Not sure what thread to place under, but I have a projection question? 

I looked at Phil Hughes and found two obvious levels of performance (before and after injury):

http://www.fangraphs.com/fantasy/index.php/chris-parmelee-phil-hughes-and-non-save-rps-values/

For 2012, should his value be set to the average of the season?  Should he have two value sets depending of fastball speed?  Should the injured set be thrown out (the way I think it should be done)?  Thoughts

Realize you aren’t asking my opinion. Nevertheless, I would favor whichever set was the latest.  I did a two-year study on injured/poor-performing hitters (not pitchers):

http://razzball.com/forums/viewtopic.php?f=33&t=13641

Would you be willing to publicly post the consensus ranked values once they are in for 2012?  Thanks!

Since deadline for submission is opening pitch, the results would only be available sometime after that.

Fred: you need to compare the results of those players to what Marcel would show.  We need some baseline.  We need to see if your group shows bias OVER AND ABOVE any other group.

Marcel gives you that baseline, and Marcel is available for all.

http://www.tangotiger.net/marcel/