THE BOOK--Playing The Percentages In Baseball

I would use choice 3 or 4, or some similar variation (like correlation of pro’s ranking and actual season ranking).  That is, rate the systems based on all their choices, not just those players they mainly draft/buy.

Why?  Because random results will have too much influence on your ratings if you don’t do this.  In any given season, a handful of big overperformers (Price, Johnson) and underperformers (mainly high draft choices who get hurt) will have an outsize impact on the results.  The pros that picked/avoided these players may have known a little bit more about them, but not THIS much more.  If you had dozens of seasons to rate these pros, it would all wash out eventually—but you don’t.

What we really want to know is the underlying talent of the pros and their systems.  By looking at ALL their rankings, you effectively give yourself a bigger sample size of each pro’s “talent.” But with choices 1 or 2, you are essentially using only a very small sample.

I think which evaluation method you choose depends on specifically what you want to evaluate, the ranking of players for winning at fantasy or the accuracy of a forecast for a given player.

For example, choice 1 would be very good for the first goal, but weak in the latter, since the second closest forecast and the least accurate forecast for Price would be the same.

So are you interested in the bottom line (winning) or the accuracy of a ranking?

But what is the point of an accurate ranking, if you don’t win?

***

“The pros that picked/avoided these players may have known a little bit more about them, but not THIS much more. “

I like that.  Even if Marcel says “16$” for Price, he’s really saying “pick a number between 14-18$”.  And the next Pro, who says “$15” is really saying “pick a number between 13-17$”.

On that basis then, it would be unfair to say that in 1000 drafts that Marcel would almost always beat out every other Pro.  The reality is that there’s an uncertainty in the forecasts that should be modeled.

So are you interested in the bottom line (winning) or the accuracy of a ranking?

But in the long run, those are the same things.  Pro X may win a lot, but his players are only the “winning” choices this year.  If you are “interested in the bottom line (winning),” then you want to know which pro can help you win NEXT year.  And the overall accuracy of his ratings will be a much better predictor of that ability than learning he got lucky because he bid $2 more on Price this year.

Tangotiger-

I’ll have more, perhaps a lot more, to say on this later, but I did want to jump in and throw out two possibly important questions that fall under the “what are we trying to do?” heading:

1. The title reads “How to evaluate a forecasting system . . .” but this seems more like “How to evaluate a fantasy baseball draft list?” as this really isn’t just a forecasting system as a significant portion of the game is valuation. Shouldn’t we be addressing the revised question? Comment #2 seems to touch on this.

2. Related to question #1 (and mentioned in comment #1) is:  Is the objective to evaluate the system for creating the draft list or to evaluate the draft list?  In other words, are we interested in a metric that might give us a hint how:

A. a list created by the same process would fair under another set of scoring rules in another season.

B. this particular list would do competing under the same rules against new sets of lists. 

?

A. I think what “we” are trying to do is to determine which system came closest to projecting David Price’s performance spike.

B. In effect, what we are trying to do is figure out which system is best at projecting sleepers… those players that outperform their expected value. We are interested in that information so that we can pay less rotisserie dollars (or draft in a later round) players that will actually give us better value.

C. Well, if I was to pick a choice, I would say Choice 5. Rotisserie value is based on categories such as wins, ERA and WHIP. Just because David Price is the 17th best in Fantasy Points does not mean he will have that same value at the end of the season. As a comparison, Jimenez has had a rough last few starts and there’s a decent chance (especially pitching at Coors) that he falls out of the top ten pitching bracket. So, I give no credit to any system as of yet. In addition, Price is decently well known so he is not much of a sleeper and $16 seems like a decent price (and value) for many starting pitchers. Besides, Price is just one data point and doesn’t necessarily mean Marcel is better at predicting fantasy value than any other system. It might be instructive, at the end of the season, to look at how they projected breakout years for other pitchers.

Richard: you missed my point about what Choice 5 represents.

FOR DAVID PRICE, we would give 0 to Marcel and the other Pros, because in real-life, he was selected SOLELY by the Joes.

That is, we give the points based on how often he was picked by the Pros, in a league filled with Joes.

You see, for David Price, there was a pretty good chance for him to put up the year he is having.  That’s why 7 of the 21 readers had him higher than the highest of the Pros.

You won’t get that kind of results with top-tiered players, or even some of the bottom-tiered players.  Those guys who have a low standard deviation.

But Price is different, as he had a huge swing in Fantasy dollars.

Given that NO Pro went out on a limb here, and that 7 of the 21 readers did, and Price met those expectations, then Choice 5 is saying that Marcel should get no value for the Price pick.

You seem to be creating a Choice 6.

***

For purposes here, Fantasy baseball is based on these rules:
http://www.tangotiger.net/forecast/rules.html

Guy/4, I agree with you that these are almost indistinguishable if you take a really big-picture view, but I guess what I mean is this:

If you care about the bottom-line (winning), you want to predict good performers and breakout players better than anyone else. You want to be bullish on players who have the greatest chance to exceed most people’s expectations. Also, your predictions on the top 200 players are WAY more important than for the next 800. It doesn’t really matter how off you were in predicting Jed Lowrie’s performance, because Jed Lowrie is a non-factor in fantasy baseball in all but the deepest keeper leagues (and maybe not even there).

If you care about an accurate RANKING, I think you do something like a RMSE or something based on some scoring algorithm, as Tango has done. You measure your rankings versus the “actual” rankings, and it counts evenly all the way down.

I agree you want to limit analysis to players drafted in most leagues (top 300?).  But beyond that, I’m not convinced you want to give extra weight to breakout players. First, a lot of that is luck.  And to the extent the pro really saw something different, then his ranking will be much higher than average and his RMSE will be lower.  Second, avoiding busts is just as important, so you would have to give them a lot of weight too.  But again, a lot of that is luck.

Tango 7/8: You’re correct and I apologize because it seems I lacked the context of the league rules, so thanks for providing the link.

To expound then, I would do a Choice 6. Basically, the 21 readers’ draft lists would be aggregated into a “forecast system”, averaging where the 21 Joes draft list to put a ranking on each player… call it JoeCast. Then I would compare JoeCast with the Trustee projection systems and the Pros and do the snake-style draft order to assign players. Whichever projection system (or Pro or JoeCast) had the highest ranking for that player gets all the points that player earns.

A: So, what we are trying to do is to see if any of the other forecast systems would draft a better team than an indiviual Pro with their bias, or better than a select, specific group of baseball fans (JoeCast) who tend to have some access to statistics but might be adding in their own personal biases, perceptions from watching the player on video, read stories in the news, etc.

B: One reason we would do this is to see if pure statistical projection systems can outperform people with biases, hunches, and interpretations. We might also be curious to see if JoeCast would
eliminate enough internal biases but still be as informed about external information as a Pro might be. We might also see if the Pro’s individual biases are so strong that it turns out that they draft a lesser team than the projection systems or JoeCast.

a) What exactly is it that we are trying to do?

I’m going to go ahead and phrase the what as:  “Evaluate a fantasy baseball draft generation system”.  In other words, I am going to treat the 21 posted ( http://tangotiger.net/forecast/totals_ballots_counts.xls ) lists as being systematically generated and attempt to evaluate the ability of the systems that generate those lists to generate other lists for use in other seasons or under other rules sets.

b) Why is it that you are trying to do that?

I was never good with questions that ask why.  My own motives are often a bit hazy to me and I don’t even try and guess those of others.

c) How do you go about doing that?

The fun part.  First, the 5 choices given:

Choice 1.  No? [see choice 6].  This is how we would score a single draft.  It has some, but not a great deal, of predictive power when extended to multiple drafts under the same rules for the current season’s stats because it evaluates the draft list only on the players actually taken . . . which leaves a lot of useful information unexploited. 

Choice 3.  No. There is no clean way to assign error and, besides, error may be a desired outcome.  For example, Price may have been the #17 performer through the first half but the optimal ranking for him in this contest was somewhere in the 140 range. 

Choice 5.  No. Having Price ranked at #143 across all possible drafts (drafts including many ‘Joe’ lists) would likely result in many contests where Price does not get picked by Marcel.  Extending the competition to some of the Joe teams or into an auction format may move the optimal ranking near 120 or 100 or 80 . . . but placing him 17th is still sub-optimal. The draft (or auction) is a competition with resources (picks or dollars) and make the other teams burn resources is part of strategy so some credit is due for ranking a player at a spot near where cost = benefit.

Choice 4. No. This seems similar to Choice 2, except that lists will get some credit / blame for players that they would never draft. 

Choice 2. Mostly Yes.

So.

Choice 6. This really could be Choice 2 in disguise, but Choice 2 just gave Marcel “most” of the credit without clarifying “most”.  This could also be Choice 1 in disguise . . . after all, this is a sequence “full credit” events being averaged. I’d simply run the actual competition simulation and Fun Competition #3 simulation another 1,000 times each on different draft lists / draft orders and check the average results (Basically what tangotiger is doing in the actual competition).  If the results were the same, 1-22, I’d call it good and if the results were different I’d run 10,000 and then another 10,000 and compare them.

Guy/10, good point about the busts. And I agree that luck plays a huge factor.

Let me try to clarify what I mean by being bullish on breakouts. If you are about winning, it’s more important to grab the potential breakout guys earlier (however you determine potential breakouts, by peak age or years in league or your gut), so that you outbid your opponents (I’d say Justin Upton was that kind of pick this year, though he didn’t work out exactly). That way you get the most credit from high upside picks if and when they break out. In the same way, you’d want to avoid potential busts (aging veterans, guys struggling with injuries, guys who play on teams which got a lot worse in the offseason).

So if we’re scoring based on the bottom line, I would want to put a guy like Justin Upton much higher in my ranking than established veteran X with the exact same projection line, just because he has a higher probability to outperform his projection.

But if we’re scoring based on accuracy, I would rank these two players about the same, because their 50% projections are similar.

Taking on the assumption that no one will blindly follow a forecasting system because, well, some people just HAVE to believe that they know more, and instead use the systems as a guide, here are my answers:

a) What exactly is it that we are trying to do?

The Pros are trying to create a guideline for the Joes.  If I were to try to quantify that, using David Price as an illustration: if a Pro says that David Price is a 15$ player, then it expects the Joes to not price him higher than 20$, that anything higher means that the Joe is being irrationally exuberant.  Similarly, you should price him less than 10$. 

The downside pricing really becomes irrelevant, since there will be ONE other drafter that will price him above 15$ anyway, and so, you don’t have to worry about downside pricing.

b) Why is it that you are trying to do that?

Basically, the draft list stops you from getting into a bidding war with someone who is irrationally exuberant.

c) How do you go about doing that?  (Choose one of the above five choices, or create your own sixth one.)

In order to best evaluate a forecasting system, given the above assumptions and objectives: randomly create a draft list that uses the original draft list as the basis. 

Let’s say that Marcel has David Price at say 16$, the next Pro has him at 15$.  And let’s say there’s a league where it’s all Joes, but you know that one Joe has Marcel by his side, and another Joe has the other Pro.

The Marcel-Joe isn’t too crazy about Price, so he adjusts his preconceived notion of Price being 10$ up to 14$ in some deference to Marcel.  The OtherPro-Joe like Price a bit more than normal, so he settles on 17$ as his Price value.  As you can see, the Marcel-Joe is going to lose him to the OtherPro-Joe.

***

What it would take however is for The Pros to “trust me” to be able to create random draft lists based on their initial draft lists.  And then run those draft lists in the competition.

Since no one agreed to that, it won’t be part of the official competition, but I can run it in my ever-expanding series of fun competitions where we can learn something.

If I was trying to figure out which system is nominally “best”, I would be looking at which system gives projections that are the closest to actual results. As such, I think Choice 3 is the closest to what I’d want to use.

Now you say Choice 3 really diminishes the credit that Marcel gets for David Price. I say that it is exactly what it should do, because in my opinion the fact that Marcel was closest among the Pros does not necessarily mean that Marcel is a better predictor of David Price. If Price is a $28 player, then Marcel was still off by 40%, it’s just dumb luck that Marcel’s projection is slightly higher than any other Pro.

Or let’s reverse it. Say a guy who has never been injured before gets injured and misses half the season. The system that is closest according to 1 and 2 is the one that projected him the worst. If the player’s injury is random chance, then the system that wins is the one that was lucky enough to be slightly lower than the rest.

Now, choice 5 is interesting, but to me all it shows is that the Pros are more conservative than the Joes (by conservative, I mean with results more closely distributed). In turn, I think it’s clear that the more a player’s performance deviates from expectations (be it higher or lower), the more likely he is to end up on a Joe’s roster instead, because the Joes have a wider spread.

Zac/15

Choice 3 doesn’t work, in part, because all we have are discrete ordered lists.  Even if the players were ordered strictly on value the player ranking Price 143 may have had him at a lower dollar value than the players ranking him 153 and 171.  We can’t assign error based on value because we don’t know value. 

To back up farther, the lists weren’t necessarily even constructed to order value; ideally they were ordered to maximize value obtained.  For example, a player may have had Price valued as the 70th best player in the scoring system (say $20) but then subtracted $5 from all of his pitchers before building a list thinking the majority were under-value pitchers.  Or, a player may have had Price ranked as the 70th best player, but didn’t think anyone else would have him better than 200th and so moved him down 180th hoping to get him 4 rounds later and some better players in the mean time.

In other words, the information you want to measure an error from (value) does not exist in the lists.

I think that some of these projection systems were designed with fantasy baseball in mind in which case I think choice 5 (the “real” competition) is the best because it comes the closest to imitating fantasy baseball. 

Others were designed for analysis in which case I think looking at the correlation between predicted WAR and real WAR (or maybe z-scores of real and predicted WAR) is the way to go.

@ tangotiger/14

I have no intuition on why adjusting a Pro list (say by averaging its rankings with a Joe list) would gain you any more insight into a Pro list than by simply comparing it to a wider variety of Joe lists.  Can you help me on that?

And, to digress:

Perhaps next year, if you are up for this again, you can get the participants to submit either raw stats and/or a dollar value with their ordered lists.

This would, first, tell you how many participants ranked strictly by value; which, in turn, would let you know how valid list adjustments are. 

It would also allow:

0. Projection comparisons by stats or value.  Even as a sidebar error-based comparisons would be interesting.

1. Player selection by auction.  Very useful in itself for distributing value and which also can be used to compare bidding strategies.

2. Comparison of valuation algorithms.  Ever wonder how someone comes up with a $37 value for a player?  It’s a very interesting area (to me, anyway) where (I suspect) most people are guessing.

JEH: as it stands, David Price has no value to any of the forecasting systems, if we compare to the Joe lists. 

But, if every Joe was supplied with a separate Pro list (Joe1 has Marcel, Joe2 has you, etc), then perhaps we’d see enough variation that Marcel might get a sliver of the Price pie.  That Marcel’s optimistic forecast, coupled with Joe1’s somewhat optimistic forecast would be enough to vault Joe1 over some other Joe.

In the end, we’re trying to model reality.  I had presumed in my reality that a Pro would go in blind against 21 Joes.

But the other reality that others are discussing is that each Joe has a Pro by their side, whispering in their ear.  So, we want to model that.

Basically, there seems to be alot of different realities that we are trying to model, and we need to describe that model in clear and certain terms so that I can write a computer program that matches that reality.

I’ve come to appreciate Ron’s article here the more I program my models:

http://www.baseballhq.com/books/myths.shtml

Bumping…

"2. Comparison of valuation algorithms.  Ever wonder how someone comes up with a $37 value for a player?  It’s a very interesting area (to me, anyway) where (I suspect) most people are guessing.”

oh yes yes yes. that also intrigues me, and I think is where a lot of edge can come in fantasy. some valuation models walk you through what they are doing, like Last Player Picked. others are more like black boxes. others like Rotolab allow you to set your own parameters for all kinds of things, like risk preference, positional scarcity, etc. fiddling with settings drastically changes values.

in all, you can come up with systems that value an identical projection from $20 to $40. weird, right?

I’ve posted how to do it, and I presume Last Player Picked is doing it the similar to the way I showed it: you follow the replacement level model, and you value each point above replacement equally.

I think that way makes the most sense, but I think even Shandler/Baseball HQ do it differently

Again, easy enough to test…

ok I found that old thread

http://www.insidethebook.com/ee/index.php/site/comments/the_worth_of_sb_hr_and_all_other_categories_in_fantasy_baseball/#60

it sounds exactly what LPP is doing, using standard scores and forcing the “last player picked” at each position to be worth exactly $1.

the only option available on LPP has to do with the hitter/pitcher split, something that was a big topic of discussion in that thread. basically, the projected value of pitchers is nearly always higher than their real world auction value pricing due to the inherent uncertainty their projections have vs hitter projections

A. I’m trying to find the Pro system that, if I draft players according to his projections, will give me the best chance of winning my fantasy league.

B. I want to win my fantasy league.

C. I think Choice 5 is the right one, but how you generate the Joes is the key. You wrote, “I looked at the 21 Fangraphs reader submissions that had the most Rays selections.” Why did you try that? I’m guessing it’s because you assume that the Joes in a fantasy league will generally be fans of a single team? Or in your official forecasters challenge format, how did you pool the Fangraphs readers to come up with the Joes?

I would bet that the reality is somewhere in the middle of a Fangraphs pool and a Pro whispering in the ears of the other drafters. In my league at least, I’m sure everyone else is going off of some sort of adjusted Pro cheat sheet like Yahoo or ESPN.

So if I were designing 21 Joes, I would give most of them a baseline of a “popular” Pro like Yahoo or ESPN. Only a few would begin with another Pro like Marcel or Chone. Then adjust some values randomly, as you’ve said, before running the simulations. Do most of the randomizing with a mean of zero, but also assign a “home team” to each drafter and have the random adjustments for his home team players have a mean greater than zero. Each Joe would have a different variance as well, to reflect how tightly or loosely people follow the cheat sheet.

Actually my answer to A above is complicated by the fact that my league uses more and different statistics than a standard fantasy league. So I might be better off stating my goal as “find the Pro that gives the most accurate projections” where “most accurate” is defined across the board in all statistical categories. Not sure how that would work, but being accurate in OPS and ERA (which is what usually gets tested) is no good to me if the system is way off in RBI and Wins and Holds and so on. (Of course no system projects every stat we use so I have to generate my own projections for several categories.)

Sorry, keep thinking of other stuff.
Not to mention that my league is a head-to-head category league, not a rotisserie-style league. So that affects how the projections turn into values and rankings.

@22,23,24

With respect to a roto scoring system, an argument can be made for valuing (or pricing, I distinguish between the two but in a competitive scenario they should be very similar) players in different ways (SGP (standings gain points), against replacement level (or some other baseline), ADP (or some other ordered ranking) transformed into an exponential curve) and I am sure there are others). 

In the end, however, I think the best you can get is a strategy for pricing that attempts to take into account for other value models in play. 

The Monte Carlo approach Tango is using here would be, I believe, well suited for assigning a price to a stat set (i.e., building a strategy).

It could be done something like:

1. Assign prices to each player using arbitrary systems,
2. run X auctions
3. and players that consistently end up on above average teams get their $value bumped up (and players consistently losing bumped down)
4. and steps 2 and 3 are repeated until equilibrium is approached.