THE BOOK--Playing The Percentages In Baseball

Could the “year 1 was the worst” for inconsistent players selection bias? 

By needing four years of everyday play, all your EXTREME inconsistency has to be positive forward, because the negative forwards are released before year 4.  Therefore, you eliminate players who had a sudden collapse in years 2-4, and eliminating those cases gives your result.

Just a thought.

As for the main result ... maybe some of the inconsistent players were slightly injured in seasons 1-3, which caused their inconsistency (even though they played every day)?  That would explain why their Marcels were off.

You could test that by seeing if their Marcels tended to be too low, as opposed to too high or unbiased.  Let me check ... yup, the projections were five points too low.

That may not be it ... again, just a thought.

#1, sure that could be. There are a lot of selective sampling issues/problems with these kinds of studies.

The Marcels were not really off any more or less for the consistent versus the inconsistent players.  The issue is mostly the spread of the 4th (or 5th in the 5 year studies) year’s performance - higher in the inconsistent group.  I am really at a loss to explain why.  I know that it seems “intuitively obvious” to most people that a consistent player would be “easier” to project.  It was never obvious to me.  Consistent players to me were just a random subset of all players.  And we know that all players will have any and all year-to-year “patterns,” by chance alone.  Maybe there is such a thing as a “consistency” skill.  I thought that someone or several people had done studies not supporting that notion, but I am not sure.

Maybe inconsistency is largely due to injury and some of the injured players recover and others don’t, leading to a larger spread in performance in subsequent years.  Maybe it is because the inconsistency is some “external factor” and that we simply don’t know what that factor is, fail to include it in our projection, and thus our projection is less accurate, which shows up in a wider discrepancy between projected performance and actual performance.

I keep saying that the performance itself in year 4 (or year 5) was more spread out for the inconsistent players.  That may not be true actually.  I was not really measuring the spread of performance but the “average squared deviation” from the projection.  Maybe I need to look at the 4th year performances themselves, independent of the projections, to see if one group has a more spread out performance (higher SD I guess) than the other.

I hate when I get certain compelling results and I can’t explain them!

Right, good point, both sets of players outperformed their Marcels.  But: the inconsistent players outperformed by 5 points; the consistent players by 3 points.  So that would explain *part* of the difference ... probably a small part, I don’t feel like doing the math.

Your intuitive feeling was that there should be no difference between consistent and inconsistent players moving forward.  My intuitive feeling is that there should be *some* difference, but not much.  Because you do have a few players who have reason to be inconsistent—injury, say. 

And.008 vs. .006 ... that’s a pretty small effect, isn’t it?  I bet people who believe in intrinsic inconsistency would be surprised the effect is that small. 

(BTW, I assume you mean the *root* mean square difference, and not actually the mean square difference?)

Oh, wait ... sorry, you do mean squared.  So variance of .008 means an SD of 89 points of OPS+.  And .006 means an SD of 77 points.  My bad.

So the difference is 89 points vs. 77 points.  That’s not a lot, but now I’m with you, in that it’s more than I would have expected.

Yes to #5.  Fairly big difference.  Plus, as I say in the article, a much higher percentage outside of 100 points.

I would like to run the same analysis using a larger data base.

OK, I ran the same analysis on a larger database - all games since 1960.  I used 4 years for the projections, so I looked at 5-year periods with 300 PA minimum for all 5 years.  Had to be the same team for all 5 years, again, to eliminate park factor problems (it might look like a player was inconsistent if he changed parks).

If I split up the players into only two groups, I get this:

Consistent

N: 925
31.2 years of age (in year 5)
Year 1: .840 age 27.2
Year 2: .845 age 28.2
Year 3: .848 age 29.2
Year 4: .846 age 30.2
Year 5: .834 age 31.2
Marcel projection, year 5: .829
Av. Squared error: .00602

In my other study, in the Marcel projection, I forgot to do regressions, which is clearly wrong.  For this one, I regressed toward league average for the projections using X/X+300 as my regression formula, where X was the weighted # of PA from years 1-4.

Not-Consistent

N: 940
30.7 years of age (in year 5)
Year 1: .827 age 26.7
Year 2: .841 age 27.7
Year 3: .843 age 28.7
Year 4: .834 age 29.7
Year 5: .823 age 30.7
Marcel projection (no regression), year 5: .824
Av. Squared error: .00601

So now we have no difference between the two groups! I have over 900 (925 and 940) “players” in each group, so a pretty big sample.

What about if I do the extremes again (top and bottom 200)?

Consistent

N: 203
31.4 years of age (in year 5)
Year 1: .856 age 27.4
Year 2: .858 age 28.4
Year 3: .862 age 29.4
Year 4: .859 age 30.4
Year 5: .842 age 31.4
Marcel projection, year 5: .839
Av. Squared error: .00524

Not-Consistent

N: 202
29.9 years of age (in year 5)
Year 1: .819 age 25.9
Year 2: .841 age 26.9
Year 3: .845 age 27.9
Year 4: .834 age 28.9
Year 5: .827 age 29.9
Marcel projection (no regression), year 5: .829
Av. Squared error: .00645

Here we have some differences but not as much as in my first study.  And again, not only is the SD of the difference between projected and actual bigger for the inconsistent group (80.3 OPS points versus 72.4 points), but 23.6% of the time the consistent players were within 20 points of their projection versus 21.3% for the inconsistent players, and 15.8% of the consistent players were outside of 100 points versus 19.8 for the inconsistent ones.

As well, the actual performance of the inconsistent players is more spread out, regardless of how “accurate” the projections are.

The SD of performance for group I for the extreme groups above is .1043 (104.3 OPS points) and for Group II, it is .1381.

When I break all the data into just two groups (not extreme), the SD of the performances of both groups are similar, .1120 versus .1178, a little greater for the inconsistent group.

So, maybe since the advent of steroids, predictable players have gotten more predictable?  Or vice versa?

I believe that injuries are somewhat predictable among players, even the nagging kind that might impact performance.  I have no analysis to support that belief, just the fact that I know I’m more injury prone than most people I meet.

Interesting to note that you still had the Year One thing in your inconsistent sample.  No idea what that might be.

I thought injuries were somewhat predictable, until JD Drew played more games than Miguel Tejada.  Small sample size, I know, but it’s shaken my faith.

If consistent players were easier to project, wouldn’t PECOTA eventually get Ichiro right? I realize this is one projection system and one (very unique) player, but he’s consistent in that uniqueness. I can’t understand why PECOTA gets him wrong seemingly every year.

If Ichiro is as unique as Silver says he is, then no system is going to capture him well.  Obviously, Ichiro has been a puzzle that he’s considered and can’t solve in any general way.

http://www.baseballprospectus.com/article.php?articleid=3497

Any Marcel-type system should project a player just fine.  What else do you need other than a player’s past performance, especially for a player that has been around for so long?  Of course you have to use the proper aging curve for a player like Ichiro, and the proper means for regressing the sample data (even though it is not going to be regressed all that much).  There should be no players who are a “mystery to project.” A player’s past performance is a perfect, albeit sample, reflection of his true talent and thus his future performance.

Even with a perfect projection system, you will always get some players wrong.  There are only 4 ways to get a projection wrong: One, a bad methodology, two, his historical stats did not reflect his true talent that well (by luck), and three, his future stats (the ones you compare your projection to) do not reflect his true talent well, again, by luck, or four, his true talent changed a lot, for whatever reasons, and you did not include that in your projection methodology.

One is a function of how good your projection methodology is of course.  Two and three will ALWAYS happen how much and by what amount, follows a normal curve.  And 4 always happens as well, occasionally, but not often, to a large degree.

As you know, Pecota isn’t a Marcel-type system. It requires matching the baseline performance of a given player to a set of historical comparables. It uses some non-performance characteristics (body type, handedness, position) as well as performance stats to find the comparables, and it often uses peripheral stats in the process. It’s the future performance of the comparables that represents the “projection” of the given player’s performance.

I think it’s possible that the match to comparables doesn’t work as well for some types of players as it does for others—e.g., for someone with an unusual combination of performance and nonperformance characteristics. In effect, Pecota may be “fooled” in such a case because the case doesn’t fit a canonical pattern of relationships between the variables in the model.  Pecota doesn’t default to a Marcel if the similarity scores used to identify the comparables are really low (maybe it should—thus becoming a hybrid model).

Marcel wouldn’t be fooled so easily by any particular “type” of case because he’s just a dumb monkey who always says that what a player’s going to do this year is pretty much what he did the last few years. No peripheral stats, no speed scores, height, weight, or position to find “similar” players.

I can imagine Silver inspecting and probing certain individual cases, especially those of outstandingly talented players, as a way to evaluate his system, but once he’s set up the formulas he doesn’t “overrule” the system’s projection for particular cases.  I see him “disagreeing” with Pecotas for particular players in some of his articles, but he lets the system “speak for itself” in his statistical projections.

Actually, PECOTA is a Marcel-type system.  The less matching comparables, the more it relies on Marcel-type.  Say for example Bonds, Clemens, RJ, etc, etc.

On some of those nearly “incomparable” players, especially when they outlast everybody (maybe because they were juiced?), Pecota relaxes its rules on age/position matching, but it still seeks “comparables” using this relaxed set of criteria. And Pecota relies on this “relaxed” set of comparables for its forecasts. I don’t think it completely defaults to a Marcel.  But I suppose you could determine this empirically (de facto) by direct comparison of Marcels and Pecotas for such rare players. Not sure you’d have enough cases to do such a test.

I never said “completely”.  In the case of Clemens et al, I’d bet it’s 90% Marcel.  The more incomparable, the more it relies on Marcel.

A bit off-topic, but have you ever thought about a way to come up with some kind of confidence intervals for the Marcel projections?  I remember reading at one point that you thought it would be possible by using the reliability (% of regression to the mean) factor that you include.  But I don’t remember reading any ideas on exactly how to do it, and I can’t think of any way to do it.

Nate has all but said that Pecota is a combination of comps and Marcel (Marcel being a generic word for a typical historical-stats-based projection system).

Fargo, do you have any inside knowledge about the Pecota methodology?

In any case, the major weakness with Pecota, in my opinion, besides what Fargo said about making “mistakes” in the comps, is the relatively small samples of players you end up with, and then not knowing how much to regress the “future” stats of the comps.  That is a key issue.  For example, if I have player A who has a Marcel HR projection of 15, and my “comps” hit 20 HR in that projection year, how much of that 5 extra HR is noise and how much is “legitimate”?  That is a critical question and I have no idea how Pecota handles that.  Similarly, if a Marcel-type projection says that given a player’s historical number of PA and given that we project him for X PA in the projection year, there is a 20% he will under or over-perform his projection by 75 more more OPS points, but the comps in Pecota over-performed by 75 or more 30% of the time and unde-performed 10% of the time, again, how much of the difference between the “normal” (Marcel) numbers and the Pecota ones is noise and how much is real?  There has to be SOME regression on those discrepancies, given that the comps are a limited sample of players.  But for certain things, that regression could be a little or it could be a lot.

Matt, we can easily generate confidence intervals or a full distribution of likely performance (the “one number” projection is some kind of mean or median of that distribution) for a Marcel projection, based on only two things: the number of historical PA it is based on, and the projected number of PA.  Alternately, we can use the projection as an estimate of true talent and just give confidence intervals or complete distributions on true talent and not worry about the random flucs associated with the projected number of PA.  If we really wanted to get more rigorous, we could use age and what have you to tweak those numbers.

BTW, I have always said that the Pecota method of using comps to figure out the exact distribution of a player’s likely future performance is a perfect way to do projections - much better than a Marcel.  In theory.  The problems, as articulated above, are sample size and figuring out the comps.  Those are two formidable problems and may render the whole concept worse than just doing a Marcel.

But I can see why someone would go that route.  The thinking is, “Why should I use a Marcel and then try and generically adjust for all these things, like age, height, weight, defensive position, injuries, playing time, experience, etc., when I can simply look at similar players in history and see EXACTLY what they did in the future.  Again, the problem is that you just don’t have enough similar players in history (and the context of history changes of course) for the future performance of these comps to be stable enough (free enough from sample error), whereas Marcel essentially uses a gigantic sample size to make projections.

Ditto 17.

MGL, I have no inside knowledge of PECOTA methods. I’ve tried to figure out how it works, and I agree that it’s an amalgam of “Marcel” with its reliance on last 3 years for a baseline (plus aging and regression adjustments), and “similarity scores” with their reliance on comparable players for the forecast.  The biggest mystery about it, it seems to me, is the method of matching individual players to comparables in the database. The most thorough discussion of this that I’m aware of is in the BP 2003 annual, but there’s not enough information there for anyone to mimic it or reverse engineer it, it seems to me.