from Amazon
Wednesday, August 19, 2009
If a metric for a player is “all over the place” after a few years, what does that mean?
You hear a lot of sabermetric literate and illiterate people talk about a metric - any metric - giving us a reliable assessment of a player’s true talent or his future performance if it is “consistent” from year to year, while if that metric is not consistent, we are much more “in the dark” about a player’s true talent and his future performance.
At the same time, people use year to year fluctuations in a metric for a particular player as evidence that a metric is not good, but we won’t address that at this time.
So my question is this:
Let’s say that player A is -10, +5, and -4 for the last 3 years. Player B is -4, -2, and -3. Can we assume that Player B’s weighted average is a significantly more reliable indicator of that player’s true talent and future performance (a projection) than Player A’s?
I don’t know the answer, BTW, but there are lots of smart folks who frequent this site that might be able to shed some light on it. This can be tested of course. In fact, I think I did a short study a while ago, looking at two types of players: One, players like player A, and two, players like Player B. Then I looked at their performance in the subsequent year. I wanted to look at 2 things: One, is either projection as accurate as the other? And two, is the variance around that year’s performance more for one group (presumably the second, “inconsistent” group) than the other? For example, the mean performance in the next year for group A might be -3, with a SD of 4 runs. For group B, the mean might also be -3, but the SD might be 5 runs.
If that IS the case, then the mainstream thought process is both right and wrong. It is right in that most lay persons think that if you see such inconsistent numbers over the last 3 years, you are not very certain about their true talent or future performance. It is wrong in that these same people think that you can’t even do a projection on that player.
