from Amazon
Sunday, March 20, 2011
Why MLE’s are a mess…
I’ll warn you in advance (what other kind of warning is there?) - this is a long post and one that is hard to follow…
If I look at park and league adjusted AAA stats and compare them to MLB stats for the same players, weighted by the lesser of the two PA (e.g., if a player had 300 PA in AAA in a certain year and 100 PA in MLB in a certain year, I use the 100 PA to weight each of those stats, AAA and majors), I get this:
For players who had AAA and MLB league service in the same year, min of 50 PA in each year, for all players from 2006-2010 (I call these “same year” players):
AAA OPS: .846
AAA OPS normalized to all AAA players (nOPS): 1.11
MLB OPS (of those same players): .691
MLB OPS normalized to all MLB players (nOPS): .92
Ratio of MLB nOPS to AAA nOPS: .83
There were 560 of these “same year” players. There were a total of 62,604 “PA” where “PA” is the combined minimum of the two PA (AAA and MLB) for each player.
The average number of PA in AAA per player was 228 and in MLB it was 168.
The average age of these players is 23.6.
There is nothing remarkable in these numbers. We are basically saying that when a player goes from AAA to MLB (it could be the other way around – I did not distinguish time frames) in the same year, he loses 17% of his OPS.
Now, what if we do the same thing but for players that have AAA service time in year 1 and MLB service time in year 2 and compare their stats from AAA to MLB? These players could also have had some MLB time in year 1 and/or AAA time in year 2. The only requirement was some AAA time in year 1 (at least 50 PA) and some MLB time in year 2 (also, at least 50 PA).
For players who had AAA and MLB league service in consecutive years (first AAA, then MLB), min of 50 PA in each, for all players from 2006-2010 (We call these “next year” players):
AAA OPS: .824
AAA OPS normalized to all AAA players (nOPS): 1.08
MLB OPS (of those same players): .715
MLB OPS normalized to all MLB players (nOPS): .96
Ratio of MLB nOPS to AAA nOPS: .88
There were 600 of these “dual” players. There were a total of 99,174 “PA”.
The average number of PA in AAA per player was 284 (in year 1) and in MLB it was 242 (in year 2).
The average age of these players is 25.8 in year 1. So these players are 2 years older than the previous group of players above.
These players only lose 12% of the OPS value, whereas the ones who played at both levels in the same year lost 17%. Why is that?
You might think, “That’s easy. The second group is one year older in MLB than in AAA and thus they not only lose value by virtue of going from one level to the next, but they gain value because they are older.”
But…
For one thing, the second group is 25.8 in AAA. You would not expect them to gain much, if anything, from age 25.8 to 26.8 (we think that players peak from around age 26 to 28).
Anyway, what if we restrict both groups of players, the ones who played at both levels in the same year and the ones who played at AAA in one year and MLB in the next, to ages 26 or older? We should not expect to see any increase in OPS in that age range due to aging. If anything, the second group should lose a little value from year 1 to year 2 from aging.
For players who had AAA and MLB league service in the same year, min of 50 PA in each, for all players from 2006-2010, age 26 and older:
AAA OPS: .845
AAA OPS normalized to all AAA players (nOPS): 1.11
MLB OPS (of those same players): .684
MLB OPS normalized to all MLB players (nOPS): .91
Ratio of MLB nOPS to AAA nOPS: .82
So, not much difference for these players. They were about 60% of the total pool of players (and 60% of the PA) that includes all ages, by the way. Their average age is 28.3.
What about the “next year” players?
Only about half of these players were age 26 or older (average age 27.9 in AAA). Here are their translations:
For players who had AAA and MLB league service in consecutive years (first AAA, then MLB), min of 50 PA in each, for all players from 2006-2010, age 26 or older:
AAA OPS: .827
AAA OPS normalized to all AAA players (nOPS): 1.08
MLB OPS (of those same players): .718
MLB OPS normalized to all MLB players (nOPS): .96
Ratio of MLB nOPS to AAA nOPS: .89
So, older players (28.3) who play in AAA and MLB in the same year lose 18% of their OPS, while older players (27.9 in AAA) who play in MLB in the next year only lose 11% of their value. (The younger ones actually lose 13% of their value - remember all players combined lose 12%.) That seems backwards. Younger ones should lose less value than older ones because of aging (younger ones should get better, older ones should not).
So what the heck is going on? And what discount (12% - “next year”, or 17% - “same year”) should we use for AAA to MLB translations?
What if for the “next year” players, we only look at players who did not play in MLB in year 1. The previous numbers for the “next year” translations represent players who played in AAA in year 1 and in MLB in year 2. As I said two paragraphs ago, these players could have also played in MLB in year 1, although I am not sure what difference it would make in terms of the translations. In any case, we’ll only look at players who did not play in MLB in year 1 – only in year 2 (they still could have played in MLB in any year prior to year 1).
For players who had AAA and MLB league service in consecutive years (first AAA, then MLB), min of 50 PA in each, for all players from 2006-2010, no MLB performance in first year:
There are only 233 players (all ages), average age 25.6. Originally, there were 600 players, so over 60% of them had MLB service time in year 1 also.
AAA OPS: .802
AAA OPS normalized to all AAA players (nOPS): 1.05
MLB OPS (of those same players): .725
MLB OPS normalized to all MLB players (nOPS): .97
Ratio of MLB nOPS to AAA nOPS: .92
Now these players are only losing 8% when going from AAA in one year to MLB the next year, when they didn’t play in MLB in the first year. For those that did play in MLB in year 1, they lost 14% (from AAA to MLB) in year 2 (.837/1.10 in AAA and .709/.95 in MLB). Wow, what a difference!
What these numbers are saying is that:
• If you play in AAA and MLB in the same year, you lose around 17% of your OPS value (remember everything is normalized to league totals – the raw numbers mean nothing).
• If you play in AAA one year and also MLB in that same year and then MLB again the next year, you lose 14% from AAA in year 1 to MLB in year 1.
• If you play AAA in year 1, do not play in MLB in that same year, and then play in MLB in year 2, you lose only 8% from AAA to MLB (one year to the next).
What if we break these last two groups – the ones who play in AAA and MLB in year 1 and then in MLB in year 2 and the ones who play in AAA in year 1 and MLB in year 2 but not year 1 – into young and old players?
The results are almost exactly the same (older players who didn’t play in MLB in year 1 lost 7% rather than 8% for all ages). Age doesn’t matter. If you played in MLB in year 1, you lost 14% (AAA to MLB) the next year, and if you did not play in MLB in year 1, you lost 8%, regardless of age.
What is happening? I think there are two things going on. As I showed, it has nothing to do with age. It has to do with selective sampling and regression toward the mean. The AAA players are not randomly selected and thus their stats do not reflect their true talent. When we look at those players’ MLB stats, they are relatively unbiased and thus reflect their true talent.
So basically an MLE, at least the way it is computed herein (simply the ratio of MLB stats – e.g. OPS - to AAA stats), is not really a “translation” per se, but a translation AND a regression toward the mean. This is a very important point, and has some important implications.
The reason there is a regression toward the mean when we go from AAA to MLB stats is because the players who get to play in MLB got lucky on the average in AAA. That is one of the reasons why they are promoted. You have to do/have two things in order to get promoted from the minors to the majors – one, good talent or pedigree (such as being a high draft pick), independent of performance, as determined by scouts and other team personnel, and two, good stats in the minors. As most of you know, any player or group of players with good (above the average of the population you come from, where that population is independent of your stats) stats got lucky, on the average.
With that in mind, let’s look at some of the numbers above.
Let’s start by accepting the proposition that no matter when you play AAA and MLB, your discount is always the same, and see if we can make sense of these numbers even though the discounts we found are all over the board.
Remember that “same year” players, young and old, lose around 17% of their OPS from AAA to MLB, while the “next year” group only loses 12% or so, again, regardless of age. Maybe they are both actually losing the same amount in true talent, but the first group was luckier than the second such that it appeared that they lost more going from AAA to MLB. In other words, more of the 17% was regression than the 12%. Maybe they both lost 6% in true talent, but the first group lost 11% in regression and the second group only lost 6% in regression (or some other combination of numbers).
Let’s see. The “same year” players had an OPS of .846 (nOPS of 1.11) in AAA and the “next year” players were only .824 in AAA (nOPS of 1.08). So yes, we would expect the first group to regress more. If we really want to separate the regression from the loss in OPS due to the change in level, we would need to establish the true talent OPS in AAA for both groups. I don’t think it would be correct to simply regress their observed OPS (.846 and .824) toward AAA average, since these are players who are deemed to be MLB ready by the powers that be for reasons other than their stats (presumably and partially at least), so the mean OPS of these kinds of players is proably higher than the average AAA player.
What if we look at their out of sample AAA stats – ones that are not used to determine whether they get promoted or not? That should give us a good idea as to their true talent. It is not a perfect method, but it is pretty good.
First, we’ll look at the “same year” players’ AAA stats in year 2, if they have any.
For the “same year” players who had at least 50 PA in AAA in year 2, their OPS was .791 (in AAA), an NOPS of 1.05. In year 1, it was .836/1.10. (Remember that all “same year” players had a AAA OPS of .846, but some of them never played in AAA in year 2.)
In MLB in year 1, they batted .668/.88. This implies a discount of 16%, a little less than the original 17%.
Unfortunately, here we don’t have an unbiased estimate of their MLB talent, because we have a sample of players who played MLB in year 1 and then were sent back down to AAA in year 2, such that they likely had unlucky numbers in MLB. So that .88 nOPS in the majors is probably more like .91, which would give us a true discount of 13%. Granted that .91 is a guess, but it should be clear that the “true” discount for these “same year” players is less than 16%.
For the “next year” players who had a discount of only 12%, not only did they only bat .824 in year 1 (as opposed to .846 for the “same year” players) which means that the regression toward the mean will be less than that of the “same year” players, but as it turns out, they also had some AAA time in year 2 (likely before they were called up), in which they batted .845!
In fact, these players actually had a combined AAA OPS in year 1 and 2 of .835. Now the discount is 13% rather than 12%, once we include their year 2 AAA stats. But again, we likely had players who were lucky in AAA in both year 1 (.824) and year 2 (.845) such that .835 is not their true talent OPS in AAA.
Let’s look only at the stats of players who had AAA time in year 1 and 2 and MLB time in year 2. Basically this will eliminate those players who played in AAA in year 1, were called up to the majors at the beginning of year 2 and stayed in the majors for the whole year (or were injured at some point in year 2). Those might be better players.
As it turns out, the players who had AAA time in year 1 and year 2 (and MLB time in year 2) only batted .811 in year 1. In year 2, they batted .845, as I mentioned above. Combined (year 1 and year 2), they batted .831/1.10 in AAA. In MLB, they batted .703 with a nOPS of .94 (before it was .715/.96). So these players had a discount of 13%.
Let’s see if we can get some idea as to the true talent level of players who get called up to MLB from AAA. We need to find an unbiased sample of performance in AAA for players who were called up at some time. This sample must be after they were called up and not before.
We’ll use the year 2 AAA stats of players who were called up in year 1. That number was an OPS of .791. In year 1 these same players had an OPS of .836. What must the population mean be for .836 players to have a true talent of .791? Let’s see. The formula for regressing AAA OPS is % regression=589 / (PA+589). (BTW, that is almost exactly the same as the regression formula for MLB OPS!) These players had an average PA in year 1 in AAA of 243. Plugging that 243 into the above regression formula, we get a regression of 71%. So .836 - (.836-mean) * .71 = .791. Solving for the mean, we get .773.
Now let’s go back and regress all the above AAA stats toward this mean and see what the “real” discount is when going to MLB:
For the “same year” players, we had a year 1 OPS in AAA of .846 in an average of 228 PA. So we regress 589 / (589+228), or 72% toward the mean of .773. That is .793, which is a nOPS of 1.04. The nOPS of these players in MLB was .92. .92 / 1.04 is .88, so our “real” discount is 12% rather than original 17%.
For the “next year” players, those that had AAA stats in year 1 and year 2, batted a combined .831 in 530 PA in AAA. We regress that .831 53% toward .773 for an estimated true talent of .800 or 1.06 normalized. They hit .94 nOPS in the majors. That is a ratio of .89 or an 11% discount. We would expect there to be a slightly lesser discount for these guys (as compared to the “same year” guys) due to them being older in year 2 in MLB.
Finally, what about the “next year” players who had no AAA playing time in year 2? They batted .841 in year 1 in AAA in 243 PA. That requires a regression of 71%. That is an estimated true talent of .793, or 1.04. They batted .98 in MLB, so the ratio is .94 for a discount of only 6%. My guess is that these guys are a lot better than .793 players. They appear to have been promoted to MLB at the beginning of year 2 and never demoted in that year.
Let’s look at how they did in the year before year 1. In both years combined, year 0 and year 1, they batted .832 in 598 PA. Regressing .832 50% toward .773 gives us an estimated true talent of .803. That is a nOPS of 1.06. However, with different weights, these guys batted 1.00 in MLB, for a ratio of .94 again, or a 6% discount.
As you can see, coming up with a pure MLE translation is not an easy exercise. In some cases, we get a discount of 12% or 11%, and in others we get 6%, even when regressing the AAA stats.
One of the questions we need to ask when presenting MLE’s is, “What does the MLE mean and what are we going to do with it?”
If we truly want it to represent what that player would have done had he and everyone else (the defense for example) played exactly the same, and we want to include the good or bad luck, but “translated” to a different environment (MLB), then you need to use some pure coefficients, like the 12% or 6% discount that I estimated above. Unfortunately just looking at minor and major stats, either from year 1 to year 2 or both in the same year, or some combination (and perhaps other years) won’t get you that number because of the regression and selective sampling issues, as you saw in all the gymnastics above.
If you want to know what a player would do in MLB given a certain stat or set of stats in another league, and you are going to include regression toward the mean – IOW, you want to know his MLB true talent, given a certain set of stats in another league – then you can simply use the actual minor and major stats.
The problem with that – and it is a large problem – is that translation plus regression is not a linear function and thus you cannot simply use one coefficient for all players. For example, let’s say that we use the “same year” translation of 18% that we get when comparing same year AAA (normalized) OPS to MLB OPS. Well, that 18% includes a translation plus a regression because the pool of players was well-above average in AAA and thus their performance was better than their true talent (they were lucky as a group).
If you try and apply that 18% discount to a player who batted the same as the whole group in AAA, you are fine. The result is exactly what you would expect if he gets called up to MLB (he will do worse because of regression AND because of the better league).
But if you had a player who was a league-average hitting player in AAA (maybe he is a SS or C with a great glove), he is not going to hit 18% worse in MLB. His AAA OPS will not regress or it might even regress upwards. It will also shrink of course, due to the better quality of the league. Similarly, if you had a super star in AAA, say with a nOPS of 1.30, he is going to regress a lot toward some mean, perhaps the .773, and then lose some more due to the better quality of the league, for a total discount of more than 17% or 18%.
So if you want your translations to include regression (IOW, what WILL a player who hits X in AAA hit in MLB, not what would that X have been in MLB, including the luck), you had better have a non-linear equation for your translations!
At this point in time, I have no solution to these problems. Typically we use MLE’s to help us with major league projections for those players who don’t have much or any MLB service time. We also use them to wonder about who should or might be promoted in the near future. As you can see, it is not real clear, though, how to come up with these MLE’s, which ones to use, and what they mean.
If we just had minor league numbers and we wanted to project major league ones, then we would want to use that non-linear equation, which we could come up with by running a multiple regression of minor stats on major stats including minor league playing time. But if we also have some major league numbers and we want to combine the minor league ones with the major league ones and then plug those combined numbers into our projection algorithm which does regression and aging, we probably want some kind of linear translation for AAA to MLB stats that includes the luck in the minor leagues. As you saw, that is not an easy number (the coefficient) or linear formula to come up with, and it might depend on when the player was called up or might be called up as compared to the bulk of his minor league playing time.
So when you see an MLE, don’t take it nearly at face value and always inquire what it is trying to represent. A “major league translation” is not enough of a description for you to know what exactly it is and what it can be used for. I believe we have a long way to go before we come close to mastering the MLE.
Tom:
1. I think to find the ‘answer’ you really need to also look at PITCHERS moving from minors to MLB in the same way as you do for hitters. There are park effects and defensive changes between levels to consider, but I don’t think you can determine the true differences between levels without also including pitchers. When I did this for Japanese players the hitters looked like they were moving to a much tougher league (something like an 18% hit) but then pitchers performed almost as well in MLB as you would have predicted if they had stayed in Japan, which led me to some very interesting MLE conclusions.
2. It’s also helpful if you look not only at guys called up but at guys sent down - guys that played in MLB only in one year, then played in the minors the next year. I think the last time I looked at this group, looking at only unadjusted performance as you’re doing, they had a HUGE spike in performance the second (minor league) year vs. their first year.
3. Not sure I would recommend combining MLE and regressions together. They are two seperate animals, and combining them only makes things more difficult and complex to figure out. You first have to figure out the correct league difficulty adjustment, and that should give you a ‘true’ MLE (VALUE of the minors performance translated to the majors environment), THEN I think you apply the same types of regressions that you would normally apply to any MLB player from one year to the next to get your ‘projected’ performance.