Monday, February 27, 2012
Value of knowing performance from 4 years ago?
Great study here. Go read it, because it came off a challenge I put out.
Now, we expect to see some difference, some tiny difference. It’s not like data from year T-4 is irrelevant. But, HOW MUCH does it mean is always the question.
Here is what Rob got:
1. He started with his two matched groups, and he had one group at 0.06 FIP lower than the other.
2. In the out-of-sample, we expected to have the same difference, but instead, the gap was now 0.19 FIP
That is, the gap was 0.13 more than expected.
Why is there a difference? Because in T-4, one group had a FIP of 2.75, and the other had 4.55 for a gap of 1.80 runs.
Therefore, knowing there is a 1.80 larger delta in T-4 will yield a 0.13 larger delta in year T. That is, you need to weight year T-4 at 0.13/1.80 = 7%
That is shockingly low. I was expecting an even higher weight. I use a decay rate of 30% each year.
So, in year T-1, I give a weight of 100%, year T-2, I give 70%, in year T-3, I give 50%, and year T-4, I give 35%. As you can see, year T-4 is 35/(100+70+50+35) = 14% of the total weight. (There’s also some regression which would bring it down.)
But perhaps a 40% decay rate makes more sense. In that case, the weights would be 100%, 60%, 36%, 22%. In that case, T-4 is 10% of all the data. Add in regression, and, well, that explains Rob’s findings.
Conclusion: year T-4 should be severely underweighted, and its weight is consistent with a 40% annual decay rate.
Great job Rob! I’ll put this as one of the top research pieces of the year already.


Great work.
In all fairness to BP, Colin was talking about batters and not pitchers when he said that they used almost equal weights for prior years. He said nothing about pitchers, I don’t think.
Surely they don’t think that pitchers should have their prior years get equal weight.
Hopefully, Rob will do the same thing with batters. I’m not sure why he chose pitchers when BP only discussed batters, but it is great work nonetheless.
Also, I’m not sure, but I think that if pitchers in the “high” group had higher FIP’s in year x-4, they would also tend to have higher FIP’s in year x-3 (and x-2, etc.). I think.
So, even though both groups had a similar 3 year FIP, their 3-year Marcel or other projection would have a smaller gap between the groups.
IOW, Rob got this:
Group FIP IP K/9 BB/9 HR/9
Low 3.46 162 9.0 3.4 0.8
High 3.52 160 8.3 3.5 0.8
However, if the high group had higher FIP’s in the earlier years, a forecast based on only 3 prior years might be something like this:
Low 3.5
High 3.48
Since he found that the higher group was .19 more in FIP in year X, it would be actually .21 more than expected rather than .13, giving even more weight to year x-4.
Again, I am not sure if a higher FIP in year x-4 means a higher FIP in year x-3 (and x-2 and x-1), but if Rob is reading this, perhaps he can tell us the average FIP in each year for each group…