from Amazon
Tuesday, November 29, 2011
The existence of outliers doesn’t preclude luck as the cause
Given enough trials, and given a large enough coins, and we can always find outliers… and the same applies to pitchers.
Using this fact, it follows that in our first year, if we have 100 pitchers, we expect half to outperform their FIP. This means that there are 50 players that outperformed their FIP in year one. Of those 50 players that outperformed their FIP in year one, we would expect 25 (.5*50) of them to outperform their FIP in year 2 by pure chance. Of those 25 players that outperformed their FIP in year two, we would expect 12.5 of them to outperform their FIP in year 3 by pure chance. Similarly, we can continue down this path halving the number from the year before. In year four, we would expect about 6 pitchers to have continued to outperform their FIP, and by year 5 we would expect just over 3 pitchers to have consistently outperformed their FIP by pure luck.
Because we started with 100 pitchers, we expect that about three of the pitchers would outperform their FIP in 5 consecutive years, by randomness alone. Many people point to those three pitchers and say, “Clearly, FIP is not accounting for something those three pitchers do.” We can now completely discount that argument for the “simulation”, because we have assumed FIP to be perfect. Thought experiments are nice because they easily allow you to comprehend and visualize a phenomenon, but there is not a lot to glean, if the experiment is completely incongruent with reality.
To put it simply, to “win” something 5 times in a row, just by pure luck, and you have a 50/50 chance of winning each time, then you will win five and lose zero a total of 0.5^5 = 3.1% of the time. Start with 100 coins or 100 pitchers, and you will flip heads, or beat your FIP, by flipping five times in a row, a total of 3 times.
And in reality?
giving us a not so unexpected final total of 3.6%
...
and finally 4% of the original starting pitcher group
This is not to say that FIP is perfect. But, just relying on the fact that you’ve been able to identify 3 or 4 extreme cases, when that’s exactly what you would have expected to find if it was all luck, doesn’t prove your point.
You need to find MORE than the expected extreme points, and NOT just “some” extreme points. Some extreme points means nothing, unless you know how many you expected to find by luck.
What about the magnitude of the outliers’ distance from tje mean? Say you find the expected number of extreme cases, vut some of them are more extreme outliers than you’d expect to see by chance alone.
I remember someone at THT saying Mike Scioscia seemed to have a skill at beating his team’s pythag because their xw-l wad better than their w-l 6 years in a row or whatever. Then Greg R commented that if you do the binomial distribution with 30 teams you’d expect to get at least one team doing that. I was happy to hear that criticism because i think mike scioscia is the most overrated human being on the planet. But then i thought: surely you have to also a count for the fact that he beat his pythag not jusr by a couple games each year, but often by quite a lot.
So am i correct to think that magnitude is an important factor in considering whether chance is agood enough explanation? And if so, how do you go about accounting for that mathematically?