from Amazon
Monday, September 19, 2011
Fans Scouting Report and the position-neutral aspect
While I have no qualms with his basic point, his conclusion misses the larger point. If it is impossible for Jason to evaluate Mark Ellis and Luke Scott’s throwing, then this only invalidates the results of the data if you end up comparing Mark Ellis to Luke Scott.
HOWEVER, and this is important, this does NOT invalidate comparing Ellis to Brandon Phillips. If let’s say everyone is having a hard time following the instructions, then this bias applies to all secondbasemen, to some similar degree. Which is why getting 20 or 30 evaluators for each player is important: if my instructions to insist on position neutral is too complicated, then those instructions are NOISE. (Random noise, within each position.) And how do your counter random noise? With sample size.
So, it works on two levels:
1. If you believe that fans can go the job on a position-neutral sense, then you get great results.
2. If you don’t believe they can do that job on a position-neutral sense, then random noise of the instructions is reduced by sample size, and you get great results (if you stay within position).
Jason ignores the FSR for whatever reason. But, if you actually look at the results, are you left scratching your head thinking “that’s totally off”? No, you don’t. Well, you shouldn’t in most cases. Indeed, I’ve asked a few teams in the past to evaluate their players (and they do follow a position-neutral aspect to it, as they should). And guess what? They always say the same thing: most of it looks really good.
The noise introduced by the instructions is not necessarily random. In attempting to compare players across positions, it seems quite possible that bias is introduced depending on what other players are on a team for comparison.