THE BOOK--Playing The Percentages In Baseball

Tango,

Thanks for the pub!!  I have no good reason why I chose the ordinal rank (your presumption was correct) vice the actual score.  Good news is I still haven’t gotten around to doing other positions, so I can do them using the actual score.

Now why didn’t I think of doing this...the half year off from school has made me slow.

Good work.

We’ve talked about “regressing” UZR towards Fan Scouting values, and in thinking about it, I don’t like the terminology ("regressing").  The Fan Scouting report is simply another data point and I don’t think that we should be referring to it as a “regression.” Plus, we still NEED to regress toward some population mean (where the population is NOT dictated by any of the data) and the Fan Scouting report does NOT represent a population mean.

So, for example, after Steve uses his method II (method I is not correct, Steve), after that, the numbers need to be regressed toward some population mean.  The one I use is the average UZR of all players at that position with the same speed score.

The reason that the Scouting Report value is NOT a number that you regress toward in the typical use of “regression toward the mean” when doing projections is that it is not an unbiased estimate of defense. You need an unbiased estimate to regress towards.

The Fan numbers, like UZR, will be too high for the good players and too low for the bad ones. If nothing else, over the course of the observations by the fans, the good players will have played a little better than their true talent and the bad players will have played a little worse.

If you were just using the Fan numbers for a projection or estimate of true talent, you would regress those (toward some population mean) as well.  If you were just using UZR, you would of course do the same (regress toward some population mean).  If you use both UZR and the Fans, you still regress - only you regress less than either one separately because you have more data and hence more “effective opportunities.”

What MGL is saying would be like this:

TrueValue = .50*UZR +.30*Fans + .20*Average

And 20% of average is zero, since Average by definition is zero.  That 20% represents the regression toward the population mean.

Of course, the more data you have the more UZR counts, the less Fans count and the less regression you have as well.

Basically, it’s more like saying: how many BIP do I give the fans and how many BIP of average performance do I give.  For example, you can add 500 BIP of average performance and 1000 BIP of Fan observastions and whatever actual BIP of the player you have.  So, if UZR is based on 1000 BIP, it counts as much as the Fans.  If UZR has 2000 BIP, it counts twice as much.

And of course, you can include Pinto’s PMR and Dewan’s +/- IF AND ONLY IF you suspect those metrics contain information not existing in UZR.

All data for illustration purposes only.

What MGL said, plus you need to take into account that these aren’t independent metrics.  As hard as we try not to be, we are definitely influenced by UZR when doing the surveys.  So, someone who is overrated by UZR would most likely also be overrated by the fans just because he is overrated by UZR.  I personally wouldn’t use +/-, since they are using the same data and similar methodology.  So .50*UZR +.30*Fans + .20*Average sounds good to me.

First off, I would say most people who do the surveys aren’t even aware of a player’s UZR.  I’d say about 10% of the surveys originate from my site, and maybe another 20% are sabr-savvy.  Over half, I’d say, are your normal folks how aren’t sabr-savvy.  Furthermore, even those who are aware of UZR will distrust UZR for players of their own team.  Simply put, they would trust their eyes much more than UZR, especially the way I laid out the surveys (7 categories, which are not readily apparent tied-in to UZR).

And, until this year, there was no UZR for rookies.  So, if you want a base for UZR/Fans correlation, just run it against rookies.

Ryan Zimmerman, for one, has ALWAYS ranked high by the Fans, even though UZR was not always high for him.  UZR has come around, and it was apparent that it would have to. 

I’m sure you can find several dozen, if not hundreds, of players that were rookies, or had very little UZR data when the surveys came out.  And I have no doubt you’ll get a decent correlation, even though both are completely independent.

AND, I would bet that the correlation with the veterans are the same. 

And that will prove, once and for all, that there is no bias from UZR.

I’ve been hoping someone else does the study, but I’ll have to do it one day.

MGL,

Thanks for the input, especially that Method I is not correct.  That method was what I set out to do at the beginning, but it didn’t feel right at the end.

My plan was to identify a population whose UZR mean I could regress to, and clearly I chose the level of shortstop as identified by the fans.  Unfortunately I neglected to consider the idea of an unbiased population for the reasons you and Steven mentioned.

Again thanks for the comments.  In my next iteration I will experiment with the speed factor that you mentioned.

Tango, MGL, et all,

Pardon my dense question, but if someone does what Tango proposes in 6 and finds no bias, what then are the pitfalls of regressing towards a population mean where the population (not the actual values, just the population) is determined by the fans? 

I’d liken it to the example MGL did the a little while ago about regressing towards an actual scouting report (I believe he used an “above average” qualifier and “equated” that to runs).

Thanks for putting up with a relative newbie!!

I’m really glad Steve got a link here, he does some great work over there. Good to see him get exposure!

Steve, whether you are “regressing” toward the Fan Scouting (or any other scouting evaluation) or just including it as another data point is a matter of semantics.  The main point is that you still have to regress towards a “real” population mean, which these scouting evaluations are not.  The reason they are not is not necessarily because they are not accurate or because they are subjective.  It is because they are biased.  I explained what I mean in my last post.  Basically, they include a “luck” factor.  Any player who has played at an above-average level for any period of time likely was a playing a little better than his true talent (and how he will play in the future) and vice versa for players who played at a below-average level.  Also, there is sample error in the Fan evaluation just like there is sample error in UZR.  By that I mean that if a player is rated as above average by the Fans, it is actually likely that he didn’t play quite as above average as the Fans thought, for whatever reasons, simply because there are many more average players (and worse) than above average players, so the chances of a random player being rated as above average but really playing at an average level is greater than the chances of a player being an above average player (that is the essence of “regressing toward the mean” - it is a “short cut” for Bayesian probabilities given some assumptions of normality).

That may be a little confusing and I did not explain things well, but suffice it to say that you want to include any data sets (hopefully that are somewhat independent of one another - or at least they ADD something to one another) you can, including scouting, and THEN regress toward some unbiased population mean.  That mean doesn’t have to be generated objectively (you COULD use the Fan Scouting reports as a “mean,” if that was all you had), but it is better if it is.

Remember that we don’t really have a population mean when we do any of these “regression toward the mean’s.” A population is the ENTIRE set of elements you are choosing from.  The means that we use are usually only samples drawn from that population and we use the mean of those samples as an unbiased estimate of the population means.  Scouting reports are usually not unbiased estimates of a mean, but as I said, they COULD be used as such if that is all you got.  But, as I also said, it is semantics whether you consider the scouting data a mean to regress towards or another data point.  Either way, it is handled the same, although you want to make sure that you include another mean in your formula, either zero for everyone or the mean of all players with a certain speed score, as I do.

I should have added, that is great work, Steve.  I am so sick and tired of seeing one year UZR numbers, it makes me want to throw up.  And adding in the Fans Scouting Report is a great touch.  The last piece of the puzzle is the actual regression toward the mean.  Then you basically have a great defensive projection for each player (and an estimate of his current defensive talent/value).  People who want to estimate, for example, which team, the Yankees or the Phillies, have the better defense, can use those numbers and not the ridiculous one year numbers that you see all the time even on the sabermetric web sites.

As I have always said, if you EVER see me quote a one year number in the context of discussing someone’s current true talent or future value/performance, just shoot me on the spot.  I’d rather quote nothing to tell you the truth (just in principle).  And, at the very least, if you absolutely must quote a one-year number for anything, it is meaningless unless it is regressed given that one year sample.  That is the thing that doubly pisses me off when they do that (quote one year numbers).

What I do when combining UZR and the FSR is based on earlier discussion I read from here re: Rally’s defensive projections from last year and other stuff. It’s crude, but I think it works okay.

The basic idea is this.

4 years weighted average of UZR, regressed to average.

I convert each season of FSR to runs.

Infielders:

4 years weighted/regressed UZR times .75 + weighted average of FSR times .25

Outfielders:

I’ve read in various places that it’s a good idea to regress OF defense more, which I usually remember to do. Technically I don’t “regress” it more to the fans scouting report, but I give the FSR data point more weight. I do regress the UZR a bit more as well, but following some stuff I read from MGL, instead of regressing it to simple average (0), I do it based on speed score.

So something like

4 years weighted/regressed (to Spd) UZR times .6 + weighted average of FSR times .4

Nothing brilliant. I took what I’d read in earlier discussions and messed around with it.

One nice thing about using the FSR (as may have been mentioned) is that if a guy has switched positions, one can take the “raw” numbers from the older position and re-weight them to get a position-specific skill for the new position.

If I could figure out an easy way to automate it in MySQL [maybe when the new “universal player ID matcher” is ready for 09-10?) , that would be great, but until then I do each player separately ina spreadsheet when I’m curious.

Mike #9 -

Agreed.

Is the data in the fan scouting normally distributed?  Is UZR?  Could you assign +0 value to the mean score within a position.  +15 to the highs and -15 to the lows and then use a standard deviation to assign the values in between?  I assigned the values in between linearly, but that did not seem to give enough credit or blame to truly excellent or poor fielders.  Thanks.

Whenever I do defensive projections, I more or less do the same thing as devil_fingers points out. I do include TotalZone data whenever I can, since it has a different data source (Retrosheet instead of BIS), and give it equal weight with UZR, though I’m not sure if that adds as much to the measurement as I think it does. I also echo the ability to switch players from different positions to see how well they would rate via FSR.

Part 2:

http://playahardnine.wordpress.com/2009/11/21/defensive-projections-take-2/