THE BOOK--Playing The Percentages In Baseball

I have to admit, I did chuckle at “r error” being left.

This is overall a good idea, though. People don’t realize the difference between sample sizes with batting and UZR.

My concern is that a running total of UZR doesn’t capture changes in a player’s defense.  Sometimes when there’s talk of a player working hard in the offseason and spring training to improve their defense it’s not just talk, they actually make a noticeable improvement in their defense.  Conversely, players generally get worse at defense as they push towards the wrong end of their prime and beyond.  The biggest problem with this latter trend is that not all players age equally.  The plodders are likely to see their defense deteriorate at a far more rapid rate than the rabbits.

These are concerns that I’m not sure can be solved within the statistic.  It seems good old fashioned scouting is needed to correct for this.  As always, a perfect answer isn’t really available.  Career UZR is a preferable approach than year to year UZR.

Totally agree with the misuse of the stat, BUT… what if a guy really IS having a bad year? If we’re lengthening the observation period, would it make sense to weight recent performance more heavily? Or does that make the whole thing too complex?

Well, you can do whatever you want to address whatever the issue happens to be.

If the #1 issue is someone talking about “trends” and trying to come up with reasons for -3, +5, -2, then address those.

If the issue is something else, address those.

Why not just Marcel then?  Or a simple exponential weighting (in practice to do this, just take last period’s weighted value and count that as x% and use this period’s value as 100-x%, then use this new value as “last period’s weighted value” next period)?  I think these other methods weight all periods equally.

I have to agree with the A team (lol). First, having a career number takes away the perspective one gets from the aging curve. As a play gets older one would assume his defense would get worse (but I guess that isn’t always the case).

If a career number was added I guess one could always do UZRCareer09-UZRCareer08 to figure out the change.

In the end though, I would probably wouldn’t want to see the career number weighted because this isn’t a projection it is a data sample that has already occurred. One could weight it for a projection or for their own purposes, but to alter the data doesn’t make sense to me. 

In the end, career UZR exists.

http://www.fangraphs.com/statss.aspx?playerid=1281&position=1B#fielding

It is the line that says total. lol.

Right, it already shows career.  What would be good is “last 3 years"… David ALREADY shows this in his leaderboard pages.  He could, in fact, show the “last 3 years” for all the stats, not just fielding.  And then, people can use whatever they want.

My projections are just simple marcels based on a 5-4-3-2 weighting regressed to 125 games with an aging factor of -0.7.  More for sure can be done using the FSR or minor league stats, but it is a basic Marcel for UZR.

http://www.beyondtheboxscore.com/2009/11/14/1157186/2010-uzr-projections

I agree with an exponential weighting type system.  David updates the numbers every week, so it wouldn’t be to hard to code it in.  That would solve the misuse of the stat issue, as well as potentially capturing actual changes in defensive ability.

As I am reading this, several thoughts and questions come to mind.

1. The year has significance to people.  I think it might be is a mistake to try and eliminate that, rather than just accepting the variance for what it is.  Is this not more of an education thing, rather than a problem with the stat itself.

2. The year marker is entirely arbitrary, but that is just as true with batting as it is with defense.  By pure luck, really, a year’s worth of batting statistics are, generally, a useful quantity.  But even those suffer from the same issues (just to a lesser degree).  Would you want to lose all annual statistics?  Is there a principled distinction that can be drawn? 

3. The decimal presentation is a real issue.  If your 1SD is 5 runs, not only should you not be presenting it as AB.C, but also B should probably not be presented as if 11 and 12 were actually different numbers either.  The concept of significant figures is basic and should be applied to UZR as it is with any other measured quantity.

4. Isn’t the problem of not being able to just average that numbers over the last X years because each annual UZR represents different numbers of opportunities is solved by UZR150.  Is this really different from any offensive rate stat?

The answer to my last question is no.

I think there’s an important question that needs to be asked before deciding what one does with UZR, and it’s an obvious one:

“What are you trying to do with the numbers?”

If you’re trying to tell a story of what a player accomplished in the past, having the yearly UZR numbers handy is quite useful. It allows us to incorporate fielding into WAR, and to see how a player’s value has has changed over the years.

Now, if you’re trying to project things forward, then you’ll need the larger sample sizes. But can’t we account for that by putting sample sizes into a projection system, and then using that to give us uncertainties in the projection of UZR?

To me, the problem isn’t posting yearly UZR numbers or UZR numbers that come with small sample sizes. The problem is people are mis-using the statistic. (I’ve probably been just as guilty as others in this regard.)

Personally, I think it would be more useful to give +/- numbers next to projections… or if you wish, the career numbers and/or 3-year averages people often use as representative of “true talent level.”

For 2010 Oliver (coming soon to a THT near you) I list the yearly totals of runs saved (they happened, it’s history although interpreted) but also a Marcel projection of runs saved.

They will also have letter grades (Ex,Vg,Gd,Av,Fr,Pr,Bd) of skills like range, hands, throwing and DPs based on a regressed value of rates weighted as per Marcel. With the grades, instead of bickering over 3.2 vs 5.1 runs saved, you will have an option of a bell shaped distribution of 7 grades.

"The problem is people are mis-using the statistic.”

And hence the title of this thread: taking the UZR out of user error.

How do you stop users from misusing UZR.  (say that 5 times fast)

That’s a good question, Tango. The need is to summarize the uncertainty in the data somehow, which is why I suggested posting +/- numbers. Perhaps the databases could indicate the sample size usually needed to equal a season’s worth of PA’s in their respective glossaries?

I don’t know what the answer is, but I’m never a fan of limiting the available data just because it is being mis-used or mis-interpreted. To me, that says we need to do more on the explanations and context given for the data. It would be one thing if the single-season UZR numbers didn’t have a historical use. But they do, so I’m against any proposal to do away with them. That’s not what you’re suggesting, but it is what Kris threw out there.

I do like your solution (running totals over some timeframe of “equivalent sample size"), if used in addition to (not in place of) the yearly numbers.

What I think is misused even more is the UZR/150 for guys with less than 150.

“Although he only has 40 games, Joe Blow has a UZR/150 of +30!”

The UZR rate (per game or chance) should be regressed before extrapolating over a given number of games.

I think the various suggestions for adding options for the display of stats are pretty good.

I would just add in that we should leave the current options as they are. The truth is, any stat can be “misused,” and given the relatively trivial “real world” consequences of this sort of misuse, we simply have to do as well as we can telling people what number (yearly, tri-yearly, projection, etc.) means what. People are going to make mistakes anyway, let’s not “hide” any information from them.

I am excited for you revamped Oliver Brian.  Studes has told us they should be great.

Thanks Nick.

I have been coding everyday since before Thanksgiving, and finally yesterday got the last of the startup in to David Gassko. But, of course, there are still upgrades that just couldn’t fit into the deadline. It’s still very good, just have ideas for making it even better that take more research and tweaking.

There is one thing that lots of people fail to understand about metrics like UZR.  When we say that a player, for example, had or has a UZR of +10 in 155 games for 2009, that is NOT an actual record of his defense.  It is not the same thing as saying that a player had an OBP or an OPS of X, which IS an actual record of his offensive performance (although it does not necessarily reflect his “true talent” even for that period of time because not all of the “same” events are created equal).  That +10 UZR is an ESTIMATE of his defensive performance for 2009 AND an estimate of his defensive talent.  The reason for that is fairly obvious and has been discussed before.  It is because the data (the hit location, speed of the batted ball, etc.) and the location of the fielder is imperfect and/or unknown.

So, for example, when a player has a +20 UZR in 100 games (+30 per 150) that does NOT necessarily mean that he has performed spectacularly on defense, although it it likely that he has performed very well.  That +20 is likely a combination of very good play and some “sample error” plus bias in the data.  He may have even played very average defense over that time period (for example, plays that he made and that appeared to be difficult according to the data were NOT actually difficult).

Contrast that with a player who hits 45 HR in a season.  Even if that does not represent his “true talent,” such as for a player who averages 20 HR per season otherwise, he still hit 45 HR by hook or by crook.

So let’s stop saying that although a sample (one year or whatever) UZR does not necessarily represent a player’s true talent defensive talent, that it did in fact represent his actual performance. It doesn’t! That is one of the differences between offensive and defensive stats and one of the reasons why defensive stats like UZR have to regressed more heavily than offensive ones.  It is NOT only because of sample size!  It is because of sample data error.

As I have said many times before, when you see a sample UZR, be it one season or 5 seasons or a half a season, do not think in terms of that number for ANY (either his true talent or his actual performance) reason.  Always think in terms of that number regressed.

As far as the general discussion about being “deceived” or “misled” by fluctuations in sample numbers, as I have also said in the past, I have trained myself over the years to pay very little attention to those individual numbers.  They usually mean next to nothing and trying to attach some meaning or importance to them will get you in trouble more times than not.

So when it comes to that last paragraph, does that mean that the individual that insists that player defense fluctuates greatly from year to year is really just trying to rationalize the fluctuations and perhaps not correctly?

In other words, the argument that defense fluctuates greatly from year to year is not really any better than those that say that it doesn’t fluctuate that much. Both sides may be misreading the fluctuations and reading into it something that shouldn’t be read. The truth is probably somewhere in between?

Depends on what you mean by “fluctuating.” If the mean (defensive true talent in this case) remains pretty much the same from time period to time period, then there will always be random fluctuations caused by sample error and those can and should be completely ignored.  If the mean changes from time period to time period, then how much of the fluctuations represent a change in the mean and how much is what we call “random fluctuation” (which can be ignored) depends on how much the underlying true talent tends to change and the sample size of the time periods.  Without knowing those things precisely or at all, you are always better off ignoring those fluctuations especially when they are close together temporally and when each sample represents a small sample size.  And again, using some kind of Marcel (a weighted average) takes care of all those unknowns pretty well and is usually the best we can do anyway unless we have some pertinent information like injuries, etc.  When we look at things like “banner years” we find very little evidence that patterns and trends mean much of anything (which suggests that changes in true talent are dwarfed by random fluctuation at least from one season to another (as opposed to, say the first 3 years of a player’s career and the last 3 years, where aging can and does create a drastic change in true talent).  That suggests that ignoring those fluctuations and just focusing on some kind of Marcel is usually the best way to go.

To what offensive stat is UZR most similar, based on year-to-year consistency/fluctuations? BABIP comes to mind, but I would think that UZR is more consistent than BABIP.

Zach/23: 2B+3B

Actually, BABIP is probably right.

MGL - How hard would it be to do a split season correlation with UZR?

#26, not hard of course.  Are you suggesting that that is better than say a year-to-year or even an intra-class correlation because the true talent is less likely to change from the beginning of the season to the end?  The problem is that the sample sizes of the number of players is still going to be small enough that the confidence intervals of the ensuing “r’s” are going to be pretty large.  I will guess that the “r’s” are going to be in .3 to .4 range (depending on the position and the number of opps or games of course) but that even for a .4 “r” it will be like .3 to .5 at the 95% level or something like that.  We have already done the year-to-year ones (for a min number of defensive games) and they are in the .5 to .6 range I think.

I was thinking more in terms of what Pizza did with BABIP, and other metrics that never “stabilized” over a full season.  I doubt you would see much of a correlation using single season split correlations due to the massive sample size issues, but if you were to do it over a span of 4-5 years, you could see how many games it took to “stabilize”.  Then compare it to other stats to see how UZR ranks in terms of predictiveness.  That might not be worth all of the trouble to code it all, but I think it could be interesting.

MGL/20,

That’s a really great point. Do you think that the “measurement error” will be reduced to the level of most offensive statistics once the fielding version of pitch/hit F/X? (I can’t remember what it’s called.)

Basically, is the uncertainty in the measurement itself something that will be solved with technologies we expect to be online in the near future?

"It is NOT only because of sample size!  It is because of sample data error.”

I am having some trouble with this, as least as a means of distinguishing it from offensive stats.  Hopefully you can explain my error.

I my estimation a batter does not hit singles, doubles, triples or Home Runs.  A batter does Walk and Strike Out (I think; I am pretty sure), but when it comes to hitting a batter swings and makes contact with a ball such that it imparts to the ball a velocity and angle.  The batter himself also runs the bases.  Everything after that, to my estimation, whether it is recorded as an out, or one of the four possible hits (or in some cases, both), is a combination of sample size and sample data error.  I do not see it as different from defensive stats.  Why is the data source for URZ qualitatively any different?

I can understand the answer being “mistakes and biases in recording defensive data are more prevalent.” I also understand Tango’s assertion that a seasons worth of defensive data is the equivalent to half a season worth of batting average data.  But right now I am having difficulty understanding why they are categorically different in kind and therefor subject to completely different concerns.

Paul, I disagree.

Derek Jeter hit 18 HRs in 2009. That is a fact, as well as his 212 hits.

Many of those hits were hard, clean, no doubters. Some were cheap and the result of luck or bad defense, but not Jeter’s hitting skill. Given enough time, those things will even out, and we can use the factual record to estimate Jeter’s true talent as a hitter, his true rate of singles or homeruns, as well as walks or strikeouts.

We do not have a complete record for fielding. How hard would it be to evaluate batting if we knew how many hits each player had, but not how many plate appearances?

We know how many balls Jeter made into outs while he was playing SS, as well as how many errors. The available written record does not tell us, factually for sure, how many ground ball hits to the outfield were his primary responsibility. I can watch film clips and give you my opinion, mgl can give you his opinion or statistical estimate, but they are estimates and reasonable people will come up with different numbers, especially when small samples do not allow the luck to even out.

Even infield singles are not for sure. I have seen balls hit between the third baseman and the bag, go down the left field line and bounce off the stands to be picked up by the SS in short LF. Gameday says “single to SS”. Yes, the shortstop retrieved the ball, which is what Gameday chooses to record, but if I have to rely on that as a proxy for who was responsible for fielding the ball, then I consider it an error in the data. In this case, Gameday and BIS or Stats could very well give different fielders on the play. If I look at the angle of the ball, and see that it was hit over the bag, I can overrule the recording of SS and award it to 3B, but there may be times (non zero probability) when that test will change a correct record into an incorrect one. More estimation, still chance for error.

Brian,
I understand that there are different sources for data error in fielding and hitting, but I still don’t see how it is different in kind.  Mirroring your example:

Gameday might say single to left field and put up a location where it was hit.  We can know it belongs into the arbitrary bucket “hit:single” and, yes, we can with 100% certainty assign the PA to the batter.  But we cannot assign the degree of credit for that hit to the batter, the fielder, or luck (from an accounting perspective - that is a difficult bucket).  Right now, for hitting statistics, 100% of the credit for a hit is assigned to the batter.  (Any credit that might be assigned to other players, using other statistics, is simply additive - from the perspective of a batter’s hitting statistics, he gets 100% of the credit for any result). 

That is data error, no?  It may well be data error that “comes out in the wash” with a substantial sample size, but then how is that different from the examples you gave, above, for defensive statistics? (Apart, of course, from the sample size issue).

Also, as to “Derek Jeter hit 18 HRs” being a “fact.” Sure, it is a “fact”, but it does nto describe Jeter’s hitting talent.  We could just as easily say “Derek Jeter had 8 errors last year.” That is also a “fact” (though the official determination of an “error” is somewhat more dubious than the official determination of a “Home Run"), but that does also not describe Jeter’s fielding talent.

I am still not seeing the difference (apart from that I can accept, though not really know, that the data errors in fielding stats take a larger sample size than with some (most?) hitting statistics.

Carrying the examples a bit further, and hopefully adding some clarity to what I am saying:

1.  A HR is hit and recorded in the record books.  We could go back to the game tapes (for those that exist) and look at the Home Runs.  Some we would overrule.  Thereby introducing exactly the same sort of error you suggested in you last paragraph.  Likewise, we could go back an look at all foul balls and determine that some should have been recorded as Home Runs, and make changes accordingly.

2. A hard hit ground ball is hit to short stop.  The short stop fields the ball but throws wide.  The batter does not advance to second base.  The scorer calls it a hit.  We may not agree.  that is a source of error (both ours and the scorer’s) in both hitting statistics and fielding.

It seems to me these sorts of error in data are replete in both hitting statistics and fielding.

I am not seeing the difference that lets us call one a “fact” and one “interpretive error.”

Then there was a misinterpretation of your original post.

“Also, as to “Derek Jeter hit 18 HRs” being a “fact.” Sure, it is a “fact”, but it does not describe Jeter’s hitting talent.”

This is taking the historical record in order to determine the hitter’s true talent, and I addressed that in my first response. “Given enough time, those things will even out, and we can use the factual record to estimate Jeter’s true talent as a hitter, “

The difference is that in hitting or pitching, we have pretty much all of the ‘factual’ historical record that we need to do the analysis. In fielding, the historical record is incomplete or open to interpretation before we even start to determine true talent. In batting and pitching, we know how many total opportunities (plate appearances and batters faced) each player had. We do not know that for fielding. We generally have the balls a fielder reached, but not the ones he didn’t. We have most of the numerator, but even less of the denominator.

As I said before, try determing a batters true talent when you don’t have anyone’s number of PAs.

And, in the Teixeira fielding discussion, I pointed out that even with roughly 300 ground balls in a first baseman’s zone, the difference in a Marcelized run projection from the best to worst 1b is around 15 runs, maybe 20 plays a year. This allows random chance to be a larger percentage of plays not made each year than at other positions who get more chances and where the rate of success is not as high.

"As I said before, try determining a batters true talent when you don’t have anyone’s number of PAs.”

But no one is doing that.  It is, of course, impossible.  In the same way that “try determining a batter’s true talent when you don’t have anyone’s [you pick the numerator].”

What is happening is a number of opportunities is being determined.  It may be that in defensive stats, the denominator is subject to more error than the numerator and for offensive stats the reverse is true.  I still don’t see this great difference so that one stat (Batting Average) is more “real” than another stat (UZR).

I will repeat again, I completely understand that UZR, particularly annual UZR, may be less reflective of a player’s true talent - but if that is the case it is one attributed entirely to sample size.  I have not been arguing this at all.

But saying that one stat requires more opportunities before it starts to reflect true talent than another stat is not what I interpret as being claimed.  I am 100% on board with the sample size issue.

“It is NOT only because of sample size!  It is because of sample data error.”

That is what I want to understand.  Why is there sample data error in UZR but not in BA?  I see error in both.

Because there is not a proper level of detail in the scoring record to definitively define the number of opportunities for each fielder. And I gave an example of how different data recorder can and likely do record different players as responsible for the same ball.

Or Gameday tells where the fly ball was retrieved, not where it landed, making it harder to do determine each fielder’s expected value of flies caught per opportunity.

‘Split zones’ refer to balls between fielders. We either have to run statistical models or watch video to assign the ball to the 3b, the ss, or maybe neither.

This is all the massaging of the original data that has to take place before we start determining true talent.

With batting and pitching, I take SI, DO, TR, HR, HP, BB, S) - park adjust, league adjust, weight last three season, regress to mean, and it’s done.

"As I said before, try determining a batters true talent when you don’t have anyone’s number of PAs.”

But no one is doing that.  It is, of course, impossible.

====

We don’t do it with batting, but it is what we have to do with defense. That’s my point.

I’d like to see a fielding metric index. For example, if you were to combine Texeira’s UZR, TZ and Dewan for 2006-2008, you’d get something like +7, +6, +6 vs. -2, -4, +10 for UZR.

We wouldn’t have to argue about whether Texeira falls down a lot while making plays. 

The numbers would be more stable and probably a better overall reflection of the defense played.

I’m sure I’m off base but it seems like a good idea to me.

It shouldn’t be the job of the creators and publishers of statistics to ensure they’re properly used.  If a writer doesn’t understand the limitations of a particular statistic, it isn’t MGL’s job or Tango’s job or Fangraphs’ job to correct them.  As Bill James said, there will always be people ahead of the curve.  Abolishing the 1-year samples doesn’t do much to cure ignorance--people will still reside in the thick of the curve.  Sure, it abolishes one way to abuse UZR, but, they’ll find another way.  I don’t have a problem with abolishing 1 year samples of UZR.  I don’t ever use anything less than career or 6+ years.  However, expecting this to be a solution to ignorance is probably folly.

I’m trying my best to find holes in Catshirt’s argument and I cannot. Is it because he has postulated a sound statistical paradigm, or is it because all those math classes I took to fulfill my minor were worthless?

One great point: arrows are excellent graphical ways for us to easily understand how a player’s career is progressing. Franklin Gutierrez: number two with a bullet!
lee mckenna scam