THE BOOK--Playing The Percentages In Baseball

Just wanted to say that the Newark Star-Ledger has had a fantastic sports section for decades now.  Having moved out of the area, I really miss it.

I don’t see anything wrong with reporting UZR to one decimal point.  Would it be ok to report to one dp if they also listed the standard deviation?

I think it’s up to the reader(s) to be aware of any small sample size issues, and FanGraphs lists sample sizes on UZR for the most part.

vr, Xeifrank

I’m with you all the way, RedRobot.  i stopped reading it regularly after high school (10 years ago, crap) but i used to read every inch of it back in the day (except for the copious harness racing coverage).  i still get wisftul thinking of those old Frank’s Chicken House ads.

"I think it’s up to the reader(s) to be aware of any small sample size issues”

The writer introduced UZR to the general audience for the first time, and the reader then needs to somehow be aware on his own of the sample size issue?  Uh-uh.

I’d say most people who use UZR don’t even appreciate the sample size issue.

I’d much prefer to see a 5-point scale, and leave it at that, as Tippett did in presenting his fielding metric in his game.

I also don’t see any problem in reporting UZR to the number of decimal places that the methodology spits out.  After all, that is the most likely true number.  Rounding it off to the nearest whole number would be giving LESS accurate information.  I get Tango’s point, that it gives the impression of more precision that it is, but still…

“...one of the most important areas of contribution, scooping balls and tracking wayward throws.”

Actually, that is one of the LEAST important areas.  When I do a WOWY for scoops, the spread per year among 1B is 1-2 runs, I think.

And “tracking” wayward throws?  Interesting use of words.  Do they use radar for that?

I really hate when anyone takes a small sample within a large sample (like a player’s career or the last few years) and tried to make “sense” of it.  There is absolutely no reason to be using 60 games of UZR for anything when you have 400 or 500 games.  To say that someone is playing better or worse defense because the numbers are better or worse is not right.  We don’t know that a player is playing bad defense because his UZR is minus. That is the whole point of using as much data as possible.

In fact, I’ll throw out this interesting question:

Say a player has a UZR per 150 of +15 over the last 5 years - IOW, most likely an excellent defender.  For the last 10 games, he has a UZR of -1 (-15 per 150).  What are the chances that he actually played below average defense over those 10 games?  How about 50 games?  100 games?

Okay, I have sort of a genereal question about UZR, or really any fielding stat.

Let’s say I’m just looking at value for, say, picking the MVP winner or whatever.

A player is +15 runs by UZR over a full season. Let’s assume, based on his past performance, age, regression, etc., that his true talent estimate is like +5.

If I’m just looking for value over those 160 games, is it better to go with +15 or +5 (or something else)?

For an MVP vote you should use just that season’s data without regard for true talent.  If you did it for fielding, you might as well do the same for hitting.  A-Rod doesn’t deserve any MVP consideration for 2009.

Not the same thing Rally.  For hitting, we are actually measuring a player’s contribution to runs and wins, at least to some extent.  UZR does no such thing.  Just because a player has a +5 UZR does NOT mean that he contributed to saving any runs.  Because of measurement error, all of those “+5 runs” could have been the easiest plays in the world.  In fact, if you had watched the player on defense, he might have contributed to lots of runs for his opponents even though he had a +5 UZR.  Completely different thing, UZR, from an offensive stat which reflects what “actually happened” in a runs/wins sense.  A sample UZR in the short run would NOT be a good MVP measure, IMO, although if that is the only thing you got, then fine.  But I am not sure that a +15 in 100 games for a player who is a -15 historically is actually a better performance in an MVP sense than a +10 for a player who is a +15 historically.  That is why I asked the question above.

Thanks, guys.

MGL, yeah, you explained perfectly what I was thinking, with regards to being skeptical using a season’s worth of fielding stats for MVP/value discussions. The question you posed at the end of your last comment got me thinking about it, though it is something I’ve thought about before.

Yes, for example, a single is a single.  It describes an even perfectly.  An RBI is an RBI.  It is relatively easy to use that type of data to assign “MVP” value (although certainly what constitutes “valuable” is debatable).

No such analogy with UZR.  It describes an amorphous series of events almost like a wave or a probability distribution for the location of a subatomic particle, or something like that.  Not so easy to go from UZR to MVP.  That is why I even hate using UZR or some other advanced metric to evaluate or criticize gold glove awards.

I’ve said this before and I’ll say it again. There are two sources of noise in a metric like UZR. One is simply random variation in sample data, such that a player may have made some bad or good plays even though his true talent may be better or worse than the net of those bad and good plays, and two, measurement error, such that a player with a plus UZR may have actually made worse plays than a player with a minus UZR. (A third source of noise, but maybe it is not technically noise, is human error and bias introduced by the persons recording and processing the data.)

I have to disagree.  I’ll speak for TotalZone, which I know all the details of, instead of UZR, which I don’t.

Quite simply, players with high ratings make more plays, fewer errors, and allow fewer hits through their area of responsibility.  It adjusts for some context, but there is a ton of noise in the data and certainly there is a lot of room for error in the difficulty of chances.

So a player with a high TZ might not be a good fielder, but he made more plays and allowed fewer baserunners.  He’s contributing to wins in the same sense that a hitter who drives in 3 with a 2 out perfectly placed bloop double does.  He contributes more to winning in that specific circumstance than a guy who crushes a line drive that is speared down the line and turned into a double play.

"So a player with a high TZ might not be a good fielder, but he made more plays and allowed fewer baserunners.”

But I don’t think UZR is actually saying that.  UZR is saying he mades plays which, on average, are difficult.  But they could have been gimmies that the data lack the precision to classify as such, and only appear to be difficult.  Right?  They’re not a “record” in the same way as hitting stats.

Zack, you are exactly right, and that is the difference.  UZR or any of the other defensive metrics, other than regular fielding average I guess, is like recording a “hard line drive” on offense.  It likely was a hit and on the average was a good thing, but it might have been caught and it might not have been caught.

I’m not really sure what Rally is trying to say here.  Of course on the average a player with a higher Total Zone or UZR or whatever “made the plays” but obviously he didn’t necessarily make more plays on a percentage basis, than a player with a lower rating, because one of the things we don’t know is where the ball was actually located.  You may SAY in Total Zone, or simple ZR, or UZR, that a player made 10 plays out of 30, but you don’t KNOW about the 30.  Some of those could have been just out of his reach and some of those could have been nowhere near him which no fielder could have gotten to.  You NEVER know the true denominator (number of chances) in any of these defensive metrics, therefore you can never know the true percentage of plays made. You can only estimate and report, “on the average” or a “likelihood.”

So, I don’t see the argument that that information allows us to evaluate “MVP-type performance” like offensive stats do.  It’s not even close.  If a player has 120 RBI and scored 100 runs, or hits 45 HR, they contributed to exactly the same number of runs/wins regardless of the quality of those events.

I think Rally wants us to say that because we know how many plays were made or not made, we have the same type of information for those defensive metrics.  As I explain above, that is simply not true.  We know how many plays were made, but don’t KNOW how many were NOT made by any particular fielder, because we don’t know exactly where those “not made” plays were and who could have fielded them if they were a better fielder.  It is implausible to think that we can evaluate MVP-type performance simply on knowing how many plays a fielder makes, without knowing the quality of those plays and without knowing what plays they could have made but didn’t. Again, these metrics tell us all of those things “on the average” but not with any degree of certainty, which makes it difficult (not impossible) to use that metric for an MVP-type of evaluation.  On offense, it tells us exactly what happened in terms of what is important for MVP-type evaluation.

If fielder A fields 30 balls out of 100, but most of them were easy - i.e. an average fielder would have fielded all 30 also - and fielder B fields only 25, but many of them were difficult - i.e., an average fielder would have fielded only 20, if you knew that, would you still award the MVP trophy to fielder A?  I don’t think so.

I think the test of how “MVP friendly” a metric is, is this:

“If you knew what really happened, would you change your vote?” For example, if a player made 10 absolutely easy plays and all of them were the last out of a one-run game with the bases loaded, would you still award him an MVP versus if those 10 plays were spectacular plays?  I think not.  Rally would have us believe that we would, I think.  On the other hand, if you knew a player hit 10 walk-off singles in the bottom of the 9th to win a close game, would you much care whether they were line drives or bloop hits?  I don’t think so. That knowledge might be a tie breaker, but that’s about it.

That is my point, and Zack’s as well.

Let me try to make the analogy using hitting stats.  We know that say the Expos had 1400 hits, 150 HR, 250 SB, and 4000 batting outs.

For whatever reason, we only track the positive events at the indidivual level, so we know how many hits, HR, and SB Tim Raines had.  But, we don’t know how many batting outs he had.

We DO know the number of games he played, but we don’t know what spot in the batting order he played in.  We try to estimate how many of those 4000 known batting outs were created by Tim Raines.  We might have access to his runs scored and RBI to which we can infer batting order, to which we can infer PA, from which we can deduce outs (since we have hits and walks).

We try to infer as much as we can, and we end up with an estimate of 320-340 batting outs for Tim Raines.  We’ll say “330 outs, +/- 10 outs”.  And since a batting out is -.3 runs, that means that we might end up saying that Raines has “+40 runs above average, +/- 3 runs”.

UZR has access to more granular data (say in my illustration we know exactly how many games Raines hit in each slot, and how many PH as well), while TZ has less granular data.  So, UZR’s estimate of “opportunities” (which is really the quality of opportunity), might make UZR say +/- 5 runs, while TZ would say +/- 10 runs.

This is the error estimate that MGL is talking about in terms of measurement error.

So, I would use this for MVP, with the appropriate uncertainty level.

Rounding it off to the nearest whole number would be giving LESS accurate information.  I get Tango’s point, that it gives the impression of more precision that it is, but still…

By that reasoning, then why not show it to 2, 3, or 4 decimal places?

The fact of the matter is that we report things in player runs, which itself is an integer.  I can understand if you are trying to explain one particular run, and you need to show it to one decimal place.  But, if a player has say 100 expected outs (why not show 100.354 expected outs?) and 120 actual outs, why not say that he’s at +20 outs or +16 runs, rather than saying +15.754 runs?

If for example it would not matter to a single person on the planet if I report something as +15.7 runs or +15.8 runs, then why bother doing that?  I can see someone out there possibly being bothered by +15 and +16 runs.  No one is going to care about +15.7 and +15.8.

Those decimals drive me nuts!

I understand there is more error in defensive metrics, and not having a clear picture of opportunities to make plays.

My issue is just that 2008 performance should not affect an MVP award that is based on 2009.  At least it won’t with my votes.  It certainly does affect what you’d pay the player for 2010.

So if Raul Ibanez ends up with an above average fielding rating for 2009, I’m not going to believe he’s really a good fielder.  But I’ll assume that more likely than not he made the plays for that year, and rank him accordingly on my ballot.

"My issue is just that 2008 performance should not affect an MVP award that is based on 2009.”

Ah, but you are wrong!  No one has attempted to answer my question yet, which I think is pertinent to the discussion.

Let’s say that Player A was +15 per 150 for the last 5 years and Player B was -15.

And let’s say that both Player A and Player B are +10 in UZR in 2009.  Again, we don’t really know that either player actually performed well.  We really don’t.  They probably did.  On the average they did.  But there is a significant likelihood that either (or both) player really performed like a zero or a +5 or -10 or a -5.  The data could just be bad.  The quintissential example of how “bad” the data is, is when we compare STATS and BIS UZR.  There are a significant number of players who are plus in one system and minus in the other!  So clearly a plus in one system does NOT necessarily mean that a player actually played well. How could it when the other system might have him at minus?

So let’s get back to my example.  Two players are +10 for 2009.  One was historically very good and one was historically very bad.  You (Rally) don’t think those historical numbers tells us something about how accurate the +10s are?  You bet they do!

If we were to go to video and look at every play for Player A and Player B, what do you think we would likely find?  That Player A, the one who was historically good, likely actually played better in 2009 than did player B!  Even though they both had the same +10 UZR!

So in evaluating MVP-type actual performance, you bet that past performance is helpful.

Tango, you swayed me on the decimal point thing.  What got me was when you said, “Why not report it to 5 or 10 decimal places?” (I am paraphrasing of course.) I realized that he is truncating the number to two decimal places anyway.  Why not truncate it to none, which is cleaner and makes more sense from a presentation standpoint and from the standpoint of not “suggesting” that it is more accurate than it is.

My problem with that is it doesn’t allow for the possibility that players can have good and bad defensive seasons.  You just have their multi-year estimated true talent level, and regardless of how the player performs he gets the same credit every year.  If that were true it would be the only aspect of baseball where players don’t have ups and downs.

I’m not going to come out and tell you flat out you are wrong because you make some good points.  But I don’t agree.  For you to tell me flat out that I’m wrong - that I think is dead wrong.  This isn’t a settled issue, there are good reasons to prefer either approach.

In some cases an abnormal defensive season is obvious - Edgar Renteria with the Red Sox because the difference wasn’t in the crude estimate of chances, but in his error total.  Three years before that he makes 19, 16, and 11.  Three years after he makes 13, 11, 16.  In between he makes 30.  With range fluctuations it is not as obvious.

"As long as I made on person change his mind, it was all worth it.”

Or some such

***

Let’s try to put some real faces to MGL’s examples.  Let’s see that Derek Jeter was historically -10, and this year he’s at +10.  Let’s say that Jimmy Rollins was historically +20 and this year he’s at +10.  Who had the better fielding season?

Let’s say Derek Jeter was historically +40 offensive runs, and this year he’s at +20.  Let’s say that Jimmy Rollins was historically +10 and this year he’s at +20.  Who had the better offensive year?

By MGL’s reckoning, even though both Jeter and Rollins are +20 hitting this year and +10 fielding this year, then Rollins is having the better year, because the fielding component is more real in Rollins case because it (tries to) handle one of the two errors in fielding metrics.

And I say:

Since the ONLY uncertainty that we have of the fielding metrics is the quality of plays, then it is that part of the fielding metric that needs to be looked at. 

Say we have these two examples:
1.  that Jeter made 400 outs last year and 400 outs this year.  He had the same pitchers in both years.  And yet, somehow, his expected outs last year was 415 and his expected outs this year is 385. 

2. that Jeter made 400 outs last year and 430 outs this year.  He had the same pitchers in both years.  His expected outs in both years is 415.

These two cases are different, correct?  Jeter indeed made 30 more outs in the second situation, and as best we can measure things, NOTHING in his environment changed.

In the first case, we think that even if his environment did not change, something about it did change, enough that his expected outs were not that tough this year.  So, even though he recorded the same number of outs, it was because of his better performance at getting at those tougher plays.

***

The problem is that we don’t know which of the two situations Jeter is in.  Did Jeter really perform better in both situations?  Or, does situation 1 really show the same guy doing the same things in two years, but we have a measurement error of the context?  And does situation 2 really show that he performed better even though he recorded an indentical number of outs?

To this end, MGL is cutting his losses, and presuming a bit of both, and therefore, needs to know how he’s performed historically.

Rally is saying (paraphrasing): f- that.  He did what he did based on how I think his context was.  My measurement errors are not going to be influenced by his past performance.

UZR and the framework with which MGL uses it works wonderfully to estimate a player’s true talent level, or for projections.  Let’s just say that isn’t the goal.  Let’s say you are trying to design a framework to estimate who well a player did in a specific season (say there’s a money award for best fielder, you are in charge of the application and integrity of the award).  What would you do differently?

Another question, Say a new shortstop comes over from Germany, call him Jerek Deter.  In his first season Deter has a -15 UZR.  In year 2 his UZR is +15.  Which season did Deter play better?

A) his true defensive value is the same both years
B) year 2

If so, how much better did he play in year 2?

Rally, you can make the question tougher on MGL by saying this:
in year 1, he made 50 errors
in year 2, he made 10 errors

(I got the idea from your earlier post.)

See, now it because very clear that it’s not a measurement error that we’re faced with, but actual performance difference.  He has a 40 error drop, and Rally is saying a +30 run gain.  So, we are left to conclude that he was indeed better in year 2, and likely a 30 run difference in performance value.

However had he made the same number of errors, then MGL might say that there was a measurement error somewhere, and that while he did perform better in year 2, that performance was not +30 runs better, but likely +5, maybe +10 runs better.

Rally, of course players have good and bad defensive years.  I said you were dead wrong when you said that past performance cannot influence our estimate of whether a player had a good or bad defensive year.  I think that is dead wrong. 

If all we have is this year’s UZR then we are stuck assuming that a player who is +5 this year actually performed better than a player with +4.  The +5 guy gets the MVP.  But, we can all agree that it is entirely possible, probably even 49% likely, that the +4 guy actually had a better defensive season than the +5 guy.  Heck, STATS UZR might even have the +4 guy as +10 and the +5 guy as +3.  The reason there is almost no effective difference in performance between the +4 and the +5 guy is because there is so much measurement error in those numbers.  If you had watched both players every day, and you have a keen eye for defense, it is likely that you would know exactly who actually had the better year.  And it is almost as likely to be the +4 guy as the +5 guy.  Past performance can help us to identify who likely had the bona fide actual better year.

Rally, if the +4 guy was +20 in the past and the +5 guy was -20 in the past, don’t you think that it was much more likely that the +4 guy truly had a better year and the +5 guy was only +5 due to measurement error?

If your answer is, “Yes,” then you have to agree that you were dead wrong with your statement that past performance can have no influence on how a player actually performed (not his true talent) in a current time span.

Maybe.  (on the +4 vs +5 guy). I’d want to look more at the numbers.  Errors vs Range numbers may influence my decision.

I never said it can’t have any influence, I said that it probably shouldn’t in that I think it intoduces as many problems (plus complexity) than it solves.

You have two players who are +20/+4 and -20/+5 in two seasons.  You need to answer which one is better in year two, and you think multiyear data is applicable for this purpose.  How much better is player 1?  Make the multiyear adjustment too small (say all it does is flip-flop who is the +4 and who is the +5) and it’s a waste of time and effort.  Make it too big (such as a straight weighted average) and you are ignoring the possibility of good and bad defensive seasons.  What is the right balance for doing this?

It’s not like there’s much difference anyway, 1 extra fielding run is not going to convince me in an MVP vote anyway.  If they were the top two candidates and even in batting baserunning, etc, I would look to non-statistical factors (leadership, contributing to pennants) before I decided based on one fielding run.

Rally is right.

In his example, if you have someone who is -15 runs in the first year (with 50 errors) and is +15 runs in the second year (with 10 errors), then the reason is pretty clear as to why he jumped +30 runs in performance.  This is not an issue of a possible measurement error (in this case).

Well, there are two questions.  I only addressed one of them.

1) Should/should we use prior data to help us tease out the noise in the current data so that our “MVP-vote” is more accurate?  I say, “Yes” and you lean towards agreeing.

2) How (and how much) should be use the past data? 

I don’t know the answer to the second question, but it does not change the answer to the first one. 

However, I don’t think the answer to the second one (the “how much") is so small as to be de minimus, such that we can dispense with it altogether.  I would guess that the +20/+10 player actually had a better current season (the +10 season) than a -20/+15 player.

Again, for those of you that are not quite following this, I am not saying that we should not give the +15 player the award because he had a fluke great season and his long term true talent is clearly worse than the +10 player.  I am saying that the +15 is likely not a great season - that the +15 has a lot of measurement error in it - while the +10 season from the other guy is likely indeed a very good season since we are pretty sure that he is a great fielder.

Here is a good analogy which I think will help with the skeptics:

Let’s say that we have 10 stop watches and that one of them is broken and gives us totally random, incorrect times.  Now we time a bunch of runners.  Each time we time a runner, we randomly choose a watch.  Obviously there is a 90% chance that the time is accurate and a 10% chance that it is not (it is a random time).  Let’s also say that the random time will look reasonable such that we cannot tell right away that it is a broken watch with a random time.  This is exactly what UZR or any other metric with measurement error is like, by the way - 9 good watches and one broken one.

When all is said and done, we are pretty happy with all the measurements. Because of the 10% chance of any runner being timed with the broken watch, we simply regress everyone’s times slightly towards an average time.  Again, we are happy with the times but not 100% certain that they are accurate.  On the average a runner with 9.0 is faster than a runner with 9.5, etc.  Again, this is exactly like what happens with UZR or any metric that requires and imperfect measurement.

Now, let’s say that we have a player who was timed at 9.0 and that is very fast.  We know there is a 10% chance that that is a random time and that we have no idea how fast the runner is.  But there is also a 90% chance that this is an accurate time and this is a fast runner.

Well, what if I told you that this runner was timed 100 times before with one of the same 10 watches and his average time was 9.02?  What do you think the chances are that he was timed with a good watch this time versus that he was timed with the broken watch?  Now we are like 99% certain that he was timed with a good watch, rather than only 90% certain.

But what if we knew that his time was 9.99 before in those other 100 timings?  Well, we are pretty certain that he got timed with a broken watch this time, aren’t we?

I hope that helps to explain the UZR thing.  It should.

The runner example is not helping me because we aren’t comparing 1 random timing with a sample of 100 times, 90 of which are accurate.

We’re comparing one season where the runner ran 10 races, each with a 10% chance of having an inaccurate time, with the previous 10 races he ran.

“I would guess that the +20/+10 player actually had a better current season (the +10 season) than a -20/+15 player.”

The second player, considering weighting of seasons, is probably an average defender.  The 1st guy is more like +14 I guess.

So you think a +10 season from an above average defender likely represents better real performance than a +15 season from what we consider an average defender.  To each his own, but to me that seems like too big of an adjustment.  The race is pretty much over before the starting gun fires.  How good would the previous -20 guy have to play before you thought it was likely he had a better season than the +20/+15 guy?

"How good would the previous -20 guy have to play before you thought it was likely he had a better season than the +20/+15 guy...”

... and was not a victim of the (possible) measurement error?

This goes back to Rally’s example of also introducing the 50 errors one year and 10 errors the other year.

What if, in MGL’s stopwatch example, you also show that one guy stumbled out of the gate, and the next time, he tripped over his own two feet.  Now, we won’t need to rely on the regression to past times, because we have additional data points that tells us with more clarity what happened.

There is a “numerator” issue and “denominator” issue.  The numerator tells us exactly what happened: how many outs, and how many errors, and how many whatever actually happened. 

The denominator we must estimate in terms of the quality of the opportunities.  About how many plays, about how tough those plays are, and other estimates.

MGL is really arguing that we need to regress the denominator to some extent.  We obviously have no need to regress the numerator, because if a guy made 400 outs, he made 400 outs.  And if he made 50 errors one year and 10 errors the next year, that really happened.

MGL is regressing the success rate, when really he should be regressing the opportunities (the denominator) per total balls in play (or something else).  You can’t take away what a player actually did, which is what his regression technique would do.

Rally, again, you are arguing “how much” rather than “yes” or “no.” If you agree that past performance (i.e., a player’s estimated true talent) can help us to figure out how much measurement error there likely is any other out of sample performance, then there is no disagreement between us.  If you are arguing over how much influence that past performance/true talent has on informing current performance, it is a one-man argument, because I don’t know the answer to that.  I don’t know whether the +10 player likely had a better performance than the +15 player given their respective past performances (I guessed that he would, but I don’t really know) - I only know that past performance informs the answer.  That was and remains my only argument, which I think you now agree on.  If you don’t, then you are still wrong.

I’m not going to give a yes or no answer. (Especially with the position that if I don’t agree with you then I have to be wrong).

My answer is that it can inform the answer (does that count as a yes?), but it also could distort, and I still think it’s better to just leave it be.