THE BOOK--Playing The Percentages In Baseball

I wonder about the data I’m using for TotalZone - retrosheet.  Since 2003 they have virtually complete G/F/L/P batted ball coding for all balls in play.  Does retrosheet get this data from STATS, BIS, or another source? 

That is the only potential bias in my system, if one scorer had more of a tendency to code a flyball or line drive than another, as the difference in out rate is enormous.  The other parameters used are who fielded the ball, whether the batter and pitcher were lefty or righty, whether or not 1B was occupied.  Those are about as close to objective facts as you can get.

Right, that’s why in my WOWY, I only deal with facts.

Given the choice between knowing the identity of the pitcher (and we know how so very high the correlation is in their ball in play distribution year-to-year), or knowing how a scorer treats something as a GB, FB, LD, Pop: I prefer the former.

As I have said several times, I am going to really into why so much difference.  For one thing, I noticed that I was not using any bunts for sUZR and I was using all bunts (and treating them like any other grounder), for bUZR.  This should mostly affect 1B and 3B of course.

There is no question, though, that once you have different characterizations for LD and FB (and even whether a ball is a LD or GB), different “speeds” (even though both use, soft, med, and hard), and different distances and vectors, there is necessarily going to be large differences in the results, especially in the short run.  You would hope that most of the differences are not systematic biases and that the rate differences should shrink as the samples get larger.  We’ll see.

I would not be surprised that because Andruw plays a short CF, that plays that one marks a LD, others mark it as a FB.

Rant: This could all be cleared up if they tracked hang time.  Who cares how they mark the FB/LD; if you tell me it’s in the air for 2.5 seconds and it travelled 250 feet, that’s what I need to know.

Perhaps MGL’s eventual study will be the impetus that BIS, STATS and MLB.com needs to get revamp some of their data recording operations.

Tango, MGL, et al:

I am an analyst with STATS, and I will certainly raise these issues with those in the company who have more power than I.  I think tracking hang time would be a great addition, and hopefully we will have it in place for next year’s data.  If there are any other ideas for further improvements, feel free to chime in.

Thanks Mike for being willing to take these ideas back to the powers that be at Stats.

One thing that would be amazing to get would be the position of the fielders at the start of the play.  I figure it might be tough, but we could do a lot with that information.

Mike: I’d be happy to know the starting position of the CF for sure, and then the SS.  You don’t even have to do the rest.  Just those two would be great for starters.

Hang time for sure.

I want to know when the batter squares to bunt.

I want to know, on every play, not just the outs, the “grade” of the play: 1 to 5, where 1 is routine hit, 5 is routine out, REGARDLESS if the fielder makes the play.

How serious is your operational team about making changes and performing quality control?  I’ll bet you that at least half a dozen of my readers would perform quality check, for free for you. 

But, I know how it works: to be serious, it has to cost you something.(*) So, we’ll charge you $10,000 for the quality check.  You pay us if you are satisfied with our work.  Hint: pretend to be unsatisfied.

(*) It’s true guys.  I’ve seen it alot.  Given a choice between free and paying, a company would rather pay.  They want a binding contract so they have someone to blame!

Oh, and when there’s the shift, I want to know where everybody is playing.

At a minimum, even if we don’t have coordinates for each fielder, we could do general classifications

3 infielders on right side
3 infielders on left side
of swung left
of swung right
1b/3b playing lines
infield in
corner in for bunt
wheel play on bunt
maybe a few more

One thing I had always wanted to do but never got the time with the records being on paper - back in the 80’s when I did stats for a college summer league my scoring system assigned every batted ball to a fielder. I wanted to see balls hit thru the infield, how they varied in the different base out states - how many more balls thru the holes when 2b/ss where in a dp situation, how many thru the hole to rf when 1b holding runner etc.

As I have previous written st StatSpeak, one problem with what I’ve seen with Stats & BIS (unless subscription versions are more complete) is that they have split zones - when a ball is hit between fielders, the location is noted but not which fielder had the best chance at the ball. So we are left to guess on assigning those bip. Please, assign a responsible fielder for every ball in play, out or not. And (if not doing it already) if a ball is unfieldable, such as off the Green Monster, label it as such so no one is assigned blame.

Brian’s list is good.

For the infield, I also want to know “where’s the daylight”?  If both the 3B/SS are swung to one side and 2B/1B to the other, then I want to know there’s the “5 hole” up the middle (Zone 46).  If the 2B is swung over up-the-middle, then the “5 hole” is in zone 4.  If they are all in their, more or less, standard zone positions based on batter handedness, then tell me that.

The key is that SOME information is better than NONE.  Just give it to us as complete as possible.

Here’s a question. How do the various fielding systems/data providers handle a deflected ball? I’m cross-checking my MLB Gameday and Retrosheet databases for 2008 right now and that’s one issue that’s cropped up so far.

Oh yeah, which reminds me: count the number of times the ball hits the ground, prior to the fielder picking it up. 

Tell me the distance that the ball first hit the ground.

If the ball is rolling, then tell me the distance of the last time it hit the ground before rolling.

If it hits anything other than the ground, mark it, and tell us in which direction it deflected.

Basically, deliver FIELDf/x for us, but as much as you can with your eyes, as opposed to the cameras’ triangulation.

Well this is interesting. I’ve finally finished mapping Retrosheet to Gameday for 2008 (batted balls only), and I have 538 plays where Retrosheet and Gameday disagree on who fielded the ball in play.

I looked over all the plays, and they agree on who the batter was, so I think I’ve finally weeded out all potential sources of error on my end of the deal.

I went to MLB.com and watched one of the plays - a Ronny Cedeno single in the 3rd inning of the 9/03 Cubs-Astros game - and sure enough, it was a ball that deflected off Randy Wolf’s leg and ended up an infield single for Cedeno.

Retrosheet scored the play as S14/G, and the output from Chadwick into my database lists Wolf as the responsible fielder. The Gameday verbatim on the play is: “Ronny Cedeno singles on a ground ball to second baseman David Newhan.”

There are some plays where Gameday explicitly notes a deflection in the verbatim, and I’ve updated my script to reflect those plays.

I’ve been working on putting together a fielding system using Gameday hit location data - I’m sure if I was better at writing a parser I could do this purely with Gameday data but I’m not (in fact, I’m not any good at writing a parser) so instead I’m mapping Gameday to Retrosheet to get my results.

Right now it’s infielders only. I’ve prepared two versions so far - one using Retrosheet batted ball types and one using Gameday batted ball types. The biggest discrepancy is six plays for Jimmy Rollins.

I’m working on writing the system up for THT for later this week, but I though the differences in systems was interesting and relevant to the discussion.

http://www.editgrid.com/user/cwyers/infield_czr_retro_vs_gameday

Colin,

Did you adjust the average plays made for pitcher strikeouts (more strikeouts = fewer BIP), batter handedness, and the GB/FB split of the pitching staff?

We have data for each individual BIP; we don’t need to control for how many there are because we can simply count them.

The only variables I’m accounting for right now is where the ball was hit (based on the Gameday coordinates data) and what kind of ball it was (in this case, ground ball or bunt). I’m going to add other adjustments incrementally (batter handedness is the next step).

Sounds good. I look forward to reading the article.

Good luck with your project Colin.  Its not as easy as it might seem at first, but it can be done.

This was recently linked on BBTF, and I wanted to BUMP to ask if there has been any updates in our understanding of when and why sUZR and bUZR differ.

No updates, but I can tell you that any time you have human beings watching games and trying to record exactly where a ball lands or is fielded, how hard it was hit, as well as whether it is a fly ball, line drive, or in the case of BIS, a fliner, you are going to have a considerable amount of variability in terms of the data.

And once you have that variability, since UZR creates so many buckets, you are bound to have lots of variability in the “numbers” after all the data is crunched.

Does that mean that neither bUZR or sUZR is particularly good in evaluating defense?  Absolutely not.  The same would be true for offense if it were not for the fact that the scorers and the rules of the game tell us exactly how to categorize each offensive event.  That certainly does not make the recording of those events any more accurate.

Imagine that there were no such thing as a single, double, triple, etc.  Or at least imagine that all you could see was where a batted ball was hit and how hard, but you couldn’t see whether it was fielded or not, or how many bases the hitter obtained if it wasn’t fielded.  And then you had to come up with an offensive evaluation system from that.

Would it be better or worse than our current ones, where you simply used whether it was a single, double, etc.? 

Assuming that you have a pretty reliable and accurate record of those batted balls, it would actually be better than a traditional one, I think.  Yet, you would have the same “problem” as you would with UZR and similar systems, in that depending who was recording the data and how it was being classified, you would have lots of variability in the final numbers until you had really large sample sizes.  And even with large sample sizes you would have considerable variability if one or both methods of recording the data had some biases (rather than just random differences), which would likely be the case.

It is simply a fallacy, and a common one at that, that because two systems might have considerable variability, that one or both systems is not very good.  That simply is not true.  It merely means that neither system is perfect, which we already knew, and that likely the combination of the two systems is better than one or the other.

Unless you work with this kind of data and the crunching thereof, you do not realize how sensitive the outcome is depending on the exact inputs.

Even if we used the exact same system of recording the information (say, STATS, or BIS), if we had 10 different persons recording the data, and they were all reasonably good at it, we would still have considerable variability in the outcome, again, until we had a real large sample of data, and even then we would still have some variability.  That is just the nature of the beast (because no one person or entity tells us exactly what the outcome of the event is, as with offense), and it does not - I repeat - it does not necessarily make the outcome any less accurate or informative than, say, for offensive events.

I tire of the argument that because BIS UZR or STATS UZR (or plus-minus, or some other PBP metric) has a particular player or players with vastly different defensive numbers, that it must mean that we can’t trust any numbers from any of the systems.  Those people that proffer that argument have no idea what they are talking about because they don’t understand where these numbers come from and what they mean.  As I said, that happens because of the qualitative nature of the data being recorded.  It does not happen with offense because the data and the methodology are necessarily better - it happens because with offense, all of the outcomes of the events are precisely categorized.  But - and here is the most important thing - those categories are not very good.  They are worse than the ones we have for defense.  What would you rather know in terms of a player’s offensive evaluation - that he hit a single or that he hit a scorching line drive or a lazy fly ball?  I think the latter even though we may not know the outcome of those two events (although the fist one was likely a hit and the last one was likely an out).  That is what we get with defense - data that is much more granular and valuable.  It is just that without knowing whether it was a single or an out, and because there is going to be some overlap between bloops and non-bloops and screamers and non-screamers there is going to be lots of variability in the outputs when we have different people recording the events.

I’ll concede that defensive metrics are overall a little less reliable and accurate than offensive ones, but the fact that different systems sometimes give you vastly different numbers, especially in the short term is NOT good evidence of that.  As I said, that would be the case for offense if it were not for the fact that someone tells us how to categorize each offensive event (and they don’t do a very good job of that at all).

MGL, you were asking recently on another thread how much measurement error there is in UZR. The more measurement error, the less we trust the stat for any particular sample size. The differences between sUZR and bUZR—which do persist over the long term, see Tango’s original post—indicate that there’s quite a lot of measurement error.

So this:

I tire of the argument that because BIS UZR or STATS UZR (or plus-minus, or some other PBP metric) has a particular player or players with vastly different defensive numbers, that it must mean that we can’t trust any numbers from any of the systems.

is obviously overstated, but it does mean we trust the numbers less than we would if BIS and STATS agreed more closely.

I agree with everything you say about how offensive stats have false precision. (Three True Outcomes excluded.) But we know that offensive stats aren’t systematically biased. Defensive stats, I don’t think we are there yet.

”...but it does mean we trust the numbers less than we would if BIS and STATS agreed more closely.”

I am not sure that is the case, but, as I said, I am not sure.  Again, to use the offensive analogy, the only reason that most offensive stats are in agreement is because they all use the same “data,” but that data is poor (but it gives us a false sense of security because we are used to characterizing offense in terms of singles, doubles, etc., and there is really no counterpart in defense, other than the error).

Let’s take that a little further.  Let’s say we have a series of offensive stats like wOBA, lwts, base runs, and EQA.  They will all pretty much agree.

Now let’s say that rather than using singles, doubles, etc. (there is probably not a whole lot we can do with the walks), we use fairly precise batted ball data.  I think we can all agree that using this more granular data is probably going to yield an OFFENSIVE stat which has more predictive value and is better at representing a player’s true talent, if “crunched” properly.

Now imagine that for all the stats mentioned above, we use that granular data, but for each stat, the data is recorded and collected by different persons, much like BIS and STATS.  Now we will definitely have a lot more discrepancy among the stats, simply because there is going to be a lot of variability among the data by virtue of the fact that the same data is being collected by different persons.

So we have two sets of stats.  In one set, we use singles, double, etc. and all the stats basically agree with one another because they are all using exactly the same “data.” In the other set, the stats sometimes do NOT agree with one another because they are using “different” (but more or less the same - as with the defensive stats) sets of data.  But because the data is so much better, all of these second group of stats are much better, even though they don’t agree with one another.

So, as you can see, agreement among the various metrics does NOT necessarily allow us to have more faith in them.  It depends upon why and how they agree or disagree.

I really think we can infer little about the integrity of the various stats based upon the variability among them, which was my principal point and one that I think is little understood among the general public and even by some analysts.

The two most important things with regards to the quality of the stat is the granularity (with respect to relevant aspects only of course) of the underlying data and the methodology used to “crunch” that data.  The advanced defensive metrics get a B+ with respect to those attributes, I think.  The accuracy of the data is fairly important also of course.  I say “fairly” rather than “very” because I am assuming some minimal level of accuracy.  And of course there is plenty of overlap as far as accuracy and granularity.  But really, I am not too concerned with the accuracy, especially if there isn’t too much bias.  What we need to get to the A or A+ level is more granularity and a little more accuracy, like speed/hang time, position of the fielders, and more precise parameters for location.

In any case, I tire of the attacks on any metric that takes a black or white position.  Tell me what you do or don’t like about a metric, but don’t tell me that it is “bad” or “good” at least in any serious discussion, although I suppose it is OK to use words like that at the extreme ends of the spectrum.  And again, without belaboring the point too much, to say that a general methodology is “bad” because two or more of them that use slightly different data bases sometimes don’t agree on the outcome or because a player can have vastly different outcomes in different time periods, is just plain wrong, and evinces a lack of understanding of hos these things work in general, so it is not a discussion that I am going to take seriously any more than a discussion explaining why a team is or is not winning based on team chemistry.

And BTW, even the 3 true outcomes have some “false” precision.  A HR next to the Pesky pole surely is not nearly the same as a HR to center field in Comerica Park.

I agree that those who make their conclusions regarding fielding metrics like “they’re not perfect” or “they still have a long ways to go” simply are saying that so that they can pretty much ignore them.

There are legitimate concerns in either the data collecting (see for example how Andruw Jones has alot of tough fielding plays with one stringer company but not another) and with the data processing (see for example how Carlos Beltran is considered by both version of UZR to have the same number of expected outs, has the same number of actual outs, and yet, he comes out with a huge difference in the two systems).  These are systematic or random biases that need to be understood and given a certain uncertainty level.

Fascinating discussion by some of the top minds in the field...what a treat!

MGL, nice post. I think we are pretty much on the same page.

So, as you can see, agreement among the various metrics does NOT necessarily allow us to have more faith in them.  It depends upon why and how they agree or disagree.

This is exactly right. In the particular case of BIS versus STATS, I was thinking that the main source of disagreement was inaccurate data. If the error is random then it’s not such a big problem. We just need a bigger sample size to draw conclusions with the same level of certainty. If there is systematic bias then we need to be very careful, especially if we don’t know how much there is.

The case of Andruw Jones told me that there is potentially a fair amount of systematic bias. But now, reading Tango’s comment about Beltran, I am even more confused. What is going on there?

(Re: false precision in HR, I guess I was looking at it from a value perspective. Then the Pesky pole shot is the same as the dead center Comerica HR except to the degree that the run environments are different. From a true talent perspective, of course there is a big difference.)

With the Beltran case, where the expected outs of both systems are the same, and yet the actual minus expected outs are far different, there can be two reasons:
1. bug in mgl’s program
2. mgl’s actual outs is different for the two systems

It may seem odd that Beltran has fewer actual outs in one system than the other, but if mgl is ignoring certain BIP (say line drives), and Beltran catches a ton of them that one system considers LD that the other doesn’t, then that answers that.

In order for his expected outs to match, the the system that is dropping the extra liners must have alot of tough non-liners to compensate.

It’s very convoluted, and if mgl has the ability to easily show us what is going on here, we’ll be able to see what is going on.

For OF’ers all batted balls are being used, however, I do ignore all pop flies shorter than a certain distance (I don’t remember off the top of my head - maybe something like 150 feet), so maybe that is the problem as some of those could be caught by OF’ers and there is probably a good deal of discrepancy between the two data sets as far as fly ball distance is concerned, especially on short flies where you don’t have the fence as a reference point.

I still don’t understand what Tango is talking about with Beltran.  If two data sets have the same expected outs and actual outs, they can still have very different UZRs, I think.  Maybe not. I’ll have to think about it.

I’ve said this before and I’ll say it again.  UZR is really sensitive to the extact parameters of the batted balls until you get a really large data set for each player.  If you take any two data sets, even if they are pretty darn similar, you are going to have lots of players who will differ in their UZRs for one or even more years.  That is just the way it is.  Again, that doesn’t by any means mean that it is a “bad” metric.  That is not even close to being true.  It is just the nature of the beast.

Imagine you have a line drive that in one system is recorded at 150 feet and in another 160 feet.  That really is not such a big difference and won’t matter at all in the long run.  But if you have only a couple dozen or so of those line drives, it is going to make a BIG difference in terms of UZR.  That is just one of many examples.

It is actually a GOOD thing if you had two systems like bUZR and sUZR.  Combining them is going to be better than using one or the other, assuming that one data set is not appreciable better than the other.  Basically it is like having two persons record the data and then combining those records, which is probably what each company should be doing anyway (having multiple people record the data and then combining the results).

I still don’t understand what Tango is talking about with Beltran.  If two data sets have the same expected outs and actual outs, they can still have very different UZRs, I think.  Maybe not. I’ll have to think about it.

At its most basic, all metrics (fielding or otherwise) is:
metric = actual performance minus expected performance

Beltran is +86 in STATS and +9 in BIS (2003-2008).  Since his “defensive games” are virtually identical with sUZR and bUZR, that would mean his “expected outs” would be almost identical.  Since his actual performance is identical (by definition), I would have expected that his UZR to be very similar in either system.

I don’t see how he could have a different UZR if you are not dropping any of his actual outs.

I guess the only other way would be that his “allowed hits” is considered to have been more extra base hits allowed than expected in one system than the other.  That is, maybe Beltran gives up alot of extrabase hits in zones where BIS does not have alot of extrabase hits allowed, but that STATS does think it does.

If you want, to make it easier to try to have a discussion on this using real data, I can look for a particular player/year that has the biggest gap in bUZR/sUZR, but has virtually identical “defensive games”.  This way, we can focus on one particular year and player.

Kind of along the lines of what Tango (#28) was saying…

MGL, do you use different run values for different types of fly balls, liners, etc?  For example, is a deep fly ball worth more runs than a shallow fly ball?

If so, it’s perfectly reasonable to get a different UZR even with the same number of expected and actual outs.  If the locations and types differ a little bit through the entire data set, the buckets could have different percentages and run values throughout.  Right?

If BIS called it a fliner and STATS called it a fly ball, you’re applying different percentages and run values, even if the outcome (caught or missed) is the same.

Posts #20 and #22 by MGL are very good.  A big revolution in baseball analysis is coming when people begin to understand that the highly granular data sets shouldn’t be analyzed in the same way as the traditional yes/no, black/white data sets baseball has had for the past century or more.

It reminds me of something I encounter in my day job, where engineers who deal with purely digital circuits don’t understand the analog behavior of the real transistors underlying those digital circuits and thus come to faulty conclusions.

The advanced fielding data, PITCHf/x, HITf/x, Trackman, etc., data sets are like the real analog systems underlying the simplified binary constructs by which baseball has been understood for so long.  Analog data is usable, understandable, valuable, but it has to be treated differently than digital/binary data.

I’m using analog to describe one aspect of the data.  Clearly, these new data sets are digital (and that has also sorts of powerful consequences for analysis), but in terms of having nearly continuously varying parameters, they behave much more like analog systems in comparison to the more digital/binary language of ball/strike and safe/out that we have been used to conversing in.

#29, yes the value of a batted ball is the average hit value plus out value of that kind of ball in that location.  So you are right, although in actuality, those hit values don’t vary all that much.  IOW, a line drive to a certain location is not going to have much of a different average hit value than a fly ball in that same location, or a fly ball hit to location X is not going to have a much different run value than a fly ball hit to location Y, near location X.

Bumping.

I’d still like to know from MGL how Beltran’s UZR can differ in sUZR and bUZR, if his expected outs in each system are nearly identical.

And, since Andruw Jones’ expected outs from STATS and BIS is as fantastically different as UZR is showing (via bUZR and sUZR), and the top list is replete with CF, then clearly we have a batted ball bias at least among CF.