THE BOOK--Playing The Percentages In Baseball

Tom, can you give me an idea of the resolution on these numbers?  Specifically, for each guy could you recalculate their UZR for this period as if they had made 1 more play, or one less play?  I’m trying to get a sense for the granularity of the number.

And if it’s not too much trouble, could you do the same for a single season worth of one (or each) guy - is it simply 3X the resolution for a 3-season number, or is is different somehow?

Yes, one more play would mean 0.8 runs.  One play per season would be 2.4 runs over the 3 years.

The difference between Pujols and Fielder is 48 runs, or about 60 plays over 3 years, or 20 plays per year.

That’s at 1B.

At SS, the gap is much wider.

I’m not sure I asked that in the right way to get the answer I’m looking for, so let me try again to make sure.

For a particular player (let’s say Giambi), to figure his UZR over the three years, you have one bunch of numbers for the league (plays made, hits in each of several zones), and another bunch of numbers for Giambi himself.  Based on how many plays Giambi actually made in those three years, in each of those several zones, compared to the league, you get to an overall UZR number for Giambi.

Here’s my question: what if Giambi, instead of having made X number of plays during those three years, had made X+1 plays? (and for a 1st baseman, what is a typical X for three years?  500?  more?  less?)

So, what if instead of 500 plays in three years, Giambi had made 501?  Would his UZR, as figured per the usual method, be -9.2, instead of the -10 listed above?

Oh, and if I can fit two questions into one post, can you tell me if UZR accounts for 1st basemen holding runners on in any way?  Seems like a 1st baseman who played for a team with bad pitching would be holding runners on more often, and thus have his range limited more frequently…

Yes, to the 0.8 difference.

And MGL does use the base/out state as one of his parameters.  That’s one of those “intangibles” that is quite tangible if Mike Silva would simply reach his hand out.

OK, let me run something by you then.

I ran a query recently, and found that in the first 4.5 months of the season, 1st basemen for the Yankees have made 214 outs on fielding plays.  That scales to 856 outs for a full three seasons.

If I assume that the outs are about 2/3 of the chances, then that makes a typical 3 years worth of chances for a 1st baseman about 1250 chances (rounding 1284 down a bit, makes sense since most players have days off, etc.)

Now, in the data above, Kevin Youkilis is +16 runs, and Jason Giambi is -10 runs.  That’s a delta of 26, or 33 plays if you use the 0.8 runs per play value.

So, on about 1250 chances, one player makes 33 more outs than the other.  Is that significant?

I ran a two-proportion test to see, and got this:

********************************
Test and CI for Two Proportions

Sample X N Sample p
1 856 1250 0.684800
2 823 1250 0.658400

Difference = p (1) - p (2)
Estimate for difference:  0.0264
95% CI for difference:  (-0.0104039, 0.0632039)
Test for difference = 0 (vs not = 0):  Z = 1.41 P-Value = 0.160

Fisher’s exact test: P-Value = 0.173
***********************************

This says that there is a 16-17% chance you could get such a result from random chance alone, or in other words, you can only be 83% sure that Youkilis was a better fielder than Jason Giambi over the last three years.  Or to generalize it, this says that a difference of 26 runs in UZR over three seasons is possibly an illusion.  Now I’ll grant you that it is five times more likely that Youk is better, but there is a 1 in 6 chance (that’s a roll of a die) that even this huge difference in UZR isn’t real.

Now suppose I look at Mark Teixeira 2008 vs. 2009.  To do this, I drop the number of chances by 2/3, and use his difference in runs (currently 11.4) 11.4 runs converts to 14 plays.  Here’s the analysis on a 14 play difference at first base over 1 season:

**************************************
Test and CI for Two Proportions

Sample X N Sample p
1 278 417 0.666667
2 264 417 0.633094

Difference = p (1) - p (2)
Estimate for difference:  0.0335731
95% CI for difference:  (-0.0311338, 0.0982801)
Test for difference = 0 (vs not = 0):  Z = 1.02 P-Value = 0.309

Fisher’s exact test: P-Value = 0.345
**********************************

The high p-value (30-35%) is telling me not to believe that Teixeira’s defense has changed, as his performance in 2009 is statistically indistinguinshable from his perfomance in 2008.

Now, could you or MGL tell me what, if anything, I’ve done wrong here?

Greg - Neither Youkilis nor Giambi played close to full time at 1B during 2006-2008.  Youkilis averaged about 115 full games for those 3 years and Giambi abouot 55. Consequently, Youkilis had only about 672 total chances and Giambi about 322.

Peter,

Thanks, I will certainly go to the actual numbers once I’m sure I have the whole thing straight in my head.  Can you comment on whether I’ve gotten anything conceptually wrong in my example here, i.e. if those numbers were the correct ones, does the rest of my post hang together?

Greg - And for your purposes you have to consider that the numbers I gave you are defining chances in the broadest possible sense, i.e. every ground ball that reaches the first baseman in his approximately 28 degrees of responsibility.  About 20% of these hit balls would be impossible for any human being to field for an out and therefore should probably not be included in your analysis. Likewise, there are many hit balls that would be fielded for an out 99% of the time by any major league first baseman and thus provide almost no information to you about the differences in fielding ability.  The actual number of N that differentiates a good defensive first baseman from a bad one is quite small.

I’m actually not trying to do any analysis of my own here, I’m just trying to understand the “volatility” of the UZR metric. 

It’s really the same thing as figuring out whether a scale has enough granularity to accurately weigh something: for a “bathroom” scale, a readout in whole numbers of pounds is fine, while for measuring out ingredients in a pharmaceutical pill, a readout in hundredths of a gram would be too crude to use.

In any zone-based defensive metric system, the basic currency, the smallest unit of measure, is the play.  Any changes in UZR therefore can only occur in increments of one play.  That’s why I’m trying to figure out how much UZR changes when one play (out of however many are in the sample) turns out differently.  Tango is telling me it’s 0.8 runs.

The stuff I included in previous posts was meant to illustrate that with a given number of chances, and a given rate of turning those chances into outs, there will be some variation (based on the binomial distribution) around that rate which you should expect, and therefore there is a certain minimum difference in UZR scores that we ought to demand before concluding there is any certainty of a true difference.  What I’m trying to satisfy for my own brain is what that minimum difference is, for various sample sizes.  This would then help me decide whether to put any stock in any particular UZR number for a player in a given period, or whether to regard the number as an approximation (and how much of one, of course).

Now, I’m sure Tango and MGL have explained this all before many times, and I don’t want to make them do it still another time, so I am trying to just assemble my thoughts and then just let them either ratify my thinking, or correct me where I’ve goofed…

Just to add Dewan’s Runs Saved numbers from 2007 through 2009 to date (Bill James Online):

Pujols 50
Kotchman 34
Youkilis 19
Overbay 19
Teixeira 17
.
.
.
Giambi -16
Sexson -20
Young -22
Fielder -22
Jacobs -27

The scale is a little more extreme (we’ve been through that, though), but it looks like a general agreement.  As you might expect/hope.

I’m not sure that the binomial distribution will necessarily work here, like it would for OBP.

What I can tell you is that the correlation of UZR of 400 BIP of two datasets is similar to the correlation of OBP of 200 PA.

So, if you are trying to create a model, and a “rate stat” for UZR, you would want it to adhere to those constraints.

Tom,

Maybe you don’t use the binomial distribution to calculate UZR, but it is definitely lurking in the background of this situation, because of the model UZR chooses to represent defense.

UZR counts opportunities in zones, and classifies the outcomes of those opportunities as either plays made or plays not made.  The number of opportunities is not fixed.  A difference between two data sets is calculated - with the two sets being league average, and the player in question.

Aside from a few after-the-fact adjustments to the data, the data types and situation are textbook binomial, and the two proportion test is the exact right tool to evaluate the significance of the calculated difference between the two data sets.

From Minitab Help:
********************
2 proportion test
A hypothesis test for two population proportions to determine whether they are significantly different. This procedure uses the null hypothesis that the difference between two population proportions is equal to a hypothesized value (H0: p1 - p2 = P0), and tests it against an alternative hypothesis, which can be either left-tailed (p1 - p2 < P0), right-tailed (p1 - p2 > P0), or two-tailed (p1 - p2 ≠ P0).

For example, suppose Candidate A is running in a local election.  You want to determine whether a person’s gender affects their vote, so you take two samples – one of men and one of women – and observe that 55% of men and 45% of women support Candidate A. You can use the 2 proportion test to determine whether this observed difference is statistically significant.

********************

So, anyway, is the raw data for plays in zones available anywhere for free, or is this the data that BIS sells?

Greg, the best source of freely-available zone data is the Project Scoresheet data that is available from Retrosheet from 1989-1999 or so. The zones are a bit larger than the STATS zones (which are in turn larger than the BIS zones), but it works for research purposes, I’ve found.

I agree that UZR (or UZR rate) is essentially a binomial - really it’s a set of several binomials, as MGL (IIRC) computes a UZR rate per zone and then figures +/- per zone and then adds everything up.

Let’s think of UZR like OBP for a second. If Albert Pujols goes a stretch where he puts up a .310 OBP, we obviously think that he’ll do better in the future, but nobody thinks that he really got on base at a .280 or a .350 clip.

UZR isn’t like that. We have a very good measure of plays made, but are simply estimating chances based upon the zone data. It’s as if we knew a players hits and walks for OBP, but didn’t know the number of PAs, just games or games started or something like that.

(What makes it more difficult is that we’re weighting chances based upon difficulty, so there’s a sort of cascading uncertainty going on with our estimates of chances.)

I think the more interesting question is how uncertain our UZR estimates are of actual performance, not actual talent.

"UZR counts opportunities in zones, and classifies the outcomes of those opportunities as either plays made or plays not made. “

This is only half the equation.  It is then compared against what an average player would have done.  It would look something like this:

playsMade,averagePlayer
1,0.72
1,0.90
1,0.44
0,0.05
0,0.20
-- ----
3, 2.31

That makes this player as +0.69 per 5 BIP.

And even that is a simplification.

Indeed, you could even consider all 4000 BIP on the field as an “opportunity”, as I do, and as Pinto does.  It just so happens that most of those BIP are not fielded by a particular position, so you’d have a huge number of:
0, 0.00

All that is pure noise.

Nonetheless, the fact is that the correlation is roughly r=.50, when “balls in zone” is around 400.  Whatever binomial model you want to create has to adhere to this fact.

Tom, thanks for the reply.  Last question for a while - I’m not sure I understand the correlation part: 0.50 is the correlation of what to what, exactly?

Take all player with 300 to 500 “BIP” in back-to-back years.  Their UZR will correlate at r=.50

You will get a similar r=.50 if you take each player’s first 200-250 PA this year and last year, and compare their OBP or wOBA.

I define a BIP as the following per 9 IP:
5 SS, 2B
4 CF, 3B
3 LF, RF
2 1B

So, if you have someone with 900 innings at SS, he probably has around 500 BIP.  Just my little way of doing things.

OK, thanks Tom, I think that gets me where I need to go…