THE BOOK--Playing The Percentages In Baseball

Well, JC, rather than refuting my arguments chose to simply refute me!  This is my last post there:

=======================================
My post #9 clearly shows that it makes a huge difference for the extreme GB and FB hitters, if you use GB/FB or FB/GB.  However, there is no change whatsoever if you use GB/(GB+FB) or FB/(GB+FB).

There is no justification for using one ratio (GB/FB) over the other (FB/GB), even though they absolutely give you different results.  In fact, you are not even justifying it.  Just deciding to use it.

There is zero opportunity cost to changing from ratios to rates, since I was able to generate 3 different regression equations in 5 minutes.

Your thread started with a comment about fans being skeptical, and here I am, giving you a thoughtful and legitimate beef, and you are dismissing it with “if you don’t like it, don’t use it”.

======================================

Hey, I know I can be tough, but when I’m right, I’m right.  JC is wrong, Clay is wrong (about EqA), Woolner is wrong (about Leverage Index), Forman is wrong about using all those James/Palmer metrics.  Heck, I was wrong about BaseRuns, until David finally convinced me to open my eyes.

Either you open your eyes, put your ego to the side, and learn, or remain wrong.  My attitude is completely irrelevant.

You make a good point about the use of a ratio.  But that seems far down the list of what’s wrong with PrOPS.  First, the premise of PrOPS is that departures from the mean hit rate on each batted ball type are luck.  So if you control for BIP type, you have a ‘true’ measure of how well a player hit.  However, we know this isn’t correct—hitters vary greatly in terms of their BABIP and SLGBIP on a given type of BIP.  So when PrOPS says a player was lucky, it may not be true at all.  This is really an unfortunate over-extension of the DIPS concept.

Now, it’s true that players with a high OPS-PrOPS will regress the following year.  JC reports this as evidence that PrOPS ‘works.’ But these players are also high OPS players—especially high BA players—who will regress in any case.  JC doesn’t examine whether the regression of his PrOPS overperformers is any more than you’d get from any similar group of high BA/high OPS hitters, so we really don’t know if it’s giving us any new information.  My guess is it’s pretty marginal.

PrOPS is also a great example of using regression for no reason at all.  JC creates separate regression models for BA, OBP, and ISO—using regression to ‘predict’ the impact of HRs on SLG, and the impact of BBs on OBP!—and adds the three together.  What’s the point of this?  If you want to regress by 100% a player’s BA and SLG for each batted ball type, then it would be both easier and more accurate to just attribute the mean performance and create a new batting line for each hitter.  PrOPS uses actual BB and HRs, so all you need to do is calculate the number of expected 1b, 2b, and 3b for the # of GBs, OFs, and LD, and you’ve got everything you need.  (Of course, it would make more sense to regress less than 100%, based on PAs.)

In the end, it’s a whole lot of work for a tiny boost in R2 from .43 (OPS) to .46 (PrOPS), one that simple regression to the mean would probably match or exceed.  I can see why JC isnt’ interested in reworking the model…

I consider JC a friend, and I’m always interested in his work.  But I find that he’s usually not interested in discussion of his work on his blog, except within certain parameters of his choosing.  That’s fine, of course.  It’s his blog.  I think he expressed his attitude well in the salary thread when he said (and I’m paraphrasing), “If owners aren’t maximizing their MRP, then I’m not really interested.” It sort of cut the conversation short.

And, I want to add, I wish GB/FB ratios would disappear entirely, starting tomorrow!

I think Tango is quite right, for exactly the reasons he says.

Another way to phrase the problem with using an A/B or an (A+B)/A instead of an A/(A+B) is that, the first way, if B is small, you can get an infinite ratio (say, 5 GB and 0 FB).  That can screw up a regression pretty good.

Even if it’s not zero, you can get some pretty high numbers that don’t deserve to be high.  For instance, 1 HR in 500 AB is 1/500 using A/(A+B), but +500 using (A+B)/A.  If Jose Oquendo hits 1 HR in 500 AB, but Barry Bonds hits 50 in 500, Jose has an AB/HR of 500, while Barry has 10. 

Do you really want to say that Jose is 50 times the non-power-hitter that Barry is?  Even if you did, that would screw up a regression too, where Jose gets a “500” and Barry gets a “10” and A-Rod gets a “15” and Ozzie Smith gets a “250” and so on.  The regression would try to figure out the value of Jose’s 250 extra “somethings” over Ozzie, which would have to be 50 times as high as A-Rod’s 5 extra “somethings” over Barry.

Another benefit of using A/(A+B) is that if you make that your talent, the distribution of performance is normal.  That’s not true for the other formulations.

Phil: I’m not sure your example is really parallel.  You’ve basically reversed the numerator and denominator, looking at AB/HR rather than HR/AB. The problem with GB/FB ratio is its non-linearity.  Say you have players with these three GB/FB rates:  30/50, 40/40, 50/30.
The GB/FB ratios are:  .6, 1, and 1.67.
The GB/(GB+FB) rates are: .375, .5, and .625.

No matter what coefficient your regression model comes up with, if you use ratios you will be attributing much less value to the 10% gain moving from 30% to 40% than to a 10% gain going from 40% to 50%.  If you believe that each additional 1% gain in GB should add (or subtract) some fixed value to OPS—which is the premise of the model—then you have a problem.

That said, this does seem like a relatively trivial problem relative to everything else wrong with PrOPS.....

Hi, Guy,

Agreed ... but the particular aspect I’m talking about there affects both cases, one where you do A/B, and one where you do (A+B)/B.  It’s just easier to see if I explain it using the AB/HR.

The important point is that your denominator must be *opportunities*.  To use anything else is wrong, unless your intent is to intentionally skew the results because you found a pattern to that effect.  That is, you do not want a linear relationship. 

However, no one has proven that GB/FB is more reliable an indicator than FB/GB.  That is, if you choose to not use opportunities in the denominator because you have found the relationship to not be linear, then you have to show it.

Is the .46 Props/.43 OPS an r or an r^2?

If that’s and R^2, r=.68 as Tango posted on JC’s site, and despite its flaws its right up there with the best projection systems.  I’m kind of doubting that, as one year, nonregressed OPS = .42 would be as good as using MARCEL.

My problems with PROPS:

1. The name.  Call it underlying OPS or true talent OPS if you will, but the name as is implies that this is how a player will do in the future. 

2. If you want to use it for projections, add an age factor in.  Am I supposed to believe that a 22 year old and a 38 year old having the same PROPS in 2006 are equal bets to play well in 2007?

3. We have multiyear data for batted balls now.  I can’t take any projection seriously that ignores anything other than the most recent year.

The correlation is of course highly dependent on the number of trials (number of PA).  And the uncertainty level around that correlation is dependent on the sample size (number of players). 

In my opinion, Marcel should be used as the baseline, so that if Pecota were to claim an r of .71 and Shandler were to claim an r of .69, it would be nice to know that Marcel’s r in the first case was .73 and in the second case it was .67.

Rally:  JC did make projections based on PrOPS, using multi-year data and age.  The 2006 projections are here:  http://www.hardballtimes.com/main/stats2005/propsproject

The methodology is here:  http://www.hardballtimes.com/main/article/props-2005-and-beyond/

Let us know how the preditions stack up.

Interesting.  I hadn’t seen that, so I must retract most of my criticism.

Sometime tonight I’ll let you know how it stacks up.

I had to remove 5 players from my sample that Props did not project: Fielder, Cedeno, Willingham, Zimmerman, and Youkilis.

Just 5 players from the 114 sample shows how volatile this is.  The R’s jump all over the place.

Here’s the new standings:

Pecota .733
Shandler .702
James .699
CBS .699
Chone .693
Zips .685
Marcel .678

And Props?  .666

The sample size and differences are too small to call this significant, but at the same time, there’s no reason at all to see this as any kind of breakthrough in predicting hitters.

Can you find players who are lucky/unlucky and unlikely to repeat by comparing season totals to Props?  Yes, but you can do the same thing and better by comparing their season totals to their Marcels.

Out of curiosity, what’s the r for OPS?

And is the actual formula posted somewhere?  I’ve seen explanations on J.C.’s posts, but not the full method.

By the way, and in fairness to J.C.: while I think using FB/GB ratio is a bad idea, it might not affect prOPS all that much.  As Tango points out, even the most extreme players are off by only a few (10-15) points either way.  Of course, if there were *really* extreme players, like 9:1 or something, then you’ve got a much bigger problem.

So I don’t think there’s any need to recall J.C.’s book from the publishers, or anything.

I think the 2006 THT Annual has the most complete description of the method. Also look at the article linked above....

The most extreme players are a 10-12 point GPA difference, which would be the equivalent of 30 OPS points.  That’s 20 walks over a full season, or 10 singles.

And of course most players aren’t in the extreme camps, but there is still a systematic bias.  I have 231 players in my sample, and the top 20% were estimated at .282 using GB ratio, .287 using FB ratio, and .286 using rates.  And in the bottom 20% the corresponding numbers are .267, .268, .266.  Since there is no reason to justify GB ratio over FB ratio (unless the intent is to introduce such a bias), why not remove the problem?

"why not remove the problem?”

Because it’s not worth even a little effort.  As Rally tells us, PrOPS is not a step forward in player projection.  Over at Sabernomics JCB said to Tango:  “This is why it’s necessary for someone to take the next step and make PrOPS obsolete.” Well, it turns out those steps have already been taken, by many analysts, before PrOPS was even created. 

PrOPS poor predictive performance is not surprising.  It essentially tries to regress factors in the following way:
LD% 0%
GB/FB 0%
Ks 0%
HRs 0%
BBs 0%
BA-LD 100%
SLG-LD 100%
BA-FB 100%
SLG-FB 100%
BA-GB 100%
SLG-GB 100%

Now, one thing we know is that every number here is wrong—none of these performances are pure luck or pure skill, so the correct regression will be somewhere between 0 and 100 for each element.  Then you have the possibility of additional errors introduced by using multiple regression to guage the value of BBs, HRs, Ks, which can be determined empirically.

Batted-ball data may eventually help improve projections on the margin.  I don’t think we know that yet.  But PrOPS clearly doesn’t take us there.

I don’t know if I’ll have the time but I’d like to see if I can work batted ball data into projections.  It obviously won’t change W, HBP, or K totals at all.  It might have some effect on HR, if you use FB% somewhere in your formula.

Its biggest effect would be on BABIP.  Maybe it can be used to get more accurate future BABIP than using the last 2-4 years data and regressing.

Maybe not.  We may need to wait for the next step in batted ball data.  It would be nice to have average speed off the bat for line drives, for example.  I know Chone Figgins and Vlad Guerrero hit line drives at about the same rate.  Something tells me a Guerrero line drive is worth more.

"Something tells me a Guerrero line drive is worth more.”

I think the current data can give you that.  Studes has done it at this site:  http://www.baseballgraphs.com/battedballs/index.html

Your intuition is right:  Chone’s LDs are worth .33 runs, Vlad’s are worth .40.  Hopefully, Studes will add the 2006 data at some point.

Why is J.C. trying to stifle discussion about the prOPS methodology in an article titled “Reviewing prOPS”?

Guy, I don’t plan to update that library.  The 2006 data is just in the Annual, for now.  However, I’ve started talking with Bryan Donovan about posting those stats on THT during the year next year, to include previous years.

I know I’m prejudiced, but I think they’re fascinating.  For some players, I’d rather see their batted ball line than their regular batting line.

First, the premise of PrOPS is that departures from the mean hit rate on each batted ball type are luck.  So if you control for BIP type, you have a ‘true’ measure of how well a player hit.  However, we know this isn’t correct—hitters vary greatly in terms of their BABIP and SLGBIP on a given type of BIP

Sorry for dredging up this thread again but I have a question about the above, something Guy said about PrOPS.

Is this true? I thought that the dependent variables for PrOPS were LD% etc so players with a higher LD% would have a higher PrOPS. As I read it PrOPS assumes that each hitter has the same ability to translate a batted ball into a base (be it a single, double ...).

PrOPS has many problems, I’m not convinced that this is necessarily one of them

It is absolutely true.  Batters have significantly different outcomes on batted ball types.  Over the last few years, Chone Figgins has hit more line drives than Vlad Guerrero, however anyone who watches Angels games knows there is quite a difference between a Vlad liner and a Chone liner.

Props is not a bad idea for a start - I might try to incorporate batted ball type into my hitter projections next year - but as is it does not project future performance as well as Marcel does.

John: There are three regression models in PrOPS, that predict Avg, ISO and OBP. The THT stats page shows this for all players.  Add the three, and you have PrOPS.  (However, since BB-rate is itself one of the independent variables, once you have PrAVG you also have PrOBP, so I have no idea why you’d need a 3rd regression for OBP.  And using HR-rate to “predict” ISO is kind of silly.  But whatever....)

LD% and GB/FB are independent variables in some or all of these models.  So a LD% of X, or GB/FB ratio of Y, is indeed assumed to have the same value for all hitters.  However, because HR-rate is itself an independent variable, assuming a constant OPS impact for a given GB/FB ratio shouldn’t lead to huge errors (other than the rate-ratio problem Tango raises). 

A hitter who over-/under-performs his PrOPS is basically a guy with a very high or low SLG-BIP.  That they regress the following year is not surprising.  What we don’t know (yet) is whether the batted-ball info adds a little predictive power on the margins.

Guy,

Thanks—that is what I thought. The issue is that, as Rally says, PrOPS assume that a Vlad liner is worth the same as a Figgins liner.

By the way, completely agree on the PrOBP thing. JC’s equation uses HBP and BB as inputs so you can actually work out what PrOBP should be using (PrAVG*AB+BB+HBP)/(AB+BB+HBP) ... which doesn’t match. Crazy.

The right approach here would have been to work out the value of a 1B, 2B, 3B in terms of batted ball types and use that to re-engineer the batting line.

One other thought on PrOPS is that not only would we expect players who have a naturally high slg-BIP to look lucky, hitters who have more speed and are ground ball hitters are more likely to outperform their PrOPS.

If I have some time I might run a quick analysis later ...

"The right approach here would have been to work out the value of a 1B, 2B, 3B in terms of batted ball types and use that to re-engineer the batting line.”

John, you should know that you can’t do valid analysis without using regression.

More seriously, I agree.  But in fairness to JCB, he would argue the regressions may pick up predictive relationships beyond the direct impact of an additional BB, HR, etc.  (e.g. if hitters who draw lots of walks tend to hit harder LDs, or whatever).  But there’s no evidence this is true, and to the extent you pick up anything it seems likely to be offset by the inaccuracy introduced by using regression to estimate empirically-known relationships.

* *

On speed: it’s clear that fast players tend to overperform.  Just look at the list of top 25 over/under-performers in 2006 THT annual: there are 6 catchers among under-performers, none on over-performer list.  JCB claims there is little or no correlation btwn over/under and speed measures.  He also claims there is no y-t-y correlation for PrOPS itself.  I suspect both of these findings result from using too low a PA cutoff, which drives down r.  If you look only at hitters with something like 400 PAs, I think you’ll find a clear speed connection.  And with 400 PAs in consecutive years, you’ll find over/under has a y-t-y correlation, which would mean that PrOPS doesn’t just capture luck.

Did JC ever publish the full regression analysis? I’d be quite intrigued to see the std error on the LD% coefficients in particular. That in itself (small sample size) could be a cause of potential uncertainty