THE BOOK--Playing The Percentages In Baseball

"If we focus on the 25th and 75th percentiles, PECOTA tells us that it’s 50% sure that Wainright’s K minus BB per 9 IP will be between 4.53 and 3.97, or a gap of 0.56.”

The percentile forecast isn’t supposed to be used in that fashion.

“The percentile forecast is designed to work for the Key Statistic (EqR/27 and EqERA) only. If a player’s 90th percentile forecast for home runs is 42, this should not be read to mean that he has a 10% chance of hitting 42 home runs (or more). Rather, it means that he has a 10% chance of having a season as valuable as the line represented by the 10th percentile forecast, whether this comes from the particular combination of peripheral statistics listed in the percentile line, or an equally valuable (but different) combination of statistics.”
http://tinyurl.com/2trvhu

The BETA score is more akin to MARCEL’s “reliability”. 

BETA is “[A] measure of the relative volatility of a player’s EqA or EqERA forecast, as determined from his comparables.  The Beta for an average major league player is 1.00; players with Beta’s higher than 1.00 have more volatile forecast than others ("riskier"), while Betas lower than 1.00 represent less volatile forecasts ("less risky")."
http://tinyurl.com/36q6vx

Waiwright’s BETA is .82, Carpenter’s 1.09.  Johan ‘s BETA is 1.00, while Halladay is at .82

Wouldn’t PECOTA’s somewhat unreliable forecast of Carpenter be due to his relatively strange career path, and list of comps ranging from guys like Clemens to Erik Hanson.

On a side note, it’s also interesting to look at Carpenters BABIP’s when in Toronto(usually well over .300) compared to St. Louis(about .284 so far).

Anyway, good stuff. Keep up the great work.

The percentile forecast isn’t supposed to be used in that fashion.

Yes, it is.  A peripheral (or component) ERA is made up of ... components.  And the two most important ones are K and BB, which is why I chose them.

In any case, the gap in peripheral ERA still stands.

***

As for the Beta, there’s no possible reason that Wainright’s Beta would be lower (less risky forecast) than Santana’s.  You just proved my point.

***

Carpenter’s last 3 years represents the bulk, if not the totality, of his performance data.  How he got to his Cardinals days seems to be irrelevant to Pecota (and Marcel, and most other forecasts).

In any case, you can go up and down the line for any pitcher or any team, and you will find that the peripheral ERA at the 75th percentile will be around 70% of his peripheral ERA at the 25th percentile.  (Picking Cards pitchers at random: Izzy is 68%, Looper 78%, Mulder 69%, Reyes 74%, Wells 70%)

If that’s the case, then what’s the point?

***

Check out this player with zero MLB experience:
http://www.baseballprospectus.com/pecota/GARCIA19860708A.php

The 25 to 75 percentile peripheral ERA is between 4.20 and 4.94! 

How is it even possible that the certainty level for this pitcher is so much stronger than for any other Cardinal pitcher?

This is just somewhat of a guess, but look at the difference in BABIP with Carpenter between the 25 and 75 percentiles. It goes from .273 to .299. That seems to be a lot wider gap than pretty much all of the other guys I’ve looked at. Take a guy like Santana(.272 to .282) or Oswalt(.286 to .300). Perhaps, PECOTA doesn’t think he can keep that kind of BABIP because of his former struggles in Toronto, or something else? It seems that may be causing the wider variance in expected performance?

I have the best projection for Wainright only because I made the decision to project him as a reliever.  Some of the other projections - certianly PECOTA - may be looking at him as a starter.  PECOTA has him at 151 innings.

I agree with your reservations about the percentiles.  I don’t think there’s any way we can have as reliable a forecast for a 19 year old who’s never pitched above A as we can for a major league veteran.

If you read the blurb from Jerome in #1, the percentile is based SOLELY in the EqERA (i.e., peripheral ERA).  PECOTA is not doing percentiles for each of the components.

In any case, it seems the BABIP is based more on the team’s fielders, since almost all the Cards pitchers shows the huge split you are describing.... except of course for Garcia, he of no MLB experience.

We should make a fine distinction between the mean forecasts, which are good, from the percentile forecasts, which are unproven, and therefore, unreliable.

Timely article on forecasters:

http://sports.espn.go.com/espn/page2/story?page=keri/070214

I’m probably not going to get involved in this debate at length, but a couple of points to note:

1. Wainwright is an odd data point to use because we’re having the system perform a relief-to-starter conversion on him.  We don’t try and look at other pitchers that made the relief to starting jump, but instead project Wainright as a reliever and then basically apply a linear multiple to his forecast based on his peripheral statistics.

2. EqERA is not the same as PERA.  Rather, it’s ERA normalized for league and park effects.  PECOTA can and does account for systematic differences between ERA and component statistics.

3. The Beta scores perform intuitively far more often than not.  Of the 15 pitchers with the highest Betas, 12 are pitchers who have never pitched in the majors.  The others are Randy Johnson, Jon Lieber, who is old and has a BB rate that is fairly historically unprecedented, and Jesse Foppert. 

4. I think people are overestimating the amount of information that a well-designed forecasting system needs.  Numbers like strikeout rate, walk rate, and groundball rate stabilize very fast.  If you plot information on one axis and forecast accuracy on the other, the curve is very steep at first but becomes flat sooner than you’d think.

5. PECOTA looks at usage data as well as rate performance when selecting its comparables, including major league career length.  Many of Wainwright’s comparables, for example, were pitchers entering their second or third full big league season.

6. Very good pitchers like Johan Santana and Chris Caprentner are inherently more unusual than garden variety pitchers like Adam Wainwright.

7. When we’ve actually tested the percentile ranges for PECOTA, we’ve found that they’ve done a very good job by and large (though better for hitters than for pitchers).

Nate, thanks for stopping by. 

I think your #5 is important.  Wainright is rather old to start being such a good pitcher in the majors (he’ll be 26 in August).  So, it’s a given that his comps will be guys with multiple years of MLB experience behind him. 

However, that doesn’t mean that his forecast range should be similar to everyone else.  The uncertainty level around a forecast is, predominantly, based on how large your sample is.  And Wainright, regardless if he scored comparable to lots of guys or few of them, still only has 318 TBF.

However, if you include minor league data, where a guy gets to add tons of TBF prior to joining the bigs, and therefore, gets a larger sample, this the uncertainty will go down.

At the same time, any minor league data behind used will itself have greater uncertainty than MLB data.

***

In your testing of the PECOTA ranges, did you test between rookies and sophs, compared to the rest?  I’d love to see whatever testing you have done.

behind used = being used

Another test would be to look at all those guys with a narrow peripheral ERA range (the Garcia) and and look at all those guys with the widest peripheral ERA range. 

And when you look at the actual, did the peripheral ERA of the guys in the narrow range actually produce ERAs much narrower than the guys in the wide range?

***

Your statement about comparing hitters and pitchers seems to be curious.  If you test for 25th to 75th percentile, you should expect to get 50% within that range, regardless if you are a hitter or pitcher.  If you don’t, you adjust the percentile levels to ensure that.

In short, hitters or pitchers, experienced or inexperienced, wild thrower, or contact hitter, speedsters, or power pitchers, all should fall within the percentile ranges, at the group level.  That’s the real test.

Nate,

Have you tested percentiles based on comparable players versus percentiles based on a player’s component statistics (you could choose many different methods here, from a simple binomial random error, to some more complex regression-based analysis)? Intuitively, it doesn’t seem clear to me that the comparable players would do a better job, given the small sample size that’s inherent in your comparable samples, that’s exacerbated when you divide it into smaller groups for percentiles.

"The uncertainty level around a forecast is, predominantly, based on how large your sample is.”

See, this is where I disagree.  Or at least I potentially disagree.  There are a whole host of factors that can impact variability above and beyond sample size: for example a player’s age, what type of player he is, his age, and the robustness of his comparbles set.  In fact, once you get past a certain minimum level of sample size, I suspect that these considerations tend to be more important than the sample size itself (especially if PECOTA is also able to model the sample size in its comparable selection to a certain extent, which it does).  I don’t have any real problem with Adam Wainwright’s sample size, for example, as long as you’re able to account for his minor league numbers in a reasonable way.  The bigger problem with Wainwright is unrelated to sample size, which is his starter-to-reliever-to-starter conversion.

When we looked at the percentiles, this was focused on MLB data.  The hitter percentiles were almost perfect.  The pitcher percentiles underestimated the number of extreme performances at the 10th and 90th percentiles by about 5%, but this had to do with the way that PECOTA was “mapping” the comparable performance and should since have been corrected.  I don’t know whether they held up for any given subgroup of pitchers or hitters; we didn’t get that far.  But there definitely are systematic differences in the empirical data.  For example, power hitters have larger forecast ranges than contact hitters, and players with speed have less variability.

Good stuff.

I think we probably disagree.  The sample size is the largest determinant, but you also have age, since, as we know, the slope of the aging curve is steepest as a player is young (pre 25), and is steep the other way when he is old (post 35).  So, the really young and old players have an extra adjustment that expands on our uncertainty.  But, nothing, I don’t think, to match the uncertainty of the sample size.

As for accounting for the size of minor league performance, this would be interesting to determine.  For example, if you have 300 PA in MLB and in minor leagues, they can’t possibly both provide the same level of uncertainty.  The minors have the extra adjustment factor (quality level is different, plus an additional translation to go from one league to another, and an age adjustment if you go beyond same-year).  So, these adjustment factors adds to our uncertainty level.  How much, I don’t know.  Maybe someone else can tackle this as a project.

I suspect that 300 PA in MLB will be equivalent to 450 or 500 PA in minors, when accounting for the uncertainty level.

Fun stuff for sure.

***

Can you explain the narrow band for such a young and inexperienced pitcher like Garcia?

FYI:  to clarify for those who haven’t bought the book, the pitching projections in the Bill James Handbook are not Bill James, per se, though are in his book.  It notes in the section’s introduction that Bill James does not believe that pitching numbers can be projected and so he had nothing to do with the pitching projections.  They are the “brainchild of John Dewan” with help from others at BIS.

Nate Silver responds in a chat, of likely some reader of my blog posting:

Marcel (The Zoo): No one understands or uses the percentiles in Pecota. Why do you keep them?

Nate Silver: Marcel,

A lot of us do use the percentile forecasts—they’re useful for example in describing just how unusual a certain performance might have been. People thought that A-Rod had a really terrible season last year, for example, but his performance was right in line with his 25th percentile forecast, hardly anything too out of line.

FWIW, when we looked at the percentile distributions a couple of years back, we found that they did a nearly perfect job for hitters. That is, a player would exceed his 90th percentile forecast about 10 percent of the time, his 75th percentile forecast about 25 percent of the time, and so forth. Pitchers did nearly as well, except that guys were exceeding their 10th and 90th percentiles a little too often—e.g. there were a few more extreme performances than PECOTA was giving credit for. I think I’ve found the reason why this was the case and it’s since been corrected.

Nate is correct that it does describe how unusual something is.  However, I disagree about the need to have an individualized range, since I believe that range is predicated on the amount of historical data far, far more than on the profile of the player.  This is the “agree to disagree” until someone--him, me, or someone else--comes along to set the other straight. 

Alf (Cambridge, MA): Will you be comparing how Pecota fared against other projection systems again?

Nate Silver: Basically, every study I’ve seen, whether its ours or someone else’s, has shown PECOTA at the top of the heap. But I’m less interested in looking at this stuff than you might think. The differences in forecast accuracy for major league players is going to be relatively small, provided that a system is able to do forecasts with some minimum level of competency. From a research standpoint, I’m more interested in things like the prospect forecasts and the five-year forecasts, where there’s much more room for marginal gains, as well as the valuation elements of PECOTA.

I agree with Nate’s position here.  The real test is the guys on the periphery.  For example, Marcel has Pujols at .331/.424/.635, while ZIPS has him as .320/.423/.643.  Regardless of how Pujols does, neither Marcel nor ZIPS will lay a claim on having better forecast his season.

A great test would be Hanley Ramirez: .298/.361/.482 for Marcel v .277/.338/.439.  Marcel intentionally ignores minor league stats, because he’s lazy.  ZIPS, Pecota, Chone, Bill James use minor league data.  Each uses them in different ways.  That’s the fun part.

Since Nate was kind enough to participate in a project I’ll be studying (which will include the above named forecasts, plus Pete Palmer, and MGL), we’ll see just how each system treats the young and inconsistent ballplayers.

Part of Nate’s position on caring though belies this:

COMING SOON Complete depth charts and forecasts for AL and NL pitchers and hitters using Baseball Prospectus’ deadly-accurate PECOTA projection system--

If someone is making such a grandiouse claim, it would behoove them to back it up every year.  So, either you care and you need to back it up, or you don’t care and don’t need to back it up.  Either way works for me.

Some other factors that I have never seen an attempt to be quantified into projections , but theoretically could be:

1. Quality of Coaching - Is there a Duncan factor?
2. Situational Stress - Are you the 7th batter for a crappy team or the third place hitter for a low scoring playoff contender?
3. Defense to Hitting Carryover- Error rate and K rate: to what degree do they go hand-in-hand?
4. Weather- Some like it hot
5. Synergy - Playing in front of a good defense, should not only decrease your BABIP allowed, but decrease your walk rate and consequently increase your innings pitched.

I have to disagree that sample size is the key to ‘reliability’, because sample size in actuality is always 0. Since you have no players with the exact same statistical record to make comparisons, you’re already fudging your sample size by relaxing the standards.  If sample size was the most important component, just make your sample every pitcher who ever played.  At that point, you may as well just use 3-year averages.

You don’t need “an exact”.  Your sample size is your sample size.  Even if you have 10 guys who were “exactly” like Chris Carpenter, and none who were like Roy Oswalt, your sample size is your sample size.  And, the uncertainty level, which is really what you are talking about, would almost certainly be very similar, since having 10 guys exactly like Carpenter still won’t trump your sample size.

Anytime you use statistical analysis to predict human behavior, you’re on shaky ground.  Did you ever read The Foundation Series by Isaac Asimov?  He thought it was possible to predict trends in human behavior, given a sample size of 10’s of billions.  But now we’re talking about samples of 10’s and 20’s, the using of which would get you an F in a college statistics course.  That’s why it’s fair to give the probability percentiles, set aside from confidence intervals.  Of course when looking at them as a whole they tell you, if the player has a good year he is going to be good, if a player has a bad year he is not going to be good.  Which most people know anyway.

All that being said, I think any local baseball fan who follows the daily press clippings and has a basic understanding of how chance plays a part in a player’s season is likely to give you a better forecast for an individual player than any of these ‘mass’ systems will.  They’ll certainly give you better counting stats predictions than Pecota, because Pecota is terrible at projecting innings and PA’s.

We’re about to find out, when I launch the Community Forecasts next week.