THE BOOK--Playing The Percentages In Baseball

If Strasburg’s ERA is 1.78, and league average is 4.5, isn’t that 2.7 runs, or a loss of .30 runs, which is better than a loss of .255 runs?

And won’t the large crowds get mad if the star they came to see gets pulled early?

BF, I don’t think that Strasburg going forward is 2.7 runs better than a reliever.  First of all, a reliever in the 6th inning is probably around .2 runs per 9 better than the average pitcher.  Second of all, the best starters in baseball, Jimenez, Lincecum, Halliday, are around 1 to 1.5 runs better than average. No starter can ever (within reason) be more than 1.5 runs per 9 better than average.  Ever.  I have been doing projections for 22 years.  Maddux, Pedro, Roger, and Randy, all in their absolute prime, were 1 to 1.5 runs better than average.

Let’s give Straburg the best ever, just for laughs.  1.5 runs better than average. That is 1.3 runs better than the typical 6th inning reliever.  That is .144 runs per inning.  Again, not even close.

“And won’t the large crowds get mad if the star they came to see gets pulled early?”

I am not commenting on any other reason to keep him in there.  For all I know, Strasburg may have promised to give Riggleman a b*** **b if he let him pitch at least 6.  Not my problem. I am only pointing out how much a team loses in WE or RE when it does that.  If I were advising a team, that is what I would do.  Once they have accurate information, they can make their own decisions based on all of their utilities, goals, other information, etc.  It is always better to make decisions based on all the correct information, right?

If you are able to ask Riggleman if the team had a better chance of winning the game with him hitting and then pitching another inning or him being lifted for a pinch hitter, and he responds, “Yeah, I know we had a better chance of winning with a pinch hitter (and it is not even close), but I chose not to because...”

Well, I’ll give him a **** ***…

"No starter can ever (within reason) be more than 1.5 runs per 9 better than average.  Ever.  I have been doing projections for 22 years. “

We’ll start with Pedro:

% of league ERA, using B-REFERENCE
1998 .61
1999 .41
2000 .34

Use the Monkey, 1 2 3
Go .61 * 1 + .41 * 2 + .34 * 3 =2.46 / 6 = .41

.41 of League ERA
2000 ERA used in Pedro’s BRef = 5.06

.59 * 5.06 = 2.98 runs per 9 innings

That is DOUBLE 1.5 - how do you get him at “only” 1.5?????

At a 4.5 run average, 1.5 runs is .67 runs allowed, or 150 ERA+ in B-Reference

From 1988-1990 to 2007-2009 is 20 3 year spans. 14 guys were at least 150 ERA+ in 486 innings for a 3 year span

Clemens, Rijo, Appier, Madduc,Johnson,Cone,Brown,Pedro,Glavine,Oswalt,
Schiling,Halliday,Santana, Lincecum

You are underestimating how good a pitcher can be.....................

If you count statistics since 1900, then 22 years is about 20% -

So you are also underestimating how long ever can be.

Other than Pedro, I think almost my top pitchers via Marcel are at 65% of league average, since I’ve published starting in 2001.

So, if the average pitcher is giving up 4.7 runs per game, then 35% of that is 1.65.

Pedro I think came in at 50% of league average (dude… you forgot regression.... you can’t take sample data and not regress).  That is the limit pretty much.

I know that MGL has said that he’s never had a pitcher at below 65%, but I think he is wrong.  I believe that if he ran his forecasts for Pedro, that Pedro will at some point have hit 50% of league average.

Perhaps Pedro is the exclusion to his “within reason” provision?

And I presume that we are removing relievers, since Mo’s career RA Index is around 50% of league average.

By the way, while we may be able to reasonably conclude that Pedro’s true talent level was 40% of league average at his peak, we can only make this determination a few years after the fact.

When we make a forecast for Pedro, the best we can say is 50%, but that comes with a higher uncertainty level than if we look at his career, and conclude that at his peak his runs allowed was at 40% of league average.

I hope that made sense.

Of course, I am talking about starters.  And I stand by my assertion that I have never had a projection better than 1.5 runs better than league average. And I am not talking league average for a starter. I am talking league average for all pitchers.  That would be 1.6 to 1.7 better than a league average starter. 

Again, we are talking about Strasburg.  NO ONE will argue that he is already the best pitcher of all time - ever.  I think that 1.5 is more than generous for him.

And yes, you cannot look at a pitcher’s peak in the past any more than you can come up withe a projection with no regression.  And you cannot even cherry pick a pitcher’s peak and then include regression.  None of that amounts to a legitimate projection.  You can’t even take the best pitchers in history and choose a random number of years from them and then do a regular projection with regression.  That is still cheating a little because the best pitchers in history are chosen after the fact, and believe it or not they are a biased sample of lucky pitchers.  Believe it or not.

So yes, 1.5 runs better than league average is the best you can EVER have for a projection. And that is generous. The only pitchers I have even close to that is Jimenez and I think Greinke, and neither one of them is quite 1.5.  I’ll put my money where my mouth is, as always, if anyone ever wants to project any pitcher (starter) going forward.  I’ll take over 1.5 any time for any starter.  Any time.

"And I stand by my assertion that I have never had a projection better than 1.5 runs better than league average. “

MGL: What did you have for Pedro entering 2001?  Marcel had him at 55% of league average, or 2.0 runs better than league average.

I never ran Marcels for the 1990s, but I would bet Maddux would be close to that as well.  Doing a quick back-of-envelope calculation, entering 1996, I also have Maddux at 55% of league average.

I used 486 ip in 3 years, so I am at starters
only.

“you cannot look at a pitcher’s peak in the past”

“And you cannot even cherry pick a pitcher’s peak”

“and choose a random number of years from them”

3 is not a random number; it came from Marcels. The monkey told me to use it.

“So yes, 1.5 runs better than league average is the best you can EVER have for a projection..... have even close to that is Jimenez and I think Greinke, and neither one of them is quite 1.5.”

So are you saying that Jiminez is as good as any pitcher ever? He didn’t make the list above of the 14 guys who were at least .67 better than league average

I am trying to determine how you would project Pedro in 2001. Wouldn’t you look at his last 3 years as a starting point? I dont think using his last 3 years to determine his talent level is cherry picking.

How did you project Pedro in 2001?

"I’ll put my money where my mouth is, as always, if anyone ever wants to project any pitcher (starter) going forward.”

Quickie Marcels:

Lincecum last 3 years .59, give it 6/7 (3 2 1) and regress 1/7 and you get .63

4.5 League Average puts him 1.66 runs better than average

You can send your money to either of these:

http://www.cancer.org/docroot/DON/DON_0.asp

https://shop.stjude.org/GiftCatalog/express-donation.do?fnl=don_sin&plt=STJGENLKALSAC1000002

bf: it’s fine to use Pedro’s 1998, 1999, and 2000 to forecast his 2001.

And giving him 3 parts for 2000, 2 parts for 1999 and 1 part for 1998 is fine.  And doing that, you do get around 40%.

But, you ALSO need to include 2 parts of league average performance.  And once you do that, you end up at around 55%.

Actually, we’d add 1.2 parts of league average for Pedro.  (It depends on the IP he’s had.)

But, we also have to age him, since he was 30 entering the 2001 season.  In any case, his Marcel was 2.62 for the 2001 season.

Here are Pedro’s Marcel’s, starting in 2001:

2.62 2001
2.48 2002
2.66 2003
2.84 2004
3.49 2005
3.22 2006
3.74 2007
3.83 2008
4.85 2009

So, for at least 3 seasons, Marcel forecasted Pedro for much better than 1.50 runs per 9IP better than average.

Interestingly, the simple (unweighted) average of the Pedro forecasts above (2001-2009) is 3.30.

His actual ERA (unweighted) average for the 2001-09 seasons was… 3.32.

***

His actual (weighted) ERA was 3.23, compared to a forecasted ERA (weighted by actual IP) of 3.06.

These are the top 15 ERA forecasts from 2001-2010, from Marcel.

ERA year nameLast nameFirst
2.48 2002 Martinez Pedro
2.62 2001 Martinez Pedro
2.66 2003 Martinez Pedro
2.84 2004 Martinez Pedro

2.88 2010 Lincecum Tim
2.92 2006 Santana Johan
2.93 2003 Johnson Randy
3.01 2010 Carpenter Chris
3.01 2002 Johnson Randy
3.06 2007 Santana Johan
3.09 2005 Santana Johan
3.10 2006 Clemens Roger
3.10 2007 Clemens Roger
3.12 2002 Brown Kevin
3.15 2004 Prior Mark

League average ERA is anywhere from 4.30 to 4.70, depending on the season. 

MGL’s “1.50” rule, if changed to “1.50ish” is reasonable enough… except for Pedro.

Pedro Martinez, for his career as a starter facing 11,000 batters, with all the ups-and-downs, has allowed runs at 1.7 runs per 9IP fewer than league average.

Obviously, we need little regression (3%) to get to his overall true talent level.  That’s 1.65 runs per 9IP better than average, over his career, as his (average) true talent level.

Clearly, at his peak, he must have been way over 2 runs per 9IP better than average, true talent-wise.

And a forecasting system should be able to see most of that.  And, Marcel did see most of that.

So, let’s just call the MGL/1.50 rule as having a provision of “except Pedro/Maddux”, and move on.

"I’ll put my money where my mouth is, as always, if anyone ever wants to project any pitcher (starter) going forward.”

“So, let’s just call the MGL/1.50 rule as having a provision of “except Pedro/Maddux”, and move on.”

So is he donating to St Judes or the American Cancer Society? Either is an excellent choice.

bf, you need to do a little work on your reading comprehension.  MGL specifically said (wisely) “going forward.” To bet, you have to pick a pitcher now and bet that he will in fact be >1.50 runs better than average over some agreed upon timeframe.  MGL’s bet, implicitly, is that there’s at least a 50% chance that anyone you pick will not actually be quite that good.  If you’re ready to bet on Lincecum, Strasburg, or whomever, go for it.  Otherwise, this is tedious.

Guy is right of course.  Any time as long as I am alive anyone wants to project a pitcher more than 1.5 runs better than league average (park and defense adjusted of course), say for one whole year, I will take the over for any amount of money!

You simply cannot cherry pick a pitcher in the past and his peak years in order to come up with a reasonable, limit for a pitcher projection going forward.  Tango, you should know that!  I can’t believe that you can seriously think that, after the fact, asking me or anyone else to come up with a projection for a pitcher based on his 3 best years, is a measure of true talent or can be used as proxy for the limit of a starting pitcher’s projection!  You don’t think that 98, 99, and 00 were “lucky” years for Pedro and thus his projection for 2001, even after regression is “incorrect?” It is!

Any pitcher or group of pitchers in the past who have the best park adjusted ERA in any time period are “suffering” from:

1) Great true talent.
2) Above average defense behind them.
3) An ability to take advantage of their home park better than the park adjustment thinks (whether you want to call that part of their “true talent” is not clear).
4) Good fortune (luck)!

If you want to get an idea of the limit of any starting pitcher’s true talent, you can do this, although the answer is by no means definitive:

Take the 5 or 10 best pitchers in modern historical baseball.  Average their park adjusted ERA below average between ages 25 and 39 for all of them.  Then add some number (I have no idea - maybe .25 runs) to account for #2-4 above.  That will give you some idea as to the limit of any starting pitcher’s true talent.

I’ll add one more thing about Pedro, Maddux, and these guys, as compared to projecting a pitcher.  All pitcher projections include chance if injury, be it tomorrow, or next year.  And injury is not just something that caused him not to pitch.  It is all those times that the pitcher pitches with a sore arm or elbow, or a bad hamstring, etc. All of these little injuries cause a pitcher’s performance going forward to be depressed which is one of the reasons why you can’t project such great performances for pitchers (and one of the reasons why we regress so heavily).

So, do you think you can take pitchers in the past who were either completely (Maddux) or relatively, at least during their peaks (Pedro and Clemens), healthy during their careers and assume that their numbers are good projections for someone in the future, when we KNOW that the average health of a pitcher going forward is NOT like Maddux or Pedro in his prime? Of course not!

My 1.50 rules includes Maddux, Clemens, and Pedro, but certainly not after the fact!  They were the greatest AND the luckiest!  Again, Tango, you should know that instantly.  You have to subtract something from their numbers to account for luck, defense, and ‘extra’ park effects if want.  And you CAN’T do that by cherry picking some 3 year sample and then doing a Marcel or any other kind of projection.  Do you think we can take McGwire’s 97, 98, and 99 home run totals, do a projection on that and then declare that as the limit for a player’s true talent HR total per year?

This:

“between ages 25 and 39”

should be “between ages 25 and 30...”

MGL: I don’t understand.  You said that you never have forecasted a pitcher to be better than 1.5 runs per 9IP better than league average in 20 years.

And I have asked you what you forecasted Pedro entering 2001.

I have to believe you had forecasted him for something close to, or even better than, Marcel.

"They were the greatest AND the luckiest!”

What are the odds that the 3 greatest players were the 3 luckiest?

One had injury problems.
One played for the Cubs
One was Roger Clemens.

“Any time as long as I am alive anyone wants to project a pitcher more than 1.5 runs better than league average (park and defense adjusted of course), say for one whole year”

Don’t know your age, but let’s say you live 50 more years- in the last 50 years it has happened 90 times.

You should factor in the chance of an injury in a projection of course, but the chance of an injury does not affect what the player’s true talent is.
His true talent is injury free. You might say it is .70 of league average, but Im projecting .74 due to chance of injury, but that chance of injury is not part of his talent -it’s an adjustment from his talent to his projection.

So, if Pedro was 2 runs better than average by Marcel, you are saying 25% of that is luck?

bf: let me handle this please.  You’re bringing in confusing issues, and in a fairly antagonistic way.  If I wasn’t trying to get to the bottom of what MGL was trying to say, I wouldn’t have otherwise interfered.

MGL: I guess the issue is “why 1.5”?  Why not 2.0 or 2.5?

Let’s take Pedro, 1997-2003, where he faced 5542 batters, with an ERA+ of 213.  Presuming your standard ERA scale is a league average of 4.00, that translated to a normalized and league/park-adjusted ERA of 1.88 (47% of league average).

What was Pedro’s likely true talent level between 1997-2003?  We need to regress his observed 47% at 5% toward league average.  That puts his estimate true talent level at exactly 50% of league average (normalized ERA of 2.00, or 2 runs per 9IP bettter than average).

So, why 1.5?  Why not 2.0 as the threshhold that no one should surpass?

Now, back to the matter at hand: what did you, MGL, have as Pedro’s forecasted ERA entering 2001?  And what did you have as Maddux’s forecasted ERA entered 1996?

And how does your answer to these two questions remain consistent with your statement here:

I have been doing projections for 22 years.  Maddux, Pedro, Roger, and Randy, all in their absolute prime, were 1 to 1.5 runs better than average.

Basically, I want to see your forecasts for these pitchers in their prime.  I want to see what you actually forecasted them.

OK, we have two different questions going here. 

One, what is the limit to how low you can forecast a pitchers’s ERA?  And guess what the answer to that is?  Pretty close to zero even though we KNOW that is not any pitcher’s true talent ERA.  And why is that?  Because let’s say that you have a league where the true talent of any pitcher is never less than .5 runs (or .25 runs, or 1 runs, or 1.5 runs, or whatever you want) better than avg.  Well, eventually, given a large enough sample size (number of years), you will have a pitcher who pitches 3 straight years with an ERA of zero.  And what would be your projection for him after those 3 years?  A lot less than his true talent!

So, as I said, in any given sample, especially a large one, your best projections are always going to be less than the best true talent of any of your pitchers, as I illustrated with the example above. And the larger the sample, the more your best projections (and not just best performances) will be better than those pitchers’ true talent and true talent of ANY pitcher in your sample.

How do you get around that?  You set a reasonable limit on all projections, both on the high side and the low side.  I do that.  You have to.  Especially on the high side.  Just because your projections are “correct” on the average, does NOT mean that they will not be “wrong” at the tails.  They will be.

The second question is, what is the limit to any pitcher’s true talent, regardless of his projection?  That is the question I was referring to, since the answer to the other one, as I explained above, is close to zero!

If your sample was large enough, you would have a pitcher who had a zero ERA for 5 straight years, regardless of the limit of all your pitchers’ true ERA.  What would be your projection for him?  Pretty close to zero, even with regression.

So whatever I would have Pedro projected as for 2001 would likely be a lot better than his true talent.  For the record, I have his normalized, component ERA for those 3 years as 2.48, 1.38, and 1.59.

Where do you get a 5% regression using 3 years of ERA+?  That seems terribly low. I would think the regression after 3 years full-time is closer to 30% or something like that, when dealing with ERA+.

I use normalized component ERA to do my projections. I get a year to year correlation of around .500 after 950 PA, which is almost a full year of starting pitching.  So after 3 full years for Pedro, the regression is around 22%.  Again, I don’t know where you get the 5% from.  And of course, if we are weighting each year, the regression is even more since our effective number of PA is less.  But we’ll forget about that.  We have:

2.48
1.38
1.59

A simple average of 1.82, regressed 22% toward 4.00, which is 2.30. I always have an aging penalty, no matter what the age of the pitcher,but certainly at age 30 (Pedro in 2001) a pitcher deserves a healthy aging penalty.  Call it .15 runs.  That leaves us with a projection of 2.45, which is 1.55 runs better than league average.

But, as I said, if we look at enough years of pitching performance, even if the best pitchers are still 2.5 pitchers (with a 4.00 average), we will eventually see 3 or 4 years of pitching way better than that.  Does that change our estimate of the limit in true talent?  No!  If a pitcher pitches 3 years of shutout ball, we now have a projection of 3.12 runs better than league average.  With aging, we’ll call it 3.00. So what?

bf, lots of pitchers will have a year where their ERA+ (or whatever you want to call it) is less than 1.5 runs fewer than league average.  But YOU won’t be able to pick which ones with greater than 50% frequency.  The bet I am willing to make is NOT that SOMEONE (anyone) will post that low of an ERA.  The bet is that you have to tell me before the season starts which pitcher will do it.  Presumably you think that he has a greater than 50% chance of doing so, and thus his median projection is better than 1.5 runs less than avg.  I contend you will never be able to do that. If you think otherwise take me up on that bet.  We may have to wait 30 years for you to find a pitcher that you think projections that well, but if you think that Strasburg is that good, let’s do it for next year!  Of course when I say “avg” pitcher, I mean for both leagues combined.  An NL pitcher right now would probably have to be 1.75 runs better than the avg NL pitcher to be 1.5 runs better than both league’s avg.

I think this discussion would be clearer if you agree on what run environment is assumed.  MGL’s projection of +1.55 for Pedro translates into about +2.0 in 2001 AL. 

I believe Pedro really was about a .50 ERA (50% better than average) pitcher at that point, if—and this is a crucial IF—you assume he stays healthy.  He did in fact stay healthy the next 3 years, and was about a .50 pitcher.

On the other hand, if you projected his WAR using a simple Marcel approach, you would have been way off:  projecting about 8 WAR per season, while he delivered 6.  Playing time needs to be heavily regressed for pitchers. 

But even leaving aside playing time, it seems to me that injury can reduce performance rates, as MGL suggests above.  Perhaps Marcels are accurate in the aggregate because of the high correlation between performance and IP.  If we ignored IP (except some minimal threshhold like 50 IP), would we find that more than 50% of starters do worse than their Marcel in any given year?  I suspect we might.  If so, it would mean Marcel is really projecting the pitcher’s performance IF he remains healthy.

Where do you get a 5% regression using 3 years of ERA+?

Actually, I said that if you observe Pedro’s 1997-2003 (5500 PA), you would regress 5% toward the league mean to get his true rate *for that time period*.

***

As for the bet, Marcel said this for Pedro:

ERA year nameLast nameFirst
2.48 2002 Martinez Pedro
2.62 2001 Martinez Pedro
2.66 2003 Martinez Pedro
2.84 2004 Martinez Pedro

Therefore, he was forecasted at better than +1.50 for 4 straight years (based on the league average ERA of around 4.50 or so).

MGL’s scale is a 4.00 ERA, and therefore, he is setting the limit at 2.50 ERA, or 62.5% of the league average.  He is saying that he would never forecast any pitcher at better than 62.5% RA index.

In ERA+ speak, he would never forecast anyone at better than 160.

Therefore, if Pedro, in 2001-2004 had 3 or 4 seasons of 160 or better ERA+, I win.  If he’s at 2 seasons of 160 or better, it’s a push.  If he had 0 or 1 season at 160 or better, MGL wins.

Let’s check b-r.com:
2001: 190
2002: 202
2003: 211
2004: 125

Therefore, in the 4 seasons where Marcel forecasted Pedro to have a better than 160 ERA+ (equivalent to MGL’s +1.50 ERA), Marcel did in fact get 3 of those 4 seasons correctly.

I didn’t run Marcels for the 1990s, but doing a quick back-of-envelope calculation, Marcel would have forecasted a better than 160 ERA+ for Greg Maddux for each of 1995 through 1999.  In 3 of them he beat that easily, in 1 of them he barely beat it (let’s give that one to MGL), and in 1 he didn’t.

So, for (presumably the only) 9 seasons in which Marcel forecasted a pitcher with an ERA+ of 160 or better, Marcel did in fact get 6 correct and 3 wrong.

Now, it’s possible there are other seasons taht Marcel did forecast for better than 160 ERA+.  RJ might have had 1 and Clemens might have had 1 or 2, etc.  And maybe if I look at their seasons, I might end up losing a couple.

***

If MGL’s point is that if I look at say the last 20 Marcel forecasts of greater than MGL’s threshhold (ERA+ of 160, which is +1.50 ERA in MGL-speak), and if Marcel is correct in 10 of them to beat 160 and 10 of them to not beat, then MGL has proved his point: don’t forecast anyone for over 160 because you just won’t be able to beat that threshhold consistently.

Lots to think about on that point.

Doing back-of-envelope Clemens, and he twice comes right on the nose with a 62.5% RA Index forecast (160 ERA+, or +1.50 MGL-speak).  So, he satisfies MGL’s rule of not ever forecasting him at better than +1.50.

RJ had two seasons (2002-2003) where he did forecast at better than MGL’s threshhold.  And he ended up at 1-1 there, which basically supports MGL’s assertion.

I think MGL’s rule might be a reasonable one, that there’s really no upside to forecast anyone at better than +1.50, in terms of betting over/under.

I mean, we can quibble about exactly where to draw that line, be it +1.50 or +1.60 or +1.70 or whatnot.  But, it’s a reasonable position to take.

Is there a maximum above which you should not forecast a batter’s wOBA?  Would you have broken your rule for Babe Ruth when he came along or remained a slave to it?

I was thinking the same thing. We’d have to work it out, say look at the top 50 wOBA forecasted ever from Marcel, and draw the line where you have 15 over/under.

And same thing for the top 50 ERA forecast ever, and draw the line.

Once you do do, look at the top 30, top 20, top 10, and see if that threshhold still holds.

If MGL is right, at some point, that threshhold should remain stable.

It’s an interesting thought.

"15” = “25”.

You probably don’t care, but Pedro beat his Marcel each year from 2001-2003, and was way over in 2004.  For the 4 years of the greatest Marcels combined, he pitched almost exactly as well as the forecasts.

Are you saying, with your limit, that if you see a pitcher as great as Pedro you need to regress him more to the mean that a typical pitcher with 600 (or whatever) innings?

I don’t think I can buy it.  I’ll be happy to take the bet, assuming we ever see a pitcher who projects past this limit again.  (Maybe if Strasburg keeps up this pace for 3 years).

"Are you saying, with your limit, that if you see a pitcher as great as Pedro you need to regress him more to the mean that a typical pitcher with 600 (or whatever) innings?”

I don’t know what you mean.

Yes, there is also a reasonable limit for batters, although it is not as clear what that is.  There is a qualitative difference between the limit for batters and for pitchers.  Theoretically a batter can hit some ridiculously high number which is only limited by where the ball happens to drop.  I can forsee a once in a hundred years batter who has virtually a perfect swing and a perfect eye who pretty much hits almost every pitch hard and doesn’t swing at any bad pitches.  Kind of like a man among boys, or a college player playing in a senior Little League.

For pitchers, the reasonable limit does not include a pitcher who throws 105 mph.  I suppose some day that might be possible.  But for a starting pitcher who throws 95-100 with great off-speed pitchers, which seems to be the reasonable current physical limit to pitching, surely there is an ERA limit.  And, as I have said a number of times already, I am quite certain that that number is around 1.5 runs less than average. And BTW, I probably should have said 62.5% of average rather than 1.5 runs. In fact, I’ll stick with that presentation from now on.

Again, this is for a reasonable all-time great pitcher.  And I include Strasburg, Clemens, Pedro, Halliday, Maddux, Seaver, etc. in that group. If someday a few players come along who are not in my current group of “reasonably all-time great pitchers” I might amend that limit.  But for now, I am quite certain of it. 

If ONE person comes along in the next 50 years or so who is better than that, well, it is not real clear if that “reasonable limit” should be changed.

I mean, come on guys, there really is no limit, per se, to any player, pitcher or batter, other than 0 ERA for a pitcher and whatever walking or hitting a home run or line drive almost every time is for a batter.  So we have to have a “reasonableness” standard which can eventually be surpassed by someone.  That standard right now for pitchers is 62.5%, in my opinion, and I am pretty darn certain of that.

Again, if anyone ever wants to take me up on that for one year of pitching, just name your price and consider it done.  The only qualification is that we will have to have an impartial person do the park, league, defensive, and opponent adjustments, since I am obviously talking about a completely context-neutral number.

It is also possible, of course, for the league average pitcher to be quite bad one year, just because of random fluctuation.  In that year, of course, it is going to be easier to surpass my limit.  That is the “bonus” I’ll give to anyone who ever wants to take the bet.

"I don’t know what you mean.”

OK.  We’ve got a pitcher who’s been a bit better than average.  Allows .80 runs compared to league average for a few years.  We put his numbers into the Marcel, add some regression to the mean, and get a projection of .87 for next year.  And we’re comfortable with .87 as an estimate of his true ability, as an over/under for next year’s ERA.

Now Pedro comes along, pitches at a .35-.40 level.  Yeah, of course he has to be a bit lucky to do that.  Put his numbers through the exact same Marcel and the regression brings his projection to .55.

Are you saying that’s not a good over/under, that you won’t project anyone better than .625?  If so, then you are regressing Pedro more than you are for Joe Aboveaverage.

Noe your later comment seems to suggest you wouldn’t, that the limit that works for everyone else can be exceeded once a century or however we’ll find a prime Pedro.  We’ll see.  I’ll be happy to take the bet, but first I have to find another Pedro.  Don’t see one out there, not Lincecum, not even Strasburg, at least not yet.

So we have to have a “reasonableness” standard which can eventually be surpassed by someone. 

I consider Pedro to be the greatest pitcher of all time.  So, when I first said that if you included a provision of “except Pedro”, then fine, let’s move on.

But, you included Pedro at his peak in one of your posts!  So, if you are backing off on that rigidness, then, alright, I think proposing a 62.5% limit (might as well make it 60%, no?), is reasonable enough, and if someone wants to forecast for something lower, then they better be darn sure of their forecast.

Why do need a “reasonableness” standard?  Haven’t we done quite fine without one till now?  It’s not clear to me what problem this is proposing to fix.

If it’s just an observation, then I guess I can take it for what I consider it to be worth.

2 questions, then I will keep quiet:

27/paraphrased:
draw the line where you have 1/2 over and 1/2
under

1.
Shouldn’t you theoretically always have 1/2 over and 1/2 under in a projection? If you project 20 pitchers to have a 4.00 era, wouldnt 1/2 be over and 1/2 under? Or are you saying at some point it will be less than 1/2 and 1/2?

2.
If you project Pedro, Maddux and Johnson to have a combined 11 seasons of .625 and they have 8, then why is that the bottom?

Thanks

Some of these questions are difficult to answer. 

This is the important point to remember:

You can be positive that there are pitchers who are better than 1.5 (we’ll assume from now on, as in Rally’s post, that we say a number like that, it means “runs less than average per 9”.), but you still cannot project anyone to be that good.

“Or are you saying at some point it will be less than 1/2 and 1/2?”

And I have easily proven that already, but I’ll repeat myself, by restating my important thought experiment:

Given a large enough sample, and using your regular, “works just fine on the average” projection system, you will have pitchers who project to be way below any possible level of true talent.  But you KNOW that is not a correct projection.  That is why you need a limit, period.  And that is why, yes, even with the “prefect” projection system, you have to have a limit otherwise some of your projections will be wrong.  That is really all that needs to be said.

If you don’t believe or understand it, try doing this on a computer, or imagine doing it if you can’t actually do it.

Set up a world where there is a limit to how good a starter can be.  Call it even 1 run less than average.  Make sure that no one pitches better, in true talent, than 1 run better than average. Now simulate one million seasons of 100 (or 300, or 600, or whatever) of these pitchers and then do projections each year based on the last 3 or 4 seasons. You are going to get lots of projections better than 1 run better than average - and guess what?  All of them will be incorrect of course.

You get the same thing at the upper end of course.  I get plenty of those with my projections.  I ignore them and assign my “cap” to those players.

If you look at Chone, Oliver, ZIPS, etc., you will see those too. They are projections using the same “perfect” (works just fine on the average) methodologies that they use on all pitchers. But when you see park adjusted ERA projections in the 6’s (which you will), they are simply wrong.  They are based on pitchers who simply got incredibly unlucky the 2 or 3 previous years.

Here is another way to look at it.  The regressions are based on the “average luck” that a pitcher can get in X number of PA.  Now, when a pitcher has a 3-year ERA of 3 or 4 or 5, we don’t really know how much luck there is in those ERA’s, so the usual regression is fine.  But if a pitcher has a 3 year ERA of zero (which we all agree is possible given a large enough sample), then we need to regress more because we KNOW that there was A LOT more luck in that zero ERA than we are account for with our regression.  Same with pitchers who have 3 year ERA’s of 6 or 7.  That have likely had way more bad luck than we are accounting for with the regression.

Now, you mathematicians will say, “That isn’t true.  Regression is regression.  The same regression that works on a zero ERA pitcher works on a 4.5 ERA ERA pitcher or a 10 ERA pitcher.”

But, you are forgetting about one very important things.  These are really Bayesian problems!  These are not “gronks” (pitchers) posting “lisnets” (ERA’s). If they were, then you would be 100% correct.

But since these are human pitchers posting ERA’s, and we know a little about both, we KNOW that a pitcher who throws less than 110 mph cannot have a true zero ERA and also that a pitcher who throws 88 with not horrible secondary pitches or control, and is allowed to pitch in the major leagues, cannot be a true 6.50 ERA pitcher!  Those change the regression equations.

Remember that the kind of regression we do with these projections is a short-cut for a full Bayesian probability analysis, which is, “Given an observed ERA of X, and given that we KNOW the distribution of true talent in the population, what is the mean or median best estimate of this pitcher’s true ERA.” The regression we do is merely an approximation, and not a very good one at the extremes for this.

For example, the normal regression we do for projections assumes that all values are possible for the distribution of true talent and that this distribution is normally distributed.  Both of those assumptions are fine for most projections, but both of those assumptions are wrong of course - especially the part about all true talent levels being possible.

That is why the projection algorithms do NOT work at the extremes and one of the very rough fixes for that is to simply apply a limit.  A better fix of course, is to scrap the simple regression we do, and use a full Bayesian analysis where we have to make some assumptions about the distribution of true talent in the population.  Of course that becomes kind of circular, but even if you assume a very generous distribution of true talent, such that there is a slight chance of a true talent 2 pitcher (rather than 1.5), but no chance of a true talent 3 or 4 pitcher, you will find that the regular projection algorithm will overvalue a pitcher who has 3 or 4 incredible years as opposed to the more rigorous and correct Bayesian model.

“If so, then you are regressing Pedro more than you are for Joe Aboveaverage.”

Yes, that is absolutely correct for the reasons stated above.  The regression in the projection model is a rough stand-in for the correct Bayesian model, and it does not work very well at the extremes, for pitchers who have terribly good or bad years on which the projections are based.

Now, while there may be pitchers in history or in the future that are better than 62.5%, it doesn’t matter.  You can’t ever project one like that. And we are not even addressing the injury problem, which obviously manifests itself more over a long-term future projection (like even one whole year) than a short-term one (like today or tomorrow only).

With the injury factor, I am going to win my bet probably 60% of the time no matter how good you think your pitcher is.  And don’t even think about going back in time and choosing only those pitchers whom you know remained healthy for at least a few years. You’ve got to also include pitchers like Gooden and Koufax.  How would those projections have turned out after a few years?  Maybe I used bad examples, but you know what I mean. You have to include pitchers who had incredible 3 or 4 year runs and then had arm problems or blew out their arm.

If you want to test my theory, go back in history and do a projection on any pitcher that had at least a chance to be projected at better than 62.5% in any given year, including Koufax in 1967 (of course he didn’t pitch in 67, but his last 4 years’ average FIP was less than 2).  And Clemens in 1993, and in 2007. Etc.

I am pretty sure that you will easily lose that bet (less than 50% of the time will the projection that is better than 62.5% turn out better than 62.5% for the whole projected year. I am also pretty sure it won’t even be close (like I said, probably 60% in my favor). In practice, it is going to be hard to do that, because I am also contending that the best pitchers in sample performance also have better than average defenses behind them, and you would need to adjust for defense if you want to take my bet. I am also contending that they faced worse than average offenses, and you would have to adjust for that as well.  But, I might be willing to make the bet even without those adjustments, even though my claim is “true context-neutral talent.” And I am definitely willing to let slide my contention that all great pitchers, on the average, took advantage of their home park beyond that of the park adjustment.

I stand by my contention that the practical limit of a pitcher’s true talent is 62.5%, but even if there was or is someone that exceeds that, which is always possible of course, you can NEVER project that, at least until we get to the point, if we ever do, that there are a significant percentage of pitchers who are like that in our population.

To answer the question, “Why does there have to be a practical or reasonable limit,” I think I already answered that, but I’ll say it again.

Because technically there is almost no limit of course.  If in the next 100 years, they have surgeries that can allow a pitcher to throw 120 mph, like in that kid’s movie, then of course there will be pitchers who have true talents near zero in ERA.  So technically there is almost no limit, but what is the point of stating the obvious?

As I said, because of the short-cut we take in the way we do projections we have to have a limit. If we use the full Bayesian method, we don’t have to.  We can simply assign a percentage to all values.  We can say that the chance that any random pitcher is a true zero ERA is .000000000000000000000001, the chance that a random pitcher is a true 1.50 ERA is .00000000001, etc.  Just assign reasonable values to all of those ERA’s, and do the full Bayesian analysis, and I think that you will find that even the pitcher who posts an ERA of zero for 1-0 consecutive years will have a projection of 2.00 (or thereabouts, depending on your distribution), and not close to zero which he would have if you used the regular method of a weighted average plus a regression.

Doesn’t this just mean that the regression factor should be say, an exponential one. So that a pitcher who has put up a 0.50 ERA in his prior 3 seasons is regressed considerably more than a pitcher who has put up a 3.50 ERA, even if they have the same batters faced? Kind of like a pythagenpat thing.

Maybe.  I have no idea.  I suppose it is something like that.

You can be positive that there are pitchers who are better than 1.5 (we’ll assume from now on, as in Rally’s post, that we say a number like that, it means “runs less than average per 9”.), but you still cannot project anyone to be that good.

“Or are you saying at some point it will be less than 1/2 and 1/2?”

And I have easily proven that already, but I’ll repeat myself, by restating my important thought experiment:

Given a large enough sample, and using your regular, “works just fine on the average” projection system, you will have pitchers who project to be way below any possible level of true talent.  But you KNOW that is not a correct projection.  That is why you need a limit, period.  And that is why, yes, even with the “prefect” projection system, you have to have a limit otherwise some of your projections will be wrong.  That is really all that needs to be said.

If you don’t believe or understand it, try doing this on a computer, or imagine doing it if you can’t actually do it.

You haven’t proven anything.  You’ve done a thought experiment.  No one has shown that projection systems are actually overvaluing pitchers at the top margin (or undervaluing at the other margin).  You’ve made a case why we might expect to see that, but you and Tango have provided no evidence.  The one piece of evidence provided (Pedro) turned out not to fit the theory.  Around here, Tango has a word for opinions without evidence.

You say we should do the work to prove or disprove it if we don’t believe.  I don’t believe you because haven’t provided any evidence.  I don’t see why I should have to do the work to examine your proposed theory just because I’m skeptical of it.  It seems shaky to me + the one example provided gets shot down + you provide no further evidence = I have to do the grunt work to investigate the theory?  Pardon me if I still remain unimpressed with the theory.

That’s why I said earlier that if this was simply being thrown out there as an interesting thought for people to consider, that’s fine.  But it seems as if it’s seriously being proposed as an amendment to existing projection systems, in which case it falls far short.

Thanks MGL.  I’m not saying I’m convinced, but you’ve explained your position well on the low end of pitcher ERAs.

I do take issue with this:

“But when you see park adjusted ERA projections in the 6’s (which you will), they are simply wrong.  They are based on pitchers who simply got incredibly unlucky the 2 or 3 previous years.”

This is incorrect.  They are not wrong projections, they are projections for pitchers who are not good enough to pitch in the major leagues.  I’m projecting 1500-2000 pitchers per year. Only about 400 of those are in the big leagues at any given time.

Now if we know that a pitcher with a 6.00 projection is well liked by the teams, and is allowed to pitch in the big leagues, then he is probably better than the projection.

For the typical pitcher in my dataset with a 6.00 ERA, he’s probably not going to pitch in the big leagues.  If 10-20 of them were picked randomly and forced to, I stand by 6.00 or whatever for what that group would have for a MLB ERA.

As for an upper limit to what MLB equivalent ERA you can project for a pitcher who can through 85-88 in the vicinity of home plate, I look at how hitters who take the mound in blowouts do.

From 1993-2009 I get an ERA of 8.84.  So I’d try not to project any professional that high.  Full numbers:
75.3 IP
91 H
14 HR
66 BB
9 HBP
31 K

And strangely enough, a .292 BABIP.

Rally, interesting #s on hitters-as-pitchers. Do you have ability to run that for earlier decades (1950s-60s, and/or 1920-30s).  Would be interesting to see how hitters fared against minimum talent pitchers back then.

I think I originally put that together because I was interested specifically in their BABIP.  So I set the minimum year at 1993, since the league average was much lower before that.  I’ll take a look.

It’s done with the BDB database.  The way to find hitters pitching, if anyone wants to, is look for players who have more BFP than plate appearances.

If I go back too far I’d have to be a bit more careful, not include a guy like Babe Ruth.

For 1969-1992, from the lowering of the mound to the juiced ball/Coors Field/ steroid or whatever you call it era:

IP 121.7
H 159
HR 20
BB 103
K 44
HBP 7

ERA 7.62
BABIP .312

Tango and MGL, I’m not intending to be insulting or rude with my post #38.

It’s just that it seems to me what you guys have here is an untested concept.  It may be right, it may be wrong.  But it’s not been tested.  Whereas, it seems to me that you guys are taking it as proven fact, not needing testing, and ready to move on to implementation.

In addition, I don’t see where the problem is that this is trying to fix, unless it’s the Oliver Strasburg projection.  If it’s it Strasburg, I’d be loathe to redesign a projection system simply because someone thinks it might be projecting one player too optimistically, especially before there is any evidence of that. 

Also, there’s no known flaw that’s causing the Strasburg projection to be optimistic in Oliver, as far as anyone has found, to my knowledge.  Whereas with Wieters and PECOTA a couple years back, Colin identified a problem with Davenport Translations for a couple specific leagues.  To my mind, that’s a very different issue than saying that Strasburg just looks to be projected too high based on historical norms.

If you’re going to adjust a projection system that has a lot of thought behind it, you ought to either (1) identify the weakness in the projection system and propose a fix for that weakness, and/or (2) show how the projection system would have historically missed on a group of players.

Simply wishcasting that Strasburg is over-projected isn’t valid science.  It’s opinionating taking the guise of science.  I would rather know that the projection system says Strasburg is as good as it thinks he is, unless someone can show how that approach would have been overly optimistic for actual pitchers in the past.  MGL’s theoretical mind exercises are poor evidence, IMO.

They’re interesting examples at the concept level, but they’re far from leading to a change in implemention, if I were making a projection system.

I took a lot of time to digest MGL’s #35, and he makes many thoughtful points.

I went to Wikipedia to brush up on Bayes.
http://en.wikipedia.org/wiki/Bayesian_probability
and read the article there with great interest.

****
In the 20th century, the ideas of Laplace were further developed in two different directions, giving rise to objective and subjective currents in Bayesian practice. In the objectivist stream, the statistical analysis depends on only the model assumed and the data analysed [14]. No subjective decisions need to be involved. In contrast, “subjectivist” statisticians deny the possibility of fully objective analysis for the general case.

For objectivists, the rules of Bayesian statistics can be justified by requirements of rationality and consistency[1][4]. Such requirements of rationality and consistency are also important for subjectivists, for which the state of knowledge corresponds to a ‘personal belief’ (rather than the objective state of knowledge in the world)[5]. For subjectivists however, rationality and consistency constrain the probabilities a subject may have, but allow for substantial variation within those constraints. The objective and subjective variants of Bayesian probability differ mainly in their interpretation and construction of the prior probability.
****

I feel that I, as well as Rally and Mike Fast are objectivists, while MGL and probably Tango are subjectivists. Not that there’s anything wrong with that, but it’s a different way of looking at the data.

From the beginning, I have designed Oliver as a set of rules, a model, which analyzes the data. The only constants I define are regression values, and they can in the end also be empirically derived. This probably falls under the definition of ‘machine learning’ and I intended for the model to get ‘smarter’ and more granular with every piece of data that is added daily.

When testing the model, absolute error needs to be held to zero (the positive cancel out the negative) while minimizing the rms (mean) error (lack of granularity produces practical limits). Now we are discussing the third limit, making sure the range of output values falls under a probability distribution. Being objective, I’d write a mathematical model to describe the range of data and test the parameters until I’m satisfied.

We’ve been discussing projected ERA, where I will say I’m comfortable with Oliver’s projection of Strasburg, his true talent at this moment in time. Of course pitchers are ticking time bombs, and his arm could go the way of Dwight Gooden’s with any pitch.

Where I’ve looked in more depth at performance on the edge is in projecting batter’s HR%, especially for young hitters several years into the future. When he was recalled last week, I saw that Mike Stanton had Oliver’s highest projected HR% of any 20 year old since 1998, with a sample of 800+ weighted PAs. He could be expected to increase his HR rate for another six years, so where does he end up? In looking at matched pairs of consecutive ages, I know that the group rises in performance, but is that true for the person on top (or on the bottom?) I’m going to look at t-scores, doing projections first at each age, and then see how players at different levels of standard deviation from the mean change in later seasons. As the SD is defined by the probability of being a certain distance from the mean, this seems to me to be a way of fitting the projections under the expected probability distribution of outcomes.

I also downloaded an introductory paper on Markov Chains/Monte Carlo and will be studying that process.

Mike, my post 27 should be enough so that you don’t treat me in tandem with MGL.  I’m still in the testing phase, and I find MGL’s thoughts very interesting.

Right, I don’t think you should be treating Tango as the same as me.  (And BTW, as I have always said, you can say anything you want about what I have to say, in any way that you want, and I will never be insulted.)

That being said, I think you are misconstruing my thesis, which is quite simple (my thesis that is).  Actually it is two-fold.  And I don’t know where you are getting the idea that I am attacking someone’s Strasburg projection.  I don’t even know what Oliver, Chone, or anyone else has for him. If they do have him as better than 62.5% of average pitcher, context-neutral, then I would attack it/them, but I highly doubt that they do.  So why are you (Mike) accusing me of attacking someone’s Strasburg projection? Did I say that somewhere?

Anyway, I said two things, maybe three:  One, the “reasonable” (again, technically there is no actual limit) limit for a pitcher projection is around 62.5% of league average, given what we know about starting pitchers (they can only throw 95-100, etc.).  Two, it is not likely that there are any pitchers better than this, but there could be.  And three, most, maybe all, projection models are short-cuts for a more rigorous and hence, accurate, model, and therefore they will be “wrong” sometimes, in particular at the extremes, because most of them (maybe not Pecota) assume a normal distribution of talent with no limit at the tail ends, whereas in reality, of course, there is a limit at the tail ends.

And when I said, “you can do or figure out my thought experiment,” I did not mean that literally - I meant that facetiously, because no one needs to do it because the results are obvious.  In case you forgot what I said, it was that no matter what limit we set on pitcher true talent in our population, and I hope you agree that there is a (practical) limit, if have a large enough sample, someone will have a zero ERA for as many years as you want, and using any of these Marcel-type projections systems, their true talent estimate will be close to zero, which has know to be wrong (because we can set the limit beforehand).  All of those things are self-evident so no one really needs to “do the work.”

Rally, yes it is true that most of those 6 and 7 ERA’s are pitchers who will never see the light of day, but you are actually agreeing with me that if they did see the light of day (scouts and teams think they have major league stuff), that you would have to change their projections.  That is exactly what I am saying!  If we know that a pitcher has major league stuff, then those 6 and 7 ERA projections are wrong!  How do they even come up?  Because there will always be, again, given a large enough sample of pitchers, those who had horrible luck in the prior years.  Now, if we knew nothing about them, then the projections with the regressions would be just fine.  But if we know things about them, then it becomes a Bayesian problem, and we are not using a Bayesian method in those projections utilizing those things that we know about each pitcher.  We should but we are not. At least most projection systems are not.  So, yes, if those 6 or 7 ERA pitchers get to pitch in the majors, it is likely that the projection was “wrong.” But even if you were to watch the pitcher and you saw he threw 89 or 91 with not horrible control or terrible secondary pitchers, those projections would also likely be wrong.

And, even if you knew nothing about these pitchers, there is still a limit to the projection at the high end for the same reason I am claiming there is a limit at the low end.  Again, given a large enough sample of major league or minor league pitchers, you will eventually get a pitcher who has a 12 ERA for 3 straight years (if he were allowed to actually pitch that long with such terrible results, which he probably wouldn’t). Your projection system might peg him as a true 9 ERA pitcher, but there is no such thing in the majors or even AAA as a true 9 ERA pitcher, so, again, that would be wrong.

So, yes, Rally, most of those 6 or 7 ERA pitchers are not going to pitch in the majors and you would stand by your projections for them, but not all.  Some of them will pitch in the majors or at least could and your projections would be wrong, and I guarantee that if you took all of your higher than 6.5 ERA pitchers and let them pitch, that their mean ERA would be a lot less than the mean of your projections.  Marcel-type projections simply don’t work well at the tail ends!

I’ll give you a perfect example:  I have Suppan currently projected at 2.12 runs worse than league average, yet he was picked up by STL and is starting for them.  There is little chance that he is a true 2.12 pitcher.  Even if he were not picked up by STL, that is simply beyond the limit of a pitcher who recently pitched in the majors and has not gotten hurt or lost 3 mph in velocity or some such thing.  I would never use that kind of projection.

Keep in mind that this thread starting with someone saying that so-and-so was like a 2 (runs better than average) pitcher going forward, and I simply said that was impossible - there is no such animal.  And I have tried to explain my reasoning as much and as detailed as possible…

"I think you are misconstruing my thesis, which is quite simple (my thesis that is).  Actually it is two-fold.... Anyway, I said two things, maybe three.”

“NOBODY expects the Spanish Inquisition! Our chief weapon is surprise...surprise and fear...fear and surprise.... Our two weapons are fear and surprise...and ruthless efficiency.... Our *three* weapons are fear, surprise, and ruthless efficiency...and an almost fanatical devotion to the Pope.... Our *four*...no… *Amongst* our weapons.... Amongst our weaponry...are such elements as fear, surprise.... I’ll come in again.”

If you want a summary of what MGL is saying, and something that I agree with:

Remember that the kind of regression we do with these projections is a short-cut for a full Bayesian probability analysis, which is, “Given an observed ERA of X, and given that we KNOW the distribution of true talent in the population, what is the mean or median best estimate of this pitcher’s true ERA.” The regression we do is merely an approximation, and not a very good one at the extremes for this.

The implication is that your true talent population is going to follow some sort of distribution.  Which means there’s no “jump/outlier”.  Even Pedro, which seems hard to believe.

So, you’d then have to ask if you even can match the true talent level to some distribution pattern.

There are two possible consequences of MGL’s argument:

1) there is a systematic error in forecasts, such that very high (and low) forecasts are under-regressed (as David S suggests above), or

2) pitcher projections should all be capped at 62.5%, because that’s the current limit of human talent (we think). 

The second isn’t very interesting (IMO), as only a handful of pitchers have had projections below 62.5% over the past 30 years (and as best I can tell, they hit the projection about half the time).  But I suppose someone could check to see.  We should find that the 57-62% pitchers fall short quite a bit (60%+), and those below 57% should infreqquently hit their projection (as often as any pitcher should exceed their projection by 8-9% or more by random chance). 

The first version would be very important, if true.  But it should be fairly easy to determine if there is a systematic bias in which higher projected pitchers miss their forecast more frequently.

Guy, right, that’s why I said it should be easy to test.

And remember, regression is a SHORTCUT to Bayes.  Shortcuts always break at the extremes.  Our job is to figure out what IS the extreme.

MGL, I think we’re close to the same page.  If you take a minor leaguer with a bad projection and see’s he’s promoted, then there’s a good chance he’s better than a 6.00 ERA.  But I’d still like to know what the scouting report is on him.  If he’s not well though of as a prospect and is called up for a few games after 6 other pitchers suffer injuries, then I’d stand by the projection that says he’s bad.  If he’s a guy like Kevin Jepsen, who had great stuff but limited results until the second half of 2009, then yeah, the projection doesn’t know enough about him and is wrong.

"Marcel-type projections simply don’t work well at the tail ends!”

Marcels assumes pitchers regress to league average, and as a result it doesn’t really project pitchers much higher than 6.00 or so. So, by default, Marcels does cap the higher end.

There is no cap at the high end (other than in a practical sense).  There is a cap at the low end, but that cap is much lower than any pitcher can reasonably be, true talent-wise.

If a pitcher has a 20 ERA in 500 IP (which won’t happen of course because he is not going to be allowed to pitch that long), then Marcel will give him a projection of around 16 or 17 in ERA.  Where’s the 6.00 cap?  If a pitcher has a zero ERA in 500 IP, which as I said, will definitely happen given a large enough sample of even ordinary pitchers, his Marcel will be an ERA of 1.00.  Do you think that is a good cap on the low side?

Rally, yes I think we are (mostly on the same page).  I think the idea of a “cap” both at the high and low end, is a fairly important idea, but it does not come up very often, so you really don’t have to pay much attention to it.  With me, it actually comes up much more at the high end (virtually never at the low end). I get pitchers like Suppan and Bush, and some minor league prospects, who have projections 1.5 to 2 runs worse than average, and I do NOT use those projections if these pitchers ever get a chance to pitch in the majors, at least until I see them pitch (for example, Charlie Haeger has a horrible projection which I assumed was wrong until I watched him pitch - a lousy knuckleball that he can’t keep down and can’t get over the plate and no fastball which he throws a lot in hitter’s counts because he can’t throw the knuckleball for a strike - why this guy pitches in the majors I have no idea). 

Remember how this thread started though. Someone suggested that Strasburg going forward, or some other pitcher going forward, could be 2 or 2.5 runs (they might have even said 3) better than league average, and I simply said, “Nonsense - no pitcher can be that good going forward.” That’s all.

"which won’t happen of course because he is not going to be allowed to pitch that long), then Marcel will give him a projection of around 16 or 17 in ERA.  Where’s the 6.00 cap?”

You just said where it is. The 6.00 (or so) cap is that he won’t be allowed to pitch that long. The cap is basically about where replacement level is, because if you are below that you won’t pitch for any length of time in the majors.

I just looked it up here http://www.tangotiger.net/marcel/

The worst ERA Marcels shows 2008-2010 is 6.12 - and it is the only one over 6 for those 3 years-with a pool of replacement level players, you’re not going to see a Marcels projection much worse that 6

Strange, I checked my Haeger projection and he’s not horrible.  4.85 updated and 4.89 preseason.  Well below average for a pitcher in Dodger stadium, but nowhere near where a ERA cap might be.  You know this, but pitchers are usually going to look bad from a scouting perspective when they are pitching poorly.  Haeger’s knuckleball probably looked pretty good when he struck out 12 Marlins in 6 innings for his first start.

I had Suppan at 5.52 before the Brewers released him.  I’ll have to knock 1.5 runs off that now that he’s with Duncan and LaRussa

I’m still reading through all of the comments, so forgive me if this has already been mentioned…

The experimentalist in me would say: No. There is no cutoff point above or below which one should cut off their forecasts of a given metric.  However, error bars (+X/-Y) should certainly be given along with the measurement.  These would be asymmetric errors when you are near the extrema, but could realistically be symmetric away from the extrema.

These errors could probably be simplistically estimated by examining the spread in performance as a function of forecast.

"There is no cutoff point above or below which one should cut off their forecasts of a given metric.”

So you are OK with a forecast of a 1.00 ERA for a starting pitcher if one happened to have had a sample ERA near zero for the last 3 years or so?

Rally, sure, when a pitcher is pitching badly, he looks terrible from a scouting perspective and when he is pitching well, he looks great.  That is especially true for a pitcher that does not throw hard and relies on something else.

MGL/58

Absolutely...provided that the appropriate error bars are given as well.  If thats what your forecasting system gives you, then thats what it gives you.  It’s either a good system or it’s not, and that’s defined by the likelihood that given a particular forecast, a players performance winds up in line with that forecast.  If the forecasts seem “wrong” at the extremes, then either the model does a bad job of handling the extremes, and the error bars should be larger in that region, or you should determine why your model fails near the extremes.

"If the forecasts seem “wrong” at the extremes, then either the model does a bad job of handling the extremes, and the error bars should be larger in that region, or you should determine why your model fails near the extremes.”

We don’t have any idea if the model fails at the extremes.  Because no pitcher has ever been as good as the hypothetical case.

The closest we have is Pedro, and in the 4 years where his projections were the best they seemed to do quite well.

I doubt we’ll live long enough to see a MLB pitcher have an ERA close to zero for 3 straight years.  If we do, sure, I’ll take a projection of 1.00 for year 4.

MGL:
While your hypothetical might happen once every several hundred billion years, I don’t understand why we should worry about it.  As best I can tell, only Pedro and Maddux have had forecasts significantly belowe 62.5% in the past 30 years.  And they hit those forecasts about half the time.  So where is the evidence that any pitcher has ever been projected better than the “known” (by you) limits of human pitching potential? 

In real life, we dont’ face your hypothetical.  We are very occasionally faced with projections that tell us we are looking at one of the 3 or 4 best pitchers of all time—or someone who is extremely good but also very, very lucky. 
Is there any actual evidence that your method of arbitrarily capping the projection is more accurate?

Yes, in practice it makes almost no difference.  A Marcel type model does nor work well at the extreme extremes by definition - by that, I mean that the “regression methodology” is an approximation.  It assumes that there is no physical limitation as to how good a pitcher can be, which we know is wrong.  It assigns a finite (albeit infinitesimally small) chance that there are pitchers who never allow a hit, walk, or run. So we KNOW that it does not work well at the extremes.  So let’s be accurate in what we say:  Yes, in practice it makes almost no difference.  Yes, the conventional (Marcel-like) projections systems don’t work well at the extremes.  What those physical limits are, be it 62.5% or 55%, no one knows.  My opinion/feeling is that it is around 62.5% but if it is 58% I won’t kill myself.

"My opinion/feeling is that it is around 62.5% but if it is 58% I won’t kill myself.”

Good to know you aren’t on a ledge somewhere.

It’s a good topic, and an interesting though experiment.  I feel I’ve learned something here.  As to where the limits are, I think they are a bit less than what Pedro did.  But you have convinced me that somewhere around there should be a limit.

I wonder if Strasburg will wind up exceeding what we’ve seen so far.  So early to tell, and of course we have to hope he can stay healthy and keep this kind of stuff.

He’s so far throwing 97.5 average on his fastball, checking Fangraphs leaders back to 2002 the best average for a fulltime starter is in the 96 MPH range shown by guys like Verlander, King Felix, and Ubaldo.

If Strasburg can maintain this kind of stuff, I would put his limit around what we get from hard throwers like Bard, Broxton, and Zumaya as far as K and hit suppression goes, with much better control.  Except he’d be doing for more than an inning at a time.  No idea if he will, but that’s where I’d put his upside.

I would like the approach using FIPS or something similar to determine the
potential lowest/practical ERA limit.

Using FIPS:

Let’s use runs allowed =(homeruns*13+bb*3-strikeouts*2)/ip + 3.00 as an estimate for a 4.00 run league.

Let’s say that we determine that per 9 innings, the lowest you can get with a pitcher who is a great strikeout pitcher is .50 hr per 9 ip and 2.0 bb per 9 ip. That’s pretty close to the lowest we see.

Given that, if we state that maintaining a strikeout rate greater than 11 ks per 9 ip (or around 30% of all batters) is not possible, even given a 100 mph fastball with great control/command, we get a 1.94 runs allowed, or 2.06 runs below average.

I’m talking what someone’s true talent is, not what is observed. If you want to project next year, there will be some regression. If I regress 1/6 to the league average of 4.00, I get a runs allowed of 2.28, or 1.72 runs allowed, or .57 of league average.

You can change the assumptions obviously, but you are going to wind up with that as pretty close to the limit, which is close enough so MGL won’t kill himself.

In order to get to 1.00 runs allowed you have someone striking out major league batters a little over 15 times per game. I don’t think anyone believes that there is a human capable of doing that.

Certainly not a human.  But some species of Marmol might be able to, for an inning at a time at least.

I was playing around with some assumptions using baseruns formula and I get about the same thing, about .50 to .55.  I was assuming a strikeout rate around Broxton’s career rate and a walk rate about half the league average, babip about .290, flyball rate around .25 and average HR/FB.

I wonder how good these assumptions work once we consider strategy changes.  Bunting and stealing often make sense against such a dominant pitcher, and strategies that reduce run expectancy against a normal pitcher might maximize the few baserunners this guy allows.

I’d agree that it’s very unlikely that we ever see someone who is less than 50% true talent (more unlikely still that there’s someone we can project to be 50% the next year since someone so close to perfection can only get worse) but what makes us think that there’s a limit of human ability here and that the chance of someone better is 0% rather than something really really small?

Pedro and Maddux are so different that it strikes me as unlikely that they are the limit of what’s possible from a human.  Why couldn’t there be someone who combines Maddux’s best attributes with Pedro’s and is substantially better than either?  Heck, let’s make them 7 ft. tall while we’re at it and equally capable with both their left and right arms.  This person is obviously extraordinarily unlikely but not impossible.

Isn’t it easy enough to imagine a lefty with Pedro’s stuff?  Wouldn’t they have been even better?  There are fewer lefties so such a person is less likely but is there any obvious reason why they couldn’t exist?

Here’s my top 50 projections with sample size >= 950

last    first    season class_cd  size    ip      era    woba
Martinez, Pedro    2002    MLB    1445    173    1.88    0.223
Martinez, Pedro    2001    MLB    1877    222    2.03    0.230
Darvish, Yu        2010    Int    1620    189    2.14    0.235
Martinez, Pedro    2003    MLB    1515    178    2.20    0.240
Darvish, Yu        2009    Int    1616    186    2.34    0.246
Maddux, Greg       1999    MLB     987    253    2.37    0.249
Martinez, Pedro    2004    MLB    1528    175    2.46    0.252
Martinez, Pedro    2000    MLB    1501    223    2.46    0.252
Strasburg, Stephen 2010    MLB     990    128    2.49    0.253
Brown, Kevin       1999    MLB    1032    258    2.56    0.256
Uehara, Koji       2001    Int    1057    159    2.61    0.262
Santana, Johan     2006    MLB    1848    213    2.61    0.262
Brown, Kevin       2000    MLB    1740    257    2.64    0.261
Brown, Kevin       2002    MLB    1619    185    2.66    0.263
Kim, Byung-Hyun    2004    MLB    1022    115    2.68    0.262
Prior, Mark        2004    MLB    1619    183    2.69    0.263
Schilling, Curt    2004    MLB    1937    224    2.70    0.267
Santana, Johan     2005    MLB    1664    189    2.72    0.265
Johnson, Randy     2003    MLB    2231    255    2.72    0.266
Schmidt, Jason     2005    MLB    1906    215    2.72    0.264
Darvish, Yu        2008    Int    1413    158    2.75    0.264
Lowe, Derek        2003    MLB    1326    152    2.75    0.269
Johnson, Randy     2002    MLB    2234    251    2.76    0.266
Brown, Kevin       2001    MLB    2150    243    2.78    0.268
Johnson, Randy     2001    MLB    2263    256    2.80    0.269
Carpenter, Chris   2010    MLB    1173    165    2.81    0.270
Naruse, Yoshihisa  2009    Int    1239    140    2.82    0.270
Millwood, Kevin    2000    MLB    1430    209    2.83    0.270
Clemens, Roger     1999    MLB     961    233    2.83    0.266
Santana, Johan     2007    MLB    1959    224    2.83    0.272
Martinez, Pedro    1999    MLB     951    234    2.84    0.269
Uehara, Koji       2008    Int    1078    125    2.84    0.274
Johnson, Randy     2005    MLB    1867    211    2.84    0.271
Lincecum, Tim      2010    MLB    1977    222    2.85    0.269
Schilling, Curt    2005    MLB    1919    219    2.85    0.273
Perez, Odalis      2003    MLB    1226    181    2.89    0.275
Uehara, Koji       2005    Int    1605    185    2.91    0.277
Dotel, Octavio     2003    MLB     964    106    2.91    0.271
Lewis, Colby       2010    MLB    1587    188    2.91    0.274
Soriano, Rafael    2004    MLB     982    110    2.94    0.275
Maddux, Greg       2001    MLB    2164    243    2.95    0.276
Iwakuma, Hisashi   2009    Int    1136    126    2.95    0.274
Schmidt, Jason     2004    MLB    1741    194    2.95    0.274
Uehara, Koji       2004    Int    1657    189    2.96    0.278
Matsuzaka, Daisuke 2007    Int    1630    183    2.96    0.276
Beckett, Josh      2003     A+    1048    116    2.97    0.275
Smoltz, John       2006    MLB    1275    145    2.97    0.279
Maddux, Greg       2003    MLB    1975    223    2.97    0.277
Brown, Kevin       2004    MLB    1308    148    2.98    0.278
Martinez, Pedro    2005    MLB    1821    204    2.99    0.277

Jeez, Brian, now MGL is back out on the ledge!

J.Cross:  why do you assume a LH Pedro would be better?  He would enjoy the platoon edge less often, so it seems likely he would be a bit worse.  (In fact, it’s rather suprising that there are any LH starting pitchers at all.)

Guy, I think a RHP needs 2-3 mph more than a LHP to have the same K-rate (I do have a data table to support this).  That’s all I’m going on.  Most of the hardest throwers are RHP’s which I interpret as a result of the fast that there are fewer lefties (in the population) so hard throwing lefties are very rare.  Most of the softest throwers are lefties which I interpret as meaning that lefties have an inherent advantage and can get by on less stuff (this would also explain why lefties are overrepresented among pitchers compared to their population rate).  I haven’t looked at movement and location though and I could easily be interpreting this wrong.

Brain, I assume those are ERA’s given the player’s park and defense?  Any idea what they would look like in terms of runs below league average in a neutral context?

Don’t forget what I said about defense.  Any pitcher with an above average ERA is likely (greater than 50% chance), on the average, to have above average defense behind him, such that his context-neutral ERA would be a tad higher, again, on the average.

Those are from the park neutral table, but it does not account for the quality of defense behind the pitcher. That’s on my to-do list.

I wanted you guys to get a feel for the distribution of the projections.

Also, I do not directly calculate ERA - it’s not a weighted mean of past ERAs, but instead calculated as a non-linear function of the projected wOBA allowed. the wOBA projection is, like a batter’s projection, calculated from H,DO,TR,HR,BB,HP,SO, which have all been derived by way of weighted means/regression/aging etc.

When looking at a player’s page on the THT Forecasts, there’s a box for actual raw stats, and just below that the seasonal MLE’s. For MLB pitchers, the difference is luck, sequencing and park factors which are used to calculate MLEs. So Ubaldo Jimenez had an actual 3.47 in 2009, but MLE of 2.97 - to be expected as he plays his home games in an extreme pitcher’s park. In the opposite direction, Jake Peavy had an actual 2.85 in 2008, MLE of 3.52. Jimenez is bucking the trend this year, with an actual of 1.60 and MLE of 2.52, showing a lot of luck.

Brain, what is the league average ERA that those numbers are based on, and is it the same for each league (NL and AL).  A while back I expressed my frustration at people presenting ERA’s without telling us the league average ERA.  Kind of meaningless aren’t they, other than to compare one pitcher to another?  I mean we can get some idea as to how good a pitcher is supposed to be, but if I see a projected park-neutral ERA of 2.76, I really don’t know if that is 2 runs better than league average or 1.5 runs.  Quite a spread…

There were all generated at the same time on the same baseline, mean ERA of 4.45 based on a mean wOBA of .332.

Future plans include determining the baseline as the weighted mean of the league totals using the same seasons and weights that the player’s projection was based on.

So all these guys were projected as at least 1.5 earned runs better than league average?

Martinez, Pedro 2002 MLB 1445 173 1.88 0.223
Martinez, Pedro 2001 MLB 1877 222 2.03 0.230
Darvish, Yu 2010 Int 1620 189 2.14 0.235
Martinez, Pedro 2003 MLB 1515 178 2.20 0.240
Darvish, Yu 2009 Int 1616 186 2.34 0.246
Maddux, Greg 1999 MLB 987 253 2.37 0.249
Martinez, Pedro 2004 MLB 1528 175 2.46 0.252
Martinez, Pedro 2000 MLB 1501 223 2.46 0.252
Strasburg, Stephen 2010 MLB 990 128 2.49 0.253
Brown, Kevin 1999 MLB 1032 258 2.56 0.256
Uehara, Koji 2001 Int 1057 159 2.61 0.262
Santana, Johan 2006 MLB 1848 213 2.61 0.262
Brown, Kevin 2000 MLB 1740 257 2.64 0.261
Brown, Kevin 2002 MLB 1619 185 2.66 0.263
Kim, Byung-Hyun 2004 MLB 1022 115 2.68 0.262
Prior, Mark 2004 MLB 1619 183 2.69 0.263
Schilling, Curt 2004 MLB 1937 224 2.70 0.267
Santana, Johan 2005 MLB 1664 189 2.72 0.265
Johnson, Randy 2003 MLB 2231 255 2.72 0.266
Schmidt, Jason 2005 MLB 1906 215 2.72 0.264
Darvish, Yu 2008 Int 1413 158 2.75 0.264
Lowe, Derek 2003 MLB 1326 152 2.75 0.269
Johnson, Randy 2002 MLB 2234 251 2.76 0.266
Brown, Kevin 2001 MLB 2150 243 2.78 0.268
Johnson, Randy 2001 MLB 2263 256 2.80 0.269
Carpenter, Chris 2010 MLB 1173 165 2.81 0.270
Naruse, Yoshihisa 2009 Int 1239 140 2.82 0.270
Millwood, Kevin 2000 MLB 1430 209 2.83 0.270
Clemens, Roger 1999 MLB 961 233 2.83 0.266
Santana, Johan 2007 MLB 1959 224 2.83 0.272
Martinez, Pedro 1999 MLB 951 234 2.84 0.269
Uehara, Koji 2008 Int 1078 125 2.84 0.274
Johnson, Randy 2005 MLB 1867 211 2.84 0.271
Lincecum, Tim 2010 MLB 1977 222 2.85 0.269
Schilling, Curt 2005 MLB 1919 219 2.85 0.273
Perez, Odalis 2003 MLB 1226 181 2.89 0.275
Uehara, Koji 2005 Int 1605 185 2.91 0.277
Dotel, Octavio 2003 MLB 964 106 2.91 0.271
Lewis, Colby 2010 MLB 1587 188 2.91 0.274
Soriano, Rafael 2004 MLB 982 110 2.94 0.275
Maddux, Greg 2001 MLB 2164 243 2.95 0.276
Iwakuma, Hisashi 2009 Int 1136 126 2.95 0.274

Not including those minor league pitchers like Darvish and Uehara and the relievers like Soriano and Dotel, I did a quick check of how many of these performed less or greater than 1.5 ER better than league average.  The count is:

10 better

16 no

2 ties (around 1.50 runs better than average)

And I think most of the yes’s were Pedro.

I think that the mean collective performance was way above 1.5 runs better than league average.  IOW, the mean performance of all of these pitchers, no necessarily weighted by IP, was way worse than their mean collective projection.

Doesn’t mean much, but I thought it would be a fun exercise…

And I ignored the 2010 projections as well.

By the same logic whereby guys with great ERAs are (on average) lucky, guys with great Oliver predictions are probably (on average) seen more favorably by Oliver than Marcel or Chone.  An example of winner’s curse. 

This would be enough to explain why the best projections made by any system are likely to, as a group, be too optimistic.  We don’t need a breakdown of the normal distribution to explain that. 

But, mostly, there aren’t enough data points here.

Maybe we could pose the question this way.  If Pedro really were just a 60% pitcher during his peak, how unlikely would his real numbers be?  Roughly how many 60% pitchers would there need to have been for it likely that we’d see a peak like Pedro’s? 

This is a little tricky to answer because unlike batting average, ERA isn’t a simple binomial thing - but I remember correctly Tango figured out how to treat wOBA as a binomial so maybe we can do that here as well?

"Maybe we could pose the question this way.  If Pedro really were just a 60% pitcher during his peak, how unlikely would his real numbers be?  Roughly how many 60% pitchers would there need to have been for it likely that we’d see a peak like Pedro’s?”

That is not going to tell you that much.  What if there were a 10% chance that a 60% pitcher would pitch like a 50% pitcher in 3 straight years, and there were 5 60% pitchers and one made it?  Big deal.  How about 3 pitchers and one made it?  Again, big deal.  20 such pitchers and 3 made it?

We are dealing with such small sample sizes and we are cherry picking something we know to be true (basically “publishing bias")…

Agreed, but I was thinking that the chances of a 60% pitcher pitching like Pedro did over a 5-year span might be much smaller than that.  Granted, given that we’ve selected Pedro Martinez after the fact, as you point out, this chance might have to be *very* small indeed to make you rethink the ~60% wall of human limitations theory. 

Really, I’m suggesting that the wall could be significantly further along than 60% and that it would take such a freakish individual to approach it that we simple haven’t seen this guy yet.

If the point is that to project someone better than 62.5% they have to have a skill set the likes of which we haven’t yet seen (or maybe have only seen once), well, I’ll more or less buy that.

Right, the whole thing is about the skill set.  If we see a pitcher who throws 102 with the control of a Strasburg or a Pedro, then we probably have to rethink the 62.5% limit.

I see a pitcher like Strasburg and I see a 62.5% pitcher.  I’ve been watching baseball with a keen analytical eye for almost 25 years.  Honestly, when I see a 50% pitcher, I’ll know it, although I doubt I’ll see one in my lifetime.  And even then, if you are talking about a projection for a year (as opposed to tomorrow), the chance of injury or some minor arm problem or just a loss in velocity or control for no apparent reason is significant enough that you probably cannot extend or extrapolate that kind of skill set beyond some short period of time.

In fact, that is one important thing that we never talk about.  Given a certain chance of injury or just some unnamed or unknown arm trouble (or head trouble), shouldn’t a short-term projection (like one start or one week) for a pitcher be better than a long-term one (like one year)?  I mean we expect Strasburg to be the same pitcher we think he is in his next start, right, with 95% or 99% certainty.  But for next year, what is the certainty that he is the same pitcher with the same velocity and command, including the chance of a serious injury such that he doesn’t even pitch?  80%?  90% at the most?

I was getting the next batch of projection updates ready last night.  Right now I don’t have a single starting pitcher projected as better than .70 of league average.  Three guys were below .75, and Strasburg vaults into the top ten.

Wherever the lower bound limit is, CHONE doesn’t think any current pitcher is particularly close to it.  We’ll see how it views Strasburg if his rate stats still look like this after a year and a half of major league pitching.

Call it signature significance for Strasburg.  Last projection update had little to go on, just his outstanding AA and AAA starts.  My system does not know he throws 100, was the #1 overall pick, or cares at all what he did in college.  It looked only at those minor league starts, and while outstanding, considers that he might have been lucky and regresses him to a typical AAA pitcher.

I love Chone (the projection system that is), but…

You should be regressing Strasburg toward the typical major league pitcher and not AAA.  Once a player is promoted, he is a different animal, due to what the scouts/teams think of him.

Before the season starts, you even have to specify whether your projections are assuming that a player makes the majors or just an estimate of his true talent not knowing whether
he does or does not.  It makes a big difference!

And of course you should be regressing pitchers toward the typical pitcher whose fastball is the same speed. I just started doing that this year.  A pitcher who has Strasburg’s numbers after 5 starts who throws 95-100 is different from a pitcher who has those same numbers after 5 starts and throws 85-90, I am sure you will agree.

He’s being regressed to major league pitchers now.  Before his debut he was being regressed to a AAA pitcher.  That will make a big difference when people see the projection improvement for just one month.

I agree on fastball speed.  But I’m not at that implementation stage yet.

"He’s being regressed to major league pitchers now.  Before his debut he was being regressed to a AAA pitcher.”

Right, it’s a funny thing.  A pitcher’s projection can change radically as soon as he throws one pitch in the majors!

If we used Bayes, we wouldn’t need to set the caps.

But, because we are using an approximation (regression), we DO need to put in some caps, because we always break at the extremes. 

Our job is to figure out those extremes.  62.5%?  55%?  30%?

That’s where the question should be.

"Right, it’s a funny thing.  A pitcher’s projection can change radically as soon as he throws one pitch in the majors!”

I don’t have it set up quite to that extreme.  I’ve got a sliding scale for guys who get a cup of coffee in the majors.  For about the first 50 innigs or so they are gradually regressed to a better mean, until my system sees them as having earned MLB status, and then they are regressed to MLB pitcher from there on.

So one pitch won’t radically alter the projections, but for the first 50 innings or so the projection will move more than it does for other pitchers.

ditto what Rally said.

My regressions for each player are based on the pct of the stats used in the projection that were compiled at each level. So, for a player who’s stats were 5% MLB, 55% AAA, 40% AA I take those same percentages to construct a weighted mean.

Brian, I think that is wrong.  The idea of regressing toward the major league mean is that you now have a pitcher who is deemed to be major league material.  It should have little to do with the percentage of opportunities at each level.  That is especially true for AA and A. If a pitcher is promoted from single A to the majors, like Porcello last year, it would be ridiculous to regress his stats toward the single A mean.  Maybe ridiculous is not the right word, but if you do, you are never going to get a good projection for a player.  And that can’t be right.  That is especially true if a team promotes a player from single A to the majors. In most of those cases, they love the player.  Going from AAA to the majors, you can at least make the argument that they might just be giving him a shot or he is a temporary fill-in.  Not so of course for A or even AA players who get promoted to the show.

Imagine a single A pitcher throws for a half a year in single A and has a 1.90 ERA and great peripherals in 80 IP.  Let’s say that he is a great prospect and gets promoted to the bigs.  If you regress him toward the single A mean, he is still going to have a horrible projection since the mean MLE ERA of a single A pitcher is probably like 6.50.  So, again that can’t be right.

If nothing else you have to regress him toward the average numbers of a single A pitcher who indeed gets promoted to the majors.  Certainly not the mean of the average single A pitcher. That makes no sense.

Rally, your method is wrong too.  Imagine that you have two pitchers, each with the exact same stats in AAA, and say a 3.50 ERA.  One gets promoted and the other one no one pays attention to.  There are lots of pitcher in the minors who compile decent or even very good stats that no one pays attention to and/or do not get promoted.  Especially the low minors.

Anyway, one gets promoted and the other does not.  You would have both pitchers projected the same.  At least until the one gets some extended time in the majors.  That can’t be right.  The one who gets promoted is likely the better pitcher even though they both have the same exact (context-neutral) minor league stats.  Surely you can’t have the same projection for both pitchers. 

The way to do that is to regress toward different numbers.

Both Rally and Brian, you have to remember what we are regressing towards (and why).  We are choosing the smallest possible population of pitchers that these pitchers belong to, without too much emphasis on the stats.  A pitcher that gets promoted and one who doesn’t are part of two different populations.  You cannot regress a pitcher (one who gets promoted, even for one game) toward the mean of a different population (pitchers who do AND don’t get promoted). You can, of course, and it won’t kill you, but it is much better not to.

It is wrong to regress a pitcher who gets called up for a cup of coffee or to fill in after a bunch of injuries to the same mean that Tim Lincecum and Johan Santana regress to.  That’s why I do it the way I do.  A player has to prove himself a bit, show me he belongs.

At the risk of repeating myself, it is also wrong to regress a pitcher who gets called up for a cup of coffee to the same mean of a similar pitcher (stat-wise) who does NOT get called up. In any case, it is easy to determine what to regress that pitcher to, isn’t it?  Just figure out the mean of all pitchers who get called up for at least one game.  I guarantee that will be better than the mean of all AAA (or whatever minor league he is called up from) pitchers.

It is not a big deal either way, and I am not sure that your sliding scale is not the best way to go, but surely you have to change the mean the second the pitcher is called up, as I explain above.  The problem with the sliding scale being tied to a pitcher’s tenure in the majors is that whether he “proves himself” or not is too correlated with his major league stats.  And as you know, using a population significantly determined by stats to establish the mean to regress toward is pretty much a no-no…

"Just figure out the mean of all pitchers who get called up for at least one game.”

But that’s not his population.  His population—at that time—is guys who get called up for ONE game.  It’s not obvious how much better such pitchers are than guys not yet called up, if at all. (And even if you could calculate the 1-gamers’ performance, it would be a biased sample since guys sent down after one game performed (I assume) quite badly.) And I think that when Rally/Brian say “regress to AAA or AA,” they’re effectively regessing to players at that level who DO some day perform in MLB. Isn’t that what the MLEs do?

Say AAA is level 4, and MLB is 5.  A guy gets a cup of coffee and a few innings, he’s at 4.1.  A few more and he’s at 4.2, until he pitches enough to top out at 5.

So the cup of coffee guy gets regressed to a higher mean than his AAA twin.  But that is after he pitches the way the CHONE system works.  If you think he should be regressed to 4.1 or something as soon as he’s called up, you’ve got a decent argument, but that would be too much work to keep track of for me.

"But that’s not his population.  His population—at that time—is guys who get called up for ONE game.”

Right.  What I meant was you simply find out the mean in their first major league game of all pitchers who got called up from AAA (or whichever minor league).

“And I think that when Rally/Brian say “regress to AAA or AA,” they’re effectively regessing to players at that level who DO some day perform in MLB. Isn’t that what the MLEs do?”

That is actually a very good point.  Which is essentially doing what I say above, since that is how the MLE’s are computed in the first place - from a few games in the major leagues either the same year they get called up or the next year.  And even those are going to be optimistic since they include players who play more than one game in the majors.

Rally, right, as I said, it really doesn’t make much difference one way or another, especially, as Guy says, since our MLE’s are really based on players who make the majors anyway, and thus don’t really apply to players who don’t make the majors, which is an occasional (and legitimate) criticism of MLE’s (that we don’t really know how a minor league player would do in the majors if he were forced to play there as opposed to deemed worthy to play there).

All in all, the best way to establish the means is to simply look at how each group of pitchers actually performed in the majors, historically.  You might have sample size problems though as you have to do it on a game by game basis. IOW, you can look at the mean stats of all AAA pitchers their first game in the majors.  But, after that, you can’t look at the means of the first 2 games in the majors since that is a biased sample (only those that have a lucky first game will pitch a second game).  You have to look at the second game separately.  Then the third, etc.  So, to establish your means, you can only use one game at a time, which is quite interesting actually…

"that we don’t really know how a minor league player would do in the majors if he were forced to play there as opposed to deemed worthy to play there”

I think we have a pretty good idea.  For one, you can set a floor.  That would be pitchers hitting. 
1. I say that anyone worthy of be kept in professional baseball, at any level, as a hitter is no worse a hitter as the average MLB pitcher.

2. We know that AAA hitters are better than AA, and so on down the line to Short season A being better than the rookie leagues. 

If your MLE adjustments put the level of play in each league at levels that do not violate those two constraints, you shouldn’t be all that far off.

I don’t think that was what I meant. What I meant was that the MLE coefficients only apply to players who are eventually deemed worthy of being in the major leagues.  They do not necessarily (and probably don’t) apply to players who are NOT deemed worthy.

IOW, let’s say we have 2 players in AAA with the exact same stats and their wOBA MLE were .300. If Player A gets promoted, we expect him to hit .300 or so, and he probably will (assuming that the .300 MLE was a projection).  If Player B is not deemed worthy of being promoted but we force him to be promoted, he probably will hit less than .300 even though he has the exact same historical profile (and age and everything else) as Player A.  That is what I meant.

I don’t do projections, but I do like to look at data.
These are raw totals. The aggregation is for the season totals for pitchers who pitched in AAA and in the specified # of MLB games that year
2006-2009:
MLB Num AAA MLB
Gms Plrs ERA ERA
0 1826 4.79 none
1 55 4.71 10.01
2 59 4.46 9.09
3 67 4.07 8.37
4 70 4.21 6.24
5 66 3.81 5.89
6 61 3.65 6.13
7 50 3.87 5.85
8 61 3.90 6.11
9 51 3.52 5.94
10 47 3.73 6.16
11 46 3.84 5.44
12 43 4.09 5.89
13 32 3.61 5.80
14 44 3.63 5.55
15 43 3.99 5.30
16 39 3.54 4.87
---starting here, more TBF in majors than AAA---
17 25 3.54 5.48
18 25 3.40 5.21
19 33 3.93 5.05
20 38 3.28 4.98
21+ 597 3.02 4.60

The pitchers who appear in exactly 1 game in a season were 27 or older the majority of the time in the season in which they pitched exactly one MLB game
[before and after seasons with age and MLB ERA]:

Jesus Colome 2005 (27) 4.57; 2007 (29) 3.82
r.a. Dickey 2005 (30) 6.67; [2007 (32) 3.72 in AAA only], 2008 (33) 5.21
Chan Ho Park 2006 (33) 4.81; 2008 (35) 3.40
mike smith DNP in majors immediately before or after (age 28 in his 1 G MLB season)
travis smith 2005 (33) 6.75; DNP in majors after
Luke Hudson 2006 (29) 5.12; DNP in majors after
Jon Adkins 2006 (28) 3.98; 2008 (30) 2.45
Chad Mabeus DNP in majors immediately before or after (age 27 in his 1 G MLB season)
Tim Harikkala DNP in majors immediately before or after (age 35 in his 1 G MLB season)
Elizardo Ramirez 2007 (24) 7.71; DNP in majors after
Joselo Diaz DNP in majors immediately before or after (age 28 in his 1 G MLB season)
Clint Nageotte DNP in majors immediately before or after (age 25 in his 1 G MLB season)
Mike Gosling 2005(24) 4.45; 2007 (26) 4.91
Jose Capellan 2007 (26) 5.54; DNP in majors after
Alex Serrano DNP in majors immediately before or after (age 27 in his 1 G MLB season)
Enrique Gonzalez 2006 (23) 5.67; 2008 (25) 10.80
Juan Lara 2006 (25) 1.80; DNP in majors after
Aaron Rakers DNP in majors immediately before or after (age 30 in his 1 G MLB season)
Neal Musser 2007 (26) 4.38; DNP in majors after
Jeff Bajenaru 2005 (27) 6.23; DNP in majors after
Juan Morillo 2007 (23) 9.82; 2009 (25) 22.50
Anderson Garcia DNP in majors immediately before or after (age 26 in his 1 G MLB season)
Anthony Lerew 2005 (22) 5.63; 2007 (24) 7.71
Kevin Barry 2006 (27) 5.61; DNP in majors after
Nate Field 2006 (30) 4.00; DNP in majors after
Phil Barzilla DNP in majors immediately before or after (age 27 in his 1 G MLB season)
Lee Gronkiewicz DNP in majors immediately before or after (age 28 in his 1 G MLB season)
Jeff Fulchino DNP in majors immediately before or after (age 26 in his 1 G MLB season)
Jim Johnson 2006 (23) 24.00 [1 G]; 2007 (24) 9.00 [1 G]; 2008 (25) 2.23
Charlie Zink DNP in majors immediately before or after (age 28 in his 1 G MLB season)
Michael Bowden DNP in majors before; 2009 (22) 9.56
Ty Taubenheim 2006 (23) 4.89 2007 (24) 9.00 [1 G]; 2008 (25) 3.00 [1 G]; DNP in majors after
Pat Misch DNP in majors before; 2007 (25) 4.24
Francisley Bueno DNP in majors immediately before or after (age 27 in his 1 G MLB season)
Abe Alvarez 2005 (22) 15.43; DNP in majors after
Tim Stauffer 2005 (23) 5.33; 2007 (25) 21.13
Mike Schultz DNP in majors immediately before or after (age 27 in his 1 G MLB season)
J.A. Happ DNP in majors before; 2008 (25) 3.69
Jae Kuk Ryu 2007 (24) 7.33; DNP in majors after
Zack Segovia DNP in majors immediately before or after (age 24 in his 1 G MLB season)
Drew Carpenter DNP in majors before; 2009 (24) 11.12

The more a player is allowed to pitch in the majors, the better he performs, both in AAA and the majors. I see no magic threshold at which a player starts to regress toward a Major League mean instead of toward a AAA mean. (A similar pattern is true for hitters.)
I think it would be instructive to compare the projection systems which make different assumptions about “regression”, and look at bottom quartile players only.

You can’t look at pitchers who pitched EXACTLY one game.  If they pitched exactly one, chances are, they were not good enough to pitch a second (or they had bad luck in the first outing).  According to Joe, 10.01 ERA for those guys.  No surprise.

What you need to do is look at the pitchers’ FIRST game. You are not allowed to look at future outcomes beyond that one.

"What you need to do is look at the pitchers’ FIRST game.”

Exactly.  Of course you probably have an advantage for a pitcher his first game in the majors because no one has seen him, so his numbers are probably a little less than his true talent.

Of course you are going to get the numbers that Joe Arthur gets when you look at players who pitch exactly 1 game, 2 games, etc.  As Tango says, if a AAA pitcher comes up and pitches one game and gets killed and he is not a great prospect, he never pitches again. If he is allowed to pitch 2 games and never again, he probably did poorly in either game 2 or games 1 and 2 (and for some reason was allowed to pitch game 2).  Etc.

As Tango and I said, if you want to know what to regress a AAA pitcher to once he gets called up and you know he is going to pitch at least one game, you can simply look at all first game performances by all AAA pitchers. If you want to know what to regress to on a pitcher’s 2nd start, simply look at the average ERA (or whatever stat) of all pitchers who pitched at least 2 games in the majors. Etc.It is as simple as that.

There is no need to do any kind of “sliding scale” or what have you, although that might work, sort of accidentally, since obviously pitchers who are allowed to pitch 2 games are going to be slightly better, true talent-wise, than all pitchers are called up for at least one game, pitchers who are allowed to pitch 3 games are slightly better than all pitchers who pitch at least 2 games, etc.

Interestingly, if you look at Joe’s numbers, you can also see that pitchers who only get one or two games (or 3 or 4) in the majors not only got unlucky in their major league starts but also have relatively high ERA’s in the minors, suggesting that they were not very good in the first place.

If you look at the ones who get lots of starts in the majors, you can see that they have very good ERA’s in the minors, and presumably most of them are great prospects and very good pitchers.

So basically, the fewer starts you get in the majors, the unluckier you got in those starts AND the worse pitcher you are, all on the average of course.

Which is exactly as we would have expected.

So, to a large extent a “sliding scale” is quite appropriate.  The only quibble I have with that method is that there should be a gap between a player who does not even get one start in the majors (which is all minor league pitchers before they get called up) and one who gets his very first start, in terms of the mean of the population.

So, in terms of ERA, the mean to regress toward might be 6.00 for all AAA pitchers before they get a start in the majors, 5.50 for their first start, then 5.40 for the second start, 5.30 for their 3rd, etc.

Let me recap here - Strasburg is a problem case for projection because he has progressed so fast that he doesn’t have much performance data. Thus his projection is heavily influenced by the data shrinkage strategy chosen. As part of this discussion (which population to consider for the prior expectation in an Empirical Bayes - type approach), these statements were made, probably with the intent being the projection of prospects:

MGL #84 - “A pitcher’s projection can change radically as soon as he throws one pitch in the majors! “
MGL #90 - “It is also wrong to regress a pitcher who gets called up for a cup of coffee to the same mean of a similar pitcher (stat-wise) who does NOT get called up. In any case, it is easy to determine what to regress that pitcher to, isn’t it?  Just figure out the mean of all pitchers who get called up for at least one game.  I guarantee that will be better than the mean of all AAA ... pitchers.”
Guy #91 “...guys who get called up for ONE game.  It’s not obvious how much better such pitchers are than guys not yet called up, if at all. (And even if you could calculate the 1-gamers’ performance, it would be a biased sample since guys sent down after one game performed (I assume) quite badly.)”

I didn’t provide data about prospects specifically, or about players’ performance in their first game in the majors ever. I only have 4 years of minor league data in my database right now, so my data was just performance of AAA pitchers who pitched 1 major league game that particular year. Some pitched a little in the majors in prior seasons, a few pitched a lot. As a group they are not generally “prospects” as I would use the term, usually being 26 or older.

I assume we’d agree that the individual player’s actual talent does not change because he switches levels, prospect or non-prospect.  I think MGL’s claim (and he may be thinking of projecting prospects specifically [with less professional history behind them]) was that the information that the pitcher reached the majors, no matter how much or how well he does there, is by itself significant additional information , and justifies changing the projection substantively.

I think that the data I posted suggests these unsurprising conclusions - if pitcher A pitches more in the majors than pitcher B, on average he pitches not only more but better in the majors. He also pitches better in AAA. And I think this is a predictive difference between MGL’s approach and Brian and Rally’s. We don’t have to rely on comparing actual MLB performance to projected MLB performance to tell which of them is right about these borderline players. If MGL is right that reaching the majors should change the estimate of a players’ talent, it should show up in their future AAA performance as well. Usually we’ll have a lot more observational data for these players at AAA.

Using Joe’s data, the average for all pitchers who get even one MLB game is 5.70, or -1.30 vs. league average.  This strengthens my long-held skepticism that replacement level pitchers are as good as -1.0 R/G.  If we exclude those who get more than 20 games, who mostly are obviously above replacement pitchers, the average is of course much higher. But that includes the very biased 1- and 2-game pitchers.  If you look at the 5-20 game pitchers, again they are about 5.70 on average. 

On top of that, Joe appears to be including relief innings as well as starters.  If we adjust for the reliever advantage, I imagine this pool of pitchers is more like 5.90-6.00.  I think that’s closer to what a team really gets, on average, when they put a replacement pitcher on the mound.  Certainly, I can’t see how we can say that replacement pitchers are any better than the average of all pitchers who get to pitch in MLB, which is about -1.3 R/G.

I have always contended that replacement level starters are right around -1.3 runs, at worst.  A replacement reliever is around -.5 runs (I use around .8 runs difference between relievers and starters - I don’t use Tango’s 1.0 runs).  I think that any data you use to test that will confirm it, more or less.

In a league that allows 4.70 runs per game (runs, not earned runs), I set the repl level for relievers at 5.00 and starters at 6.00.

So, that -1.30 is exactly what I’ve got.

1st career game as a starting pitcher 2004-2009

April 42 games, 218.66 ip 4.90 runs per 9 ip

May-Oct 178 games, 898.6 ip 5.82 runs per 9 ip

TOtal 202 games, 1117.33 ip 5.64 runs per ip

So, assuming April starters are not replacement players, 6.00 looks about right

I don’t think that first games in the majors tells you much about replacement level.  Some of them are good prospects, right?  Simply identify which players you think are replacement level at any time and then look at their results after that.  It is really quite simple.  Maybe look at the 180th to 210th best projected starting pitchers. Or maybe look at all FA making less than a million per year. I am not exactly sure how to do that, but I think it can be done (identifying replacement level players before the fact).

And please, please, pretty please (I have asked this at least 2 times before), when anyone gives us ERA or RA, or some such thing, tell us what league average is.  4.90 rpg means nothing to me when I don’t know what the league average is over that time frame, which I don’t, unless I look it up.

"4.90 rpg means nothing to me when I don’t know what the league average is over that time frame, which I don’t, unless I look it up.”

I looked it up and came up with 4.72

I thought the 2004-2009 time frame would be enough of a reference point.

NP. I could have looked it up of course, and off the top of my head, I probably have some idea, but when someone is making a point, any point, about ERA, they need to always express it in terms of “as compared to league average.”

That is pretty good for first games.  4.90 compared to league average of 4.72 is pretty close to league average for a starter. I have to think that a lot of that is due to never having been seen before. Maybe .25 runs or more.