THE BOOK--Playing The Percentages In Baseball

That’s a very strange lesson to take from Strasburg’s performance.  He put up a 2.10 FIP and 2.17 xFIP, according to Fangraphs.  But because his BABIP was .338, you tell us that the lesson we should take is that rookies definitely cannot be projected at better than a 75% run rate?  I don’t know what lesson, if any, I’d take from such a small sample, but it certainly would not be the lesson you are proposing.

There’s two things that FIP doesn’t account for:

1. average on balls in play
2. sequencing of events

Strasburg’s BABIP was a horrible .338, and his sequencing was terrible (.193 / .240 / .298 with bases empty, .272 / .317 / .380 with runners on base).

Perhaps I got lucky that his non-FIP performance was so abysmal (relatively speaking) that it allowed the 75% forecast to come in so well.

That his K/PA rate was so phenomenal, that had we known that from the outset, if god herself told us that he would K 33% (!!) of his batters, then perhaps we would have forecast a 60% or 65% forecast on runs allowed.  That’s possible.

I still would not exceed my threshhold though.  Given that we’ll never see another Strasburg for at least 3 lifetimes (what does that mean?  12 years), then I think the Tom Seaver Rule is safe.

Rules of thumb would save us from Wieters I and Wieters II, and Wieters III.  It’s a sanity check.

Agree with Mike. I think Strasburg exceeded even the loftiest, hypy-iest of predictions

Yes, the Sportsbook predictions were way way off, but he basically was an elite pitcher right from the get go. which I think was exactly what you were railing against.

2.91 ERA, 2.17 xFIP, 12.19 K/9 (!!!!), 2.25 BB/9, 47.8% GB %, the combination of all that makes him perhaps the best SP in the majors.

I don’t see how you can beat your chest on this topic, if anything I would expect you to post a “my bad”

And the 2.10 FIP doesn’t change anything.  That puts his RA index (using FIP) of close to 50%, correct?

I said this:
“If you want to give Strasburg a 50% index as his upside, you probably give him 100% as his downside.”

Isn’t it more reasoable to suggest that his 2.10 FIP was on the lucky side of his mean forecast, rather than it being the mean forecast? 

Look at Josh Johnson.  His ERA is 2.27 with a FIP of 2.46.  And that was done on 100 more IP than Strasburg. 

When it comes time for the ERA forecasts next year, is someone going to forecast Strasburg better than Josh Johnson?  I doubt it.  And Johnson’s forecast will probably be 70% of league average.

I agree with Mike and Ken as well Tom.  Technically, you are correct that Strasburg’s ERA is higher than his forecast and fits into your Tom Seaver rule, but Strasburg’s peripherals suggest an ERA in the low 2’s (with the static nature of xFIP and the like, it really should be below 2).  He’s actually pitched better than Brian’s forecast.

PECOTA though, forecast only a 10% chance of being better than 3.94.  Big ouch on that one.

Well, no. PECOTA forecast only a 10% chance of being better than 3.94 in 170 or more innings, in the 2009 run environment.

Obviously the spread of ERA goes down as the IP goes up. So if you look at the PECOTA percentiles and compare them to a player with less than half the projected playing time for the lowest percentile - of COURSE the observed spread will be greater than the spread predicted by PECOTA (or anyone else). It is impossible to project the spread of a projection so that the percentiles function equally well for an arbitrary amount of playing time.

Looking at the weighted mean projection, compared to actual, for ERA:

4.45-2.86=1.59

Is that good? Is it bad? Looking at 70 IP of one player? Ellifiknow.

"is someone going to forecast Strasburg better than Josh Johnson?”

Yes, probably. Marcel will probably come out around 2.8 or 2.9 ERA I would guess for JJ. Which is were THT predicted Strasburg would be and is right where he ended up.

Tango, the problem I have is that you are using Strasburg’s performance as proof that your rule of thumb was accurate.  If I were in your shoes, I’d be using Strasburg’s performance to seriously question the wisdom/accuracy of my rule-of-thumb rather than gloating about how I got very lucky it wasn’t blatantly disproved.

Could your rule of thumb still be right, despite Strasburg’s performance?  I suppose it could be.  But to try to use his performance as proof that you were right is involving some major arm-twisting and severe avoidance of sound sabermetric principles.

Yes, you can say that it’s possible he got as lucky in the FIP/xFIP categories as he was unlucky in the BABIP/sequencing categories, but do you have any reason to say that, or is that just BS based on how you want to the numbers to come out?  The SO, BB, HR, GB categories are in general much more repeatable and correlated to persistent skills in a shorter sample than the BABIP and sequencing, right?  But you’re throwing that out that window so that you can say you were right?

I’m going to be very interested to see if Brian’s forecast of Strasburg is going to be better than the 66% RA Index forecast he had coming into 2010.

Follow me here: with NO MLB information whatsoever, Brian forecast Strssburg for an RA Index of 66%.

Now, with his unquestionably fantastic toolset and overpowering performance on the books, Brian’s forecast for Strasburg entering 2011 must be better than 66%, correct?

MGL has already offered that he NEVER forecasts ANY pitcher better than 65%, and that INCLUDED Pedro.  PEDRO!  The greatest pitcher ever, having the greatest peak ever.  Even at his very peak, MGL dialed it back, and even though Marcel forecasted Pedro at 50%, MGL kept it at 65%.  And MGL’s hard-rule of 65% served him well (if I remember the little mini-study I did a few months back).

Put another way, which pitcher are you going to give the better forecast for next year?  Josh Johnson or Strasburg?  Felix or Strasburg? 

I’m going to guess that Strasburg’s RA Index mean forecast will come in at around 70% for 2011.  And, you couldn’t possibly have forecasted Strasburg at 65% entering 2010, and then 70% for 2011, after seeing the great pitching in 2010.

Mike makes a good argument, that Strasburg’s real performance was better than 75%.  But an interesting question is whether injury risk should be treated as totally separate from our forecast.  Perhaps not.  Strasburg must be operating at the very outler limits of what human ligaments can survive in terms of stress.  The fact that no other starter throws, or perhaps has ever thrown, this hard would suggest that is true, as would the high injury rate for many others (Wood, Gooden) who threw nearly as hard. 

As a Nats fan, I hope Strasburg bounces back and enjoys many years of throwing 99mph fastballs.  But it may be that we should set a maximum forecast of 75% (or somewhere in that range) because anyone significantly exceeding it is very likely to get hurt, and will either return as a less effective pitcher or not return at all.

Brian has Strasburg forecast for an ERA of 2.55 should he pitch any more in 2010 and an ERA of 2.47 in 2011. He has Josh Johnson at 3.11 and 3.25, Felix Hernandez at 3.24 and 3.40, Cliff Lee at 3.24 and 3.48, and Roy Halladay at 3.03 and 3.21.

I’m not sure what run environments those are in, which is obviously an important piece of information.

isn’t it likely that prediction systems “break” around the historical outliers? that for players like Pedro 2000, peak Bonds, etc, projection systems just aren’t as accurate as they are for everyone else?

Even at his very peak, MGL dialed it back, and even though Marcel forecasted Pedro at 50%, MGL kept it at 65%.  And MGL’s hard-rule of 65% served him well (if I remember the little mini-study I did a few months back).

Isn’t that the opposite conclusion of what was found?  Didn’t we find that Pedro was the only one that broke the rule, and his performance actually more closely matched the un-limited forecast rather than the hard 65% rule?  Where is that thread, anyway?

You are ignoring what I said here:

I said this:
“If you want to give Strasburg a 50% index as his upside, you probably give him 100% as his downside.”

Isn’t it more reasoable to suggest that his 2.10 FIP was on the lucky side of his mean forecast, rather than it being the mean forecast? 

Here’s what you have to answer: given that you’ve seen a starting pitcher with an FIP relative to league average of around 50%, over 274 batters, and throwing 98mph fastball, and striking out one third of the batters, is it more likely, or less likely that his true mean RA Index is greater than 50%?  Surely, we are not suggesting that his true mean is better than 50% are we? 

No, of course not.  His true mean is somewhere north of 50%, and he got lucky to get it to 50%.  This is nothing more than Bayes.  We are extremely certain that Pedro at his peak had a true mean of 50%, and that was based on a few THOUSAND batters.  Even if you want to argue that Strasburg’s performance approached Pedro, our uncertainty of 274 PA compared to the 3000 or so batters that Pedro faced in order for us to estimate Pedro’s true talent at 50% means that Strasburg’s mean was worse than 50%.

Therefore, given that the observed 50% is on the high side, we are left to wonder whether that is one SD or two SD or three SD from his true mean.  And, I think it’s very reasonable to say that Strasburg’s true mean in 2010 was around 75%, and he pitched at 50% for 274 batters.

And now, entering 2011, having observed what we observed for 274 batters, then our guess for his true mean will probably be 70%.  And, if he has a full year like we saw, then his true mean entering 2012 will be 65%.

This is how it works guys.  That’s why the Tom Seaver Rule is needed.

Brian has Strasburg forecast for an ERA of 2.55 should he pitch any more in 2010 and an ERA of 2.47 in 2011. He has Josh Johnson at 3.11 and 3.25, Felix Hernandez at 3.24 and 3.40, Cliff Lee at 3.24 and 3.48, and Roy Halladay at 3.03 and 3.21.

How does this make any sense?  Seriously, after what Cliff Lee has done, what Felix has done the last 2 years, what Josh Johnson has done, how could Strasburg’s forecast for 2011 (2.47) be as far from Felix (3.40), as Felix is from the league average? 

How different could Josh Johnson and Strasburg’s performance in 2010 have been?

Nice piece by Pos on fragility of pitchers:
http://joeposnanski.com/JoeBlog/2010/08/24/the-pain-of-pitching/

Tom/14, I agree that his true mean/true talent was very likely somewhere north of 50%.  We had a projection coming in of what, 65% or something like that, from Oliver, which is the only system using his college data.  So maybe we’d say we think his true mean was really more like 60% and he got lucky to get it to down to 50%.

I don’t know if that’s accurate.  It’s completely off the cuff.  But it seems more reasonable than you picking his true talent at 75% for no reason that I can tell other than that you are skeptical it could be any lower.

If you are using the reason that it has to be around 75% because of the Tom Seaver rule, then that can’t be used as a proof of the Tom Seaver rule.

Tom/14 makes a lot of sense, and I think he’s right that Strasburg’s 50% FIP this year does not necessarily mean Brian’s forecast was correct (obviously). 

However, why do the deviations around the forecast have to be symmetrical?  Couldn’t Strasburg have a mean forecast of 65% with an upside of 50% and a downside of 100%?

I don’t know the inner workings of Brian’s system, but I’m pretty sure each of those numbers are forecast within the pitcher’s particular park and possibly their defense as well.  So Lee in Texas and Strasburg in Washington, for example.  Which is why I mentioned that knowing the run environment is important.  But that’s not listed in the THT forecasts.

How different could Josh Johnson and Strasburg’s performance in 2010 have been?

Brian has their major league equivalents for their 2010 seasons as 2.25 ERA for Strasburg and 2.76 ERA for Johnson.  That includes Strasburg’s 49 innings in the minors this year.

We may well never be able to say how good Strasburg “really” was this year.  Even if he manages to avoid surgery, there’s a good chance he will modify his approach—maybe bring down his FB speed 2-3 mph, throwing fewer hard breaking balls, or whatever else the Nats think/hope minimizes injury risk.  And if he’s lucky he still ends up pitching like Halladay/Johnson.  But then we’ll never be able to say if that was always his true talent or if—briefly—he really was Pedro.

Tom,

No, no, no, no, and once more, no. You CANNOT say that Strasburg was lucky, because we are not having this argument ex-post. The question of how Strasburg would do came up before he had ever thrown a major league pitch—therefore, there is NO reason to “correct” bias in his numbers. That would be like regressing to the mean twice. If a pitcher posts a 2.00 ERA in a season, maybe his likeliest true talent projection is 3.00. If a pitcher posts a 3.00 ERA, maybe his likeliest true talent is 3.75. But if a pitcher posts a 2.00 ERA, it does not follow that his likeliest true talent is 3.75. Which is what you are currently trying to argue. Strasburg was a pre-selected subject. Therefore, there is no reason to expect bias in his numbers. What happened happened. Oliver was right. You were wrong. End of story.

End of story.

You can say all the rest, but don’t say that.

How can I be wrong if his runs allowed rate was 74% of league average?  You may want to shut down the discussion, but I’m keeping the dialogue open.

I understand you may be frustrated and exasperated by my “logic”.  That doesn’t mean you can try to end the discussion.

Well, no. PECOTA forecast only a 10% chance of being better than 3.94 in 170 or more innings, in the 2009 run environment.

Obviously the spread of ERA goes down as the IP goes up.

Colin, it may be obvious that this should happen.  And yes, I agree it is obvious that it MUST happen.

But, that is NOT how PECOTA works.  The range of ERA forecasts is independent of the number of innings forecast.  I’ve been talking about this for 6 years, and hopefully you are the one person at BPro who cares enough about this to actually fix this.

How can I be wrong if his runs allowed rate was 74% of league average?

I agree with many of the points you make, but this is incredibly disingenuous.  When Strasburg’s peripheral stats indicate an ERA in the low 2’s, it’s clear that he deserves a lower ERA than 74% of league average.  I’m not sure why the almost entirely random events (such as his BABIP and sequencing of events) should be held up as evidence that Brian’s forecast was too optimistic.

Mike, posts 24 and 25:

http://www.insidethebook.com/ee/index.php/site/comments/is_there_a_minimum_that_you_should_forecast_a_pitchers_era/#24

You can reasonably argue that the limit of any forecast should be at 60% of league average.  MGL uses 62.5%.  Marcel has not fixed limit.

Colin, it may be obvious that this should happen.  And yes, I agree it is obvious that it MUST happen.

But, that is NOT how PECOTA works.  The range of ERA forecasts is independent of the number of innings forecast.  I’ve been talking about this for 6 years, and hopefully you are the one person at BPro who cares enough about this to actually fix this.

That’s not true, though. The PECOTA percentiles are based upon the performance of the comps. The thing is, so are the projections of playing time. So no, the percentiles are not independent of the innings forecast.

What they are independent of is the number of innings used to arrive at the baseline forecast. So every forecast - for the purposes of the percentiles - is treated as being equally accurate. We can get into a lot of reasons why this is not a correct assumption, etc. I have not spent a lot of time yet thinking about the percentile forecasts for next year’s PECOTA (once I finish up the WARP revisions I plan to start moving on PECOTA work, and I’ve got a pretty big list of priorities to look at and work on between now and when the first set of PECOTAs will be released).

So all I can really tell you right now is that:

1) I’m in charge of the PECOTAs this year.
2) I will be working to make them the best I possibly can.
3) We will be informing our readers about what we’re doing with the PECOTAs in greater detail, similar to the way we’re treating the changes to WARP, as we start getting closer to the offseason.

75%: I’m saying that there’s still a discussion to be had.  Strasburg could very well have pitched much differently with men on base than not.  Indeed, didn’t Feller say something about it, and Mike Fast was pointing to that as well? 

You can argue that the observation should be say halfway between the 74% that his runs allowed is showing and the 50% (or whatever) that his FIP is showing (or 62% observation).

If you want to say that we observed a performance of say 60%, fine, I’m ok with that.  Even if you want to call the performance a 50% observation.  Fine.

The question is if that is 0, 1, 2, or 3 SD from his true mean.  Or even if it’s worse than his true mean would have estimated.

And I’m saying that we observed 75% keeps the conversation open.  Telling me “end of story” is the same thing as telling me to shut up.  I’m talking, and I’ll keep talking, thanks.

I’m saying that a reasonable forecast for his true mean could not exceed a certain threshhold without any MLB competition.  And I’m saying that after 274 batters of MLB competition, you’re going to have another threshhold that he can’t go above in terms of estimating his mean (probably 70%).

No one here has taken on any of the challenges I have put out there, in terms of estimating for me what your forecast would have been for Tom Seaver, Dwight Gooden, and Vida Blue, guys that not only came highly acclaimed, but were fantastic right off the bat.

So, go ahead, do that for me.  Find me the guys who had the best performances off the bat AND who were very highly acclaimed before showing up.  And tell me what you forecast for them after they faced 1500 or 2000 MLB batters.

And then tell me how you can possibly put Strasburg better than that after 274 batters.

Strasburg made 12 starts and pitched 68 innings.  If Justin Verlander had a 2.10 FIP over 68 innings, would we conclude that a forecast of a 2.86 ERA for a full season was correct?  There isn’t enough here to draw any kind of conclusion.

It’s a shame he got hurt, because this would be much more interesting if we were debating his prediction going forward after 68 innings.  I’d be on the side betting that the regression to the mean on Strasburg would be a much stronger force than the force of converging ERA and xFIP.  From the comments, I get the impression that a few of you would be betting on the latter force.

IDEALLY, the perfect forecast would be one where we require NO OBSERVATIONS.  It would purely and simply be based on the scouting of the player.  You hook him up, get his fastball speed, get his movement numbers, get his location, presume he can sequence randomly, etc.  And then, you’d say he has a 65% true talent level for Runs allowed.

And then the next year, your scouting tells you that he improved due to age or experience, and you forecast 60%.  And then the year after that you forecast him for 50%.

And, this is the important part, REGARDLESS of the number of PA, whether it is 100 PA or 1000 PA, your forecast DOES NOT CHANGE!  That’s because you scouted him perfectly.

If you guys are saying this is where we are with Strasburg, that we are learning nothing about him based on his performances because, well, all this is expected.  Maybe you are right.

I’m saying no.  I’m saying that the scouting only gets you so far.

Tango, this doesn’t feel right. I think that you should take a few steps back from this argument and start running some numbers. Brian’s projection of Strasburg is something he should take pride in, and it seems like you’re summarily dismissing his work without evidence of your own.

I don’t see where I dismissed anything.  This is what I said from the outset:

Good for Brian for being really out there on the forecast, and coming in pretty well overall (with some luck do to a depressed run environment).  PECOTA though, forecast only a 10% chance of being better than 3.94.  Big ouch on that one.

I think that’s a pretty fair assessment, no?

A good example of what I’m talking about on the scouting side would be Ozzie Smith. From the outset, before even playing a guy in MLB, he was considered a defensive whiz.  He may have even been considered a +30 runs per season player for all I know.  That would be his mean forecast.

And then, whether he played 1 game or 2000 games, it didn’t matter.  He had the perfect forecast because we were able to translate his toolset exactly into what we’d expect from his performance.

So, if we rewind 30 years to before he was a rookie, and someone came in and said “Ozzie will be the greatest fielding SS of all time”, I would say “b.s., the best you can forecast him is at +15”.

And then if the first year he shows +30, I’d say that in his second year, the best we can forecast him is +22.  And then he has a second year of +30, and THEN I would say his forecast is +30.

So, I would have been dead wrong about Ozzie, that I would have set that threshhold because it was too restrictive, that I didn’t allow enough for the scouting portion of the forecast to move that threshhold level.

That’s what we are talking about here for STrasburg, that we would move the threshhold level based on the scouting.

The problem is that Brian did NOT use any scouting data for his forecast (I don’t think).  He based it on the performance numbers.  Indeed, had he used the fact that he touches 100mph with command, he’d have no choice but to give an even more optimistic forecast. 

In order to shut me up that allows you to give me a forecast with no MLB data that is better than a 75% forecast of runs allowed relative to league, you MUST include scouting data.  With no scouting data, then I’m happy to set the best estimate for a MEAN forecast of 75%, meaning that we’d observe such a player to perform at 50%-100%.  And that’s what we got.

***

Where I *am* dismissive, or at least highly questioning of Brian, is the reported estimates for 2011 for the top pitchers.  The huge gaps in those between Strasburg and the other top pitchers is unfathomable.

Tango, you seemed to have dismissed Brian’s forecast when you said this:

“When Stephen Strasburg comes up, and when you see a forecast that does not come with an uncertainty level, you can ignore that forecast.  If it does come with an uncertainty level, make sure that the upper boundary of that forecast does not already make him the best pitcher in the league (even if you really really really think he might be).  If it does do that, ignore that forecast too.”

I feel that you should revise that statement.

You also said this, which seems relevant given that these figures are higher than Brian’s 2011 forecasts for Johnson, Felix, and Lee as well:

“That’s a FIP of around 3.10 or 3.20.  Indeed, that line likely represents Strasburg’s somewhat optimistic mean forecast for his first year, and likely a reasonable forecast for his second year.”

Colin:

Here’s an example of what I’m talking about. 
http://www.baseballprospectus.com/card/card.php?id=RIVERA19691129A

Rivera has a mean forecast of 2.77, with a 90/10 OBSERVATION range of 2.12/3.13.

Pettitte:
http://www.baseballprospectus.com/card/card.php?id=PETTITTE19720615A

Mean: 5.01
Observed 90/10: 4.16/5.49

That can’t be right, can it?  No, of course not.

And I understand exactly why it happens.  Because Nate did the percentiles wrong.  I’ve told him how to test it, and it’s never been done.

You take all the pitchers forecasted with an ERA of 4.00 of 60-80 IP, and show me how often they exceeded the 90th percentile forecasts and were worse than the 10th percentile. 

Now, repeat with pitchers of 160-180 IP.

You won’t get the right answer.

I’m pretty sure you understand what I mean, and I’ll be happy to leave it at that until you get your hands dirty in this.

It’s so weird that I’ve offered my help for free on this, and all I got in the past is pushback.  Truly, sometimes giving advice as to how to fix things is like telling someone how to drive: it’s very unwelcomed.

“When Stephen Strasburg comes up, and when you see a forecast that does not come with an uncertainty level, you can ignore that forecast.  If it does come with an uncertainty level, make sure that the upper boundary of that forecast does not already make him the best pitcher in the league (even if you really really really think he might be).  If it does do that, ignore that forecast too.”

That was directed at ANYONE and didn’t I say that before today?  So, you are quoting me out of context here.

As I said in my initial thread today, I gave Brian his props for sticking his nuts out there. 

I pre-dismissed it, as I should have, because it came with no uncertainty level. I stand by the above statement, that you MUST offer the uncertainty level of the mean because then it would show that the estimated true mean was something like 45% to 85% or something.  That’s how you evaluate the forecast. 

I was fair then in that thread, and I was fair in this thread.

Brian’s preseason projection, at least as quoted in one of the linked threads (Oliver only has updated projections on THT):

120 IP, 150 K, 34 BB, 10 HR

His actual numbers, prorated to 120 innings:

162 K, 30 BB, 9 HR

So Brian was too conservative all around.  Just got lucky that Strasburg had a high BABIP, saving the ERA projection.

Using my standard projection algorithm, I have Strasburg currently projected at 65% of a league average pitcher in RA, based only on his performance this year.  That is better than JJ and better than Felix.  It is basically the best in baseball.  Lincecum (if healthy), Jimenez, Lee, and Holliday are close, and certainly with the uncertainty around all of those projections, they could all be the same or any one could be better than the other.

I am not sure what the argument is here.  A lot of people are talking past one another.

Tango, you never answered my question in post 18.  Why does the upside/downside have to be symmetrical?  Isn’t it quite reasonable to give Strasburg a 65% projection, with an upside of 50% and a downside of 110%?

OK guys, I’ve caught up with the comments.

I’ve been running some numbers, but my first comment is that although all the analysis here has been of ERA, I will note that ERA is the last step in my projections process. Basically, the pitcher projections mirror what I do for batters, compiling a weighted mean of the past three seasons (or four if in season) in each component, then adding the regression, aging, park & league factors. I currently do not account for which players are on defense behind each pitcher, but that’s something I am working on for the future. From the batting stats off each pitcher, I calculate the wOBA allowed, then use a formula I devised to convert wOBA allowed to expected ERA.

Here are some numbers
‘Pre’ is the pre-season projection
‘Act’ is the actual, raw MLB performance

The pre-season is based on his college stats and the 2009 Arizona Fall League.

Strasburg
Type Size  ERA wOBA   BA   OB   SA  _bh  _hr  _bb  _so
 Pre  817 2.81 .268 .219 .275 .330 .306 .032 .072 .307
 Act  274 2.91 .263 .221 .266 .328 .323 .031 .062 .336
 Now 1049 2.58 .258 .213 .265 .314 .297 .029 .067 .303

Strasburg’s current in-season projection is better because of the extremely few hits and homers he allowed in the minors this year, where I calculated a MLE line of 175/238/224, babip of .232.

His actual MLB BB & SO rates this year were 62/336. The translated minor league numbers this year were 72/250, when added to the MLB give a total season MLE of 66/305, where the pre-season projection was 72/307.

Three other guys, who were selected not because of their performance, but that because I had been questioned early on about the projections for Colby Lewis (two years in Japan, no previous MLB success), Ian Kennedy (no previous MLB success) and I added Mike Leake (no professional experience).

babip looks tough to predict without accounting for defense.

Lewis
Type  Size  ERA wOBA   BA   OB   SA  _bh  _hr  _bb  _so
 Pre  1479 3.17 .285 .247 .284 .370 .302 .030 .041 .228
 Act   643 3.37 .286 .224 .289 .353 .278 .035 .071 .241

Leake
Type  Size  ERA wOBA   BA   OB   SA  _bh  _hr  _bb  _so
 Pre  1247 3.83 .310 .259 .311 .397 .290 .033 .057 .162
 Act   596 4.24 .339 .287 .342 .436 .316 .038 .076 .152

Kennedy
Type  Size  ERA wOBA   BA   OB   SA  _bh  _hr  _bb  _so
 Pre   898 3.99 .311 .248 .323 .379 .295 .024 .095 .190
 Act   646 4.41 .322 .243 .314 .430 .273 .054 .084 .199

Lewis has not kept the extremely low walk rates he had projected from his Japanese performance. His babip is down, HR and SO up slightly. Overall projection looks very good.

Leake allowed more BB, SO rate is good, HR close.

Kennedy’s BB and SO are right on, but his HR rate exploded.

In #39, the ERA’s are actual, not the one calculated from the wOBA allowed.

Below, the wOBA is adjusted for ballparks played in (not neutral), and wERA is from that wOBA. This corrects for abnormal sequencing as well, what the ERA would have been expected to be given the componets, which is the same way it’s calculated in the projections.

     adj  adj    prj  prj    act
         wOBA wERA   wOBA wERA    ERA
Lewis    .283 3.16   .285 3.17   3.37
Leake    .331 4.39   .310 3.83   4.24
Kennedy  .313 3.91   .311 3.99   4.40

Tango’s post 32 really helped me to understand his argument; I hadn’t realized that scouting information wasn’t being factored in.  While reading this I kept thinking how Steve Nebraska from the movie “The Scout” would have broken the Seaver Rule before throwing an MLB pitch.  For those who don’t know, Steve Nebraska was basically the perfect baseball player...he threw a 112 MPH fastball and pitched an 81 pitch, 27 K perfect game in his MLB debut in the World Series).  However, if we are discounting scouting data then of course we wouldn’t have projected him so well.

However, what if Steve Nebraska would have pitched in the minors for say 10 starts?  Had he struck out 270 minor leaguers in 90 innings, I think it would be clear that the Seaver rule should be violated in this instance.

Now of course we won’t ever see a pitcher quite like that, but the mathematician in me isn’t satisfied with a rule if I can conceive of a counter-example...no matter how contrived.  However, if there is some pitcher with a better minor league track record than we’ve ever seen before (but not as good as Nebraska’s 10 perfect starts), it seems reasonable that we could project him better than Tom Seaver.  I don’t know enough about Doc Gooden, Vida Blue, etc. to know if such a pitcher has ever existed, but I felt the esoteric need to point out that such a player could exist.

Using my standard projection algorithm, I have Strasburg currently projected at 65% of a league average pitcher in RA, based only on his performance this year.

This is Pedro, 1999 first, and then Strasburg, 2010 this year:

835, 274: PA
37.5%, 33.6%: K/PA
4.4%, 6.2%: BB/PA
1.1%, 1.8%: HR/PA

So, Pedro, compared to Strasburg had more K, fewer walks, fewer HR, and faced 3 times as many batters.

MGL, using ONLY 1999 Pedro, what would be his forecast entering 2000?

Because I get the feeling you are getting the forecast you are getting by not setting PA=274 for Strasburg, in light of past comments you made where you’ve never had a forecast for anyone ever that was better than an ERA of 2.50 in a league of 4.00.

***

Nick/38: I didn’t realize the symmetry question was directed to me.  It would be mostly symmetrical for wOBA, not ERA. 

***

Rally/36: those component forecasts are great.

Here’s Oliver’s one year projections of Pedro

season age  size   ip   era   woba     ba    ob    sa    _bh   _hr   _bb    _so
1999    27   951  234  2.89  0.272  0.214 0.275 0.339  0.267 0.042 0.068 0.263
2000    28  1501  223  2.50  0.254  0.209 0.261 0.307  0.297 0.029 0.057 0.320
2001    29  1877  222  2.06  0.233  0.189 0.238 0.279  0.266 0.031 0.048 0.326
2002    30  1445  173  1.92  0.225  0.184 0.232 0.266  0.279 0.026 0.045 0.358
2003    31  1515  177  2.24  0.242  0.195 0.249 0.290  0.270 0.026 0.049 0.306
2004    32  1528  174  2.50  0.254  0.208 0.265 0.298  0.287 0.019 0.057 0.287
2005    33  1821  203  3.04  0.279  0.220 0.282 0.349  0.283 0.031 0.064 0.259
2006    34  1848  208  2.96  0.277  0.220 0.273 0.358  0.270 0.035 0.060 0.240
2007    35  1531  172  3.01  0.279  0.216 0.274 0.363  0.255 0.045 0.063 0.234
2008    36   971  106  3.60  0.303  0.239 0.292 0.411  0.278 0.051 0.061 0.225
2009    37   906   94  4.57  0.335  0.273 0.330 0.449  0.310 0.050 0.074 0.194
2010    38   706   71  5.27  0.357  0.287 0.345 0.490  0.313 0.058 0.071 0.174

Brian,

There’s something wrong there.  Those aren’t one-year forecasts (forecasts using only one year of data) are they?

Also, you need to show the league numbers, so we can normalize things.  Why not just show us the ERA relative to league average?

It looks like what you did BRian is that you started Pedro in 1998 as being his rookie year, and then went from there.  Is that the case?

One year forecast means looking one year into the future, but based on three years of data. However, my Oliver database starts in 1998 (the last expansion.

1999 = 1998
2000 = 1999 + 1998
2001 = 2000 + 1999 + 1998
2002 = 2001 + 2000 + 1999
etc

So, yes I prefer starting projections in 2001 as that’s the first where I have three seasons available, but you were asking about 2000.

> ERA relative to league average

because I currently don’t have queries set up to report that, I pasted in what I had on hand.

I noted in an earlier thread these MArcel forecasts for PEdro:

ERA year
2.62 2001
2.48 2002
2.66 2003
2.84 2004

Let me tack on Brian’s just published numbers to the end:

Marcel year Oliver
2.62 2001 2.06
2.48 2002 1.92
2.66 2003 2.24
2.84 2004 2.50

It seems pretty obvious that Oliver takes a much less regression-toward-the-mean approach than Marcel.

Let me tack on the actual ERA for those seasons:
Marcel year Oliver Actual
2.62 2001 2.06 2.39
2.48 2002 1.92 2.26
2.66 2003 2.24 2.22
2.84 2004 2.50 3.90

These are the differences for Marcel and Oliver:
2001 0.23 (0.33)
2002 0.22 (0.34)
2003 0.44 0.02
2004 (1.06) (1.40)

***

So, when I see the following for rest of season 2010 and 2011 from Oliver:

2010 2011
2.55 2.47 Strasburg

3.03 3.21 Halladay
3.11 3.25 Johnson
3.24 3.40 Felix
3.24 3.48 Cliff Lee

You just can’t have a 0.74 run gap between the #1 and #2 pitchers in estimated talent.  Not with the guys we are talking about.  And not based only on the performance data we’ve seen.  And not while excluding scouting data.

Oliver is sticking its nuts out there in the really extreme cases, and Marcel does not.  My objections is based on the extremeness of the forecast.

IDEALLY, you WANT that, if you are going to be more right than Marcel.  AFter all, we know the spread in true talent is greater than what Marcel estimates.  Marcel takes a Bayesian position and says “ehhh, maybe Pedro is 30% chance of being the best pitcher”, while Oliver seems to say “ha!  PEdro is like a 90% chance of being the best pitcher”.

Something like that.

So, he’s doing the same thing with Strasburg, taking a very very very strong position that STrasburg entering 2011 has no peers whatsoever, that he is to Halladay and Felix and Josh Johnson MORE than what Pedro was to his peers (perhaps exclusing RJ).

And he’s basing that on 274 batters, while Pedro would have had thousands of batters to establish his claim.

And he’s basing that on 274 batters, while Pedro would have had thousands of batters to establish his claim.

Oliver is basing its Strasburg projection on ~1400 batters he has faced since 2007.

Tango/25, thanks.

Mike/48: ok, he’s basing it on 274 MLB batters, and 1100 college batters and minor league batters, while Pedro’s claim is being established against 3000 MLB batters.

***

And don’t forget, Pedro pitched better in 1999 than Strasburg did in 2010, while facing 3 times as many batters (meaning that we are much more sure that he was in fact better).

Compare Pedro entering 2000 to Strasburg entering 2011.  Oliver is saying Pedro was 2.50 and Strasburg is 2.47.  Relative to run environment, that’s probably 55% to 59%.  What did we know about Pedro entering 2000?  He pitched better and against more batters (in 1999) than Strasburg did in 2011.  We also know that Pedro in 1997-1998 almost certainly pitched better and against more batters than Strasburg in 2008-2009.

And yet, using NO SCOUTING INFORMATION, the estimate for Pedro entering 2000 is just a bit better than Strasburg entering 2010?  (Not that the scouting information would necessarily help Strasburg more than Pedro… it might.)

***

Is anyone going to agree or disagree on this:

2010 2011
2.55 2.47 Strasburg

3.03 3.21 Halladay
3.11 3.25 Johnson
3.24 3.40 Felix
3.24 3.48 Cliff Lee

***

Forget about all the frustration you have with me personally, and focus on the merits, and make the case as to how Strasburg already deserves an almost Pedro-like forecast using NO SCOUTING INFORMATION.

"I was fair in this thread.”

You’re getting pushback here, Tango, because the tone of your post was inappropriate, given the data.  Especially this statement:  “Anyway, if what we’ve experienced here is not enough to dial back forecasts to something reasonable, then nothing will ever prove it to some people.”

You said this based only on your rough RA comparison.  If you had known that Brian’s forecast was actually slightly too conservative, based on FIP, I don’t think there’s any way you would have written that. 

Brian made a forecast which was basically dead on, both for FIP and ERA.  You could make two reasonable responses: 1) good for Brian, maybe I was too conservative, or 2) good for Brian, but it’s a small sample, and I still think it was too optimistic a projection given the data we had.  But you went for door #3:  “nyah, nyah.” And the data just can’t back that up.

Now, you have backed away from this in various days, sliding from a 75% ceiling to MGL’s 62.5% (without quite committing to either).  But it would be much more gracious in situations like this—which don’t happen very often—if you could just bring yourself to say “I got this one wrong, and I retract statement X.” (And I say this as someone who largely agrees with your substantive points in this thread.)

You said I should say this:

or 2) good for Brian, but it’s a small sample, and I still think it was too optimistic a projection given the data we had.

And I started the thread by saying this:

Good for Brian for being really out there on the forecast, and coming in pretty well overall (with some luck do to a depressed run environment).

We are splitting hairs if you are suggesting that how you think I should have said it, and how I actually did say it, is any different.

***

Now, you have backed away from this in various days, sliding from a 75% ceiling to MGL’s 62.5% (without quite committing to either).

I obviously didn’t write it well enough: the ceiling remains at 75% for any pitcher with no professional experience.  The 62.5% (or 60% or whatever we finally end up coming to an agreement as the best threshhold to maintain) is for a pitcher that has established experience. 

***

You said I implied this:

door #3:  “nyah, nyah.” And the data just can’t back that up.

Based on me saying this:

Anyway, if what we’ve experienced here is not enough to dial back forecasts to something reasonable, then nothing will ever prove it to some people.

I am quite capable of saying “nyah, nyah” if I wanted to.  I didn’t say that here, nor did I imply it. 

***

I’ve issued many statements and challenges in this thread.  Only Brian has seen fit to take me up on it, the one guy who I would understand if he wanted to pushback on me.  Everyone else is focusing on however they want to read the undertones of my post.

Brian’s forecasts seem to be very optimistic for pitchers with very high K rates (as evidenced by the Strasburg forecast and the Pedro forecasts).  There’s a good reason to do that, since I’ve just shown in another thread that K/PA stabilizes the quickest, and therefore you DO want to overweight, substantially, recent K/PA.  So, that’s where you should be telling me to shut up about this, since Brian correctly nailed the K/PA for Strasburg, and so, everything else is going to follow from that.  And that’s how Brian can put his nuts out there, that his K/PA is so Pedro-esque and RJ-esque that even a Bayes process would say that, yup, Strasburg deserves the lofty forecast, even with no scouting information.

That’s how you take me down.

***

I still have the open question to MGL of how he forecast Pedro entering 2000 and 2001, because I can’t believe that he’d have Strasburg2011 ahead of Pedro2000 and Pedro2001.

Forget about all the frustration you have with me personally, and focus on the merits, and make the case as to how Strasburg already deserves an almost Pedro-like forecast using NO SCOUTING INFORMATION.

I don’t know whether Strasburg deserves the projection Brian is giving him.

My pushback is against you saying (1) Oliver is definitely wrong about Strasburg, and (2) there must be hard limit to cap pitcher projections.  I’m not sure about either of those, and nothing you have said in this discussion, going back to previous threads, has convinced me.

I’m willing to accept that it’s still an open question.  I agree completely with Guy/51 that a lot of my strident pushback in this thread has been because you claimed that it was now a closed case with your viewpoint declared the winner by knockout, when in reality the judges are still looking over their scorecards.

I’ve issued many statements and challenges in this thread.  Only Brian has seen fit to take me up on it

Only Brian has the data about how Oliver works.  I can’t tell you why Oliver predicts what it does for Strasburg.  I’m not a minor league equivalency or projections expert, and there are few in the world who are (Brian, Rally, MGL, and handful of others).

I have a fundamental logical issue with your hard cap.  Just because it works okay for recent history doesn’t prove much.  You’re creating a rule and “proving” it based on in-sample data. 

It’s as if you said that if we were to predict the American president’s age upon taking office, that we should have a hard cap at age 45 because Kennedy was 44, Clinton was 46, and Obama was 47.

It’s reasonable to put a hard cap at 35 because there is a rule that says the President can’t be younger than 35, but just because we haven’t had a president younger than 44, does that mean we should have a hard cap at age 45?  Or is it eminently possible that a president in the near future could be 40 or 42 years old? 

What if we had a strong candidate (a Strasburg) polling well in the primaries who was only 39 years old?  Would you say, “We definitely know he won’t win this election because he’s too young.  He’s below our hard cap that history has shown us.  The Clinton Rule.  He’ll have to wait eight more years.”

"Brian’s forecasts seem to be very optimistic for pitchers with very high K rates (as evidenced by the Strasburg forecast and the Pedro forecasts).  There’s a good reason to do that, since I’ve just shown in another thread that K/PA stabilizes the quickest, and therefore you DO want to overweight, substantially, recent K/PA.  So, that’s where you should be telling me to shut up about this, since Brian correctly nailed the K/PA for Strasburg, and so, everything else is going to follow from that.  And that’s how Brian can put his nuts out there, that his K/PA is so Pedro-esque and RJ-esque that even a Bayes process would say that, yup, Strasburg deserves the lofty forecast, even with no scouting information.”

Tango - I think that’s similar to what a lot of people are saying.

basically, all of the ERA predictors/estimators, like FIP, xFIP, qERA, SIERRA, etc, all us K rate as the key (single most important?) input. and they all say that the lofty Strasburg predictions were right on.

the only piece of data that really agrees with you is the non “luck” adjusted actual results, like RA, over what is a very small sample size.

don’t you agree that when talking about 63 innings, something like xFIP or SIERRA is a better predictor of future ERA than ERA itself? if so, then why do you care so much about 63 innings of ERA instead of 63 innings of K Rate, BB rate, GB Rate, LD rate, etc?

I have a fundamental logical issue with your hard cap.

Because it is far more reasonable to think that no rookie should be considered better than one of the best pitchers of all time in his prime.

Look at the NHL, where players routinely make it in the NHL at age 18 or 19.  Gretzky was considered a sure-fire star (he signed a 20 year contract like in his third game).  Mario Lemieux was considered a sure-fire star.  Same with Bobby Orr.  And those guys went on to become 3 of the 4 best players of all time.  And in their first year, they did not establish that they were among the best players of all time.

I don’t follow the NBA too much, but I guess Jordan, Magic, and Bird were like that too.

We can go through all the sports and I’m sure it’s all like that, that you’d have a once-in-a-generation talent, and they simply do not play to their lofty potential off the bat.

And in baseball, there is a bigger learning curve, a steeper hill to climb.

That’s why putting in a hard cap for rookies is the right thing to do.  Strasburg certainly pushed that thinking to its brink, but if it took a once-in-a-generation kind of performance to show that the hard cap was reasonable enough, then I’m happy with that.

I further think that MGL’s point to putting in a hard cap for ANYBODY is also worth exploring.  And indeed, forcing a hardcap at say 60% for any pitcher would seem to also make good sense.

Remember, all these things are Bayesian problems, and they are based on our expectation of the talent distribution.

Putting a hardcap at 75% for rookies and 60% for any pitcher sounds good to me.  Now, if that hardcap should be 72% and 58% or whathaveyou, sure that’s possible.  I’m taking what I think are reasonable positions, and I’ve laid out why conceptually I think it should work like that.

I may be wrong, I may be in the minority, and I may not explain myself well enough.  But I’ve explained myself as best I can.

What others should do is stake their positions, put out their concepts, and make their case.

Kneel/55: the best answer to your question is by Patriot/28.

***

The key is whether scouting information is used or not.

Tango/77:

ok, but Patriot saying

“There isn’t enough here to draw any kind of conclusion”

is a whole lot different than you saying:

“if what we’ve experienced here is not enough to dial back forecasts to something reasonable, then nothing will ever prove it to some people.”

those are two completely contradictory POV’s.

Earlier this year you put the hard cap idea out there as an argument that Strasburg couldn’t be that good in his rookie season.

Now that Strasburg was that good in his rookie season, you are saying that even if it didn’t work for Strasburg, the hard cap is still fundamentally a sound idea.

Okay, I guess.  It’s great if that works for you.  I have no use for a hard cap that’s going to work except when it doesn’t.

What I will give major props to Brian for is this:

If he were to not issue any other kind of lofty forecast for any rookie for the next 5 or 10 years, that he let Strasburg be the one and only guy he gave such lofty forecast for, then yes, Brian deserves major kudos.

But if he puts out forecasts like that routinely, say Wieters I, Wieters II (Montero), Wieters III (Strasburg), Wieters IV (Bryce Harper), Wieters V (whoever is next), and he manages to hit on one or two of the next ten Wieters-level forecasts, then no, he won’t deserve the major props.

Brian went all-in on Strasburg, and he saved face.  Great, fantastic.  If he goes all-in on nine more Wieters, then he better be right half the time.

Seeing how he went all-in on the Pedro forecasts makes me think that he pushes too far.

That’s how you are going to evaluate Brian’s forecasts.

(And same for everyone, by the way.)

Basically, we’ve had two cases in history where projection systems would have projected a player to break your hard caps.  Pedro and Strasburg.  In both cases, your hard cap has failed.

You can claim that it’s a success since it only fails once in a generation or whatever, but if it only applies once in a generation, that’s still an awfully poor track record.

I have no disagreement whatsoever with what you are saying in #60.  If that is the sum of your argument, I am in complete agreement.

those are two completely contradictory POV’s

They are not contradictory.  I’m saying that what Strasburg did hasn’t changed my view of the 75% hard cap.

And Patriot is saying that if anyone else put up the numbers that Strasburg did after facing 274 batters, that we wouldn’t be talking about this anyway.  I think I’d be able to find 274 consecutive batters where Felix or Verlander or Johnson or Pedro or RJ or Maddux put up Strasburg-esque numbers.

And if all we had were those 274 batters, I don’t think our estimate of their true talent would be better than 75% I don’t think.

What you can say to me, which is what Patriot is saying, is that this was a “let” (like in tennis).  It neither challenged my 75% hard cap, nor was it fairly tested because he’d have to have had a superhuman performance to knock down my cap based on only 274 batters.

Basically, we’ve had two cases in history where projection systems would have projected a player to break your hard caps.  Pedro and Strasburg.  In both cases, your hard cap has failed.

Actually, I don’t know what my hard cap is for established players.  But, I will tell you how I would put that in.  Look for the 20 best forecasts out there ever, and be right on 10 of them based on whatever the hardcap is.

So, if that means I’ve got 6 Pedro years and 3 Maddux years and 2 RJ years, and 1 Clemens year and 1 Gooden year, and 1 Seaver year, and so on, then put the hard cap right where I get half of them right.

Test that against no hard-cap, and see how well the hard cap works.

This particular hard cap is the idea of MGL, it’s an idea that fascinates me, and I think it’s an idea that works.  It’s a matter of finding that threshhold.

For the sake of discussion, I’m presuming that level will be around 60%.  MGL said 62.5%.  I’m guessing it’ll be pretty close to that 60.

Tango 63:

“I’m saying that what Strasburg did hasn’t changed my view of the 75% hard cap.”

that is far different than what you originally said, which again was:

“if what we’ve experienced here is not enough to dial back forecasts to something reasonable, then nothing will ever prove it to some people.”

One is “due to sample size issues I’m not convinced I’m wrong”. ok, fine.

but your original statement was essentially

“the evidence is very clear that all of you should realize you were wrong”

Look, I think overall you are probably right. Strasburg is not Pedro. But I see zero evidence in the Strasburg results from 2010 to support your position.

This whole idea of a “hard cap”, either for rookies or established players is silly.  It is a “hard cap” only because the likelihood of someone being better than that (or a projection being better than that) is very, very small.  So the idea of a hard cap is tautological I think.  I don’t think that the likelihood of someone being better than any reasonable “hard cap” you want to make is zero.  If 50 years down the road (or even today) there was a pitcher who threw his fastball routinely at 103 with great off-speed pitches and great command, would we adhere to the rookie hard cap?  I don’t think so.

So, I think, as Mike said, we have a hard cap until someone comes along who breaks that hard cap, which sort of makes no sense (or at least renders the notion of a hard cap silly).

When I said that my hard cap was around 62.5% I simply meant that that was the lowest projection I have ever had for any pitcher, including Pedro, Maddux, Clemens, RJ, etc., in their primes.  I did not mean that no pitcher will ever come along who is better than that, or that I will some dat have a pitcher with a better projection than that. In fact, right now, I have Strasburg at around 65%.  If he were to continue to perform at his current level (and thus, the regression toward the mean would get less and less), his projection would continue to go down, perhaps to less than 60%.

Anyway, if I said or implied that (no pitcher can ever be less than 62%), then I am changing my tune.  And I am referring to starting pitchers only of course.

So how about we call it a “soft cap” and we all go home?

And while I think that a reasonable “soft cap” for an established pitcher is 60-65%, I don’t think a reasonable one for a rookie is 75%. I think that 60-65% should be used for any pitcher, rookie or established.  Of course, for the rookie, the amateur or minor league stats AND the scouting reports have to be other-worldly.  I think that both were the case for Stras, weren’t they?

Again, whatever number you want to use (62%, 65%, 60%) is not really a cap at all.  It is merely 3 SD or whatever.  If you put a cap at 60% can you say that it is impossible for a pitcher to come along who is 59% and you will eventually have a projection for him of 59%?  Of course not.  What about 58%?  Where do you draw the line?  There is no line.  It is just that the likelihood quickly approaches zero (but never really gets there) as you go lower and lower.  What is the “cap” on how hard a pitcher can throw?  How fast a person can run the 100 or the mile?  How tall a person can be?  I guess there are numbers that are physically impossible (a 30 second mile, a 1 second 100, a 20 foot tall person) with what we know today (but maybe not in a 100 years), but certainly we don’t want to use those as our “reasonable caps...”

I think the following seems reasonable, call it a “soft cap” if you want:

If your projection system projects a rookie at better than 75% or an established pitcher at better than 65%, go back and scrutinize it carefully to make sure you’re not doing something wrong because those sorts of projections should be extremely rare.

I don’t throw my lot in with the following, which is how I understand the “hard cap” Tango advocates:

If your projection system projects a rookie at better than 75% or an established pitcher at better than 65%, it is wrong, and you need to examine it until you find and correct the mistake which must surely be there.  As a consumer of said projection system, ignore it or handle it with a large dose of skepticism, because it is by definition faulty if it gives such a projection.

FWIW, Greinke’s first 10 starts of 2009 (75 IP, 283 PA):

W/PA 4.2%
K/PA 28.6%
HR/PA 0

Seems to me this is basically an argument about how soon you can identify the true outliers.

Tango says the odds of Strasburg really being as good as he was this year (or of any rookie being a 60% pitcher or whatever) are pretty low, and that’s because hardly anyone has ever really been that good.

Others say: good golly, have you seen him pitch? Or actually, good golly, have you seen his peripherals at all levels?  What we know about the universe of all MLB pitchers doesn’t apply as much to the guy who really is off the charts.

It’s a frustrating problem because you can’t do a study of all the Strasburgs; there aren’t enough of them (indeed, that’s the point of the argument).

MGL said this:

When I said that my hard cap was around 62.5% I simply meant that that was the lowest projection I have ever had for any pitcher, including Pedro, Maddux, Clemens, RJ, etc., in their primes.

He also said this:

In fact, right now, I have Strasburg at around 65%.

I just don’t see MGL how you can end up with that.  Pedro in 1999 (facing nearly 900 batters), had more K per PA, less BB per PA and less HR per PA than Strasburg (facing under 300 batters).

So, how is it possible that using Pedro1999 can give you barely a better forecast than Strasburg2010?  I don’t get that.  Are you also using scouting data that somehow is more favorable to Strasburg than Pedro?

Are you going off memory of your Pedro2000 and Pedro2001 forecasts, or have you verified that?

***

but your original statement was essentially

For the love of god, will people please stop paraphrasing me?  Don’t tell me what I must have meant.  ASK ME what I might have meant. I’m right here.

***

If 50 years down the road (or even today) there was a pitcher who threw his fastball routinely at 103 with great off-speed pitches and great command, would we adhere to the rookie hard cap?  I don’t think so.

I said with no scouting information the idea of a hard cap is sound.  If Brian said he also used the fact that Strasburg touches 100, then it would be a different issue.  But, he did not do that.  He relied solely on the fact of his performances.

***

Anyway, if I said or implied that (no pitcher can ever be less than 62%), then I am changing my tune.  And I am referring to starting pitchers only of course.

Yes, you actually said or implied it.  You basically framed it as a Bayesian issue.  I think forcing a hardcap at 60% (or whatever research shows) makes more sense than not having a hardcap, and seeing who-knows-how-many forecasts at 5% points below that.  It’s to ensure that for an algorithm that doesn’t use Bayes, and instead relies on shortcuts and other processes, doesn’t go overboard.  It’s a sanity check.

Basically, what Mike says here:

If your projection system projects a rookie at better than 75% or an established pitcher at better than 65%, go back and scrutinize it carefully to make sure you’re not doing something wrong because those sorts of projections should be extremely rare.

***

Of course, for the rookie, the amateur or minor league stats AND the scouting reports have to be other-worldly.  I think that both were the case for Stras, weren’t they?

No, that’s the main problem.  Even if they were, the scouting reports would have to include location.

***

I don’t throw my lot in with the following, which is how I understand the “hard cap” Tango advocates:

If your projection system projects a rookie at better than 75% or an established pitcher at better than 65%, it is wrong, and you need to examine it until you find and correct the mistake which must surely be there.  As a consumer of said projection system, ignore it or handle it with a large dose of skepticism, because it is by definition faulty if it gives such a projection.

Again, love of god, don’t paraphrase me.  OBVIOUSLY, a forecast can be anything.  It could be an ERA of 0.00.

If you give me 10 rookies that you forecasted at better than 75%, almost certainly you will get 1 or 2 right, and 5 or 6 wrong.  That makes it a bad forecast.  Don’t do it.  Don’t do it because your UNCERTAINTY OF THE MEAN is so great that you can’t possibly forecast a rookie at 70%, because doing so means that you are actually forecasting his MEAN at 70% +/- 25%.

And you can’t do that while using no scouting report data.  The performance data won’t be there for you to give that kind of forecast.

***

Like I said, if Strasburg is the ONLY rookie that Brian ever forecasts at better than 75%, then I will tip my cap to him and say he did great.  That he put his nuts out there on one guy, and he saved face.

Given that he’s said this:

2010 2011
2.55 2.47 Strasburg

3.03 3.21 Halladay
3.11 3.25 Johnson
3.24 3.40 Felix
3.24 3.48 Cliff Lee

I do think he’s got a serious issue.  He’s basically put all his cards on the K/PA table.  He’s basically saying this:
Strasburg + average pitcher = Josh Johnson + Felix.

For the love of god, will people please stop paraphrasing me?

He’s basically put all his cards on the K/PA table.  He’s basically saying this:
Strasburg + average pitcher = Josh Johnson + Felix.

Wisecracks aside… I agree with MGL, let’s just call it a “soft cap”, or a “double-check-needed indicator”, if you will.  And we should really be referring to a “hard cap” as a “4 Standard Deviation line” instead.

I think it’s fairly easy to reconcile the last few Mike Fast, MGL, and Tangotiger posts when we use those alternate terms instead of “hard cap” and “soft cap”.

Given how hard I’ve been here, I should go ballistic on the PECOTA forecast for Strasburg, no?

Colin::WietersPECOTA
as
Tango::StrasburgPECOTA
?

***

I should also give kudos to Brian for sharing all his work like this.

Right, soft/hard, call it what you will.

I just know that the next Mario Lemieux who comes along, arguably the perfect hockey specimen ever constructed, I’m still not going to say he’s among the best ever as a rookie, and I don’t care how many Junior League records he broke, and how overwhelmingly stupendous his talent is, and how I would have been guaranteed that he would reach Gretzky-level status in his fourth year.

The uncertainty level of the mean is simply that much greater for rookies than anyone else.  And for him to play at +4SD from the mean, when one SD for him is double or triple that of the veteran superstar, then I have to scale back the mean forecast point, so that his 90th percentile forecast can match Gretzky’s 90th percentile forecast.

It’s really that which motivates me.  If you make Gretzky’s 50th percentile as a 200% player, with 1 SD as +/- 10%, and he plays at 4 SD, then he’s playing at 240%.

If I make Mario’s 50th percentie as a rookie as 140%, with 1 SD +/- 25%, and he plays at 4 SD, then he’s playing at 240%.

This is as clear as I can make it.

So, how is it possible that using Pedro1999 can give you barely a better forecast than Strasburg2010?  I don’t get that.  Are you also using scouting data that somehow is more favorable to Strasburg than Pedro?

I believe that MGL has mentioned that he uses fastball velocity in his pitcher projections.  So Strasburg averaging 97 on his fastball would certainly explain the projection.

If it’s the fastball, then the better comparison point would be Randy Johnson then.  Why would Strasburg get the huge forecast, but not RJ?

Tango says:

Like I said, if Strasburg is the ONLY rookie that Brian ever forecasts at better than 75%, then I will tip my cap to him and say he did great.  That he put his nuts out there on one guy, and he saved face.

Given that he’s said this:
2010 2011
2.55 2.47 Strasburg

3.03 3.21 Halladay
3.11 3.25 Johnson
3.24 3.40 Felix
3.24 3.48 Cliff Lee

I do think he’s got a serious issue.  He’s basically put all his cards on the K/PA table.  He’s basically saying this:
Strasburg + average pitcher = Josh Johnson + Felix.

Doesn’t this imply that Brian is following your advice and not sticking his nuts out on another forecast? 

I don’t think it’s a K/PA issue.  Carlos Marmol is leading the majors in K/PA at 41.7% (!), and he only projects for a 3.48 ERA for the rest of 2010. 

I think it’s much more likely it’s an age issue.  Brian has said that he has extreme aging curves for pitchers in which they peak at 24 and then decline heavily after that.  This would explain why Lee and Halladay are so “high”.  Felix and Johnson are so high because they have outperformed their peripherals in recent years.  Felix has a 3.33 FIP over the past 3 years, which when combined with aging and regression gels nicely with his 3.40 ERA projection.

Also

If you give me 10 rookies that you forecasted at better than 75%, almost certainly you will get 1 or 2 right, and 5 or 6 wrong.  That makes it a bad forecast.  Don’t do it.  Don’t do it because your UNCERTAINTY OF THE MEAN is so great that you can’t possibly forecast a rookie at 70%, because doing so means that you are actually forecasting his MEAN at 70% +/- 25%.

This is what I am questioning and you still haven’t really answered me.  Why does the +/- have to be symmetrical?

For a guy at a projected 65% of the league average, he is much more likely to be 90% then he is to be 50%.  I agree that the less major league data we have on a player, the greater the +/- on his projection should be - but that does not necessarily mean that a rookie at 65% has an upside greater than 50%, it just means that his downside is much larger.

I think you have to incorporate age when deciding where your cap should be.  Brian is projecting SS to have a K/BB ratio of 4.5 and a K/9 rate of about 11.2.  Since 1960, there have been only 9 seasons where a pitcher had K/9 of 11.0 or better and K/BB ratio of 4.3 or better—once by Schilling (age 30), 3 by Pedro (25 to 28), and 5 by RJ (31 to 37). 

And basically, the pool of pitchers who can strike out 11 batters a game at Strasburg’s age and stay healthy is the null set.  Three pitchers have broken 11 at about his age:  Gooden, Wood, and Oliver Perez.  Gooden and Perez never came close again.  Wood did a few years later, but obviously fought injuries continuously. 

You can maybe make the case that Nolan Ryan and Sam McDowell belong in this category, if you adjust for changing historical strikeout rates.  But that’s a complicated adjustment.  At least for the past twenty years, no one under 25 has demonstrated an ability to do this.  And only Pedro has done it under the age of 30.

Tango, again, you may very well be right about 65% or 70% or w/e. I’m not saying you’re not.

I am saying that these 68 innings don’t give us any reason at all to “dial back forecasts to something reasonable”. If anything, these 68 IP are evidence that the projection was pretty good. They were dominating, elite innings.

either way, it’s only 68 IP so lets not make too much out of it.

Brian has said that he has extreme aging curves for pitchers in which they peak at 24 and then decline heavily after that.

Excellent point.  Yes, that could certainly explain it, and why it doesn’t affect Pedro2001 or RJ, because those guys were much older.

So, it’s possibly a combination of high K/PA and young age.

And maybe that’s why MGL has him so high as well.

***

“Why does the +/- have to be symmetrical? “

I already said that RA won’t be symmetrical, but wOBA basically will be.

But, do you really want me to say for RA Index as 75% +30%/-20%, as opposed to 75% +/-25%?

Kneel said:

I am saying that these 68 innings don’t give us any reason at all to “dial back forecasts to something reasonable”.

I said this:

What you can say to me, which is what Patriot is saying, is that this was a “let” (like in tennis).  It neither challenged my 75% hard cap, nor was it fairly tested because he’d have to have had a superhuman performance to knock down my cap based on only 274 batters.

Again, love of god, don’t paraphrase me.  OBVIOUSLY, a forecast can be anything.  It could be an ERA of 0.00.

How can we have a conversation if I can’t say how I understand what you’ve said?  If I’m wrong about what you meant, correct me.  But one of the principles of a good conversation is repeating back what you understood from what the other person said so that they can tell if you heard what they were trying to communicate.  Simply quoting you verbatim is not going to accomplish that.

I don’t mean by my “hard cap” definition that a player could never get exceed that in a given sample of performance.  I mean that I think you are claiming that an accurate forecast should NEVER go past the hard cap because it is impossible for a pitcher’s true talent to be that good.  Impossible, as in 1 in a 100,000 chance, or whatever it would need to be such that we practically will never see a pitcher whose talent (based on past performance record) is truly that good.

You did say that, but then you use the expected high variance around a rookie pitchers projection (which I agree with) as justification for the idea that you cannot project him much lower than 75%.  I’m saying the asymetrical variance of a pitchers possible ERA (especially at the extreme low end) is a way for you to reconcile your idea of high variance around a rookie’s projection with the idea that no pitcher could possibly have an upside better than the greatest season of all time. 

Wouldn’t you agree that, especially with inexperienced pitchers, much of the variance is not tied with whether or not he will be the best in the game, but whether his stats will not translate to the higher levels or get injured?

Mike#83: Tango can speak for himself, but I think he’s not saying that a pitcher can’t be that good.  He’s saying that there’s no way to know he’s that good.  Gretsky probably really was that good as a young player, but he did not “prove” it yet.

Mike said:

I mean that I think you are claiming that an accurate forecast should NEVER go past the hard cap because it is impossible for a pitcher’s true talent to be that good.

(Bolding mine.)

Hiz replies:

He’s saying that there’s no way to know he’s that good.  Gretsky probably really was that good as a young player, but he did not “prove” it yet.

Yes, exactly what Hiz is saying.  And Lemieux is probably a better example of a young star than Gretzky, if for no other reason than Lemieux weighed 60 pounds more, and had a longer reach.

***

Nick, if we could, I would prefer to speak in wOBA terms, and not RA terms.  Then, we could leave things as symmetrical.

So, let’s see what this means, practically speaking.  If I take a wOBA of .290 at +/-.040, that translates to an RA Index of:
75.5% +25.5%/-20.0%

Given that we’re talking about some estimates, I don’t see it that big a deal for me to say 75% +/- x%

Btw, Roy Halladay does not have the second-best predicted ERA for a starting pitcher (after Strasburg), according to Oliver.  That’s Mat Latos.  Probably some park effect there.  And Wainwright and Carpenter also nose in ahead of Halladay, and Lilly is right after Halladay.  So it doesn’t look like it’s only or mainly a K/PA thing that makes Oliver spit out the top projections for pitchers.

That’s Mat Latos.

“George is getting upset!”

***

I agree, it must be an age thing.  Brian must be pretty aggressive on that front.

"I don’t follow the NBA too much, but I guess Jordan, Magic, and Bird were like that too.”

They played at the level of alltime greats from their first NBA game on.  Bird/Magic were pretty much as great as they’d ever be, Jordan started great and then became greater.

Magic took his team to a championship as a rookie, Bird turned around a 29 win team into a 60+ win team, with most of the same roster as before.

Guys like Kobe and Lebron had more of a learning curve, as they came straight from high school instead of playing college.

Yes, I am going by memory for Pedro.  I do not know for sure that I have Stras better than I had Pedro at the time, plus my pitcher projection algorithm is way different now.

More importantly, Tango, you cannot possibly know that I should or would have a better projection for Pedro or Strasburg, given their raw unadjusted numbers.

I do an adjustment for fastball velocity, I do park adjustments, I do opponent adjustments, I do defense adjustments, and I do aging adjustments where I assume that a pitcher gets worse every year regardless of his age.  My basic projections system uses 4 years of component stats with each one regressed individually.

So you really cannot tell from the raw numbers what my projections are going to be.

Even if I have Stras at 65% and Pedro at 67%, there is so much uncertainty in that, that they are essentially the same projection.

If we are talking about projecting rookie pitchers without scouting information, then what ARE we using?  If we are using minor league and amateur information, then sure, there should be somewhat of a “hard” (again, it is not really “hard” - what if a pitcher came along who had a .25 ERA in 500 IP of minor league ball) cap around 70% or so, because no matter what, those numbers are very unreliable, as compared to major league ones…

My basic projections system uses 4 years of component stats with each one regressed individually.

I’m saying it would be quite difficult to get a forecast for Pedro2000 and PEdro2001 to be worse than Strasburg2011, given how much more reliable data you have of Pedro 1996-1999 than you would have Strasburg 2006-2009.

I already showed you how Pedro1999 took Stras2010 to lunch (not even accounting for the fact that Pedro1999 had a much higher run environment than Stras2010, or for the fact that Pedro had 3 times as many MLB batters that season).

For fun, I think you should run your algorithm for Pedro, so we can see how it would have responded to him.

So, that’s why I am questioning it.

I’ve been taking notes (I sleep from 730am to 4pm)

First, here’s my current top 25 mlb pitchers, size>=1000, column ‘mlb’ is total unweighted bf in mlb from 2007-2010

 
name                 age throws org_cd  size   mlb   era   woba       
Strasburg, Stephen    22    R    WAS    1049   274  2.58  0.258       
Latos, Mat            22    R    SDN    1090   775  2.93  0.274       
Wainwright, Adam      28    R    STL    1889  3106  2.93  0.275       
Carpenter, Chris      35    R    STL    1398  1596  2.95  0.277       
Halladay, Roy         33    R    PHI    2184  3671  2.96  0.278       
Lewis, Colby          30    R    TEX    1642   813  3.00  0.277       
Lilly, Ted            34    L    LAN    1739  2994  3.06  0.283       
Jimenez, Ubaldo       26    R    COL    1939  2799  3.09  0.279       
Lincecum, Tim         26    R    SFN    1985  3124  3.09  0.279       
Lee, Cliff            31    L    TEX    2019  2995  3.11  0.283       
Johnson, Josh         26    R    FLO    1617  1968  3.14  0.283       
Greinke, Zack         26    R    KCA    1947  2964  3.19  0.286       
Hudson, Tim           35    R    ATL    1332  2377  3.21  0.288       
Sabathia, CC          30    L    NYA    2156  3702  3.26  0.290       
Wilson, C.J.          29    L    TEX    1050  1474  3.32  0.289       
Lester, Jon           26    L    BOS    1868  2665  3.34  0.291       
Hernandez, Felix      24    R    SEA    2111  3442  3.35  0.292       
Santana, Johan        31    L    NYN    1937  3310  3.36  0.293       
Haren, Dan            29    R    ANA    2062  3505  3.37  0.294       
Oswalt, Roy           32    R    PHI    1814  3183  3.39  0.294       
Kershaw, Clayton      22    L    LAN    1677  1862  3.39  0.289       
Marcum, Shaun         28    R    TOR    1130  1908  3.42  0.296       
Kuroda, Hiroki        35    R    LAN    1545  1883  3.43  0.295       
Buchholz, Clay        25    R    BOS    1622  1416  3.43  0.294       
Peavy, Jake           29    R    CHA    1307  2467  3.45  0.295

Pedro in 1999 (facing nearly 900 batters), had more K per PA, less BB per PA and less HR per PA than Strasburg (facing under 300 batters).
So, how is it possible that using Pedro1999 can give you barely a better forecast than Strasburg2010?

#######################

If Oliver’s had included 1997, Pedro’s 2000 projections would be much better. Projections will usually lag a year behind performance, and performance has randomness, which projections smooth out.

mgl mention using sd’s instead of a hard cap - I think transforming components based on their z-scores is worth looking into. Regression is also designed to rein in the outliers, but sometimes, like with Strasburg, the performance is overwhelming even in a sufficient sample size.

‘Gretzky at 18’ - it might be more a matter of age than lack of mlb experience. Yount, Griffey, and Sheffield also only performed at a league average rate playing mlb at age 18 or 19. I start tracking some players at 16. I have stats for Strasburg from age 18. He is now 22. It’s a matter of how reliable is the translation from amateur, college and minor leagues. Yes, I believe the uncertainty of projections from that data will be higher than from mlb data.

nick/78 - I don’t consider the aging curves ‘extreme’, and for pitcher are fairly flat from 24 to 27, but each component has it’s own distinct curve.

My Wieters is much better than Pecota’s, my Montero was a logical error that was fixed, I’ll wait for next year for Harper (although it looks like he might be 70-80 bb, 160-170 so, about average on both, with of course lots of hr power). I think Lewis and Leake weren’t perfect, but still good. Lewis has turned in 3.5 war.

My thoughts on Strasburg going forward - in a admittedly limited sample set of Gameday data, he had consistently had a gb% right around 66%, which is quite high. However, in mb in 2010 this dropped to 47%, and his ld%, which was 11% in AA & AA, went to 24%. This is likely the source of his very low babip in the minors, and high babip in mlb. I’ve also commented earlier in this thread about his translated minor so% was .250, while mlb was .336.

I would expect his gb% to get back up over 60% (unless they are coaching him to pitch higher in the strike zone) with reduced k’s (about .300) and about a league average babip. If he is pitching up in the zone more, that could have contributed to the higher k rate. Would need to look at pitchf/x and compare pitch locations in AAA to MLB.

Last time I looked, Pecota didn’t have a forecast for Strasburg, but they do now I would say that they it looks rather pessimistic. Only a 40% chance of being a regular?

Without looking too hard at the numbers, I don’t know how you would have Lilly ahead or close to Lee.  Lilly would be a poor man’s Cliff Lee, no?

Gretzky/18 years old: in hockey, things are different.  Don’t compare an 18yr old hockey player to 18 at other sports.

***

If Oliver’s had included 1997, Pedro’s 2000 projections would be much better.

What is more reliable: 900 MLB batters in 1999 for Pedro in forecasting 2000, or 247 MLB batters for Strasburg in 2010, plus some 1100 other minor and college batters in 2007-2010 in forecasting 2011?

Do you want to call that a wash in terms of reliability, so that Pedro’s incomparable performance in 1999 was comparable to Strasburg 2007-2010?

Because that’s exactly what you are doing.  I’m not saying it’s wrong, but it’s a very aggressive position to take.

Rally: ok, basketball skillset translates much quicker, given that.  Thanks.

I believe you were saying that because I listed a 2.50 projected ERA for Pedro in 2000, and I have Strasburg’s current in-season projection at 2.58, that I essentially have them as equal in talent. I pointed out that Oliver for 2000 did not include Pedro’s (or anyone’s) 1997, which means that using the same methodology but on a complete data set would give a lower projection for Pedro, this not Strasburg’s equal.

From 1997-1999, Pedro had ERAs of 1.90, 2.89 and 2.07. Oliver starts at 1998, so it only used the 2.89 and 2.07 seasons to generate a 2000 projection. If 1997 (1.90) had been included, the 2000 projection would be much lower than 2.50, and thus closer to the actual 1.74 he put in 2000. Back of the envelop, using only actual ER and IP, weighted 1.0/0.7/0.5, gives 2.42 if only using 1998-1999, but 2.30 when 1997 is added in.

Forget about the years.  What counts is the number of observastions of performance (PA).

Looking ONLY at Strasburg 2009-2010, how many PA do you have? 

Now, for Pedro 1999, how many PA do you have?

Let’s say they are both 800-900.

Now, what is the forecast for Strasburg2011, and Pedro2000, using only the amount of data noted in this post.

I’m not sure what good the projections requested in #98 would be.  If all you knew about Pedro2000 was Pedro1999 and all you knew about Strasburg2011 was StrasburgMLB2010, then of course you’d get a better projection for Pedro2000. 

But that’s not all you know.  Oliver knows about Pedro1998 and it knows about Strasburg2008, Strasburg2009, and StrasburgMiLB2010.  The interesting question here is how much we really can deduce from Strasburg2008-thru-MiLB2010.  (And so far, Oliver’s predictions from that data have been pretty good.)

Hiz: the point I am trying to make is this:

GIVEN that we INTENTIONALLY limit ourselves to 800-900 last batters faced by Pedro as of Feb 1, 2000 (i.e., look only at Pedro1999), and the last 800-900 last batters faced by Strasburg as of Feb 1, 2011 (i.e, the 274 MLB batters in 2010, the whatever batters from minors 2010, and the college batters from 2009), then answer this question:

1. Which data is more reliable?
2. Who pitched better looking only at that data?

Now, unquestionably, the Pedro data is more reliable.

In addition, we know that PEdro pitched better than Strasburg’s 2010 MLB.  We highly suspect that Strasburg’s 2010 MLB is a better performance than the rest of his performance.  And so, Pedro’s 900 batters really shows how he pitched much better than Strasburg’s 900 batters.

Given that, it becomes an almost certainty that Pedro was better.

The only way for Strasburg to have a better, or even close to, the same forecast as Pedro is due to aging.

That’s why I am saying, start by controlling the sample size.

This by the way will tell you how much you are regressing.

Then you change it so they are both the same age.  That will tell you how much aging affects the forecasts.

Finally, you add more years, and that tell you more more certain you are of your mean estimate.

If Brian follows through here, we’ll get to the bottom of this.

It sounds like age is being considered only for the purpose of adjusting future projections.  But shouldn’t it also play a role in our estimate of Strasburg’s likely current talent?  Since 1960, among pitchers who logged more than 300 IP before age 24, the best performances ever are Fidrych (156 ERA+) and Mussina (152).  But both are under 400 IP, and we would presumably estimate their true talent at lower levels.  The best performances with a sample over 1000 IP are Gooden (135) and Blyleven (134).  I think it’s fair to say that Gooden, at .74 vs. average, was the greatest young pitcher (under age 24) in modern times.

Now, Brian is projecting Strasburg as a 165 ERA+ pitcher at age 22. So he’s saying that Strasburg is the greatest young pitcher of the last half century, and by a huge margin.  Not “he could be the best ever,” but “our best estimate is that he is by far the greatest 22 year old pitcher.” (And nearly as good as Randy Johnson was when he was winning 4 consecutive CYs.) And this is based on 68 MLB innings, plus his earlier stats. 

I think it’s more than fair to call that an “aggressive” forecast.

I don’t know if I can run through these scenarios. I have many thousands of lines of sql code that create tables along the way, making it difficult to do things such as change the number of years being input to generate the forecast.

I pointed out the missing 1997 data in order to try to put Pedro’s 2000 projection on an even footing with Strasburg’s. The standard model for a pre-season forecast is to take the past three years. I did this with Strasburg, but arbitrarily ignored 1997 (with a 1.90 ERA) for Pedro.

Now if we would take two years of Pedro (which was 1500 BF in 1998 & 1999) in mlb, and two years of Strasburg in college, Pedro’s (mlb’s) is going to be more reliable because there are fewer assumptions made in transforming the data. When using only mlb data, there’s park factors, weighting, regression and aging. When using non-mlb, you have to add in league translations, where the methodology is not as straight forward. My first BP Idol piece last year compared my method of calculating league factors by comparing all minor league levels directly to mlb to the chaining method used by most others.

Even so, as I mentioned above, projections smooth out year to year variations in the actual performances. In a piece last year at FanGraphs I measured the normal amount of error to be expected when comparing batter projections of size >= 800 to actual performances of size >= 600, and got one SD at about .035 wOBA. So even if a batter has a true talent .335 wOBA, 70% of the time we can expect it to vary between .300 and .370.

To review, first I apply park factors so that each player is normal within ea ch league each year. The aging curve is then applied to calculate a peak value for each player (basically what he would be expected to do at age 24 based on the performance and age in the given player season). Once also normalized for age, I do a matched pairs analysis of minor league paired to
major league data. The league factor is then applied to the player’s park normal data to get a mle. These are summed to get one record per player per season. These are then weighted and regressed, with an age adjustment to project to the next season.

If I have two equivalent mle’s, one from mlb and the other from minors, college, etc, one way to devalue the non-mlb is to regress it to a worse mean. Pedro gets regressed to the average value of a mlb pitcher. Strasburg gets regressed to the average value of a college/fall league player.

Strasburg’s raw performance was better than Pedro’s, but he was facing college batters. The component rates are run through an odds ratio calculation to compensate for the level of competition, and then regressed to his peers. At that point Strasburg is not as good as Pedro, but better than anyone currently in mlb. A couple forearm strains later, maybe that will no longer be true.

Pre-season, with no professional experience other than the AFL, I projected Strasburg to allow a mle line of 219/275/330, and in mlb he actually allowed 221/266/328. I know it’s a small sample, and I know it might not hold up, but it does look pretty damn good to me.

When the season is over in five more weeks I will run some diagnostics and publish the results at THT.

"Now if we would take two years of Pedro (which was 1500 BF in 1998 & 1999) in mlb, and two years of Strasburg in college,”

Again, forget about this “2 years” business.  Only talk about number of batters faced.

Make them both equals if you can.

"Pre-season, with no professional experience other than the AFL, I projected Strasburg to allow a mle line of 219/275/330, and in mlb he actually allowed 221/266/328. I know it’s a small sample, and I know it might not hold up, but it does look pretty damn good to me.”

I agree, congratulations.  An excellent projection. I didn’t even attempt to project from his college stats. 

But I did project his MLB debut date slightly better than you

My understanding is that this is a difference in how each side believes players should be evaluated/projected.

Please, correct me if I’m wrong.

Tango is saying that we want to give a projection that the player will beat half the time, and not beat half the time. The general point being that if you run the season 10,000 times (or have 10,000 players with the same projection) they will average out at the projection.

Because of his emphasis on hitting the mean, he doesn’t think that a young player with so little information should be projected so highly on the basis of the results. Because of the huge swing in uncertainty, he suggests erring toward the mean. If you give a 65% RA projection, how can you realistically expect the player to beat that half the time given the uncertainty?

The other side seems to be much more interested in nailing what the player is capable of, or what they are expected to do. That is a much different type of projection from what Tango expects. I think that the high ceiling makes for a higher projection because of the higher expectation of success than failure. So sure, he may not exceed that projection much, but they assume a much smaller chance of failure.

The benefit of Tango’s method is that it’s easy to understand that the mean is just that—a mean. It’s not what Tango expects him to do, but rather what the true talent of a young player without much MLB data to evaluate projects to be. And it may go up more later with more data, but it shows a lot of variability.

The benefit of the other method is that it has a much bigger chance of nailing very unique talents who have significant upside but little data supporting it. Tango will likely never catch the outliers since that method is more conservative (but he’s not really trying to!). And the other method will end up overestimating the true talent of a lot of players with limited data.

Let’s say | is are the best/worst projections, and o is the mean projection.

Tango:
|---o---|

Other:
|-o-----|

I think that’s all it comes down to—a difference in evaluation. The problem is when Tango applies his error bars to the other’s mean, or vice versa. It looks like Tango isn’t giving Strasburg enough credit, and to Tango it looks like you’re giving him too much credit.

Of course, as is often the case, I could be totally wrong. In which case, please let me know, because I find the conversation interesting (although a bit dense at times).

Sal, where the f-ck were you two days ago?

A perfect post.

Sal, from that illustration it seems as if a forecast that is out there like Brian’s on Strasburg is going to be too optimistic more than half the time.

That is true if Oliver’s method produces a lot of these forecasts of league domination.  But from what I see, He’s giving that forecast to very few pitchers.  Pedro at his peak would have been better than the forecast about 50% of the time.  Our sample on Strasburg is one season, and he hit the forecast.

Unless there are a bunch of pitchers Brian is projecting to be dominant that wind up closer to league average, It seems like these extreme projection make sense.

Yes, a sample of 2 pitchers and maybe 4-5 Pedro seasons, one Strasburg season is not enough to prove that these forecasts are right, that they aren’t too optimistic.  But nobody as produced any evidence to suggest that they are not right.

Using Sal’s great technique (thanks again!), this is what we have as the likely true mean of Pedro at his peak, Strasburg in 2011, Roy Halladay at his peak:

It WILL NOT be this:

It COULD be this, but then you have to explain why the uncertainty level was able to be reduced so much given such limited performance data:

You COULD try to explain it on the basis of scouting data, just as Ozzie Smith’s fielding greatness was so self-evident that we didn’t even need him to play one game.

This is why I kept asking for the uncertainty level of the mean forecast.

I agree, Sal’s comments focused the discussion.

I estimate the means, and that’s what I have been discussing. The expected variances are more murky.

I mentioned above about comparing batter projections (of various sizes) to actual stats the following year and finding the error distribution. Here’s what I can do - repeat those empirical tests, but also group by source of the data (college, rookie, Single-A...MLB). Then I would be able to say that a sample of 1000 of MLB data has an SD of .035 wOBA, while the same sample size from college has an SD of .060 (numbers made up for illustration purposes only). This would allow me to provide error bars that reflect the reliability of the data source.

A projection is always an estimate of a player’s true talent at some point in time.  I don’t think that the uncertainty is necessarily symmetrical around that number.  In fact, I don’t think it is for a variety of reasons:  One, the talent distribution is not symmetrical, and two, the chance of injury really skews it, assuming that the projection includes chance of injury (which it should, but it doesn’t have to), which, for a pitcher, is always significant.

The uncertainty around a projection is always higher given a smaller sample of historical performance that it is based upon, as Tango’s graphics illustrate.  Always.  There are other things of course that can affect the width of the error bars (the size of the uncertainty) but the primary thing that drives it is the size of the sample that the projection is based upon.  We have discussed this with regard to the Pecota projections (and their collapse, breakout, etc. numbers).

Now, as Tango also says, the size of the uncertainty can also be affected by scouting information.  And that is simply because the distribution of possible talent gets reduced once we can use scouting information to reduce the size of the population (and the spread of talent within that population) that the pitcher likely comes from.

For example, if we only have Strasburg’s numbers, then our “Bayesian” analysis includes the chances that he is a true talent pitcher from the population of all pitchers, which includes the really crappy ones who only throw 88 and have no command.  But once we reduce the population to pitchers who throw 97-99 and have very good command and very good off-speed pitches, obviously we reduce the spread of the distribution of true talent.  Basically for the purposes of Bayes, with no scouting info, we have a distribution from 60% to 130%.  With scouting, we probably have 60% to 100% or maybe even 60% to 90% or 85%…

Haha yeah, Sal’s post is what I was basically trying to say, apparently not very eloquently.  You can still have Strasburg projected to be the best pitcher in baseball, while allowing a large error range for the projection by giving him a lot more downside than the more established pitchers.

Sorry tango, our schedules are reversed so I walked in to a massive discussion and it took me 2 days to process it.

MGL, of course you’re right. In reality, distribution isn’t going to be symmetric. But for projection, how sure can we be about what type of distribution it is, and where the mean will be? Let’s say we know the distribution has a “positive skew” like this (on the right):
http://en.wikipedia.org/wiki/File:Skewness_Statistics.svg

Big chance of being really good (on the left side in the hump), but because of injury, a chance of tailing way off into the bad side.

The question that I’d ask is how do you know how much skew/what mean to give each pitcher? Now you’re talking about not only giving a mean for each player, but also a probability density function to explain what type of skew they have. Then you could compare Pedro/Doc/Strasburg using their various probability curves.

I think that even then, even if we were to go to those absurd extremes, we would have to put the mean of Pedro or Doc higher than Strasburg—they have more data showing better performance than Strasburg has, which should put their mean higher. You can make Strasburg’s curve “flatter” in the sense that it has a lower mean but has the potential (in a very small amount of outcomes) to be better than them, but to say the probability of them being better than the elite pitchers in the game seems like a stretch.

Projection has come a lot way in the last decade. A really really long way. But we’re still wrong a whole lot. I agree in principle with what Tango is saying. Rather than trying to nail the outliers, let’s focus on getting the mean right first. Then we can worry about the various probability distributions and specifics once we know we’ve got that mean nailed. Otherwise we end up projecting Strasburg based on limited MLB batters faced as better than Pedro in his prime, and there is NOTHING in the data to indicate that we should be doing that (or at least that I can see).

Does he have a chance? Yes. But a better chance than Pedro? That’s what your numbers are saying, but can we honestly believe that as baseball fans? That Strasburg would top the best most commanding peak in baseball in his rookie/sophomore seasons?

That being said, I’d love to see every projection on a nice distribution with the mean marked as a circle and the median marked as another circle, for both their runs allowed and their innings pitched (or wOBA and PA for batters). But to do that based on data, we’d still have trouble getting the outliers (like Strasburg) right based on data, because they don’t have as many comparables (that’s why they’re outliers).

I just think it comes down to how we deal with uncertainty in our projections. Tango does it by saying, “This is what I’m comfortable giving for him based on what we know—he will be better half the time, but nobody should predict him as being better than X% RA (the hard cap) even as a best case scenario because there just isn’t data to support that type of a projection.” Others want to say “There may be a lot of variance, but based on the potential we see in his peripherals/talent level, we should be projecting him to do as well as some of the top talents in the league, otherwise we have a much bigger chance of underestimating his talent and looking like we don’t know what we’re talking about.”

If we were to put Strasburg in a group with all other hard-throwing pitchers with similar stuff, what sort of distribution would we expect? I would think we’d need to take into account the chance of injury much more seriously considering how many hard throwers blow out their arms. That would drop the mean down, wouldn’t it?

I tend to agree with Tango’s method because it seems more conservative, and I think if we had 10,000 Strasburg 2010s, his mean would probably be more on target. For the one 2010 Strasburg season we had, Brian beat him out. Do we give Brian more credit for getting this one right, or do we give more credit to the method that seems to be based on the historical data a bit better? I would think that sabermetrics should err on the side of data, but I’ve got about 20% of the brainpower of you folks on the forefront, so I could be missing something significant here.

I guess no one else buys this idea, but I don’t think we can treat rate performance and injury as totally orthagonal.  Strasburg may be a 2.50 pitcher today (or rather, a few weeks ago), but I think there’s virtually no chance he remains a 2.50 pitcher AND provides the Nats with a lot of innings over the next few years.  That is, the chance that the Nationals get 500 IP of 2.50 pitching over the next 3 years from Strasburg is close to zero.  On the other hand, 500 IP at 3.30 doesn’t seem out of the question. 

Or look at Gooden.  Was he a “real” .60 pitcher at age 19-20 who then got worse, or was he lucky his first two seasons?  Probably the former.  But the fact remains that no young pitcher has ever sustained that level of performance for very long.  So if I’m trying to forecast Strasburg’s next game, maybe I go with 2.50.  But if I’m trying to estimate his value over the next few seasons, I don’t think an expectation of 2.50 is reasonable.

Guy: If you look at Brian’s forecast for Felix and Doc and Josh Johnson, then he would agree with you.  His forecast for Strasburg is so far away from them, that it seems virtually impossible that it could be correct.  If you look at the difference between Stras and Josh/Felix, could that possibly be as much as the difference between Josh/Felix and an average pitcher?  Not too mention that your uncertainty level of Stras is much higher than Josh/Felix, that Brian is REALLY going out there on the Stras forecast.

This is what Oliver is saying:

He’s putting the true talent mean of Felix/Josh at halfway between SS and the average pitcher.  It certainly looks unreasonable, doesn’t it?  (Using only performance data.)

Furthermore, our certainty of Felix/Josh, using only performance data, is much tighter than our certainty of SS’s true talent mean (I marked it as 3 times wider than Felix/Josh.  If you think it should only be two times as wide, then it would look like this:

And remember, Brian uses no scouting information.

And suppose he did.  Well, we’d also be more more sure of Felix/Josh, so this is what it would look like if Brian also used scouting information:

At best, if you used performance and scouting, it should look like this:

And if you only used performance data entering 2011, at best it would look like this:

And, really, you guys implicitly agree with me.  When I asked early in the season about who you’d want to give a 4/100MM$ deal to, you had Strasburg between Felix and Josh Johnson.  And, if I were to ask again, and tell you that there’s no health issue entering 2011, I think you would give me the same answer.  Greinke and Lincecum would go down, Felix might take the #1 spot, Johnson would go up, and Strasburg would be somewhere in there.

That’s why it is CRITICAL to ask what the uncertainty of the mean is, because once you see it graphically like above, it’s going to be much clearer of the implications being made.

"That is, the chance that the Nationals get 500 IP of 2.50 pitching over the next 3 years from Strasburg is close to zero.  On the other hand, 500 IP at 3.30 doesn’t seem out of the question.”

I agree.  If nothing else, because of chance of injury.

A great parallel is Doc Gooden of course.  But a more recent one is Felix. 

At 19 years old, facing 328 MLB batters, he strikes out 23% of batters (which, being 19, is tremendous).  His FIP was 2.85.  He had a 96MPH fastball, plus he had a curve and changeup.  And his GB rate was 67%.  Then his next 3 years were good, not great.  And then we have his two last great seasons.

A forecast for Strasburg entering 2010 at 66% of league average would basically make that kind of path impossible, unless it was 66% +/- 33%.  A 75% +/- 25% kind of forecast would give you a chance to say that he can be as great as Pedro at his peak if it all comes together, but also that perhaps he won’t be able to get it together, like some of the struggles Felix, and all young pitchers, have had.

And Strasburg’s elbow just decided to make a large swath of this conversation moot. He’ll be getting Tommy John surgery.

(Not to say the more theoretical parts of the discussion aren’t useful - they really are. But now we’re never really going to know who was “right” about Strasburg’s true talent this year.)

Now the relevant question is whether the Tommy John surgery will decrease or increase Strasburg’s projected career totals.  (I’m not actually kidding.  Much.)

Tango, it didn’t cross my mind at the time, but when you said you thought it was reasonable to think that Strasburg would improve from year 1 to year 2, were you implying that that was the case only if Strasburg maintained healthy?

No, including chance of injury.