THE BOOK--Playing The Percentages In Baseball

Hey Tom,

J Cross (I believe) actually tested Oliver’s projections from last year on this site recently, and found that they did quite well (and most importantly, beat the monkey!). Maybe someone can find the link.

Brian looked into the Yankee forecasts, and there was indeed a bug, but Montero still looks very good. We’ll keep digging and if nothing changes, I’ll throw Brian to the wolves and have him write up an article justifying that projection.

And for total transparency, I’m keeping an updated list of known issues and planned updates to THT Forecasts right here, so that you can see what we’re working on:

http://www.hardballtimes.com/main/forecasts/todo/

J Cross tested Oliver (late addition to the group).  Oliver came in third, behind CHONE and ZIPS, but ahead of Marcel.

David, can you send me an email showing me how THT writers can access the pages?  I would think you’d need to give us access, since you say we’ll be providing updated commentary

David, the testing works ok for me, to a point.  And that point being that we trust Brian for the openness he has shown the community.

It’s unclear which version of Oliver it was that Brian sent in, and whether periphery things like park factors and component weights were changed based on in-sample data, etc.

To be fair to all the other forecasters, a late submission should count like an amateur on the PGA tour: it counts for standings, but not for prize money.  That’s why I said what I said.

***

Good job on the transparency.  By the way, SF *is* part of the official OBP definition.  Maybe it’s your treatment of IBB?  I haven’t looked.

Sean/2,

You should be able to access it using the same user name and password you did to submit player comments. If that isn’t working for you, let me know, but it should.

Tango/3,

I meant SH.

The MLE on Montero shows a better batting line in 2007 than his actual stats in rookie league.  That can’t be right.  I know Yankee stadium is good for homers, but that is beyond belief.

Is the fielding rating runs above average or something like runs above average plus position adjustment?  Posada’s defensive ratings look like Pudge in his prime.  If he’s a -1 fielder but catchers are getting a +15 position adjustment, then I can see +14 for him.

Sean/5,

Above position. I agree that it looks wrong. There’s something going on with some of the catcher defensive ratings. Brian is looking into it.

The other thing to consider is that the test I did doesn’t really work to Oliver’s strength (b/c it was a small pool of hitters and Oliver specialized in minor league equivalents).

I’m going to look at a somewhat wider pool for pitchers but what I think we *really* need to do is carry these projections into the future.

This year Steamer (in addition to having significantly better methodology, I hope) is projecting a much larger pool of hitters (something like 1250 compared to last year’s 270).  What we’re going to try to do is the equivalent of pecota’s 6-yr forecasts by projecting 2011, 2012, 2013… for young players.  Now, for Steamer, these projections probably won’t be that good.  Minor league stats aren’t our forte.  But, I think it would be good to see that Oliver is making better projections about what today’s 20 yr. olds are doing in 2013 than Pecota (or vice versa).

So, Brian, if you’re out there, is this something you have any interest in?  To really judge the accuracy of the hardest projections I think we need to apply aging curves and carry them forward.

Tango, this should be pretty easy to do with Marcel, in fact I could probably get this done myself just by applying Marcel’s age adjustments forward.

Jared: actually, I don’t think you could necessarily just apply the age adjustments.

Those work based on the player having played the previous three years for the next season.  But, that doesn’t mean that it’s going to continue to move forward.

What I’d really have to do is run a regression of year x+2 to years x, x-1, x-2, and I’d have to include some sort of attrition rate to account for the fact that the player might not even be in the league.

Basically, it’s kind of a cheap thing that the forecast tests work only on players if they’ve actually played. 

That’s why the best thing is rate+playingTime forecasts (WAR) would be the right thing to do.  But, most people agree that playingTime is really tough to forecast, so, you really get cut-off from the real test you want to do.

The Montero 8 WAR peak comes in only 445 PAs, too, but I assume that is still pre-update on the Yankee hitters.

From looking back over Brian’s MLE article on BP, it seems like the MLEs are designed to match production in the minors to what a hitter will be expected to hit once he reaches the majors, not what he would be hitting if he were in the Majors at that point in time.  As Brian pointed out in the BP article, that means the MLEs for the low minors will really be a combination of a translated batting line for that season, plus an aging component to cover the years it takes to get to the Majors.  If I’m understanding it correctly, that means that a low minor MLE for Oliver is really adding in a few years (however long it takes the typical player in the sample to play his rookie season in the Majors) of aging.

If that’s true, and Oliver is just plugging the MLEs into the weighted average for the projection, then it’s really plugging in what Montero’s line would be expected to translate into a few years down the road, but still crediting him with that performance as a 20 year old.  Then, unless the 6-year forecast is doing something other than what I assume it is, that would take his projection for age 20 using translations that give him the benefit of a few years of aging, and then keep adding improvement for aging up from 20.  If that’s correct, then the result will be that future forecasts are going to have a tendency to run wild for players from the low minors because they’re doubling up on their aging.  That seems to me to be the most likely reason for the Montero projection going so high (as well as maybe giving him credit for sticking as an average catcher along with his huge bat).

Maybe there has to be a reverse-aging done to the MLEs plugged into Oliver?  For example, if the typical player from high-A to the Majors in the sample is 3 years, then you could put whatever aging curve you are using in reverse and dial back 3 years of aging from the MLE, and then plug that into the projection.

re:#7

and Oliver specialized in minor league equivalents).

Well I have to say, not today it doesn’t.

But I’d love to see someone start assessing multiyear forecasts.  It’s something I’ve wanted to do ever since PECOTA started publishing them, but I’ve never captured the data in time and then they all just disappear.

At one point I had a small group of player projections from an early PECOTA run that I found a few years later.  As I recall, the future forcasts looked pretty lousy.  It wouldn’t surprise me if a regression monkey would do just as well.

There’s something going on with some of the catcher defensive ratings. Brian is looking into it.

David - Its not just a catcher problem.  Every Yankee MLB infielder has a 2009 fielding run value much higher than UZR and my BZM fielding metric.  Also, it is not clear whether what is being shown as a fielding run value is total fielding runs for the year or runs/150 games.  Either way the numbers seem way off.  Is there no cross checking whether the totals for individual players on the 2009 roster show any relation to the team’s total runs allowed?

Multiyear projections are close to an impossible task.  Use Marcel 2003-2005 stats, age adjust 3 years into the future, and see how well you can project 2009.  Probably not very well.

Peter/11,

It’s total fielding runs.

As for any questions about how Oliver works, you’ll have to ask Brian. I’ve e-mailed him about this thread; hopefully, he’ll come and comment himself at some point today.

As for any questions about how Oliver works, you’ll have to ask Brian. I’ve e-mailed him about this thread; hopefully, he’ll come and comment himself at some point today.

Not a very satisfactory answer David.  Brian is a good researcher and he will probably find the mistakes in his fielding system soon enough and fix them.  Proofing your own system is difficult to do and it’s not a big deal to find errors in a first iteration.  I appreciate that you are being open about what needs to be fixed and keeping a list for everyone to see.  But that should have been done before the product was offered a premium offering.  That is what beta testing is for.  And that was your (meaning THT) responsibility not Brian’s.  Somebody should have at least been checking whether the system made sense.  As it stands right now what you are offering is not nearly as good as what Fangraphs has for free.

The Montero MLE issue, which I suspect is really a Trenton or Eastern League MLE issue, really bugs me.  Either CHONE and ZiPS are badly wrong, or PECOTA and Oliver are badly wrong.  (I suppose it’s theoretically possible all four are badly wrong, but let’s ignore that possibility at this point.)

I haven’t looked at the Oliver numbers at THT enough yet to know how the MLEs look for players other than Montero.

From what I’ve seen of PECOTA’s MLEs so far, everyone who played at Trenton has an MLE nearly equivalent to the raw numbers they put up at Trenton.

Dan Szymborski’s park factors from 2005-2007 have Trenton as somewhat of a pitcher’s park, but not the most extreme park in the minors, for sure.  PF for runs 0.96, for HR 0.84, for doubles 1.03.
http://www.baseballthinkfactory.org/files/oracle/discussion/2007_minor_league_park_multipliers/

Now, it may have nothing to do with Trenton specifically and may be a general Eastern League issue.  It’s just showing up at Trenton because we have only Yankees player cards for free viewing.

Wow, some of these look “crazy”.  (Scroll all the way down.)

http://hardballtimes.com/forecasts

Yankees at 106 wins.  Toronto at 65.  White Sox at 63.  Generally strikes me as too wide of a spread.

As a test, can either Brian or someone at THT input Alex Rodriguez’ age 19-21 seasons (1995-1997) into the projection system to project his age 22 season and to give a 6-year forecast based on those 3 seasons?

At age 19, A-Rod spent half the year in AAA and half in the Majors, and then spent his age 20 and 21 seasons in the Majors.  So, that should give us an idea of a baseline for a hugely-touted, stud prospect at that age with minimal effect from MLEs in the projection.  Then, compare his 6-year forecasts to the top forecasts from players around that age in the low minors.  Players who are still in the low minors shouldn’t project anywhere near what one of the best prospects in recent memory who already had multiple very successful seasons in the Majors would project to, but if Montero is projecting to a .450 wOBA 4-6 years down the road, I suspect that might not be the case.  If prospects in the low minors or with recent histories in the low minors are projecting close to what A-Rod would have, that would likely indicate an issue with how the MLEs are effecting forecasts.

League run-scoring levels in 2009:

The Eastern League is in the low half, but it doesn’t seem extraordinarily out of line from the others.

Wow, some of these look “crazy”.  (Scroll all the way down.)

http://hardballtimes.com/forecasts

Yankees at 106 wins.  Toronto at 65.  White Sox at 63.  Generally strikes me as too wide of a spread.

They look better than they did an hour ago, when the Yankees were projected at 111 wins and the White Sox at 55.  Both of those are definitely crazy projections. 

Why is it that PECOTA/BP and Oliver/THT are having a much harder time getting their projection systems to work than CHONE did?  PECOTA so far seems to be offering only limited transparency such that it’s hard to figure out whether to give any credence to their projections.  Oliver is in the same boat with me at this point, but I’m hopeful that will not remain the case.

I think part of the issue with the Yankee MLEs is that they are being projected into Yankee Stadium, which based on last year is very favorable for hitters. Depending on how everyone is handling the park factors, that can have a large effect on the final results.

Peter/14,

Sorry you feel that way. I think Oliver is a very good system, and the projections look mostly right to me, minus a few that we are looking into. Moreover, we’re offering depth charts updated every week all season along with weekly updated projections (based on the full Oliver methodology) and player comments, which is a lot more than Fangraphs offers.

Are we perhaps not looking at CHONE (ZiPS) hard enough compared to the for-pay systems?  My initial response is no, but it’s something to consider.

And Mike, are you referring to just this year, or at their inception?  CHONE hasn’t made any significant changes recently (that I’ve heard about), while PECOTAs seen a software change (among other things) and Oliver’s getting an overhaul/expansion with THT.

I’ve looked plenty tough at CHONE, ZiPS, MGL, Bill James, PECOTA, Pete Palmer, Ron Shandler, Marcel, and… there was another one, for the 2007 forecasts.  That’s the kind of evaluation that should be done.  Check out those threads.

***

My response to the Montero/Wieters issue:
http://www.insidethebook.com/ee/index.php/site/comments/best_age_22_26_war_rookies/

#20—Is there some level of regression/weighting used for park factors for new stadiums?

"Why is it that PECOTA/BP and Oliver/THT are having a much harder time getting their projection systems to work than CHONE did?”

I have a much bigger staff at my disposal.

There certainly is a lot of work that needs to be done, and I’m very skeptical on the projected superstar performances of so many minor leaguers, but here’s what I see as selling points that my projections can’t offer, and that PECOTA does not offer:

7000+ players, including college stats.  I’m projecting about a third of that, and I think PECOTA projects a similar number of players.

The player comments.  It’s mostly the same group of bloggers that produced the THT season preview books of the last few years.  A huge number of people are not numbers oriented and would prefer reading player comments.  I’m far more numbers oriented than most but I greatly prefer reading the comments than scanning a stat line.  I don’t often pay attention to the projection, why should I, I’ve got my own system.  But I always read the comments.

Here are two selling points on the comments being more interesting than on the PECOTA pages:

1. BP has some excellent writers, but it’s a smaller group producing them.  Every THT comment comes from a blogger who is a fan of that specific team, who has probably watched that player every day.
2. BP’s 2010 comments are only in their book, their contract does not allow them to go onto the projection pages until next year.

CHONE hasn’t made any significant changes recently (that I’ve heard about).

Every year has seen little tweaks here and there.  This past year saw the migration from excel to access - this is part of the reason percentiles and multiyear have not been repeated.

I think part of the issue with the Yankee MLEs is that they are being projected into Yankee Stadium, which based on last year is very favorable for hitters. Depending on how everyone is handling the park factors, that can have a large effect on the final results.

1. I didn’t realize that MLEs were translated for a specific major-league park.  Without knowing otherwise, I would have assumed it was translated to a neutral major-league environment.  PECOTA, at least, implies they are doing this since EqA numbers are supposed to be independent of level and park.

How do Rally and Dan S. handle their MLEs?

2. Baseball-Reference reports a park factor of 0.97 for New Yankee Stadium.  At least on the surface, this seems correct to me, since the Yankees scored ~6.02 R/9inn at home and ~5.51 R/9inn on the road and allowed 4.30 R/9ip at home and 5.08 R/9ip on the road.

David, how do you get a big hitter’s park factor for Yankee Stadium?

I agree with Rally that multi-year forecasts probably won’t be very good but I still think there’s utility in trying to do them; and (like with chone v2.0) trying to see how we should weigh “scouting” information when looking forward.

Tango, maybe we could project the chance that a player is in the majors (as pecota does) and their expected performance *given* that they’re in the majors and then see how systems stack up but I haven’t really been able to wrap my mind around this problem yet.

btw, here’s the update with Oliver added (copied from the other thread):

R w/ actual:

R with actual ‘09 OPS:
0.638 Chone
0.624 Avg Projection
0.623 ZiPS
0.607 Oliver
0.590 Marcel
0.583 Fantistics
0.568 Sporting News
0.567 Steamer
0.564 PECOTA

Jeff/20,

Hopefully, Brian can answer that question. Honestly, I don’t know.

Sean/22,

Thank you. THT Forecasts, though not yet a perfect product, definitely offers many things that no one else does, and it’s good to see someone bring some focus to that. And the problems we do currently have are all things we are working hard to fix, so it’s not like they’re going to persist forever. In the meantime, I’m going to do everything I can to be as open and upfront as possible about any problems we’re having.

Yankee stadium was great for HR last year, but depressed other offensive events so the net effect is fewer runs scored than on the road.

I also think that these Oliver projections w/ comments have a lot to offer over and above free systems.  Of course, I’d pay for CHONE too if I had to.

David #20 - Let me clarify.  I am not saying that you won’t eventually get it right.  I have confidence that THT, which I think very highly of, and Brian, who I also think highly of, will work together and fix things to come up with a fine product.  Like Rally, I also think that the blog commentors are a terrific idea.  Unlike Rally, I don’t see any value in having six year projections on the 6000 of the 7000 minor league players who will never play even one game in the major leagues.  The point of my previous comments is about the timing.  I think it was very wrong to come out with a premium product that needs so much fixing when just a little forethought could have resulted in a much better initial offering.

David #20 - Let me clarify.  I am not saying that you won’t eventually get it right.  I have confidence that THT, which I think very highly of, and Brian, who I also think highly of, will work together and fix things to come up with a fine product. 

Ditto for me, and I would also like to emphasize how much I appreciate David’s commitment to transparency.

The point of my previous comments is about the timing.  I think it was very wrong to come out with a premium product that needs so much fixing when just a little forethought could have resulted in a much better initial offering.

I do find it a bit ironic to have posted my questions about Montero’s PECOTA only to have them followed up a couple days later with a more outlandish prediction for Montero by THT itself.  Eh-heh.

Point taken, Peter.  Given that people lose interest in projections after the season starts, it was either roll it out now or wait until next year.  We decided to roll it out now and take our lumps.  No one has to pay for this product if they don’t want to, and rolling it out now will make next year’s product much better.

The other strategy would have been to roll this out for free, but that would have been a mistake, IMO. Once something is free, people don’t want to pay for it in the future.

Peter/31,

That’s certainly a legitimate viewpoint. The way I see it, no matter when we launched there would have been some bugs, since all of you can do a much better job spotting problems than just the few people that worked on developing THT Forecasts, so waiting wouldn’t have meant we could offer a perfect product. With that in-mind, I’d rather put out a very good product earlier than a not-quite-perfect product later: People can feel free to discount projections they don’t like for the moment, but there’s still a lot to like and I’d rather give people access to that as soon as possible. Two weeks from now, THT Forecasts will be much better than it would have been if we had held off releasing it until then, and everyone will have had two weeks to use the product. It’s a win/win as far as I’m concerned.

Also, I apologize.  I should really start reading things before I comment on them:

For each player, we’ve listed six years worth of projections to help see how high a young player is expected to peak or when an older one might fade away

This is exactly what I was hoping for.  Excellent!

studes #33 and David #34 - I understand the need to have a projection system released before the season starts and preferably as soon as possible.  But all these problems so far have been identified by relatively few people in a matter of a few hours.  You have known this was coming for weeks, if not months.  There was plenty of time to release beta versions to a dozen people or so, gather feedback, and make the changes.  It might have delayed the introduction 2 days, not 2 weeks.  You had ample warning of the kind of critiques you were going to get from the discussions about BP’s release of a buggy PECOTA.  But enough said.  What’s done is done.  I want you guys to be a big success.  Fix it and sell millions and all will be forgotten.

I think a fair consideration along Peter’s point-of-view is to have had a limited rollout to a select few, say even just to people of this blog.  You get people to test it crazy for say 3-5 days, you can even reward them with a free subscription for a year for anyone that finds say at least 5 problems, etc.

I presume the fair consideration along David’s point-of-view is that they’ll launch full throttle, take their lumps and move forward.

***

This is the same history as PECOTA, when they asked readers do they prefer Beta or wait for production-ready.  They preferred Beta.

Of course, Beta means almost-production-ready with a few tweaks.  The readers there obviously believed the product was alpha, not beta.

And, we’ll see how THT readers respond in this case.

I posted at the same time as Peter.  Eerily similar post.

Just bought them.  Wow!  They look terrific.

Yankee stadium was great for HR last year, but depressed other offensive events so the net effect is fewer runs scored than on the road.

This is true in terms of how runs actually scored in 2009, but when I looked at the component park factors I found it boosted lefties’ wOBA by about 2%, and righties’ by about 1%.  It just didn’t manifest itself in the actual run scoring.

Also, a big part of the lefty HR park factor was Johnny Damon, who hit 17 of his 24 HRs at YS.  If we remove just him from the list, the lefty HR park factor drops from 1.18 to 1.09.

Of course, we shouldn’t make too much of a single year’s park factors anyway.

I do find it a bit ironic to have posted my questions about Montero’s PECOTA only to have them followed up a couple days later with a more outlandish prediction for Montero by THT itself.  Eh-heh.

Proof positive that THT doesn’t have a “point of view.”

By the way, Brian responded about Montero at THT:

“Some comments re Wieters and Montero

Going into 2009, Witers had three years of college and one year of pro ball. I don’t know how much PECOTA uses college stats.

In 2009 Wieters had some very favorable park factors, Bowie 1.16L/1.07R for HR, Frederick 1.35L/1.57R, and he was already 23 coming into 2009. Most of a player’s growth is by age 21. I have similar curves to what Jeff Sackmann wrote here at THT last week using college data, but not high as his numbers.

My one year projections

    BA  OB  SA wOBA  WAR
2006 Fr 260/335/392  321  1.1
2007 So 261/345/407  333  1.7
2008 Jr 270/356/424  345  2.4
2009    298/387/485  380  5.0
2010    296/372/466  366  4.2

On the other hand, Montero is only 20 coming into 2010, so has more growth potential, and he’s always played in pitcher’s parks Tampa 0.81R for HR, Charleston 0.65R, Trenton 0.85R, and he’s moving into (eventually) New Yankess Stadium, which after one year is 1.34R for HR.

I am sure Montero will rake. My high numbers put him in company with Miguel Cabrera or Carlos Delgado in their early 20’s. He is on everyone’s top 5 list. As a Pirate fan, I am disappointed by the peak projections for guys like Alvarez or McCutchen, so not everyone projects high.

I will be looking into blending in a player’s actual track record of growth or decline the past few seasons with the standard aging curves to idetify those who aren’t growing or slowing down as expected.”

While it’s less sexy than projecting stats, I’m hoping Tom can do an analysis at the end of the year in terms of pre-season playing time predictions.  Last year, Fantistics edged out BP and Fans.

If THT can do better, that would be a nice selling point.

In-season updating is nice but isn’t as valuable - it’s not hard for even the most statistically challenged to look at a week’s worth of box scores to figure out if a guy is going to play every day vs. platoon vs. spot fill…

I can’t stress how important playing time estimates are for fantasy baseball purposes.  I just ran my estimates - based on CHONE and ZiPS rates - and Tommy Hanson is worth $13 if he pitches 140 IP and $23 if he pitches the BP-projected 182 IP.  Nate McLouth is worth $20 at 632 PAs and $11 and 532 PAs.  Whatever margin of error you’re finding in CHONE vs. ZiPS vs. Oliver vs. PECOTA, I imagine playing time is a more significant factor for any counting stat (and even ratios are affected if you try and value a player’s contribution to a fantasy team - 200 IPs of 3.50 ERA are more valuable than 150 IPs of 3.40 ERA...)

Hi folks, I was sleeping the day away and didn’t get into work until 4 pm, and then had to wait for a break to do any responding.

I’ve left a couple comments at THT, and I will be re-reading the previous 42 here and will be responding over the coming hours.

Kincaid/9 - I have stated that there’s some aging built into the MLE formula, and applying the aging curves on top of that might be double counting a portion of the aging. This is something I will be looking at again over the next few days.

I will go back over the five year forecasts based on seasons from 1998-2009 (the extent of my db) to see how I can improve their accuracy.

What we wanted to find is the upside or peak performance that can be expected based on what a player has done so far and how others have aged from that same point. The more years into the future, the more variance is introduced. There will be an increasing number of misses that I don’t think anyone can avoid.

My batting projections, published last year at FanGraphs, were park neutral. I was asked to make the current version home team adjusted, as the fantasy customers will be looking to project what the player’s actual stat lines will look like.

My first thought was that you are creating a 2010 projection and then applying the age adjustment based on that to the players 2011 projection, then to the 2012 projection...etc. The 2011 projection should be based on the weighted average of 2008, 2009, and projected 2010. Each year’s projection needs to be re-marceled.

My second thought is, there is something wrong with the MLE. Using this nice little tool to compare with THT

http://www.minorleaguesplits.com/mlecalc.html

Here is Montero’s MLE from 2008;

MLS .226/.258/.319, 9 HR
THT .319/.363/.518, 24 HR

That is quite a profound difference. As was suggested earlier, THT looks more like an age 27 peak projection rather than an age 19 translation.

43 - I spent a lot of time this week on playing time estimates based on past playing time, age, and productivity. I think they look good for the vast majority of players, but I admit each player is calculated in a vacuum. Others at THT were working on depth charts. For fantasy purposes, you would want the depth chart playing time, as that would divy up PAs to players on the roster based on their relative value to the team.

45 - That makes sense, and I will look at that as well.

I was just writing to David Gassko about the long range forecasts. For the vast majority of players the numbers are reasonable. I looked at the best players six years out from base seasons of 2000-2009. The only projected wOBA of 400+ before 2009 was for Prince Fielder, but then there are five in 2009, so it appears this year is just not typical. However, there were a few players at the fringes who were breaking the system (the Tango method).

I was able to identify a place where I was double counting age. The league adjustments were written several months ago, before I was asked to produce projections past one year. In the league adjustments, I note whether the player was young, at or old for his league, and have seperate factors for each. At higher levels, where the players are in their 20’s and there’s not much growth left, there’s little difference, and I don’t even use them for AAA. Down at the rookie leagues, there is considerable difference, with the 17 year olds in rookie ball getting twice the HR factor that a 19+ player does. Doing it this way got a nice MLE, but doesn’ work when adding a multi-year aging curve on top of it.

The aging curves were derived using the delta method for players on the same team in consecutive years. If I move this calculation up front, I can use it to create a second set of age normalized league factors whch would be weighted mean of the MLE factor for each component at each age, multiplied by the remaining growth before peak. This second set would be the base for the long range aging factors. The HR factor for a 17 year old in the AZL would go from .346 to 1.1 because his HR% is expected to grow 3.2x before his peak. Then the 3.2 is factored back in, year by year, with the aging curves, but I’m only doing it once and not twice.

Or I can just use the one set of age normalized MLE factors, and use one year of aging for the MLEs, and chained age factors for the long range. More coding, but likely a better solution.

Just an update: Brian’s been fixing the problem with the MLEs, and while this is not necessarily final, the current 2010 projection for Montero is .289/.331/.455. That’s about 60 points of OPS more than CHONE says, and 30 less than PECOTA. Sound better now?

I’ll post something when we put up the updated projections.

David, that does sound somewhat better.  That makes the 2010 projections as follows:

.255/.296/.425 CHONE
.273/.315/.416 ZiPS
.289/.331/.455 Oliver
.291/.334/.481 PECOTA

I’m also curious to see the new Oliver 2009 MLEs.  Any chance we get to see those broken out by level and not just by year?  Here’s what we have so far for Montero’s 2009 Trenton MLEs:

.263/.305/.449 MinorLeagueSplits (assumed similar to CHONE)
.276/.320/.465 ZiPS
.312/.354/.529 PECOTA

It’s the latter issue (the MLEs) that bothers me more than what each system does with those MLEs and component aging curves, etc., to arrive at projections for 2010 and subsequent years.

Hey Mike,

I don’t think we’ll have MLEs broken out by level any time too soon. Not that it would be difficult to do it, but personally I like seeing a single MLE line for each season, so if we did break them out I would want to have an option to see it both ways on the site, and that would take a little while to program. We definitely have a bunch of priorities before that.

As for how the MLEs are calculated, what I’ve learned over the past few days is that what’s important is how you define what an MLE is. For example, we currently list Montero’s 2009 MLE as .320/.367/.571. What that means is that if Montero had played in the major leagues last season, he would have hit .320/.367/.571. I think that’s probably true. However, he did not play in the major leagues last season and that’s a very important piece of information. Another way of putting is that if a 19 year-old hit .320/.367/.571 in the major leagues, he would likely have .317/.370/.539 at AA Trenton and .356/.404/.583 at A+ Yankees. Actually, that’s not completely right either, since Brian’s MLEs tell us what will happen once Montero hits the majors. So what they’re actually saying is that a 19 year-old who hit .317/.370/.539 at AA Trenton and .356/.404/.583 at A+ Yankees would likely hit .314/.361/.542 (Montero’s current 2010 forecast) once he hit the majors. That sounds reasonable enough. The truth is, though, that Montero is not likely to get more than a cup of coffee this year and maybe even next. So if continues on his current pace, will .314/.361/.542 be a reasonable projection for 2012? I think so.

Anyway, the point of all this is that how you define what an MLE is will significantly affect the MLE line that you end up with. For example, let’s say that you have someone who hits .300/.400/.500 in AA. What should his MLE be? Using the traditional method, maybe it should be .250/.350/.400 (numbers for illustration only, obviously). But what if you have a guy that his .250/.350/.400 in the major leagues? What should his AAE (AA Equivalent) be? I don’t think it would be .300/.400/.500. The fact that he was given a spot on a major league team already tells us that his performance is depressed less by facing major league pitching than the other guy’s. So maybe his AAE is only .285/.375/.470. But a guy who hits .285/.375/.470 in AA wouldn’t have hit .250/.350/.400 in the major leagues—he would have done worse. When he does get called up though, he probably will hit .250/.350/.400. And so it goes—like I said, this is actually a much more confusing issue than I think people realize (certainly than I realized), and I’m not sure there’s one right answer as to what a player’s MLE was. Depends on how you define MLE…

I had a little difficulty in wording things in that MGL-sized post, so let me try to clarify one thing. Montero hit .317/.370/.539 at AA Trenton and .356/.404/.583 at A+ Yankees last year. What the Oliver MLEs tell us is that when a 19 year-old does that, when he hits the major leagues, he will hit .320/.367/.571. What ZiPS tells us is that if Montero had accidentally been called up to the major leagues last season, he likely would have hit, .276/.320/.465. Different questions, different answers.

Thanks, David.  That’s exactly the kind of thing I was getting at, and I appreciate your explanation.  I probably need to go back and read Brian’s BP Idol piece on MLEs yet again.

This all started for me when I saw that whatever PECOTA did last year with Wieters, they seemed to be doing again with Montero.  That’s why I asked about MLEs by level, for the sake of comparison.  It’s not something I fundamentally care about as an improvement to the THT forecasts.

I do think Brian is approaching the MLE very differently than CHONE, ZiPS, and presumably PECOTA.  I’m not the expert, but it seems like Brian may be onto something here.  I wish I understood it better than I do.

And maybe I should just forget about PECOTA and let them come back to us when they have their problems worked out.  Although they are currently claiming that there are no issues with the underlying PECOTA projections and the only problems are with the PFM and depth chart process, and I don’t believe that.

Interesting explanation, David.  That’s exactly the sort of detail we need to have spelled out at THT.  Put that on your list. 

Oliver was originally designed just as David explained above, that by using a direct comparison of each minor league to the majors it was telling you how the player would be expected to perform his first couple of years in the majors. I’ve run rmse tests, even for college, that come out very well. This was stuff I originally worked out a year or more ago.

After I got signed to THT, they asked me to do the five your forecast. I did a delta aging study using all levels of players who played for the same team in two consecutive seasons. I then added five years of aging past the current projection, but we did not immediately realize that when I did this it was putting the aging curve on top of a peak projection forecast.

To remedy this, I used the aging curve to identify, for each batting line input to the matched pairs analysis, where each player was expected to be in terms of standard aging. This would normalize for age and then allow me to measure only the differences in ebvironment by league.

Now I can explicitly produce a current value (if the player gets called up this year) and also a peak value. The current value is what should be input to the depth charts, and the peak value would be used for identifying prospects.

After initially somewhat confusing myself, today I worked out more of the math, so now the algorithm not only is general enough to cover everything (minors, college, indies, foreign) but turned out to be very simple to code (that stays in the black box though).

The new Montero projection sounds much better.  Comparing to CHONE catcher projections, that drops him from near-Mauer levels to around Miguel Montero (depending on how the Chase Field PF compares to new Yankee Stadium, I guess) in offensive production.  That’s still pretty close to the top 10 hitting catchers in baseball (by CHONE’s rankings, I don’t know if Oliver is higher on more catchers or not), which still sounds pretty optimistic, but it’s a much more realistic optimism than before.

Montero’s new unregressed MLE’s, his current value estimate

2007 GCL R  .221 .277 .329
2008 SAL A  .287 .321 .422
2009 FSL A+ .321 .351 .542
2009 EAS AA .289 .326 .477

I would then sum and group by level, adding regression to the mean based on highest level played in and sample size. Then do a three year weighted mean and add aging for the next year’s projection.

forgot to mention, that in 2008 Montero was three years younger than the average guy who eventually makes MLB from the Sally league, and in 2009 was three years young for the FSL and four years young for the Eastern league, so there’s lots of room for potential improvement.

@ David (#50): I think things start to fall apart logically once you assume that some players’ performance is affected more heavily by quality of competition than others’.

Your two hypothetical players:
Player A: .300/.400/.500 in AA, .250/.350/.400 MLE
Player B: .250/.350/.400 in MLB, .285/.375/.470 AAE

Now let’s add a third player, Player A’s ever-so-slightly inferior AA teammate:
Player C: .290/.390/.490 in AA, .240/.340/.390 MLE

According to this set of assumptions, Player B is better than Player C in MLB, but in AA, Player C is the better of the two.  That seems really problematic to me.  And the only way I see to avoid this sort of thing is to use the same MLE multipliers in both directions (MLB to AA and vice versa).

Anyway, you’re absolutely right that what the “right” MLE is depends on the question you’re trying to answer.  Personally, though, I don’t see the utility of the question Brian seems to be answering ("how will this player perform once he reaches the majors?") unless it’s accompanied by some assessment of how likely the player is to actually make it to the top of the ladder.  But to each his own…

DPO/58,

I agree that it needs to be accompanied by a player’s likelihood of making it to a certain point on the ladder, but the utility of this approach is obvious. If the Yankees had said Montero was their starting catcher this year, which MLE would you have wanted to use? Brian’s, obviously.

As for the first half of your post, the whole point is that the multiplier is NOT the same in both directions. A major league player by definition can better handle major league pitching than someone who hasn’t made it past AA. He therefore should be punished less by an MLE when translating his AA stats to the majors, which is equivalent to saying he is punished more by a AAE when translating his major league stats to AA. It’s not pretty, but it is the reality of the matter.

And by the way, adding in the likelihood of making it to the majors is exactly what we’re doing now.

If the Yankees had said Montero was their starting catcher this year, which MLE would you have wanted to use? Brian’s, obviously.

I don’t think that’s obvious in this case.  Has there been testing to see how the various MLEs work specifically for players who jump from the low minors directly to the Majors very quickly?  When you look at the typical performance of a player who puts up a given line A+/AA once he reaches the Majors, you are going to be looking mostly at numbers that are put up years later, so they’ll have years of improvement from aging factored in.  For someone who jumps directly to the Majors with no aging in between, I think it is likely that that translation is not necessarily appropriate.  Those cases are probably rare enough that they won’t throw off the overall results, but that doesn’t mean that the method best applies to the rare cases when someone does jump directly to the Majors without the typical aging factored into the MLE.

Also, do you believe that the converse of that statement is true:  that if the Yankees do not promote Montero to the Majors now, that this method is not the one you want to use?  If that is the case, then doesn’t that imply that for the majority of the thousands of minor leaguers being projected, that is the wrong method to use, since they are not being named MLB starters?  It sounds like Brian is addressing that and updating the process used for inputting the MLEs to the projection, which I think is a good thing, but it also sounds like that is going away from his initial method from the BP article and moving closer toward the other methods used by other projection systems (not necessarily that it is changing back into the other methods, but that it is changing the results to the point that the adjustments would drastically change Montero’s projection to where it is much closer to what other systems are projecting than when it was strictly sticking to the original MLE method).

David/60, sounds like an exciting feature.  Definitely something that could make Oliver stand out if it goes well, and it’s good to hear you’re working on such an integral piece of the puzzle.

Kincaid/61,

I agree that if Montero does not start the season in the majors, that is not the method you want to use. And you are correct that what we’re doing now is going to take that into account and therefore move our MLEs closer to what other systems find. I think we’re you’re going to see extra accuracy from us, though, is where the hitter does reach the major leagues—for some minor leaguers, that’s 2010; for others, some number of years down the line, though remember, we project those years too. Essentially, where the Oliver MLEs really shine, IMO, is once the player hits the majors, and what Oliver will be able to do is tell you when that will be and what the player will do once it happens.

By the way, the “when that will be” part is implicit in the Oliver system—we’re not actually going to be showing a player’s probability of making it to the major leagues in a given year, though I think that would be a very nice feature to add at some later date. Sorry for any confusion.

While it would be nice to see separately, using the “when that will be” implicitly in the forecasts seems like it is the most important thing anyway.  Especially given the method of MLEs used, that seems like a huge piece to have in the forecasts, and predicting how that affects forecasts years down the road is probably more fruitful than trying to show when a team will actually decide to give the player the call.

David/59: “a major league player by definition can better handle major league pitching than someone who hasn’t made it past AA.”

“Can better handle major league pitching” just means “is a better hitter”, so we’re on the same page there.  But that’s not at all the same thing as what you said initially, namely that an MLBer by definition will have his performance depressed less by MLB pitching.  I don’t see why that has to be true, and we don’t have any proper evidence for it that I’m aware of (there aren’t any major leaguers playing in AA, except for guys on rehab assignments).

Glad to hear that the revised Olivers won’t be treating reaching MLB as a fait accompli.

My original MLE process, as described in the BP Idol article, was to take performance in any given minor league and compare it directly to that player’s performance in his first three years in the majors. So, of coruse, that’s what that translation was showing, how any given minor leaguer projected to do in his first couple years, and it was very good at that.

However, until now I never explicitly accounted for age. I believe this has strengthened Oliver, as it now can do a wider variety of types of projections as well as likely being more accurate.

DPO/65 - One part of my process is to regress to the mean based on which leagues a player has played in. After calculating the league factors, I find the MLE for an average player in that league, and then players in that league are regressed to that average MLE, dependent on sample size.

So, the higher the talent level of the leagues a player was in, the better of an MLE he will be regressed to.

Brian,

I presume you mean that if you look at what a player did in A ball, and then see what he did in MLB, that 1 to 4 years elapsed in-between.  And so, in order to do the “right” translation, you need to include number of years to pro ball as a parameter.  Otherwise, you will age every A player as jumping straight to MLB.

Did I understand the issue correctly?

Tango - No. The projection estimated a players performance in his first few years in the majors if and when he made the majors in a typical number of years for the level he was currently in.

In my original (pre-THT) version, I coded players as young, at, or old for their league, and had different factors for each. Now I am normalizing the player stats for age before calculating the league MLE factors. From that, you have a current value (which I did not have before) and also a peak value (which would normally be close to my old style MLE, if the player reaches the majors in a typical number of years).

Hey guys,

THT Forecasts has been updated, and the MLE problem has been fixed (for hitters, pitchers are coming soon). I wrote up a long post explaining the problem here:

http://www.hardballtimes.com/main/blog_article/tht-forecasts-update/

Note that we’ve also added leader boards for our projections (and actually, regular season stats as well once the season starts up), which I think is a really nice feature no one else really does.

Great! Now Brian can begin to tackle why he had the Yankees as more than a +50 run fielding team when other fielding analysis had them in the range of +10 to -26 runs.

My explanation in Tango/67 matches the explanation in David/69.  I don’t know why Brian/68 said “no” other than the likelihood that I didn’t explain myself or Brian didn’t understand me.

Tom/71,

I think that’s exactly it.

A THT reader:

Dave said…

I hope you spend a minimum amount of time shilling your for-pay product as the season progresses. It’s not why I come to THT. I’m sure a lot of people feel similar.y

I won’t sully THT’s message board, but I will sully mine.

So this Dave character does not come to THT to see the gang there talk about a service you have to pay for.  But, he DOES come to THT to read for free all the work the guys there put out with their time and effort.

Hubris.

Tango/67
“in order to do the “right” translation, you need to include number of years to pro ball as a parameter.  Otherwise, you will age every A player as jumping straight to MLB. “

In my pre-THT version, and now, I do not *explicitly* count the number of years to pro ball.

In the prior version, I measured how everyone in each minor league did in that league, and used matched pairs to compare with how they did in their first couple years in MLB. It was assumed that there was little difference between leagues on the mean age of when players entered MLB, and that players in the same league would take a similar amount of time to reach MLB. I then broke down each minor league by the age of the player. Looking at those factors, I decided to use younger, at or older than average age for each league. HR, BB & SO were the only ones with significant differences, and the differences were more pronounced the younger the average age of the league.

In the current version for THT, I first developed an aging curve, of which you have a copy. Again, I did not calculate the number of years between the minor league and major league stat lines being compared, but instead looked at the player’s age in each stat line, then adjusted the stats according to the aging curve before building the matched pairs.

It would not suffice to only state that a player took three years to go from the Carolina league to MLB, as 19 to 22 would yield a different result than 21 to 24. I am looking at where the player is along the aging curve at each step.

The projection is then a three year weighted mean of the park, age & league neutral MLE, regressed, with aging added back in a year at a time.

By the way, we’ve now fixed the pitcher MLEs as well. Note that even after the fix, Stephen Strasburg projects for a 2.86 ERA.

I saw Kevin Goldstein speak last week on the BP book tour, and in response to a question stated that Strasburg had the highest scouting reports of a generation, and said the nearest comparison was Mark Prior. Indeed, my projections for Strasburg coming out of college are very similar to the ones I have for Prior, with even a few more strikeouts, but likely within the margin of error.

I just started a thread up on this issue.