THE BOOK--Playing The Percentages In Baseball

Colin acknowledged on Twitter that these are basically worthless at this point.  They haven’t done any work on the depth charts, and obviously, there are still a lot of free agents out there.

I believe it Dave, sounds like BP to me (half-joking).

I’m pretty sure that MGL will tell you that these are always useless.

I wouldn’t say NO work has been done on the depth charts. But yeah, there’s still a lot of work to be done when it comes to forecasting playing time and things like that. (Right now most teams don’t even know exactly what their 25-man roster is going to look like on Opening Day, so how should we know?) Certainly this is a work in progress.

But at the same time, some of the complaints I’m hearing about are vastly overblown, like the AL East for instance. The projected differences between the Yankees, Rays and Red Sox are much, much smaller than the uncertainty in the projection - it’s barely accurate to say we’re projecting the Rays to do better than the Yankees at all.

The one division I really want to sit down and take a look at right now is the AL West. If fans of teams want to give us input, that’s of course welcome. But input of the “player X is probably going to start the season in Triple A, so I think you’re overstating his expected playing time” sort is vastly more useful than “team Y is the bestest team evar, you haterz!” sort. Not that I need to say that around here, of course.

I just read the comments on Clay’s Unfiltered post on the PECOTA arrival. Someone brought up Stephen Strasburg’s “absurd” VORP—please don’t tell me this is Matt Wieters all over again.

By far, the most interesting result is the Angels projection, because their off-season is over, for all intents and purposes.  Unless you guys just really mis-allocated playing time (is Napoli projected as the starting center fielder or something?), no amount of roster adjustments are going to fix the fact that PECOTA hates them again.

Zach, Strasburg has a projected 31.2 VORP. Is that absurd? I dunno. That’s putting him as the 24th best projected pitcher in the league. High, certainly, but nothing like Wieters as a top-five player.

Brian’s OLIVER system is the only other one I know of that does college translations. So it’ll be interesting to see where Strasburg falls there.

Oh, and the run environment in MLB is projected at 5.0 R/G.  That’s interesting too, in a “this isn’t a very reliable projection - wait for the update” kind of way.  MLB has averaged about 4.6 R/G for most of the decade, spiking up to 4.8 R/G a few times, but hasn’t been over 5.0 since 2000. 

Barring an explanation for the surge in offense, I’ll assume something was calibrated incorrectly.

I’ll take a look at the RPG issue and get back to everyone on that, David. Thanks for pointing it out.

One thing I want to point out - last year’s PECOTA forecasted wins did spectacularly poorly in Vegas Watch’s comparison. As far as I can tell, the big issue there was the projected spread - PECOTA had an SD of 8.98 in its projected wins, very high compared to its automated peers. That has been fixed - right now the SD of projected wins for PECOTA is 7.20, which is much more in line with where it “should” be.

I’m guessing that the people who are upset about the Yankees also haven’t seen that part of the projection has Mariano having a massive decline in performance. I’m going to guess the Mariano decline has been predicted for a long time and because the only remotely comparable players declined a lot by this point (after all, how many 40 year old closers are there in the history of the game, even). Truth is, there may not be a way to predict the performance of someone like Mo because he’s such an outlier.

I don’t know why Colin would think that I think that projecting team w/l records now or just before the season starts are “worthless.”  I don’t.  If I said something like that in the past, there must be some context that is missing.

The forecasts above “look” OK to me, for whatever that is worth.  I’m not sure that you can project a team to have 95 or 96 wins though.  Obviously for a team that is that good, they are more likely to be negatively affected by injuries.  As well, when you project a team to be really good or really bad, there is a suggestion that you did not regress some of the players’ projections enough.

Also, the counterpart of a team being projected too high because of the effect of injury, is a team being projected too low, because of the much greater chance (than a good or great team) for improvement as the season goes on, from roster moves, plus the fact that a bad team does not get affected as much from injuries - in some cases, they even get better.

Colin, can you look at R/ER ratio issue too?  League wide the ratio is 1.14, shouldn’t it be more like 1.08?

Here is Cairo:  Who does these projections (Cairo) again?

Yankees   102.1   59.9
Red Sox   93.6   68.4
Rays   89.2   72.8
Orioles   69.8   92.2
Jays   66.9   95.1

W Sox   87.9   74.1
Twins   81.5   80.5
Indians   76.4   85.6
Tigers   74.5   87.5
Royals   70.8   91.2

M’s   86.7   75.3
Angels   80.9   81.1
Rangers   77.9   84.1
A’s   76.8   85.2

     
Phils   91.5   70.5
Braves   87.8   74.2
Mets   82.2   79.8
Marlins   73.8   88.2
Nats   73.2   88.8

Cards   92.1   69.9
Reds   85.5   76.5
Cubs   83.7   78.3
Brewers   81.3   80.7
Pirates   70.2   91.8
Astros   69.1   92.9

Dodgers   87.2   74.8
Rockies   86.1   75.9
D-backs   84.1   77.9
Giants   75.5   86.5
Padres   71.8   90.2

MGL has said in the past that to compare forecasters based on the team W/L forecasts is somewhere in the neighborhood of useless, I think.  Going from memory.

Who does these projections (Cairo) again?

That’s me.

Huh, I got a three for one deal there. Figuring out one question seems to have answered two others.

The reason the ERAs seem “off” compared to the team runs is because the standings page seems to be in a completely different run environment than the player forecasts. It’s not just pitchers, it’s hitters too. Add up team runs scored (remember to only count each hitter once - we list a hitter multiple times if we think he’ll play more than one position) and you get a different answer than the league totals as well.

The upshot of this is that once you “correct” the standings to the correct run environment, the spread of team wins should tighten up as well. At least, that’s what I’m getting from Pythagenpat.

I mean, I can’t tell you WHY this is happening. But I’ve sent along a note to the right people and hopefully they can find it and fix it pretty quickly.

Tom’s right about what I was referencing with the MGL comment - it was meant to be at least a little tongue-in-cheek.

As for the Rivera forecast - yeah, that’s one we talked about when we were proofing the PECOTAs for the book. Because of his very uniqueness, he’s a tough one to predict.

[On a general note there - as Clay notes, we now have the ability to do historical PECOTA forecasts. He and I have talked about using this to study aging patterns, and how well PECOTA’s career path adjustments predict aging. He and I are both too busy right now to actually DO this (he’s got PECOTA player cards to crank out, and I’ve got a defensive metric to build - first article this morning!) but once the season gets underway it’s something we’re probably going to start work on.]

I’ll comb over the Angels forecasts and the playing time projections and see what I can’t come up with there. I’m sure that one raised a lot of eyebrows, and if that’s the forecast I’m going to war with then I need to get comfortable with it.

...yeah, let me clarify one of those remarks before it becomes misinterpreted as a desperate plea for help or something. I can’t tell you what’s causing the discrepancy because I simply don’t know.

>> I’ll comb over the Angels forecasts and the playing time projections and see what I can’t come up with there. I’m sure that one raised a lot of eyebrows, and if that’s the forecast I’m going to war with then I need to get comfortable with it.

According to Halos Heaven, you’ve already gone to war - and I can’t blame em.
vr Xei

What stood out for me was the low projection for the Angels.  That is the only one that had me scratching my head.

The AL East projection seems fine, for the past two years we know that two of the Yankees, Red Sox, and Rays will emerge from the East into the playoffs, and at the beginning of the season you can throw darts to pick which of the two.

The Yankees and Red Sox look pretty close.  I would have picked the Rays as a solid 3rd, but haven’t added the numbers up yet.  They have the talent to win the division again - but I’m not sure that should be considered the most likely outcome.

Agree with Xei on the Angels.  You aren’t going to convince any Angel fans, including this one, that we should be expecting a last place finish.  But it’s not worth a war.  Any Angel fans reading this , just look at the track record, ignore this, and enjoy the pennant race.  The other teams in our division have improved to the point where there will be one, but we aren’t out of it.

Updates to PECOTA in a few hours:

http://baseballprospectus.com/unfiltered/?p=1495

The Yankees always seem to be underestimated in these pre-season ratings (although Vegas lines don’t underestimate them, they may overestimate them), because these pre-season ratings don’t (and I’m assuming here) have a variable for the willingness of the Yankees to spend money/trade prospects late in the season if they are in a dogfight for the playoffs.

The Yankees seem to be really overestimated by Vegas. I was there 2 weeks ago and it was even money for them to win the world series. Does this make sense to anyone?

being 50/50 just before the playoffs start is ludicrous, let alone before the season.

KY, being willing to add players at the trade deadline if you are in contention does not add very much to a pre-season win total, especially if you have an excellent team already (the better the team, the more difficult it is to upgrade, right?), I don’t think.

You are isolating the Yankees, so you have to do 3 things:

One, tell me how much more likely they are than an average team to add some significant wins at the trade deadline or thereabouts.

Two, tell me the chances that that happens (they might choose not to regardless of their standing, they might be so far ahead that they don’t need to, or they might be so far behind that they don’t need to).

Three, tell me the average number of wins they would get by upgrading their team, say in mid to late July (or whenever).

My guess is that the total expected number will be less than 1 win.

And BTW, Vegas always has horrible odds for many teams to win the series or pennant.  I don’t know what their “juice” is for those kinds of bets, but it is very high.

Third time’s the charm:

http://baseballprospectus.com/fantasy/dc/

This should fix all of the issues that have been noted here, as well as some that hadn’t, like the problems with pitcher BABIP projections. If anyone has any other questions or concerns, let me know and we’ll look into those as well. (And hopefully we’ll be getting a more detailed explanation of what went wrong and why on the site here as well.)

Maybe it’s just me, but as long as Colin is somehow involved, I feel pretty good.

I second #26.  Communication is a great concept.

I did my team projections over the weekend.  Have not published yet.  Maybe this week. Hate to say it but my results pretty much match the third time results.  Texas looking like the favorites in the west (only good thing about that is if it does happen I will continue to root for Vlad).

I think I had the Dodgers still on top in the NL, and Atlanta closer to Philly (possibly just ahead).

One thing in my projections that I was shocked by is how small a correction I needed between batting runs and pitcher runs allowed.  Once I assigned playing time batter teams averaged 744 runs and pitchers 738.  Despite the fact that they are done completely independently.  So all I had to do was add 6 to each staff’s projected runs allowed and then pythagpat it.  This is the removal of the “Lake Wobegon effect” where all teams are above average.  So small though, this year, that had I not done it I’d probably only have teams averaging 81.6 wins instead of 84 or 85.  It’s a product of assuming if he played well last year, he’ll get to keep playing, and if he stunk, somebody better will replace him.  But a proper amount of regression in the system does a lot to minimize that.

As much as possible was done through a script.  For pitchers assign games started until you hit 162 and go down the list from best pitcher to worst.  Then, find out how many innings you need to get to 1440, and assign relief innings until you’re there.  Only one manual change made (Josh Outman) due to a known injury.  I should get Jose Arredondo out as well.  Can’t remember how many other good pitchers are known to be out.  I could update projections and redo pitchers in a few minutes.

For hitters I would have more trouble.  I tried to assign playing time and bench roles based on how the best players were at each team’s positions, but there is a lot of manual editing needed where I know that some player is getting the time regardless of what the projections say.  My program kept trying to give Yuni Betencourt a September callup role only. 

Royal fans would probably want to replace Moore and Hillman with my depth chart robot.

24 MGL: good points, thanks

22 jarjar: there is a bit of an art to figuring out the Vegas odds.  There are different casinos, some offer closer to fair odds, others rip off their customers as much as they can get away with.  Here are a few steps you need to take to figure out “Vegas odds” when it comes to futures bets (like who wins the World Series)

1. look for the best odds in town.  if one book is offering +100 (50%) on the Yankees to win the World Series and another book is offering +300 (25%), then no one in their right mind would (or should) bet at the one offering +100.  You can safely throw that one away since it is meaningless.  The only way the +100 number could be meaningful is if the other teams didn’t add up to much more than 70% (50% on the Yankees + 70% on all the other teams = 120% on all MLB tams combined), but I can guarantee you that if some books have the Yanks at +300 and one book has them at +100, then that means their total percentage adds up to greater than 200% for all teams combined.  Otherwise, sharp bettors would simply go in and bet every other team (at the book that offers Yanks +100) with Yankees +300 at other books and take in the arbitrage profits.  The numbers don’t ever get close enough for sharp players to even contemplate doing that.

2. combine the best odds in town with the best odds online.  often the odds online are better.  whether you can access those odds or not isn’t relevant since so many others can, they are part of the marketplace.  There aren’t many places with World Series odds up right now, but one populat place is currently dealing a two-way line.  That means their customers can bet on the Yankees to win the WS, as well as the Yankees NOT to win the WS.

The current odds are:

Yankees win WS +385 (20.6%)
Yankees will not win WS -450 (81.8%)

You’ll see that this adds up to only 102.4%, this book is dealing a fairer line than others and thus I expect it to be closer to fair market value.

Thus, roughly, the market is pricing the Yankees at 20% to win the World Series right now.  Is that overestimating or underestimating them?  I don’t know, but I do know that using 50% that some rip-off Vegas casinos are offering is the wrong way to approach the subject.

adding one more comment: “Vegas odds” isn’t just the ones in Vegas, there is a world wide betting market, no reason to just limit it to the relatively small sportsbooks located in a small city in a desert.  the books offshore dominate the market.

apologies for the multiple post.  I thought I’d supply a link to the odds for all teams in case people are interested.  This book is not available to U.S. residents as they conform to UIGEA rules, but nevertheless, they get lots of action and deal fair lines:

http://www.pinnaclesports.com/ContestCategory/Baseball+Futures/Team+to+Win+2010+World+Series/Lines.aspx

Braves and Rangers look like pretty good bets at the pinnacle site.

Looks like Clay updated these again.  I don’t even know where to begin.  Now they’re claiming that the best projector of future performance for a pitcher is his 75th percentile projection, because the system has been systematically underrating pitcher performance.  Except, you know, it did pretty well for several years before Nate left, especially at projecting pitchers. 

Also, the projected team totals are just not possible.  Every single team in baseball is projected to post a lower RS total than their team OPS.  A five minute look through history shows that above average offenses outscore their team OPS total, usually by a good margin. 

2009 examples:

Yankees: .839 OPS, 915 RS
Angels: .792 OPS, 883 RS
Red Sox: .806 OPS, 872 RS
Phillies: .781 OPS, 820 RS

The current PECOTAs have the Yankees leading baseball with 821 runs scored and an .825 OPS.  The Orioles, Twins, and Rockies are all going to manage better than an .800 OPS, but score fewer than 800 runs. 

The Reds, projected for a .755 OPS, are only going to score 691 runs apparently.  The Dodgers, Indians, and Marlins all had an OPS really close to .755 last year, and they scored 780, 777, and 773 runs respectively. 

Expect another update soon.  These certainly aren’t fixed yet.

Yeah, they’re screwy.  I tried estimating team runs scored using their listed OBP and SLG and they’re off.  This is quick and dirty and almost certainly imprecise, but I don’t have the time to go through the team by team stats right now.

First, I converted OBP and SLG to GPA using (1.8 x OBP + SLG)/4 (as defined in the Hardball Times stats glossary).

Then I converted that to runs using the formula PA*1.356*(GPA^1.77), again from the Hardball Times, where PA = 4100 divided by (1 - team OBP).  I get an average shortfall of 65 runs scored, with Houston the closest at 44, and the Yankees the furthest away at 100.

Here’s the list:

Team: PECOTA RS, GPA RS, Diff
Houston Astros: 691, 735, 44
San Francisco Giants: 681, 726, 45
Milwaukee Brewers: 743, 791, 48
San Diego Padres: 663, 711, 48
Pittsburgh Pirates: 671, 720, 49
Chicago White Sox: 732, 782, 50
Los Angeles Angels: 754, 808, 54
Toronto Blue Jays: 702, 756, 54
Cincinnati Reds: 691, 746, 55
Washington Nationals: 674, 730, 56
Texas Rangers: 816, 873, 57
Kansas City Royals: 714, 774, 60
Arizona Diamondbacks: 740, 802, 62
St. Louis Cardinals: 722, 784, 62
Los Angeles Dodgers: 688, 752, 64
New York Mets: 732, 797, 65
Oakland Athletics: 716, 781, 65
Philadelphia Phillies: 764, 830, 66
Detroit Tigers: 732, 800, 68
Florida Marlins: 733, 801, 68
Chicago Cubs: 712, 780, 68
Baltimore Orioles: 781, 850, 69
Tampa Bay Rays: 800, 870, 70
Seattle Mariners: 701, 772, 71
Cleveland Indians: 740, 822, 82
Minnesota Twins: 778, 860, 82
Atlanta Braves: 730, 814, 84
Boston Red Sox: 808, 893, 85
Colorado Rockies: 769, 854, 85
New York Yankees: 821, 921, 100

This seems like a pretty embarrassing moment for BP.

BP just released its book, which according to its cover, includes “Deadly Accurate PECOTA projections.” Now, Clay Davenport is basically saying that all the pitching projections in that book are wrong.

I don’t think BP is saying the projections in the book are wrong (they are, however, always preliminary, i.e., not based on final depth charts and playing times).

Steve Goldman just posted that some of the problems seen in the weighted means spreadsheets and player forecast manager are not in the book.

Right now it looks like they’re having problems generating updates to the PECOTA estimates as well as getting the PFM to work as it should. And Clay is working on this.  I suspect these are more data management problems than problems with the basic PECOTA algorithms. But since Clay had to take Nate’s humongous spreadsheet macros and turn them into a different code, there’s certainly room for slippage. And in the past, Clay’s job basically involved getting the input data (translated stats, etc.) ready so that the PECOTA’s could be run.  Now he has the entire process in his hands. So now he could conceivably be checking the whole computational process from data input and translation to estimates to reporting.

I agree that it’s embarrassing to BP to have these problems. They should have beta tested and worked out the kinks before they released the projections online.

Saying that the 75th percentile are more accurate than the 50th indicates to me a problem with the rates that were projected. I haven’t gotten the book yet, but that seems to me like something that would be in the book. A means test should be one of the first things done, followed by rmse for variance.

It wasn’t clear whether Clay was saying this 75th percent fix applies only to the PFM settings to use when using the latest versions of the forecasts, or whether instead he was saying that the PECOTA’s in the book also have this problem. The latter would be a more fundamental error.  Time will tell.

Colin/Clay/anyone from BP want to address the issue that David and SG raised in posts 34-36?  It’s still hanging around in the data BP has on their depth charts today.

Between this issue and the 75th percentile kludge, my faith in PECOTA is pretty close to zero right now.

If it would help the BPro guys in any way, I’d be happy to start a thread dedicated to them answering questions from you guys.

It looks like the standings were updated again this weekend, with a nod towards run modeling issue.  It’s better now, with good offenses now being projected to score more runs based on the projected OPS numbers, but still too conservative.

And, of course, this update wasn’t actually mentioned anywhere on BP, so we don’t really know what they did.  And, it didn’t stop BP from turning in the last version of their projected standings to ESPN and proclaiming that the results were credible. 

Given how many members of the media are using PECOTA projected standings as a proxy for “statheads think X will happen”, I’d say BP has a responsibility to make sure these things aren’t broken.  And they still are. 

Either fix them or get rid of them.

BP has finally acknowledged the problem:

On a related topic, we’ve seen statements on the internets that the team triple-slash batting stats don’t mesh up with expected runs scored in the depth chart projected standings. I want to take this opportunity to categorically confirm these claims. The issue is not with the PECOTA projections themselves, but how they’re playing with the depth chart process. Our newest full-timer here at Baseball Prospectus will be addressing this issue later in the week… more on him in a day or two.

How’s everyone’s confidence in PECOTA these days?  Black boxes spitting out strange results and then being magically repaired is fun, isn’t it?  I’m sure the folks at BP know what they are doing and we should just trust that they’ve found all the errors that aren’t so obvious that the sabermetric public is able to help them find them, right?  If anyone still has faith in that mound of crap after the last two years, you’re a much more trusting individual than I am.

I think you have two camps at BPro.  One is the made up of guys who are, have been, or would be, free to post in this blog, and discuss things as openly as the rest of us do.  It includes, but is not limited to, Pizza, Matt, Colin, Eric, Tommy, Tim, Jeff… basically, all the new guys.

The other is those who treat BPro as a business, and so will look to maximize profit out of their property.  Who this is, I don’t know.  It could even include the new blood.  Those people THINK that the way to do this is to black box everything.  It’s worked fantastically well-enough for them so far, and so, they figure, let sleeping dogs lie.

The dog is now barking. 

Hopefully, the new blood will keep pushing for transparency, while the rest of us applaud and support them.  So, I say we give them a long enough leash to let the transition work itself out.  Dan Fox is a great example (perfect example really) of the way we (I) like things done at BPro: he maintained his own blog in parallel.

The day Colin stops publishing SQL code, the day Pizza stops posting his datasets, the day the new blood shuns this blog, that’s the day we concede.  Not before.