THE BOOK--Playing The Percentages In Baseball

Here is the link to their “playoff odds report”.  I don’t know if it is premium content.  Good luck figuring out what everything means.

Lest I be accused again of constantly criticizing people, I read BP every day, I am a loyal subscriber, and I buy and enthusiastically read every one of their books and publications. Obviously, I think highly of them overall.  Either that, or I am a masochist.

Oops, forgot link!

http://www.baseballprospectus.com/statistics/ps_odds.php

Mickey,

That’s why I pay more attention to the PECOTA-adjusted report, which is based on projected records:

http://www.baseballprospectus.com/statistics/ps_oddspec.php

You’ll see the numbers there make a lot more sense.

That’s still not quite right, David.  They are regressing the team’s current record toward their preseason-PECOTA projection, which doesn’t make any sense.  Why would you regress a team’s current winpct to what you thought their winpct was at the beginning of the year?

What needs to be done is to regress the individual players’ performance, incorporating all of the latest performance data and use that to compute the team’s winpct going forward, and then use that as the input to the simulation.

Pct3 is “third-order wins,” or the Pythagenpat of the team’s Equivelent Runs (linear weights) scored/allowed, with a strength of schedule adjustment. Don’t ask me why they feel the need to present that much data. (How many support-neutral pitching stats do they have again?)

The overall point (for me, at least) is that BP’s stat section has a bad tendency to throw out as much data as humanly possible, without giving any real indication as to what it means or what you should be looking at. THT does a much better job with their stats section, precisely because they don’t try to do as much.

Derek Jacques did a breakdown of the BP Playoff Odds report here:

http://www.baseballprospectus.com/article.php?articleid=6677

This should help you refine your argument, MGL.

#5, I agree wholeheartedly with your second paragraph.  I only have so much time in my day like everyone else.  I am certainly not going to spend much of it trying to figure out what all the data in their various reports mean.

In your first paragraph, OK, now that we know what it means, what is the point of using it to generate “playoff odds” if the results are clearly ridiculous?

They are using that number, pct3, as a team’s actual wp for the rest of the season in order to generate the results.  The Yankees are clearly not a .496 team, the Orioles are not a .498 team (better than the Yankees??), the Mets are not a .512 team, the Cardinals are not a .523 team (better than the Mets), the Pads are not a .453 team (worse than the Pirates, 20 points worse than the Giants, and the worst in the NL), so on and so forth.

I don’t get the point.

And then Jim Baker was quoting the results in his article.  He said that these results are not written in stone. Really.

IIRC, the point was reader demand - they used to wait to do the postseason odds report until much later in the season, but readers (read: paying customers) requested to see the report sooner.

I really don’t have a problem with them publishing the results. I do have a problem with them publishing the results without error bars. It may be nontrivial, but certainly you should be able to figure out what the standard deviations are on “third-order wins” based on the number of games into the season, and see how that translates into your playoff odds. So instead of presenting me with “Cardinals, 44.58173 percent” (and that’s another thing that bugs me - do you seriously need seven significant digits for this?), give me: “Cardinals, 44.6%, +/- 20%.” That way you’re presenting not only the data but a context for it.

[Obviously 20% is just for illustration purposes - I have no idea what the error bars are on these predictions. I’m guessing they’re pretty large, though.]

I don’t understand the point of having three different playoff odds reports, particularly one that is so fatally flawed as the standard one (people have mentioned their issues with the PECOTA adjusted ones here, but those are certainly a lot more sound).  There is never a time during the season at which I would rather look at the standard one than the PECOTA one.

I have no problem with running a PECOTA report from day one; after all, isn’t that one of the points of making projections in the first place?  But the standard one gives ridiculous results as MGL says.  If the readers demand that one, then the best course of action for BP would be to educate the readers on why you shouldn’t use the results of 30 games to set odds for the outcomes of the pennant races.

I really don’t have a problem with them publishing the results.

Why would you not have a problem with some method that clearly comes up with absurd (even to the casual reader) results?  This is THE sabermetric web site, not ESPN printing their power ratings or something like that!

If the readers demand that one, then the best course of action for BP would be to educate the readers on why you shouldn’t use the results of 30 games to set odds for the outcomes of the pennant races.

Yes, you could literally produce that whole page, and say, “See, this is why you DON’T want to use a method like this!”

But I really don’t see the point of producing that page.  Again, on a sabermetric web site, you don’t print something that you know is wrong, and REALLY wrong, because “the readers demanded it” (what did they do - picket the BP offices and start petitions?). That makes no sense.  And then, to top that off, one of their writers quotes those post-season odds and says, “They are not written in stone.”

OK, I’ve already said too much on this issue, but it boggles my mind.

has anyone gotten any feedback from someone who works for BP? That might shed some light on the issue.

After reading this, I found myself completely agreeing with an outraged MGL… didn’t think that would happen any time soon.

I don’t see why there is any problem with producing a report that says, this is what will happen if the teams continue to play like they’re playing. It’s not predictive—and I’m not sure it ever claims to be—it’s just a more comprehensive look at what already has happened than just looking at wins, losses and runs.

MGL: I agree completely. I was trying to respond to the argument that “the customers demanded it”. What I meant was that if they got that feedback from readers, they should not publish the page, but instead explain why they’re not publishing the page. 

Also, if they would have just started with the PECOTA report and never published the “regular” one, I doubt there would be anyone demanding to see that one.  But I agree--it’s worthless, and is basically just a regurgitation of the current adjusted standings, which are fine for what they are, but should not be taken to have predictive value.

Devo, only because this is BP I even bring all this up, but a report that “predicts” w/l records and playoff and WS odds should do just that, to some extent at least.  If what it produces is worthless, which it clearly is (I don’t think there is ANY argument from BP or anyone that the pct3 column represents anyone’s notion of how these teams will perform for the rest of the season), I don’t see the point.

It is a little silly to justify something that is clearly flawed, by saying as long as you state that it is wrong, it is OK to print it.  That does not make much sense to me.

That would be like having a legitimate sabermetric site (let alone the most famous and well-read site in the world) and having a page entitled, “player projections,” and then when you go to the site they you a goofy, incorrect method for doing player projections, leading to ridiculous, meaningless, projections, but they put in a disclaimer, “Even though these are really bad, in fact, worthless, projections, we are giving you them anyway, just to show you what they would look like using this particular method.” That is silly to me. 

I don’t think this is one of my usual rants.  This is perplexing and outrageous to me.  When I went to site and read some of the numbers, I assumed that I was reading something incorrectly, but that was not the case.

I would love to hear BP’s retort.

The other thing is that there is no disclaimer or even an explanation of the methodology that people can easily follow, so people are simply going to assume that the results are accurate and make sense.

My memory of the history of those reports is that they only started with the basic one, the one that MGL is complaining the most about, and that they published it with a few weeks left in the season.

At that point, a team’s record (or runs scored and runs allowed) for that season are a better predictor of how they will do the remainder of the season than are the compilation of pre-season player projections.  So said BP at one point in time, and it makes sense to me.

Then they started showing these reports earlier in the season, probably not May 1st to start with, but earlier than September, so they added the PECOTA version, which is more accurate earlier in the season than the original version, but less accurate in September.

I don’t remember the reason they introduced the Elo version, but Elo weights more recent performance more heavily.  That’s obviously not very relevant in May.  I’m not sure it’s relevant in September, either, but maybe some people think so.

This seems like par for the course at BP to me.  They have lots of ideas and they turn into stat reports and those stat reports get produced forever and used in ways they were never intended.  It reminds me of a garage filled with all sorts of tools lying about, some of which are incredibly useful if you can find them or have someone teach you how to use them.

I personally love having the playoff odds report available (later in the season), just like I love having PECOTA available.  But I can’t test out the assumptions behind either one of them, so I don’t use them for serious research, I use them for entertainment.

I think Mike has hit the nail on the head here: there’s no real analytical purpose to ANY form of standings right now - actual, adjusted, or Monte Carlo. But just because there isn’t any analytical value to something doesn’t mean it serves no purpose.

And definately agree about the packrat nature of BP - nobody bothers to clear out the attic, they just keep stuffing more things into it.

there’s no real analytical purpose to ANY form of standings right now - actual, adjusted, or Monte Carlo.

I love projected standings and chances of making the post-season, etc.  I update mine all during the season.

I simply take my updated pitcher and position player projections, pro-rate them for projected playing time for the rest of the season (ironically, I use BP fantasy charts for that), and “play out” the rest of the season, using each team’s projected wp and a log 5 matchup formula for each game.  I play out 10,000 seasons, the post-season, etc., and voila!  Nothing fancy. The only trick is to use a decent wp for each team based on their player projections and some approximation of playing time for each player, which includes everyone’s chance of injury, chance of getting called up from the minors, sent down, etc.

I don’t use ANY “bulk” stats for the team when updating the player and thus the team projections.  I certainly don’t use the team records to-date to estimate their wp for the rest of the season.  Why would I?  What do they tell me that I don’t already know, and then some?

Why would I?  What do they tell me that I don’t already know, and then some?

They will tell you something at some point in the season: likeliness to turn over your roster, in terms of adding immediate talent, in return for prospects.  A team in the hunt for a playoff spot has a better chance of getting more immediate talent, than a team that is already out of it.

And, the empirical data supports that.  Clay presented the data a few years ago, and we had a thread at battersbox.ca discussing it.  I’ll try to find it later…

It’s still clear that the “original” method of producing the odds report is flawed.  even the PECOTA version is flawed.  Why would you regress a team’s winning pct to a preseason projection when the true talent of individual performers and team depth is dynamic?

As long as they’ve got PECOTA, they ought to re-run PECOTA once a week and get new projections.  That takes care of the regression aspect.  Then use that for the input in the sim.

Sal, based upon what Silver has said in the past, you can’t rerun PECOTA in a week - it’s far too computationally intensive. (I’m pretty sure that’s a problem with scaling in Excel - it’s not designed for what they’re using it for.)

He’s talked about building a stripped-down mini-PECOTA for in-season use. I’ve stopped following BP for a while, though, so I don’t know what’s come of it.

Yes, I would think that since Pecota is not nearly a “Marcel” per se, you can’t just “re-run” it at any point in time in order to incorporate recent performance.

But surely you can just take your Pecota projections and “apply a Marcel” to them, or just “tweak” them, to adjust for recent performance.  Or something like that.

Right, I was going to say what MGL/21 said.  In fact, PECOTA, if it doesn’t have good comparables (say like a 43-yr Clemens), will simply resort to 90% (or so) Marcel.  Therefore, for in-season, there’s no reason not to do that.

http://www.hardballtimes.com/main/article/how-soon-is-now/

I had to reverse engineer the numbers, but the dozen teams that looked like the Cards (below-500 in year x-1, April of .621 +/- .010) look like this:

year x-1: win% = .429
April record, we selected on: .621
year x, post-April win% = .523

What to expect the rest of the way?  Well, I’d weight the 29 April games at double, the previous 162 season at one, and then a regression amount of say 100 games at .500.  (All this is pure guesswork.)

29*.621 + 162*.429 + 100*.500 all divided by (29+162+100) = .473

They posted a record of .523 post-April, which is 50 points higher than expected.

It is only 12 teams, so you’d like to expand that.  And you’d like to come up with a more general function (look at teams at .600, .580, .560… that is, all of them, but especially the extreme teams).  And of course, fill in the spots that I’m guessing at.

Also, include amount of team turnover.

Finally, include R and RA.

Or, simply look at the Apr 1 roster, and figure out what the true talent really is.

Hmmm.. that should be:
2*29*.621 + 162*.429 + 100*.500 all divided by (2*29+162+100) = .486

They posted a record of .523 post-April, which is 37 points higher than expected.

“I love projected standings and chances of making the post-season, etc.  I update mine all during the season.”

If you make those publically available and update them weekly (see, willing to work with you), I promise to never look at BP’s playoff odds report ever again.

#25, nope sorry.

But honestly, it is not all that hard.  Just find some web site that has updated projections for each player and add everyone up on a team, estimating each player’s playing time for the rest of the season.  That will give you the team’s average wp for the rest of the season (after doing some sabermetric gymnastics, like converting the team’s total offensive and defensive numbers into runs above/below average).

Then use that in a sim which pits each team against each other according to the rest of the schedule.  Use a log 5 matchup for each game.

I realize it is hard if you don’t have the programming and sabermetric know-how and experience, but what I am trying to say is that there is nothing that I have a monopoly on and there is no rocker science involved.

Tango, I hated that article about the Cardinals on THT.  It is like those, “Let’s look at history and see how teams do when they are down 3-0 in a 7-game series.” As if it doesn’t matter whether the team down 3-0 is a big fave or big dog, or anything in between, against their opponent.

Of course, that kind of thing helps us IF WE KNOW NOTHING ELSE!  But we ALWAYS do, unless we are living under a rock or we don’t know anything about baseball, in which case, what do I even care about the Cardinals expected record the rest of the season!

The overwhelming proportion of teams that have a good record in April are good teams.  If a team has a good record in April but a bad record the year before, most likely they were unlucky the year before, or their team changed significantly, or both.  Again, do I care about that with the Cardinals?  No!  They were not a good team last year, but their record was much worse than their team talent.  This year, they are not a good team, despite a good April record.

The ONLY thing I care about is the true talent level of the Cardinals right now and estimated for the rest of the season. How they did LAST year is NOT part of the equation, and how they did in April is not MUCH a part of the equation (again, unless I know little or nothing else).

Just look at their true talent level.  With a dozen or so web sites with good projections available for each player, it ain’t that hard to do!

The Cardinals are a mediocre team at best.  I DO NOT CARE AT ALL how they did last year or in April.  They are a mediocre team at best whether they went 18-12 in April, or 12-18, or whether they won or lost 100 games last year!

End of story.

I realize I am preaching to the choir.  You summed the whole thing up when you said:

“or just check their true talent” (or something like that, your post is not in front of me).

“what I am trying to say is that there is nothing that I have a monopoly on and there is no rocker science involved.”

And all I’m trying to say is that if someone did what you are proposing and made it available to the public for free, I think most in-the-know would abandon BP’s obviously flawed approach.  Personally, since I’m not willing to do the work myself (nor would I know how to perform all of the necessary “sabermetric gymnastics"), I just try to extract as much “good” information from BP’s playoff odds reports that I can. 

Heck, when the season gets old, I’ll also use coolstandings.com’s “dumb” projections (assume 50/50 chance for each game), just because they’ve at least run the monte carlo on it.

I know how to run projections, update depth charts, and run team projections, but for me the process is pretty time intensive.  It would probably take me a week to rerun everything, which makes the daily update impossible.  It would be nice to have the whole process down to pushing a button, but it’s not anywhere close to that.

I use “depth charts” which represent the estimated playing times of all players on a team for the rest of the season.  I can change those as injuries, trades, call-ups and call-downs, manager playing time decisions, etc., occur.

Then I have updated rate projections for all players, including pitching, defense, hitting, and baserunning.

I can apply these projections to the depth charts to get estimated runs scored and allowed for each team.  All the projections are context-neutral so I don’t have to worry about park effects.

So then from the estimated RS and RA for each team, I can come up with an average wp (win percentage) for each team.

So I have a file which has, for each team, their expected average wp for the remainder of the season, and their current w/l record.

I don’t use ANY team level stats or current team w/l records to estimate the rest-of-season wp’s.

I then just “play out” the rest of the season and the post-season some number of times (10,000 or whatever), using a log 5 for each game (e.g., a .480 team plays a .530 team, there is an X chance, like, .545, that the .530 team wins).

I use different wp’s for the post-season games though (no #5 starter, more games for #1 and #2 starters, etc.).

Like I said, no rocket science, nothing that Rally, Tango, etc., and certainly BP can’t do…

BP has 2 other “playoff odds” reports, as has probably been mentioned in this thread.

They are all extremely confusing as far as figuring out the methodology. I would suspect that less than 1% of their readership is able to do that (why would you print something that less than 1% of your readership is able to understand, assuming that is true?).

Their “ELO” projections are terrible and make little sense, just like the regular ones, although not nearly as bad.  By “little sense” I mean the results, and not the methodology.  If I could understand their methodology, I am pretty sure that it would be accurate, at least given what they are trying to do (which is NOT to give us any kind of accurate estimates for the rest of the season).

For example, in the ELO rankings, they have TB, which has a current record of 16-15 in their chart, having a final record of 77-85.  How is that possible, given that pre-season, BP had them winning around 87 games, a .537 team or so.  In order to start with a 16-15 record, and finish with a 77-85 record, you have to be a .466 team going forward.

Why would BP use a methodology that has TB as a .466 team going forward?

Their other playoff odds, the “Pecota adjusted” one, seems to have reasonable outputs.  Even with that one, though, they have some problems. E.g., they have a .599 for the Yankees perc3 and .562 for the BoSox.  I don’t know exactly what perc3 means, but those numbers don’t seem right.

According to the glossary, they say that perc3 is derived by taking some current season number and regressing it towards their pre-season expected wp for each team.  The example they give is the Yankees, who had a .580 pre-season expected wp, which I have no quarrel with.

So, if the .599 is based on a regression toward .580, the number they are regressing must be a lot higher than .599.  What number is that?  What about the Yankees so far this season suggests a wp of greater than .599?

These “playoff odds reports” are a mess!

30, Silver’s article on Elo:

http://www.baseballprospectus.com/article.php?articleid=5247

yes, I started to read that and gave up quickly.  I don’t normally have time for real technical, complicated articles and studies.

There was a “bug” in the PECOTA adjusted standings, according to Clay Davenport:

“...the Pct3 is supposed to be a combination of ther actual performance to date and
the performance expected by PECOTA; due to the error, it was simply stuck on the pre-season PECOTA value.”

http://www.baseballprospectus.com/statistics/ps_oddspec.php

They still make little sense, as far as I can tell.  For instance, Baltimore was projected to have a .478 win%. Their W3 win% is currently .484.  Wild guess, At this point in the season, I’d still assume an 80% or more regression of pct3 toward the preseason projection.  BP comes up with a pct3 of .460.  Ummm, huh?  Also, apparently, TB’s preseason projection is already only deservant of 10% of consideration.

Am I doing something wrong?

http://spreadsheets.google.com/ccc?key=pjOhZzPpnP_POC_zmUPMLQw&hl=en

Yeah, you’re not using this season’s projections.

Serves me right for just assuming that link was current and not looking at the date of the article.  I was wondering why they had TB so bad.

Is this the preseason projection or a more updated one:

http://www.baseballprospectus.com/fantasy/dc/