THE BOOK--Playing The Percentages In Baseball

Looks like a job for Phil B!

Did they consider the status of the possession arrow?  I don’t think it would explain the whole discrepancy, but it could be a factor.

The team that wins the opening tip guarantees itself equal possessions in the first half, and possibly will get a one possession advantage.  It is not that rare to see a half go by without a jump ball, in which case the team that lost the opening tip will start the second half with the ball.  And since the average pts/posession is somewhere around 1...of course, the downside is that the first jump ball of the second half will go the other way.

My take after an admittedly quick review:

The core finding is that teams down by 1 at the half win only about 49% of the time, when they should win 54% (according to the authors).  They attribute this to extra effort, especially soon after halftime.  They report a separate experimental study that also finds increased effort when participants in a game are told they are slightly behind an opponent.

I’ll be very surprised if this holds up under scrutiny, but we’ll see.  My concerns would include:

1) Many of the down-by-1 teams that go on to win must pull ahead of their opponent early in the second half.  Why doesn’t the trailing teams then have the same movitation to increase effort?

2) Why would this motivational impact disappear when down by 2 points?  Seems implausible.

3) I’m not sure their model of how often a +1 team “should” win is right.  They say on average every 2 points equals +.08, so the expectation is about +.04.  But each additional point both increases the chance the leading team is truly better, AND creates an independent advantage to leading team.  I would expect each additional point to be more valuable, up to some level where the marginal gains start to diminish.  Also, because FGs in basketball are worth two points, a single basket turns a -1 team into a winner, while only achieving a tie for a -2 team.  So again, I’d expect a two-point lead to have more than twice the value of a one-point lead. It could be that a +1 team “should” only be a .520 team, or something like that. 

4) Sample size is a potential problem.  They have a lot of games (>6500), but don’t report the number of +1 halftime games.  Assuming it’s around 650, 2SDs would be .038, more than what I expect the true gap is.

5) The experimental result could easily be explained by change in strategy.  The game is typing the “a” and “b” keys on a keyboad rapidly in succession.  Obviously, there’s a tradeoff between speed and accuracy.  If you’re a little behind after the first “half,” you should risk errors to increase speed.  All we learn is that players can type faster than they realize when they first begin the game (indeed, all players improve in the second half). 

6) A similar change in strategy could explain the basketball result:  down by one, teams are a bit more aggressive (on offense and/or defense), risking fouls to score or prevent scoring.  It would make sense that the optimal aggression level increases in 2nd half, since players at risk of fouling out have diminishing amount of minutes left as game proceeds.  It would be interesting though (and surprising) if teams behind find a more optimal aggression level than teams ahead.

Patriot:  They do account for possession arrow in some regressions, and the result is larger estimate of the behind-by-one factor to about .075 (which is obviously huge).

Following up, the authors focus tremendous attention on the “discontinuity” in the data when one team is +1.  But if you look at their figure 1A, you’ll see there are two other discontinuities: 
a +4 team (halftime) is less likely to win than a +3 team, and
a +7 team is less likely to win than a +6 team.  Each of these disparities appears to be as large or larger than the one they focus on.  This suggests to me that random variation could be a big issue here.  And it’s a serious omission that the authors don’t even mention the fact that two other one point gains also yield a reduced win% rather than the expected .004 gain.

You gone and done it… you are making me read the paper.  If what you are saying is accurate, then shame on the authors…

Ok, page 20, figure 1A is the one Guy is talking about.

Guy is right that when 4 points behind, they have a better chance of winning than when 3 point behind.  The same applies when 7 behind being better than 6 behind.  As is the case when 1 behind being better than tied.

The gap in each of these three cases is at best roughly the same, and if the charts are accurate, seems to show a larger gap with the 4/3 scenario.

This repeats itself when the team is ahead as well: better chance of winning when 3 ahead rather than 4 ahead, and 6 ahead rather than 7 ahead.

Indeed, the symmetry here is really the story, not the contrived “playing better when behind” story.  Or, more likely, pure randomness.

Also interesting how the expanded version of Figure 1A presented in Figure 1B stops the chart at the 3 run differential, exactly at the spot where Guy’s observation take place! 

Good job Guy…

I wrote to the author, Jonah, asking him to respond to the concerns addressed here.

We’ll see if he’s of the Brad Null / Alan Nathan mold of responding with great appreciation to the thoughts of non-academics, or ... well, you know, not.

Are you able to expand on the hockey comment?

And yet, this will probably get published in a prestigious economics or finance journal.  Let’s hear it for peer review!  :>)

* *

This paper reminds me of the famous Price/Wolfers NBA referee racism paper.  Does anyone know if that ever got published?

I don’t have my numbers handy, and it was based on the 07/08 season.

What I did was look at what the score differential was (tied, up by 1, up by 2, up by 3, by 4+), and then looked at the percentage of future goals scored.  And, in each non-tied case (except up by 2), the team that was ahead scored 52% of the goals (or something like that… maybe it was 54%).  But, when up by exactly 2, the leading team only scored 48% of the goals.

I didn’t break it down by Goals scored / goals allowed at the time (and I never revisted it).

We should expect the team that is ahead to score more goals in the future in that game, since the better teams are going to be the ones that are ahead (on average).  But, the up-by-2 was interesting to me.

I would guess that the team that is ahead by 2 will go into a defensive shell.  And if let’s say when they are up by 1 or up by 3 that they score 2.8 and allow 2.6 (just picking numbers out of thin air), then when they are up by 2 they might score 2.2 and allow 2.4 (or something like that).

I think I also broke it down by period, and it was even more pronounced in the 3rd period (which it would have to be if the theory is sound).

Anyway, maybe you have the data handy to take a look at this as well…

My own raw NCAA data shows that being up by 1 is an advantage. But this does not account for possession arrow or team strength.

I agree with everyone on the multiple discontinuities, but I would like to point out a similar phenomenon I found in the NFL.

http://www.advancednflstats.com/2008/08/end-game.html

Through much of the 4th quarter, teams with the ball and down by 1 point are more likely to win than teams with the ball and up by 1 point.

My theory was that teams generally play below the optimum risk/reward strategy balance. Trailing teams would play more aggressively by necessity in the 4th quarter, closer to the optimum. Leading teams would probably tighten up, playing even further from the optimum. It could just be statistical noise, but I think the strategy theory is plausible.

Brian:  A specific version of your theory would be that teams up 1 with 12-15 minutes left are too quick to accept a FG, rather than push for a TD, because it means a 4-pt lead and forces the other team to score a TD.  A -1 team has less incentive to accept a FG, as an opposing FG still means a loss.  Of course, this assumes teams are over-valuing the 4-pt lead with that much time remaining (seems plausible).

Right, I can totally buy into Brian’s theory in football, only because of the specific scoring conditions in football, and the way the off/def are trying to decide on pass-or-run.  Similarly in hockey, where I show that teams go in a defensive shell, and it makes total sense to do so with a 2-goal lead.

But, no way in basketball.  First of all, it’s a 1 point lead at half time, so no way are teams going in a “defensive shell” or changing strategies at that point, especially since that lead can switch to the other team within 30 seconds!  This is totally unlike baseball / football / hockey, where leads actually mean something (60-70% or whatever chance of winning when ahead by one score).

The basketball thesis has to be looked at in terms of the last 2 or 5 minutes in the game, where the strategy of a 1 point game can be more readily accepted as changing.  But, not if it’s a 1-point game where the lead keeps switching after every play or two.

The more you think about it, the more implausible it becomes that this could happen in basketball, this change in approach, with just a 1 point difference.  At 4 points, yes.  1 point?  No.

So, my suggestion to the authors is to pick a point late in the second half, where the winning team has around a 70% chance of winning.  At that point, I can imagine that teams are changing strategies.  Just taking a wild guess, but let’s say a 4-point game with 6 minutes to go will give you a 70% chance of winning.  Maybe Brian will tell us the real numbers.

I think you need to have a reasonable prior here, rather than blindly looking at the data and trying to find a pattern with a regression.  If, for example, the prior is that there is zero expectation that either team will change their strategy with a 1-point lead coming into the second-half, then guess what: all data you see must be random.  That’s the only way to conform to your prior.

Tango:  Remember, the authors aren’t arguing for a change in approach.  They specifically reject that, in fact.  It’s increased effort, they believe.

However, I agree it’s implausible.  If you think about it, the trailing team will often be either ahead by 1 point or behind by 3 after the first possession of the 2nd half.  Either way, their incentive for greater effort has already disappeared according to the authors’ own theory!  The lead will change dozens of times, and usually will only rarely return to -1. The argument really doesn’t make any sense at the level of basic logic.  I bet there is no effect at end of 1st or 3rd quarter, if the authors look.

And as I mentioned, their experimental result appears to be a logical game theory outcome.

Right.

They are arguing that starting the half down by 1, and then being up by 5 four minutes later is different than being up by 1 to start the half and being up by 5 four minutes later.

Why would someone, with 4 minutes into the second-half, have more or less effort the rest of the way, if they knew they were up or down by 1 to start the half?

As you point out, the effort can only exist in a very short time period.  And once that happens, then it’s gone.

What the authors should have done is what I did in hockey: how many points are scored and allowed in the two minutes to start the second-half, depending on the score. 

I must believe that it will be pretty even.

And, who wins the tipoff to start the second half, which must have the maximum spot where you can see the brutal raw effort come into play, no?

Hopefully, the authors can accept our peer review (from a subject matter expert level).

Hi Tom,

Those are some great questions, but we really address them all in the paper itself. Feel free to suggest that your readers actually read the paper (linked in the article) and if they still have questions, definitely let me know.

Regards,
jonah

Have not read the paper, and I don’t know why they used NCAA rather than NBA (where the stats are more easily available), but I can say that you see all kinds of crazier things in the NBA, when one team is ahead by X number of points after the first half, which goes against simple logic. Much of it has to do with effort, use of the bench, fouls, end play, not running up a score in basketball (they do it more in college than in the NBA), the need to rest players, the crowd, etc.

The lead will change dozens of times, and usually will only rarely return to -1. The argument really doesn’t make any sense at the level of basic logic.  I bet there is no effect at end of 1st or 3rd quarter, if the authors look.

Guy (and Tango), the authors address that issue of why a 1 point lead at halftime is significant.  They suggest that it is because the team has time to reflect on the point differential.  That makes sense.

The leads after the first play of the second half, while constantly changing of course, have little or no effect on the effort of the teams because they do not have time to reflect on it.  At least that is what the authors are suggesting.

I like the study, and I think that their conclusions are plausible.  I particularly like the fact that they supported their thesis with other research on motivation in competition and with their controlled experiment.

Of course, there is some chance that the incongruity occurred by chance, no matter what the authors or anyone else think.

The thing I don’t like, however, is that a one point lead is cherry picked to some extent.  What if there were no real change in effort by either team regardless of the score and what if we saw some incongruity at a 2 point lead or a 5 point lead or a tie game (where maybe the road team or the home team tends to win more often than they “should")?  The authors could point to one of those anomalies (and of course the more point differentials we look at, the more the chance of an anomaly by random chance alone) and construct a “story” like this one.  For example, what if the anomaly happened at a 2 point or 3 point deficit?  Surely you could call that a “slight lead” especially in basketball where a 2 or 3 point lead at halftime IS a slight lead, easily overcome.  So, going into this research, they could have found an anomaly at 1, 2, or 3 points at least, and drawn the same conclusions.  That changes the significance level of their results, does it not?  In handicapping analysis, they call that “curve fitting,” whereby you are trying to find “angles” in sports (like this one actually) that have predictive value. If you look at enough angles, you will find one or more that appear significant and thus appear to have predictive value.  However, if you look at enough of them, you will find some that do appear to be significant, but likely occurred by chance and have so predictive value.

Anyway, as I said, I like the study and I find their conclusions and explanations entirely plausible, especially in light of the experimental research they do and cite on motivation and competition.

The response in #17 is disappointing.  Alas, less in the Null/Nathan tradition, more in the JCB/DB tradition. 

The sample size just isn’t big enough to support these conclusions.  If you look at Brian’s data, he shows teams +1 at the half winning at a .519 clip:  http://dberri.wordpress.com/2009/03/05/modeling-win-probability-for-a-college-basketball-game-a-guest-post-from-brian-burke/
Look at his table and you’ll see there’s just a lot of random variation. To try to smooth it out, I combined 18 thru 22 minutes remaining.  It’s still not very smooth:
+1 .553
+2 .581
+3 .651
+4 .661
+5 .736
+6 .775
+7 .773
+8 .864
+9 .864
+10 .846
(I just averaged the home and visiting percentages; in fact the home team leads more often, so some of these should be a bit higher.)
In fact, that the author’s own data does not show the +1 result as significant at the 95% level.  There’s just nothing here.

MGL:  I don’t find the halftime theory plausible at all.  But obviously it’s the data that matters, not our intuition.

I don’t think the controlled experiment tells us anything about “effort.” It tells us that those who were told they were slightly behind made faster keystrokes at the risk of errors—exactly the right response to learning that A) you’re losing, and B) someone else has found they can raise their score by typing faster.  What else would we predict.  Now, if they continued to find this effect after people had played several rounds of the game and found their own best scoring speed—which is much more analogous to competitive athletes—then I’d be impressed.  But they didn’t look at that.

I emailed the friend who sent me this article, he’s a student of Jonah Berger. I told him about the issue Guy brought up with the change in W% at -4 and -7, so hopefully we’ll get a response to that from Mr. Berger.

I also agree with pretty much everything said in #19

Brian finds that the WP in NCAA basketball is a function of the point differential and NOT the time left in the game, other than in the last few minutes.  Why would that be?  That is completely counter-intuitive.  It must have something to do with the back and forth nature of basketball and also something to do with the fact that teams in basketball drastically change their strategy depending on the score.

Brian finds that the WP in NCAA basketball is a function of the point differential and NOT the time left in the game, other than in the last few minutes.

I used to do live in-game betting for basketball at the ncaa and nba level.  A 10 point lead with 10 minutes to go in the first half is hugely different than a 10 point lead with 10 minutes to go in the game.  2 of the biggest factors for odds of winning are of course point differential and time remaining.  Relative team strength (I usually used pregame point spread as a proxy) would be the third factor.

I would think so, but unless I am reading Brain wrong, that is what he said in the above-linked article on Wages of Wins.

From Brian’s article:

One thing I’ve already noticed that’s interesting about basketball is that the win probability equation is the same for nearly the entire game. In other words, a 6-point lead for the home team in the first 10 minutes of the game yields the same WP of 0.86 as a 6-point lead with 10 minutes to go in the 2nd half.

So, Brian, what say you?  I think that is obviously wrong.

MGL:

The thing I don’t like, however, is that a one point lead is cherry picked to some extent.  What if there were no real change in effort by either team regardless of the score and what if we saw some incongruity at a 2 point lead or a 5 point lead or a tie game

This is EXACTLY what the data is showing! Look at my post #7.

As far as I’m concerned, it may as well have been cherry-picked…

"This is EXACTLY what the data is showing!”

Right.  There are two variables here: minutes remaining and size of lead. Maybe the authors really did begin with a theory that the lead only matters at half time, not before or after (though it’s an odd theory).  But I can’t imagine they began with the theory that a one point deficit inspires extra effort, while two points does not.

In Brian’s raw data, you find a similar point with 11 minutes remaining:  a -1 team wins at a .514 rate.  What does that mean?  Nothing.

This post is from ubelmann.

==============================

I’m looking at the paper now, and I have to say, this all looks very fishy.

From a general standpoint, they start off by saying that it is intuitive to expect a team with the lead to have a greater chance to win, then at the beginning of “Study 1” they claim to expect the team that is slightly behind to have an advantage.  That’s maybe not a red flag for me, but at least a yellow flag.

Then there’s the matter of fitting.

To formally test whether the difference in winning percentage is statistically different from expected,” we perform a more structured analysis using a regression discontinuity model (Thistlethwaite & Campbell, 1960).3 This method fits a quintic regression line through all the data to control for the expected effects of halftime score difference on winning percentage. A dummy variable for being behind by one point or more is also included which allows for a discrete jump in the polynomial function for being behind. Such analyses are useful because they allow us to make causal inferences about the effect of being behind, but such claims can only be made about points near the discontinuity (i.e., losing by one). As one moves further from the discontinuity, other potential factors make such inferences less tenable.

Now, they limited their data set to games with a score differential of +/- 10.  That’s reasonable enough in that blowouts could give strange behavior, but since all of the +10 games give you the same information as the -10 games, and so forth, it means that they have reduced their data set to 11 points.  And then they tell me that they’re fitting a quintic(!) regression line (plus some other junk) through 11 points?  A quintic in and of itself has 6 degrees of freedom and while I don’t exactly understand the dummy variable that they are using, that would seem to add another degree of freedom in the fitting.  I admit that I’m not taking the time to read the citation on fitting, so maybe there’s something in the method that ensures fewer than 7 free parameters, but quintic regression (to me) suggests a lot of free parameters in play.

It’s no wonder that they think the data point at a score differential +1 is significant; with that many degrees of freedom, you can almost make the regression curve fit right on top of the data set.

For similar reasons, I have trouble with a lot of their data presentation.  There’s no way that their graphs should include both positive and negative score differentials--it’s totally redundant and it has the unpleasant side effect of making the “strong discontinuity” look as big as it could possible be if it was just a random fluctuation.

It’s not really clear to me that they tried a linear fit to the data. Given the 11 points and their associated uncertainties, I would love to do a least-squares linear fit to the data set to check what the chi-squared is on that fit.  That seems like the first step to me--try the naive fit to see how well it describes the data.

If this is a real effect (I don’t know that the data that they present can really confirm or deny their claim), it should show up in some testable ways.  It should be strong immediately after the half (as Tom suggests).  The positive outcomes on a per-possession basis should also show the effect.  It should also be true if you go back farther in NCAA history.  (I would suggest using NBA data, but that might be too different.)

Given that the effect they are trying to explain is non-intuitive, they certainly haven’t convinced me.

"So, Brian, what say you?  I think that is obviously wrong”

I can’t speak for Brian’s data, but why “obviously wrong?” A +6 lead after 10 minutes “projects” to a 24-point difference in talent, while +6 at 30 minutes projects to just +8.  These are crude projections of course, but I’m sure the earlier lead is associated with a greater talent disparity.  So that could roughly offset the increasing value of the lead as time remaining diminishes.  I don’t know if this is true, but it seems possible....

Guy/28:  Yes it is sometimes true that those 2 factors you mention can have an apparent “negating” effect at times.  However, this absolutely does not mean that time remaining in a basketball game has little effect on the outcome until the last few remaining minutes of the game.  Clearly it breaks down at the extremes (i.e. 20 point lead with 10 minutes left vs 20 point lead after 1 quarter, or tie game with 10 minutes left with one team a heavy pregame favorite vs. odds at the start of the game when the score is 0-0).

I suppose it is possible that, if one were to look at aggregate data for many games between teams of various strengths, it might appear that time remaining was not a major factor in determining the outcome, due to the nature of the 2 often negating variables mentioned.  However, in an individual game this is not so.

ubelman/27: ouch, I did not realize that the reason I noticed the mirror data is because the data itself WAS mirrored!

When I present charts like that, I always show it from the perspective of one team (say the home team), so that I have independent data throughout.  To show the same data points on both side of the line is… well…

Late to this discussion, so I don’t have much to add ... you guys have pretty much picked this clean.

1.  I didn’t realize the data was mirrored either, until I read Tango/30.  Yuck!

2.  I don’t trust the regression because of the quintic ... it’s possible the best-fit quintic is assymetrical about zero, and the “team behind” coefficient is just there to restore the symmetry. 

3.  Figure 1B tells the story for me.  There is a difference between where the -1 “should” be, and where it is, but it doesn’t look significant.  I think we’re just talking random chance.

4.  To convince me, run this on a new dataset and show me the same effect.  The authors’ story has some surface plausibility—that teams behind by 1 who have lots of time to reflect on it, play harder to compensate—but you have to show a much stronger effect to go beyond that.

Guy (and Aunt Bea), the “offsetting effects” makes sense, but I don’t think there is much difference in true talent between teams where one team has a 6 point lead after 10 minutes or a 6 point lead after 30 minutes.  Hardly any difference at all.  I realize that your example of +8 and and +24 was “for illustration purposes only” but I don’t think it is even remotely close to that.  Maybe +2 and +3 or something like that.

On the other hand, I should not have dismissed Brian’s claim, since that data (and the resultant WP versus point differential/time remaining curve) should speak for itself.

BTW, if the authors’ thesis and conclusions are true, we really should expect to see a similar effect in the NBA.  If there is, that would lend strong support to their conclusions.  If there isn’t, they could argue that we only see such an effect in amateur (albeit high level) sports, however, having to use that argument surely weakens (by a lot) the strength or their results and conclusions.

I have NBA half and full game data for several years, so I can run the data fairly easily.  I’ll do that shortly.

Freakonomics has some discussion in the comments here:

http://freakonomics.blogs.nytimes.com/2009/03/17/when-losing-leads-to-winning/

I tried to post a link to this discussion in the comments there, but I guess the moderator didn’t approve it.

Boy, that’s a tough crowd over at Freakonomics.  They seem to have the same take we do here:  much ado about statistical noise.

The promotional boost from Wolfers is typical, and illustrates one of my criticisms of the sports economists (in general; I’m sure there are exceptions).  Although they like to describe the academy as a rough-and-tumble world of mutual criticism, the sports economists I’ve read are almost unfailingly supportive of their peers’ work with never a critical word to say.  Maybe internally, when us non-academic plebes can’t see, they turn on each other with withering criticism.  (I tend to doubt it, given what manages to get published.) But at least in the public arena, it’s solidarity in defense of the priesthood.  I’ll be interested to see if Wolfers backs off his praise, defends the paper, or just keeps his head down and drifts away.

MGL’s terrific followup is here:
http://www.insidethebook.com/ee/index.php/site/comments/being_behind_is_a_good_thing_part_ii/

And I posted a link at the freakanomics blog.  I have the email of the moderator there if it doesn’t get posted…