THE BOOK--Playing The Percentages In Baseball

What’s really weird about Silver saying he’d bet against his model is the fact that he’s done studies that show that “momentum” in election polling doesn’t have any predictive effect. It’s actually more like a random walk.

Also, the abandon-the-loser effect is pretty well accepted, but I’ve never heard it called that. It’s usually characterized as polls overestimating minor candidates. And it’s always hard to predict who the last-minute undecideds will break towards.

"The reason should be clear: why HAVE a model that includes all the parameters you deem relevant, if you then throw away the model if you don’t like the results”

I think what’s going on here is that Nate’s polling model for primaries here isn’t that “model that includes all the parameters you deem relevant.” It is really just polling.  The one he’ll roll out for the general in a few months will be far more comprehensive, and I don’t think we’ll see the same kind of hedging.  The early primaries/caucuses function very differently and are really difficult to model.  It’s still useful to have some kind of baseline to get a sense of what to expect, but I don’t have a huge problem with outlining before the actual voting why the simple model might have big shortcomings.

But the problem is, he ended up being wrong!  One had Romney beating Santorum by 3 points and the other, the one he would bet on, had Romney losing to Santorum by 4 points.

So, he was really in no position to say what he would bet on.

Hedging against the model used to happen all the time with PECOTA, too.

Would it have made it any better if he ended up being right? Would that have changed your opinion?

In fact, I’m not sure his revised model was the one he was going to use, it was just to show that maybe he’d underestimated Santorum’s support and that he’d bet that he’d win the state. As it was he was only 8 votes off being right so his “gut” had a pretty instinct I’d say.

Also he has shown that for primary/caucases elections there is an effect of “momentum”, that is why he put it in the model. That doesn’t mean it exists for general elections, but it does for primaries.

I don’t care about the binary win/lose.  I care about the % of votes.  Hence, I NEVER referred to the ultimate winner, but rather, the % of votes earned.

So, of course my opinion will change if the results were different.

He had one model that says one thing, he had another model that said another thing, he says to bet on the one that has more uncertainty, even though the official one is the one with less uncertainty, and the end result is something in-between.

Oh, and I’m sure he messed up the math too.

For example:
http://elections.nytimes.com/2012/fivethirtyeight/primaries/iowa

He had these are the % of votes, and these are the chances of winning:

Romney: 22 / 44
Paul: 21 / 34
Santorum: 19 / 18

Now, there’s no way his uncertainty level is so low that he can have 3 point difference in the mean of Romney and Santorum result in a 26 point difference in chances of winning.

And how do I know his uncertainty level can’t be that low?  Because he tweaked his model to come up with a massive swing in the unofficial model.

He goes from a 2:1 odds in one model to (probably) a 3:1 (the other way) in another model.

I think what he ended up doing is presuming a 0% uncertainty in the mean forecast, and simply went with a binomial as if it was a true 22 +/- 0 v 21 +/- 0 v 19 +/-0.  He did the same mistake with the Chipper Jones forecast from a few years ago.

I’m sure someone out there can run a sim for us, if you plug in a +/- of the mean of 1 sd = 3 points or something.  You’d probably end up at most 1.5 to 1 or something (just as a WAG) between Romney and Santorum.

But there is good reason to maintain generality in the model and not calibrate it for every specific permutation of parameters: effort. While Silver did estimate this new model, he explicitly pointed out that the dearth of historical data makes it much more difficult to statistically extrapolate these particular parameters.

In this case, the model isn’t perfectly specified, and Silver knows this. He wants to further regress these Iowa-specific parameters, but figuring out the proper amount to regress out of sample is a serious, non-algorithmic task.

Instead, he simply pointed out why subjective reasons indicated that these extra parameters would help out Santorum to some degree. You’re right that his additional hypothesis—that Santorum would pick up the majority of the abandon-the-loser votes from Perry and Bachman—was incorrect, assuming that you do believe that the additional momentum parameters had a real effect here, but that was his impression of an exception in this specific context that I think every modeler has a right to make.

Coming up with a ‘gut’ parameter is precisely the same thing-- but I do think the person who designed the model in question does have a bit more credibility in providing this personal opinion, as he knows the shortcomings and extent of the model better than anyone.

"I don’t care about the binary win/lose.  I care about the % of votes.  Hence, I NEVER referred to the ultimate winner, but rather, the % of votes earned. “

That’s fair, and ultimately the right approach, I think.  But that isn’t what Nate was going for in his post. 

“One had Romney beating Santorum by 3 points and the other, the one he would bet on, had Romney losing to Santorum by 4 points.”

He wasn’t saying that he was betting on the alternative model or that the alternative model was better.  He was saying that in his opinion, Santorum was a modest favorite to win because there were a number of plausible reasons to expect Santorum to outperform the polling average.

When two models exist, it’s generally never a great idea to only pick one, especially if both are reasonable and somewhat diverse. In other words, they don’t have the same results but both are believable, which is exactly what we have here. This will actually decrease uncertainty! Thus, as Tango said, regardless of Nate’s gut, averaging the two models was the best choice, regardless of the final result (which just happened to confirm this theory).

This is the predominant reason why ensemble learning methods (such as boosting) are so popular right now.

How is this any different than having your projection model based around FIP components, then saying you don’t believe the Matt cain or Tom Glavine projections because you know things the model doesn’t (they outperform FIP on a consistent basis)?

pm/11, The difference is that FIP is not really a projection model...FIP is FIP.  FIP by design does not tell you about Cain’s ability to prevent hits on balls in play.  It so happens that you can do a decent job by using it as a projection system (so it often gets misused by Fangraphs readers, etc.), but I’m sure that if Tango were spending time making projections he cared about, he would formally incorporate those other factors into his model.

If you know things that the model doesn’t, update the model!

"he simply pointed out “

He didn’t do that.  He said he would BET on the alternate model.  Had he simply pointed it out, had he simply said that he’s not sure which of the two models he agrees with, then fine.  That’s “simply”.  He went further than that.

***

“How is this any different than having your projection model based around FIP components”

Well, *I* don’t have a projection model based around FIP components.  Why are you explicitly saying I do?

"He said he would BET on the alternate model.”

No he didn’t.  He said he thought Santorum was a slight favorite not based on the alternative model, but based on the combination of things external to his basic polling model that were potential positives for Santorum.  He wasn’t saying the alternative model was necessarily better.

I guess I took his headline too literally:

Why I’d Bet on Santorum (and Against My Model)

And by the way, it’s bullsh!t to say that he’d bet on Santorum because he not only rejects the original model, but also because he is not in favor of the alternate model, and so, he just puts things together to come up with Santorum as his pick, without respect to the alternate model.  I mean why even have ANY model, if you can go with your gut at will.

Nate Silver does not follow grizzled BBWAA logic.

Please.

Nate chose Santorum because he decided on the alternate model.

He says this after outlining the alternative model:

“This alone might not quite be enough to convince me that Mr. Santorum is the favorite Tuesday night.”

He then covers some other factors external to his basic model, including this:
http://fivethirtyeight.blogs.nytimes.com/2011/12/22/in-iowa-conservative-republicans-have-overachieved-polls/

He says in the article, “Our default model is designed to provide for a reasonable benchmark. But it is also designed to be simple rather than necessarily accounting for all the salient evidence.”

This his attempt to account for some of that other evidence, which includes some of the things in the alternative model.  It wasn’t an endorsement of the alternative model at face value, nor a rejection of his default model.

http://fivethirtyeight.blogs.nytimes.com/2012/01/03/why-id-bet-on-santorum-and-against-my-model/

C’mon dude.  He even says:
“ld win with a possible 27 or 30 percent of the vote.”

Given that he had Santorum at 23 and had dropped Romney down to 19, then presumably Santorum going up to 27 or 30 would mean Romney would go down to at least 17 or 16.

My head is spinning at everything he was saying, that he would allow himself to come to this conclusion:

Why I’d Bet on Santorum (and Against My Model)

He was perfectly fine to say everything he did… except when he made a conclusion.  This is the POINT.  You CANNOT make a conclusion, given all the uncertainties and how much changes in model changed the % votes so much.

"call Mr. Santorum the favorite. Not a heavy favorite, to be sure — I’m not sure that I’d place a bet on any of the three candidates at even odds — but I like his chances a little better than those of Mr. Romney and Mr. Paul.”

This hardly seems like the kind of brash conclusion that should be precluded by uncertainty.  It is qualified at every step.  He isn’t revamping the general election forecasting model based on this.  He isn’t even updating the primary projection model!  He is just introducing some evidence not incorporated in the default model in order to make a call on a binary event that even he states is very close to a tossup.

Why in the world would you want to bet on something that is a tossup?

And none of them were even odds (meaning 50% chance of winning).  I mean, duh.  You have three that are close, that means they are going to be hovering around 30% each, give or take.

He made a brash strong statement on his blog post headline.  Can we AT LEAST agree that the headline belies all the uncertainty and back-and-forth he was going through?

"I mean why even have ANY model, if you can go with your gut at will.”

I have all sorts of prediction models for gambling and find that this sort of thing happens all the time.

1. Build a model based on various inputs that works well under most circumstances

2. Find myself in a situation where I feel there is relevant info that the model doesn’t take into account

3. Try to make some sort of educated guess on how to adjust for the other info, even though I don’t have time to build a new model before game time (or in Nate’s case, the caucus)

4 (occasionally depending on the adjustment). Bet against the model

In Tango’s defense, I should add that if there was infinite time to plan in advance for every possible contigency, I would agree with him.

You could literally build EVERYTHING into your model and never have to wing anything.

All I’m saying is that Nate shouldn’t be in a position to bet on anything.  Just say: “too many variables for me to bet on anything”.

When there isn’t a play, THEN DON’T BET.

Tango,

You do realize there’s a decent chance Nate doesn’t write his own headlines, right?

To his own blog?

Blog software is pretty straightforward: first thing you have to do is create a headline.  That’s the URL of the entry. Then you put the content.

What you are suggesting is that Nate submits his material outside of the blogging system, and someone else types in the blog post for him.

Otherwise, you are suggesting that Nate creates the blog post with the automatic naming of the URL=headline, and then an editor goes and changes not only the headline, but the URL as well.

I’d give the odds of Nate not writing his own headline at 0.001%.

But would you bet on it?

I am SO glad you’re commenting on this, Tango.  When I read Silver’s post (before I read this one) my first thought was, “Oooh, Tango would HATE that.” ‘preciate you infiltrating my brain.

This really goes to show just how well-embedded the instinct to use our “fast thinking” brain is.  Most people can get behind pure objectivity in theory.  But when the rubber hits the road, the pull of including the subjective in a hedging manner can be too strong for even the best of us to overcome.

That said, I think you’re a little hard on him.  to have a best current model is not to say that you necessarily must believe you have the absolute best possible model.  There seems to be perfectly reasonable justification for keeping your official model as one based purely on variables that have been objectively verified as meaningful, if for no other reason than to prevent it from being dismissed on the grounds of Nate’s subjectivity.

That Nate recognizes the possibility that his model could be improved and is willing to adjust his specific predictions occasionally doesn’t undermine the logic of the model itself.  Absent the context of being a decision-maker, why must one have one and only one number?

RMR: Right, exactly!

This is my point.  If you must BET, if you are a decision-maker, you NEED one number.  You have to choose.

But, Nate recognizes the complexity, the uncertainty, he shows how in one case Romney is 3 points ahead, and in the other Santo is 4 points ahead.

Why, therefore, would he BET on anything? 

As I said, Nate was fine with everything, EXCEPT his conclusion as noted in his headline.

Speaking of plays, where can I put a little money on Silver not writing his own headlines at Tango’s odds?

RMR - I agree a model is more objective (or at least systematic) than simply using one’s opinion. But a lot of subjectivity can go into forming a projection model, so, as you note, humility about the limits both of one’s own intuition and of one’s models are good traits to show.

As to betting, well, Nate’s now a pundit. He’s paid not only for his analysis, but for his opinion, so I’m sure his editors prefer that he actually pick a winner in advance to simply offering hedged probabilities. Nate has to bet.

Being contrary often attracts attention, and someone as associated with modeling as Silver publicly disagreeing with his own model is probably an excellent marketing gimmick. So I wouldn’t be surprised if Nate happily wrote the headline.

But another potential downside of the headline is that it may help undermine trust in modeling as systematic analysis: look, a prominent stat-head disagrees with his own model! So why bother with modeling at all?

Re: 31.
“If anyone gives you 10,000 to 1 odds on anything, you take that bet. If John Mellencamp ever wins an Oscar, I will be one rich dude.”

Re: #7, I simmed it assuming that the true vote percentages are normally distributed with an SD of 3. Also I only looked at Romney, Santorum, and Paul for simplicity (but it shouldn’t change the relative results).

I get:
Romney: 35.4%
Paul: 33.9%
Santorum: 30.6%

Andy, fantastic!

Can you re-run trying different uncertainty levels?  Try SD = 0, 1, 2 as well.

I love it when you guys do the hard work!

The obvious points here are that political elections are not:

a) Worth modeling (you’re not estimating any innate ability/performance; you’re estimating human behavior—why?).
b) Able to be successfully modeled within any reasonable margin of error.

Unreal that a guy who showed such promise 10 years ago is now genuinely wasting time (both his and his readers`) trying to do some horrendously designed meta-analysis of opinion polls.  As far as I can tell, exactly zero value added since he launched 538.