THE BOOK--Playing The Percentages In Baseball

Wednesday, January 04, 2012

Why I’d Bet on My Model (and Against My Instincts)

Nate Silver had a headline that read:
Why I’d Bet on Santorum (and Against My Model)

I thought: NO!!!!  Nate, why?  Why?

The reason should be clear: why HAVE a model that includes all the parameters you deem relevant, if you then throw away the model if you don’t like the results?  Nate’s model showed Romney with getting 22% of the votes, Ron Paul with 21% and Santorum with 19%.  After them were Gingrich 15, Perry 10, Bachman 8, Huntsman 4.  (Without rounding, that adds up to 99.3%).

The actual results shows: Romney 25, Santorum 25, Paul 21.  Gingrich 13, Perry 10, Bachman 5, Huntsman 1.  Presumably, when it comes to actual voting, no one wants to vote for the eventual losers, so Bachman/Huntsman lose 6 points, and that moves over to the expected winners (with Santorum absorbing most of the extras, it seems).

Nate however was not talking about this “abandon the loser effect”.  Instead, he was talking about momentum, and using a model with more parameters (but also more uncertainty), he said:
23.5 Santorum
21.4 Paul
19.1 Romney
15.4 Gingrich
8.7 Perry
6.7 Bachman
4.3 Huntsman

So, in this case, we see he nails everyone except the bottom two losers and Romney, and most of their lost votes went to Romney, who Nate missed in this revised model.

Follow me here.  Under the revised momentum-based model, the one that showed that Santorum getting all the extra votes, it was actually Romney that surged ahead far above the revised model.

But, under the official momentum-free model, it’s Santorum that surged ahead beyond the model.

Basically, he over compensated Santorum with the momentum-based model, and under compensated him with the momentum-free model.

The end result?  Well, half-and-half (meaning using both of Nate’s models) would have given us this:
21.3 Santorum
21.2 Paul
20.5 Romney

And all the abandon-the-losers vote then were split more evenly between Santorum and Romney.

So, I really wish that those people who have forecasting models to NOT hedge their bets here.  Either you have a model or you don’t.  And if you think momentum should be included, then include it.  If that means you don’t know enough then underweight it.  If you want to include a “gut” parameter, then include that too if you have to.  (But do that BEFORE seeing the results.)

And if you want to have a abandon-the-loser adjustment (is there such a thing?  did I just make that up?), then model that as well.  That is, Ron Paul is not going to get those votes.

The key point is this: the personal opinion of any single person is useless.  Completely useless.  How am I supposed to distinguish one person’s guts from the other?  How am I supposed to distinguish between someone’s lock-of-the-week ("that’s a pretty big lock") to someone else’s opposite shoe-in-of-the-week ("that’s a pretty big shoe")?

So, keep your opinion consistent, or learn and improve it.  But, don’t show two models, and then tell me why you would bet on your shoe-in model, but not your lock-model.