from Amazon
Friday, January 07, 2011
Owning up to bad forecasts
I can remember all the way back to college when I’d hear forecasts and wishing that they’d revisit those forecasts and tell us how well they did. I love doing this myself, not only on my own stuff, but others too. Not that I think anyone is particularly insightful, but more in a Amazing Randi v Uri Geller way: exposing the truth. In doing that, you get a few who start to stand out, like MGL and some of the regulars around here, but even those standouts are very tempered. It’s a standout in a statistical significance sense, rather than a “bet the farm” sense. These standout guys will win say 52% to 54% of their bets. Which is great of course in Vegas, awesome really. But, that doesn’t make them Oracles. Anyone who says they are going to be right 60% or more of the time on anything that has an otherwise 50/50 chance of occurring, I would outright dismiss at first and then demand evidence.
This is why I like what Nate Silver does. He provides a good landscape here. I would have preferred that he presents it the way I do it, as a winning percentage, because EVERYONE knows what that means. His point system is not obvious, and really, in three days, no one will remember it. But, when I say that Chone won 54% of the time and PECOTA won 46% of the time and Marcel won 51% of the time, well, I’m going to remember that for a pretty long time.
This is how we should be presenting things, as head-to-head won-lost records. I think that’s how you sell a forecast, not in RMSEs and correlations and other easily forgettable and hard to understand without context numbers. All you need is some baseline, some thing that is pretty much setup to win 50% of the time. That’s why Marcel exists.
So, for Nate’s political stuff, he should either designate some pollster as “average”, or he should create an absolutely simple and transparent version of Marcel, one that should win half the time against typical competition.
Recent comments
Older comments
Page 1 of 344 pages 1 2 3 > Last »Complete Archive – By Category
Complete Archive – By Date