from Amazon
Sunday, February 05, 2012
Is Nate Silver alot more certain than he lets on?
I’ve been following Nate’s mean forecasts for the five primaries so far. So far, he’s made 25 predictions over those 5 primaries (and obviously, they are interdependent). His worst forecast result was Santorum in Iowa, where he gave him a mean forecast of 19.1, and he ended up at 24.6, for a difference of 5.5 points. His average error over those 25 forecasts is 2.34 points, with one standard deviation being 2.71 points.
However, his posted uncertainty level is much higher than that. Let’s take Mitt in Iowa as an example. He gave him a mean forecast of 24.5, with a range of 13 to 32 (a range of 19 points). In another article, he notes that his range is the 5th and 95th percentiles. Those levels are reached at the +/-1.645 standard deviations (or a range of 3.29 standard deviations). This means that one standard deviation for Romney is 5.8 points.
So, I calculated it for all 25 forecasts, and one standard deviation averaged 4.6 points as Nate’s uncertainty level. However, as I noted earlier, the actual observed standard deviation was 2.71 points. This means that Nate’s uncertainty level is 4.6/2.71 too wide, or 1.7 times too wide.
Now, either he made a calculation error of his historical data (making the width of his uncertainty level almost twice what it should have been), or this year, things simply worked out alot closer to the mean than expected, just by luck (after all, we only have 25 data points).
Here’s the data for those who want to take a crack at it:
1SD is simply the difference of the 95% and 5% columns, divided by 3.29.
Results mean 5% 95% 1SD diff State Person
24.6 19.1 10.0 29.0 5.8 5.5 IA Santorum
22.9 18.6 11.0 27.0 4.9 4.3 NH Paul
18.6 15.0 8.0 23.0 4.6 3.6 NV Paul
17.0 13.9 7.0 22.0 4.6 3.1 SC Santorum
11.1 8.1 4.0 14.0 3.0 3.0 NV Santorum
24.5 21.9 13.0 32.0 5.8 2.6 IA Mitt
31.9 29.5 20.0 38.0 5.5 2.4 FL Newt
46.4 44.2 33.0 51.0 5.5 2.2 FL Mitt
40.4 38.7 26.0 49.0 7.0 1.7 SC Newt
39.3 38.5 27.0 47.0 6.1 0.8 NH Mitt
21.4 21.0 12.0 31.0 5.8 0.4 IA Paul
16.9 17.0 9.0 26.0 5.2 -0.1 NH Huntsman
10.3 10.5 5.0 18.0 4.0 -0.2 IA Perry
0.7 1.2 0.0 2.0 0.6 -0.5 NH Perry
13.3 13.9 8.0 21.0 4.0 -0.6 FL Santorum
27.8 29.3 19.0 39.0 6.1 -1.5 SC Mitt
13.3 15.1 8.0 24.0 4.9 -1.8 IA Newt
9.4 11.5 5.0 19.0 4.3 -2.1 NH Newt
13.0 15.6 8.0 24.0 4.9 -2.6 SC Paul
9.4 12.3 6.0 20.0 4.3 -2.9 NH Santorum
5.0 7.9 2.0 15.0 4.0 -2.9 IA Bachmann
22.7 25.6 20.0 32.0 3.6 -2.9 NV Newt
0.6 3.8 0.0 8.0 2.4 -3.2 IA Huntsman
47.6 51.3 41.0 56.0 4.6 -3.7 NV Mitt
7.0 11.0 5.0 18.0 4.0 -4.0 FL Paul
1.8 SC Other
1.5 NH Other
1.3 FL Other
0.3 IA Other