Monday, January 14, 2008

Gentlemen, rest your sphincters!

This deathless line was spoken by Harvey Korman as Hedley Lamarr in Blazing Saddles, and it is good advice in many situations in today's strange and suspicious world. I use it today discussing the New Hampshire election results in which the Democratic polls released closest to the election did a very bad job of predicting the actual results. Or did they?

A few weeks back, friend of the blog Distributor Cap did a post about polls, which included a definition of margin of error WITH 95% confidence level. This means that if candidate X is preferred by 49% of the people polled and the margin of error is +/- 5%, the pollsters are saying that they are 95% confident the vote tally for X will be between (49+5)% = 54% and (49-5)% = 44%. They could be wrong about this number, but the odds of that are about 1 in 20 of being wrong and 19 in 20 of being right, which is another way to say 95%.

This works pretty well in two person contests or contests where third party candidates will only pick up some small part of the vote, probably less than 5% and certainly less than 10%. In wide open contests like both the Rep and Dem situations right now, where five candidates or more might each pick up a measurable proportion, this 95% confidence is in each number, 95% confident they get Obama right, 95% confident they get Clinton right, etc.

In a two person poll, the only ways to make a mistake are either to have one side take a massive amount of the undecided (very rare in late stages) or one side to take away from the opponent, making both numbers wrong. Here the odds are pretty much 95% right, 5% wrong, or 19 in 20, as I stated earlier.

But with five numbers to predict, and each of those predictions at the 95% confidence level, the probability of getting them all right is .95 x .95 x .95 x .95 x .95 = 77.4%. Here are all the odds in five person race.

all 5 right: 77.4%
1 wrong: 20.4%
2 wrong: 2.1%
3, 4 or all five wrong: 0.1%

One time in five these kinds of polls should have one mistake, someone doing much better or much worse than predicted. That's not really a rare occurrence. And for that I repeat the fictional evil politician Hedley Lamarr, for even evil politicians can give good advice every once in a while.

(Ladies and) Gentlemen, rest your sphincters.

Now playing: Lene Lovich - Lucky Number
via FoxyTunes


Jess Wundrun said...

As I understand it, the numbers polled for Barack Obama came in exactly as called. 40% of those polled were undecided as of the weekend. Only Hillary's numbers jumped due to undecided's deciding.

Matty Boy said...

It's expected the undecideds will "break" in numbers similar to the decideds, but in this compressed news cycle world, I don't know if that is necessarily true anymore.

My point here is that the pollsters got it wrong, but it's harder to get right in a multiperson race.