Wednesday, January 27, 2010

Robo-polls are useless.

Public Policy Polling has another poll that is making news. This time, it says that Fox News is the most trusted name in news, with a 49% positive rating to 37% negative rating, higher than any other organization.

It may be true and it may be false. The point I want to make is that Public Policy Polling (PPP) uses methods that CANNOT BE TRUSTED.

PPP does robo-polls. You are not talking to a real person, but instead a recording. Do you hang up automatically when you hear a recording at the other end? A lot of people do. If a large percentage of people fail to respond, the sample suffers from self-selection bias. The results could still be close to correct, but we cannot say there is a confidence interval, which is known to the general public as the margin of error. Any poll could be wrong, but if a sample is legitimately "random", the probability of a large mistake can be calculated and the larger the sample, the more likely the error will be minimal.

The famous example of self-selection bias is the Literary Digest Poll of 1936. The Literary Digest took a straw poll that showed overwhelming support for Alf Landon over incumbent Franklin D. Roosevelt, something like a 60%-40% landslide. There were four million responses, which would mean a tiny margin of error if the sample were truly random. The problem was that the four million responses came out of ten million questionnaires, so about 60% of the people asked simply threw the postcard away. It was in fact Roosevelt who won a 60%-40% landslide, with Landon carrying 2 out of 48 states in the Electoral College, Maine and Vermont.

I feel like a voice in the wilderness but I will keep screaming until someone notices. Don't trust robo-polls, which means don't trust PPP in particular. (There are others.)

Thank you for your kind attention. (Posted at the Smirking Chimp as well.)

No comments: