This blog is still alive, just in semi-hibernation.
When I want to write something longer than a tweet about something other than math or sci-fi, here is where I'll write it.

Sunday, November 18, 2012

Watching the watchmen:
Final grades for all polling companies with 3 or more predictions in final week


At least part of my pay is for grading papers, so it seems only fair that I grade the numbers from the polls released in the final week of the election.  I would have done this earlier but I gave three midterms so I have been busy... grading papers.

A+ companies

Company: PPP (d)
Total races polled in final week: 33
Record: 33-0
Worst miss: Predicted McCaskill by 4.0% MO Senate, won by 15.5%
Average distance from correct percentages: 1.9% toward GOP
Median distance from correct percentages: 2.0% toward GOP

It's really hard to argue with 33-0 being at the top of the list, and PPP deservedly earns the Busy Bee Medal, but notice how much PPP skewed towards the Republicans in their sampling. You will see how big a trend this is in all the companies, with damned few exceptions.

===============

Company: SurveyUSA
Total races polled in final week: 11
Record: 11-0
Worst miss: Predicted Heller by 6.0% in NV Senate, won by 1.2%
Average distance from correct percentages: 0.9% toward GOP
Median distance from correct percentages: 2.2% toward GOP

Yet another company with a spotless record, several tracking websites like Real Clear Politics are slow to publish the stuff SurveyUSA puts out. If someone has put them in the penalty box for earlier infractions, it's time to let them back in the game.

===============

Company: Marist
Total races polled in final week: 10
Record: 10-0
Worst miss: Had Obama by 6.0% in OH, won by 1.9%
Average distance from correct percentages: 1.5% toward GOP
Median distance from correct percentages: 2.0% toward GOP

Median is farther to the right than the average because of this big miss towards Obama.
 ===============

Company: Grove Insight [d]
Total races polled in final week: 7
Record: 7-0
Worst miss: Predicted Obama by 6.7% in MI, won by 9.5%
Average distance from correct percentages: 0.1% toward GOP
Median distance from correct percentages: 0.1% toward Dem


I'm giving this partisan pollster the Fair and Balanced Medal for this election cycle. Only seven races called, but their median and average away from true are incredibly close, and even more pretty, one is on the GOP side and the other on the Dem. Nobody else did that.

===============

Company: Gravis
Total races polled in final week: 6
Record: 6-0
Worst miss: Predicted Nelson by 3.0% in FL, won by 13.0%
Average distance from correct percentages: 2.7% toward GOP
Median distance from correct percentages: 1.6% toward GOP

Lots of companies were off by a bunch on the margin of victory in the senate races in Florida and Missouri, and Gravis was no exception.

===============

Company: Zogby
Total races polled in final week: 6
Record: 6-0
Worst miss: Predicted Nelson by 20% in FL, won by 13.0%
Average distance from correct percentages: 3.4% toward Dems
Median distance from correct percentages: 3.5% toward Dems

Zogby gets the In the Tank for the Dems medal. Not even the partisan Democratic polling companies skewed as far left as Zogby. Over-predicting Nelson's margin of victory was a big, big outlier.

===============

Company: Angus Reid
Total races polled in final week: 4
Record: 4-0
Worst miss: Predicted Baldwin by 2.0% in WI Senate, won by 5.6%
Average distance from correct percentages: 1.6% toward GOP
Median distance from correct percentages: 1.6% toward GOP

Another perfect score that oversampled GOP voters and never got pulled off course.

===============

Company: Mellman [d]
Total races polled in final week: 4
Record: 4-0
Worst miss: Had Obama by 2.0% in both FL and IA, won by 0.9% in both
Average distance from correct percentages: 0.4% toward Dem
Median distance from correct percentages: 0.5% toward Dem

The partisan company Mellman oversampled Democrats slightly, but only by about a half a point instead of the bigger oversampling of Republicans shown by many companies who got perfect records.

The reason this is possible is because this election really wasn't that close.
 
===============

A companies

Company: WeAskAmerica
Total races polled in final week: 12
Record: 11-1
Missed with Romney by 0.9% in FL where Obama won by 0.9%Worst miss: Predicted McCaskill by 3.4% MO Senate, won by 15.5%
Average distance from correct percentages: 1.3% toward GOP
Median distance from correct percentages: 1.6% toward GOP

A miss is a miss, but this was so close I might only take a half point off if I were grading a student who turned this in.

===============

Company: Reuters/Ipsos
Total races polled in final week: 6
Record: 5-1
Missed with Romney by 1.0% in FL where Obama won by 0.9%
Worst miss: Predicted Brown by 9.0% in OH Senate, won by 5.2%
Average distance from correct percentages: 0.1% toward GOP
Median distance from correct percentages: 1.0% toward GOP

Yet again, a compnay misses a perfect record by missing the toughest election to call by a single percentage point. 5 of 6 would count as 83%, which is a middle B, but 5.5 of 6 is 91.7%, a low A.

===============

F companies

Company: Rasmussen/Pulse Opinion Research (subsidiary)
Total races polled in final week: 16
Record: 9-5-2
Missed with Romney by 2% in FL, Obama wins by 0.9%
Missed with tie in OH, Obama wins by 2.0%
Missed with Romney by 1% in IA, Obama wins by 5.6%
Missed with Romney by 2% in VA, Obama wins by 3.0%
Missed with Romney by 3% in CO, Obama wins by 4.7%
Missed with tie in VA senate, Kaine wins by 5.0%
Missed with Thompson by 1% WI senate, Baldwin wins by 6.7%
Worst miss: Predicted Nelson by 4.0% in FL Senate, won by 13.0%
Average distance from correct percentages: 4.5% toward GOP
Median distance from correct percentages: 4.4% toward GOP

According to their websites, Rasmussen contracts out the actual polling to Pulse Opinion Research, and Pulse Opinion says they use Scott Rasmussen's model.  According to a Twitter conversation I had with Josh Marshall of Talking Points Memo, he has done research that says Pulse Opinion is a subsidiary of Rasmussen, though he did not provide a link.  In either case, these two are joined at the hip. There were a few races they polled more than once in the last week, and in that case I take the most recent, or if reported on the same day, the one with the larger sample size.

Okay 16 points possible, 9 definitely right.  This reminds me of the grade lawyers I see in class after midterms begging to have their grades changed.  I'll give a half point each for the close presidential races in FL and OH, but no way I give a half point for calling a tie in a race decided by 5%. That means 10 right, 6% wrong, 62.5% on the test.

When I am being really generous, that might be a D or D-, but looking at the massive difference between Rasmussen/Pulse and the next company better than they are, I'm calling it an F.

===============

Company: Wenzel [r]
Total races polled in final week: 3
Record: 1-2
Worst miss: Predicted Mandel by 5.0% in OH Senate, Brown won by 5.2%
Also pickey Romney by 3% in OH, Obama won by 0.9%
Average distance from correct percentages: 6.6% toward GOP
Median distance from correct percentages: 4.9% toward GOP

Most Republican partisans only did one or two polls in the last week, but Wentzel did three, so they get to do the Walk of Shame in this week's grading.

But the real shame belongs to a non-partisan poll that stunk the place up just a little bit worse.

===============

Company: Mason-Dixon
Total races polled in final week: 3
Record: 1-2
Worst miss: Predicted Rehberg by 4.0% in MT Senate, lost by 3.9%
Also called Romney by 6% in OH, where Obama won by 0.9%
Average distance from correct percentages: 7.3% toward GOP
Median distance from correct percentages: 7.0% toward GOP

Yet again, it's a non-partisan company that looks the most partisan as Mason-Dixon gets the In the Tank for the GOP medal. 
===============

Do you believe in duality, professor?

Hypothetical question asker, it's nice to have you back. And no I don't. Duality is for wimps.

Like Gaul, I am split in three parts.

When it comes to politics, I am definitely a left leaning partisan. When it comes to gathering poll data, I always play it down the middle, preferring the median to the average, which can drift badly when pulled by outliers. But when it comes to the polling industry, I have to hope the free market prevails. A ridiculous amount of money was spent on this election, including the hiring of more polling companies than I have seen in any election cycle, way more that 2004 or 2008. Some people know they didn't get their money's worth, most obviously when their pollster said they were ahead or tied when they actually were far behind.  The companies that did well should get gold stars by their names and the companies that did badly should have their bad grades posted like a weak report from the Board of Health.

The polling industry could use some new blood to bring them into the 21st Century, a company with a lot of smart people who are good with numbers and data and ideas of how to collect it in the modern age. Nate Silver has an interesting story that Google Consumer Surveys was one of the most accurate predictors of the presidential popular vote tally. Nate, may he be blessed by Odin, Vishnu and the little baby Jebus, works at The New York Times and has lots of access. I live in Oakland, far from Olympus but surprisingly close to Silicon Valley. I have a very well-placed source that tells me Google Consumer Surveys is about to take a big step in that direction and pick up the crown dropped by the people now dragging the late George Gallup's name through the mud.

Remember, you heard it here first.

No comments: