Friday, December 21, 2007

Poll watchers: When not to believe a poll

Most people have a rather simple formula in determining whether they believe the results of a Poll. They judge it relative to their personal opinion of "where the race is". For instance everyone believes Joe Biden is in the second tier group of Democrats running for President in 2008; if a poll showed that Biden was 3 points behind Hillary in New Hampshire all of our first instincts would be to discount it as a fluke or otherwise in error. Yet, if that same pollster had results that indicated Biden was 3 points behind Hillary in Delaware, we might tend to believe it and assume Clinton was so strong she is defeating the other candiates in their own states. Both could be wrong or right. The best thing to do is wait for confirmation from a series of polls. But another way to approach would be to look at the results of other questions asked in that same poll and compare them to other polls both recent and historical. If a certain demographic group is either over-represented relative to the voting population, or if the results of a certain sub-group don't match other polls, you should be skeptical of the larger question. Here is an example:

cbsnews did a poll in South Carolina among Democrats concerning the Presidential race. The topline results were:

Obama 35
Clinton 34
Edwards 13

However the internals tell a peculiar story:

among women: men:
Obama 39 30
Clinton 31 38
Edwards 10 17

There was a so-called "reverse gender gap" where the women supported the man, and the men supported the woman.

Since I've seen hundreds of polls both in the states and nationwide that consistently detail a normal gender gap, I immediately questioned this poll result.

There are a number of ways there could be errors:

methodoly in weighting different samples: by age, by region of state, by race, etc.
random sample error
statistical error

There is the possibility the results were totally accurate even if they were counterintuitive. Perhaps the large African American population caused race to supplant gender, so that blacks both male and female went more towards Obama, and whites towards Hillary.

I looked at the internals of another poll taken in the same general time:

Survey USA had poll dates of 12/17-18 there the topline number was

Clinton 41
Obama 39
Edwards 17


among women men
Clinton 50 28
Obama 33 48
Edwards 15 21

So even though top results were pretty similar with a 2 point lead for Clinton versus an Obama 1 point lead, the internals were completely inconsistent. Since the SurveyUSA's gender question/answer is consistent with virtually all other data, it is wise to believe that poll over CBS. In this case there is only a 3 point difference, but as the vote gets closer their results may diverge.

No comments: