Creating Unbiased Scientific Polls: Challenges & Solutions

[airborne18]

This is not a cynical question, though it will sound that way.

OK I have taken a bunch of stats courses, but I am not in the social or political sciences.

How do you formulate a political poll that is unbiased, in a scientific sense? Or is that impossible?

I know some build their sample based on historical demographic voting patterns. But that seems flawed. And some use census data, which I would guess is flawed for different reasons.

But I am just trying to understand it more than anything. And I trust that here I will at least get pointed in the right direction for the answer.

 

[talk2glenn]

This is not a cynical question, though it will sound that way.

OK I have taken a bunch of stats courses, but I am not in the social or political sciences.

How do you formulate a political poll that is unbiased, in a scientific sense? Or is that impossible?

I know some build their sample based on historical demographic voting patterns. But that seems flawed. And some use census data, which I would guess is flawed for different reasons.

But I am just trying to understand it more than anything. And I trust that here I will at least get pointed in the right direction for the answer.

Unbiased, in what sense?

If you mean, how are samples constructed so that they are representative of either eligible or likely voters (depending on the poll), organizations use a combination of exit polls, past election results, and census data. To correct for errors in random response rates between the sampled population and the general population, pollsters use statistical weighting, and will ask self identification questions as part of the poll.

The typical sample weights are political identification, gender and race. Geographic representation can be built into the sample by distributing your sample proportionally over the 50 states, so it usually doesn't need to be weighted (but if results aren't within expectations - more common these days with cell phones - you can still weight to correct). Once you randomly sample a population of 700+ people, you compare their self identification responses to expected values. Undersampled respondents have their answers weighted up, and the oversampled have their answers weighted down.

The result is a statistical range that is usually 95% (but sometimes 99%) likely to contain the actual population value - this is the nature of normal distributions. The median of this range is reported as the poll result (the margin of error tells you how wide it is).

Obviously, proper weighting is critical - it is difficult to keep track of party identification between elections, and sometimes the changes can be dramatic and unexpected. We saw this most recently with the 2000 election. Methodology has changed since then to keep up, as we can see with the current election cycle polling. The organizations expend considerable effort on keeping up with voter trends between cycles.

 

[airborne18]

Yes you hit it on the head. I think where I get stuck on this is that you cannot construct a sample in a vaccum. So you have to understand the characteristics of the population in relation to your survery subject matter to create a representative sample.

For example, constructing a sample is much more difficult for a statewide primary in a small state like Delaware than it is for a national election, or a larger state. Especially if the population has surged between census and election cycles.

I have read about the weighting and you have confirmed that this is the point that various polling organizations debate.

 

[loseyourname]

I think what you're getting at is that poll results get less and less reliable as predictors of actual voting the more the actual population of voters diverges from the expected population of voters. It's getting more and more difficult to predict who will actually vote.