What is the Margin of Error in Polls and How is it Calculated?

[Dale]

While people did argue "no, this is just opinion", it's pretty clearly really an attempt to prognosticate. Otherwise, why use likely voters? Why not include everyone - resident aliens, illegal aliens, those under 18, and so on. They have opinions as well.

That is one problem with statistics in general. People like to misuse them. This is similar to how a small p value is often taken to indicate a large or important effect. It is much easier to just get excited over a number than it is to understand what that number actually means.

 

[Klystron]

Living and registered to vote in a "swing state", I receive a few 2024 election poll requests per business day including CNN. Many of the pollsters ask several demographic questions before the actual poll. Counting or ignoring your (election) response depends on your demographic answers.

My daughter who lives nearby responded to several identical polls. Our demographics coincide except for age, gender and (wait for it) race. She identifies as Asian American while I mark White as there is no Ashkenazi category or Decline to State. I was dropped from all the polls while my daughter was deluged with followup questions and advanced polls.

Published poll results (NYT, WaPo) did not mention these participation filters. Selecting poll participants makes sense depending on criteria but seems fraught with preconceptions and possible malfeasance.

 

[FactChecker]

While people did argue "no, this is just opinion", it's pretty clearly really an attempt to prognosticate.

Yes. There is no point otherwise.

The bigger issue is, of course, that US presidents are elected by the states, not the populace at large. Changing opinions in California or Wyoming makes no difference.

All states except Nebraska and Maine are "winner-take-all" for their Electoral College votes. California and Wyoming are probably safe for one side or the other and their individual votes are never split.
An interesting agreement: A lot of states have agreed that they will give all their Electoral College votes to the winner of the national popular vote as long as enough states in the agreement can dominate the Electoral College. That would bring the Presidential election back to being decided by the popular vote using a "Rube Goldberg" mechanism.

So what is being measured is correlated with electoral outcome but not the same.

Right. But it is the votes of the "swing States" that really matter. So it is the poll in individual states that are important.

A must-win district for the Democrats is NE-2, Omaha.

Nebraska is probably a safe state for Trump. It would require a enormously lopsided vote in Omaha to change that.

I am hoping these are two completely eparate polls and not that a third of the people surveyed in the national poll are from Omaha.

That is a safe bet.

 

[Vanadium 50]

Living and registered to vote in a "swing state"

I'm sorry.

as there is no Ashkenazi category

I met a professor in some liberal arts field or other who went on a tear about how the Ashkenazim are really Polish, there are no Mizrahi, and the Levant was originally settled by Muslims 2000 years before Mohammed. Maybe she wrote the poll.

I don't see any skulduggery in you not being called any more (and might consider it a plus). If they are already overampling in your demographic and undersampling in your daughter's, isn't this what you expect? I also don't expect that they would drop a completed poll - just de-weight it.

 

[FactChecker]

Rather than guessing, it would be interesting to see what methods the pollsters actually use. A lot will be proprietary. I have seen some detailed descriptions, but I can not find them with a casual search. Here are a couple of general descriptions of some issues by CNN and the polling company, SSRS, that they are using now.
If anyone can find more detailed descriptions by respected polling companies, I would be interested in seeing them.
CNN: https://www.cnn.com/2021/09/10/politics/cnn-polling-new-methodology/index.html
SSRS: https://ssrs.com/research-areas/political-election-polling/

 

[Hornbein]

Selecting poll participants makes sense depending on criteria but seems fraught with preconceptions and possible malfeasance.

Truly random sampling of voters is highly impractical. I don't have any better ideas than stratified sampling.

FactChecker: A lot of states have agreed that they will give all their Electoral College votes to the winner of the national popular vote as long as enough states in the agreement can dominate the Electoral College. That would bring the Presidential election back to being decided by the popular vote using a "Rube Goldberg" mechanism.

If such a system ever made a difference the voters in the states whose votes were switched would be justifiably outraged. Such laws would be immediately rescinded. Indeed if there were ever a chance that such a law would cause Hated Enemy to win against the will of the state's voters then I believe it would be rescinded before the election. In short, the whole thing is yet another symbolic gesture.

 

[Vanadium 50]

I was afraid to bring that up, lest this devolve into a discussion on the Electoral College. The important point is that it is what the election rules are, not what they might be. It is pretty clear it is doing what it is designed to do, which may or may not be what any given citizen wants.

I agree with @Hornbein that holding the election and then changing the rules to change the outcome would provoke outrage, and will never happen.

Givem the present estimated distribution of polls and likely state outcomes, we are talking about around a ½% edge for one candidate. Not insignificant, but also not huge - it has been higher in the past.

 

[Vanadium 50]

A lot will be proprietary. I

While I'd like to see this too, this is why I think we never will. They are selling their 'secret sauce". I'd be delighted if they did a data plus analysis dump of past polls: let's have a good look at 2016.

The problem with correcting for sampling bias comes about because of correlations. I can easily re-weight the variables so that all the 1-d distributions match expectations, but what about 2-way? That is, my sample might match the parent distribution in male vs. female and young vs. old. but not in young men vs. older women. If you have 10 yes/no variables and a 1000 subjects, you have one per bucket. You'll never get that right (which is why they do something else)

 

[FactChecker]

While I'd like to see this too, this is why I think we never will. They are selling their 'secret sauce". I'd be delighted if they did a data plus analysis dump of past polls: let's have a good look at 2016.

I have seen descriptions that are detailed enough for my satisfaction. I don't remember what polling organization did that.

The problem with correcting for sampling bias comes about because of correlations. I can easily re-weight the variables so that all the 1-d distributions match expectations, but what about 2-way? That is, my sample might match the parent distribution in male vs. female and young vs. old. but not in young men vs. older women. If you have 10 yes/no variables and a 1000 subjects, you have one per bucket. You'll never get that right (which is why they do something else)

It seems like correlations of the general population like sex versus age distribution, which can be well established independently of the poll, would be possible to adjust for.

 

[Vanadium 50]

It seems like correlations of the general population like sex versus age distribution, which can be well established independently of the poll, would be possible to adjust for.

Sure. Bur when I am looking at Pacific Islander females with some college but not degree, in a particular age and income band, I have chopped the data up so finely I may not be able to correct: if I expect 0.5 in my sample, what do I do if I have two? If I have zero?

A sample of 2000 and 11 yes/no questions puts on average one entry per bin. Sometimes you'll get 1, sometimes a few, and sometimes none.

This is why the pollsters don't do this.

 

[FactChecker]

Sure. Bur when I am looking at Pacific Islander females with some college but not degree, in a particular age and income band, I have chopped the data up so finely I may not be able to correct: if I expect 0.5 in my sample, what do I do if I have two? If I have zero?

Right. So nobody should take it to that extreme.

A sample of 2000 and 11 yes/no questions puts on average one entry per bin. Sometimes you'll get 1, sometimes a few, and sometimes none.

This is why the pollsters don't do this.

I think you mean that they don't use stratified sampling to a ridiculous extreme. They certainly do use stratified sampling.

 

[Vanadium 50]

Their sampling, AFAIK involves clustering. If you remember the term "soccer moms", that is a cluster. If a respondant ends up in that category, by virtue of some of her answers, they integrate over the others. In a sample of 2000. they probably want 20-40 clusters.

If someone is a public sector worker other than a first respnder or member of the military, they have certain voting preferences, and they are more similar to each other than across groups. So you can integrate over age. If someone is unemployers, their voting preferences are very different if they are 25 or 75, so you can't.

 

[FactChecker]

Fine. Let me then ask yet again, how do you beat the √N uncertainty, as hee CNN poll I mentioned claims to.

This discussion hasn't gotten very far. I suggest reading the literature on stratified sampling, where this is discussed in detail with the proper rigor. A standard classic text is Sampling Techniques by William Cochran, although there are many other references.
If the subject is polling to predict vote results, there is one thing to remember: the final vote depends on many of the most recent events like the weather, major world events, the recent optimistic or pessimistic mood of particular voting groups, etc.
No statistical method can accurately predict those effects and it should not try. It should only try to estimate the current state when the poll is taken.