[screen 1]
“New poll shows candidate leading by 3 points!” Headlines treat this as meaningful. But with a ±3% margin of error, it’s statistically a tie.
Polls dominate political and social discourse. Understanding how they work - and their limitations - helps you evaluate polling claims critically.
[screen 2]
How Polling Works
Polls attempt to measure population opinions by surveying a sample:
-
Define population of interest (registered voters, adults, etc.)
-
Select representative sample
-
Contact selected individuals
-
Ask questions
-
Weight responses to match population demographics
-
Calculate results and margin of error
Each step introduces potential problems that affect accuracy.
[screen 3]
Sampling Methods
The quality of sampling determines poll reliability:
Random probability sampling: Everyone in population has known chance of selection (Gold standard - expensive and difficult)
Random digit dialing: Calling random phone numbers (Declining response rates as people ignore unknown calls)
Online panels: Recruiting participants for surveys (Convenience, but self-selection bias)
Opt-in internet polls: Anyone can respond (Worthless for serious analysis - highly biased)
Method matters more than sample size.
[screen 4]
Sample Size and Margin of Error
Larger samples reduce uncertainty:
- 100 people: ±10% margin of error
- 400 people: ±5% margin of error
- 1,000 people: ±3% margin of error
- 2,000 people: ±2% margin of error
Diminishing returns - quadrupling sample size only halves margin of error. Most quality polls use 800-1,500 respondents, balancing cost and precision.
[screen 5]
Understanding Margins of Error
Margin of error (MOE) indicates statistical uncertainty:
“52% support, ±3% MOE” means true value likely between 49% and 55%.
If opponent has 48% (±3%), ranges overlap: 45-51% vs 49-55%. The race is statistically tied despite headline claiming “4-point lead.”
Most media coverage ignores MOE, treating differences within statistical noise as meaningful.
[screen 6]
Response Rates Matter
Fewer people complete surveys than in the past:
- 1990s: 30-40% response rates
- Today: Often below 10%
Low response rates create non-response bias - people who answer might differ from those who don’t. Quality pollsters use weighting to compensate, but it’s not perfect.
This is why polls sometimes miss surprising results - certain groups systematically don’t respond.
[screen 7]
Weighting and Adjustments
Raw poll results rarely match population demographics. Pollsters weight responses so results match known population characteristics:
If poll reaches too many college graduates (who answer more often), their responses are weighted down while non-graduates are weighted up to match actual education distribution.
Weighting assumptions affect results. Different weighting choices explain why polls of the same race give different numbers.
[screen 8]
Question Wording Effects
Small wording changes dramatically affect responses:
“Do you support allowing women to choose abortion?” gets higher support than “Do you support abortion?”
“Estate tax” vs “death tax” - same policy, different support levels
“Assistance to the poor” vs “welfare” - same programs, different approval
“Forbid” vs “not allow” - logically similar, psychologically different
Quality pollsters test question wording. Advocacy groups design questions to produce desired results.
[screen 9]
Question Order Effects
Earlier questions influence later answers:
Asking about terrorism before immigration questions increases anti-immigration responses.
Asking about personal economic experience before national economic question affects assessment of national economy.
Quality polls randomize question order or carefully consider effects. Advocacy polls exploit order effects.
[screen 10]
Timing and Context
When polls are conducted affects results:
- After major news events, opinions spike then revert
- Polls right after conventions show “bounces” that fade
- Weekend vs. weekday polling reaches different demographics
- Campaign events create temporary shifts
A single poll is a snapshot of a moment. Trends across multiple polls over time are more informative than individual polls.
[screen 11]
Poll Aggregation
Averaging multiple polls reduces error from individual polls:
- Individual polls have random variations
- Aggregation smooths these variations
- Shows trends more clearly
- Less likely to be misled by outliers
Sites like FiveThirtyEight and RealClearPolitics aggregate polls. These averages are more reliable than single polls, though still imperfect.
[screen 12]
When Polls Fail
Polls sometimes miss badly:
2016 US election: Underestimated Trump support
Brexit referendum: Late-deciding voters broke unexpectedly
2015 UK election: “Shy Tory” effect - people reluctant to admit Conservative support
Reasons include:
- Differential non-response (certain groups don’t participate)
- Late shifts in opinion
- Social desirability bias (lying to pollsters)
- Turnout modeling errors
- Sampling frame problems
[screen 13]
Evaluating Poll Quality
Before trusting a poll, check:
-
Who conducted it? Established polling firms vs advocacy groups
-
Sample size and method? Random sampling vs opt-in online
-
Response rate disclosed? Quality pollsters report this
-
Margin of error? Typically ±3-4% for good polls
-
Full question wording? Should be available
-
Timing? Recent vs old
-
Sponsor? Who paid for it and why
-
Methodology transparency? Quality pollsters publish detailed methods
[screen 14]
Red Flags in Polling
Be skeptical of:
- Opt-in internet polls presented as legitimate
- Polls with extremely large or small MOE
- No methodology disclosed
- Advocacy group polling on their own issues
- Reporting single polls as definitive
- “Instant” polls after events (no time for quality sampling)
- Polls that seem designed to generate headlines
- Extremely surprising results with no replication
[screen 15]
Using Polls Wisely
Polls can be useful when understood correctly:
- Look at poll aggregates, not individual polls
- Pay attention to trends, not single snapshots
- Account for margin of error
- Check methodology before trusting results
- Recognize that close races are genuinely uncertain
- Understand polls measure opinion now, not predict future
- Be aware of systematic biases in polling
Polls are tools for understanding opinion, not crystal balls. Treat them as uncertain indicators, not definitive answers.