The Polling Crisis Is a Catastrophe for American Democracy
If public-opinion data are unreliable, we’re all flying blind.
Even with the results of the presidential contest still out, there’s a clear loser in this election: polling.
Surveys badly missed the results, predicting an easy win for former Vice President Joe Biden, a Democratic pickup in the Senate, and gains for the party in the House. Instead, the presidential election is still too close to call, Republicans seem poised to hold the Senate, and the Democratic edge in the House is likely to shrink.
This is a disaster for the polling industry and for media outlets and analysts that package and interpret the polls for public consumption, such as FiveThirtyEight, The New York Times’ Upshot, and The Economist’s election unit. They now face serious existential questions. But the greatest problem posed by the polling crisis is not in the presidential election, where the snapshots provided by polling are ultimately measured against an actual tally of votes: As the political cliché goes, the only poll that matters is on Election Day. The real catastrophe is that the failure of the polls leaves Americans with no reliable way to understand what we as a people think outside of elections—which in turn threatens our ability to make choices, or to cohere as a nation.
The current poll-industrial complex traces its roots to the 1930s, when George Gallup created it. But in the past few decades, it has become truly dominant within the realms of politics and policy. President Bill Clinton was notoriously fond of using poll data to guide political decisions, but the real efflorescence came with Nate Silver’s creation of FiveThirtyEight. Silver founded the site in 2007, out of frustration with the punditry he heard on TV, which he believed either misread public-opinion polls or ignored them entirely.
Silver predicted Barack Obama’s victory in the 2008 Democratic primary, then in the general election, and again in 2012, and founded a movement. His critique, which fit with the coolly technocratic impulses of the Obama years, also carried the day. Silver and his cohort felt that reporters were botching horse-race coverage because they relied too heavily on anecdotes, many of which were not representative, either because reporters were too cloistered or because they chose incorrectly.
Silver didn’t want reporters to stop going into the field; he just wanted them to contextualize what they heard. “The impulse maybe isn’t bad,” he told me last year. “But, you know, polls are also a way of talking to voters.” He hoped, however, that by combining the polls into averages and creating models of the electorate, he might free up reporters to write other stories. Instead, horse-race journalism simply became more sophisticated and poll-obsessed, and in the process, it became even more central to political coverage—now enhanced by the sheen of quantitative rigor.
The 2016 election was a shock to this new regime. Polls and poll analysts expected a Hillary Clinton victory, and instead Donald Trump won. Amid a popular backlash, the poll clique—pollsters and analysts alike—defended their results and blamed the public for not understanding polling or probabilities. They pointed out that the popular vote had closely tracked national polling on Clinton versus Trump, and while state polls were very consequentially wrong, many were not wrong by all that much.
Some observers, including me, more or less bought that defense. Pollsters closely examined their methods and promised to try to fix problems in 2020, though they noted that polls are never perfect. (Again, the only poll that matters … ) FiveThirtyEight, which smugly boasted that it had given Trump a better chance (roughly 30 percent) than most analysts did in 2016, gave him just a 10 percent chance in 2020. The Economist was even more bullish on Biden. Then came the vote. In every swing state but Arizona, Trump outperformed the FiveThirtyEight polling average. This is not to pick on FiveThirtyEight, which went to unusual lengths to ensure that its averages were accurate, but simply to indicate how far off the polls as a whole were.
The coming days and weeks will see careful analysis and less careful recrimination, but no one seems to know yet exactly what went wrong. But the answer almost doesn’t matter, unless you’re a professional pollster, because after two huge presidential flops, pollsters have lost the confidence of the press and public.
Expect two lines of defense. First, many pollsters insist that their polls are snapshots, not predictors (and bridle at people such as Silver, Nate Cohn of the Times, and G. Elliott Morris of The Economist using them to create forecasts). If their snapshots are so far off, though, where were they aiming the lens? Why bother? Second, the analysts will protest that they’re only as good as the polls, but who cares? Whatever the instructions on the bottle, the public uses opinion polls to try to understand what happens. If the polls and their analysts don’t offer the service that customers are seeking, they’re doomed.
Pollsters and analysts are unlikely to get much sympathy, especially today. But the train wreck of their industry has consequences that run deeper than its impact on their own professional lives, or even having set incorrect expectations for the presidential race. Much of American democracy depends on being able to understand what our fellow citizens think. That has become a more challenging task as Americans sort themselves into ideological bubbles—geographically, romantically, professionally, and in the media they consume. Parties are now mostly ideologically homogeneous. We no longer spend much time around people who disagree with us. Public-opinion polling was one of the last ways we had to understand what other Americans actually believe.
If polling doesn’t work, then we are flying blind. That is an especially acute problem at the moment, because the coronavirus pandemic has made the old way the media got at this—shoe-leather reporting, despite its many shortcomings—much harder to pull off. The Trumpist alternative of simply trusting gut feelings isn’t any better. Gut feelings have failed plenty of candidates before; they may still prove to have failed the president this time.
When an election can give a definitive answer to a question, by telling us which candidate or policy Americans prefer, the problems with polling matter less, though they make vote-counting more stressful. But anything that happens outside of the quadrennial and midterm elections is now murky. Earlier this summer, Trump took a hard line against protests for racial justice following the police killing of George Floyd. As I noted at the time, there was a quick and significant drop in the president’s polling, especially among white voters. Trump never recovered that support in polls, but if the polls were off, who knows whether that drop was real, or whether white attitudes about racial justice have really changed?
The inability to rely on polling also undermines a range of generic political choices. Many voters decide which candidate they favor in primary elections based in part on rational calculations about who is most likely to win a general election. Policy makers choose what ideas they support or oppose based on what they think the public supports. If poll data are not reliable, though, the candidates that triumph in primaries are more likely to be the loudest and best-funded, and the policies politicians back might run contrary to the wishes of the voters who elected them.
Without reliable sources of information about public opinion, the press, and by extension, the public, should perhaps employ a measure of humility about what we can and can’t know in politics. As wise as this may be—and even if people manage to act on it—that sort of epistemic humility risks falling prey to the same asymmetrical warfare that has characterized much of the Trump era. At the moment, the leader of the Republican Party is an authoritarian populist who claims to represent the “true” will of the people, despite losing the popular vote twice. The president is unlikely to exercise any such humility in claiming, without evidence, that public opinion is with him. He might be wrong, but without reliable polls, who’s to say otherwise?