A Cure for Mad Poll Disease

Horse race polls are opinion journalism, not science.

OMG! The election is just 411 days away, so, for sure, we’d better be spending our time obsessively refreshing the latest horse race polls. 

More than a year before the 2024 election – before any of Trump’s trials or jury verdicts, before House Republicans do or don’t impeach Biden, before another sure-to-be-controversial Supreme Court term, and who knows what else – pretty much every major media outlet has weighed in with headline-grabbing polls showing Trump and Biden to be running even. That, in turn, has created enormous panic – both from Democratic partisans, and from everyone else who dreads a second (and forever) Trump Administration. 

If you share this panic, you might be suffering from Mad Poll Disease. Symptoms include anxiety, problems sleeping, loss of affect, and feelings of helplessness about the future of democracy, which are only exacerbated by frantic Twitter exchanges about polling methodology and sample bias. 

Folk cures aren’t working. Some people try to relieve the symptoms by arguing that the disappointing polls are mistaken, either due to potential bias or methodological problems. For example, the recent Wall Street Journal survey drew fire because of the involvement of Republican pollster Tony Fabrizio. But the press responds by positioning these critics as partisans who just don’t like the results. 

Jay Rosen and many others properly argue that our political journalism must focus on covering the stakes of this election (such as the real-world harms we can expect if MAGA wins) rather than the odds that either party will win. But the political press dismisses those critiques as partisan as well, protesting that they are being asked to withhold from the public “objective” reporting about the odds in favor of a higher mission (protecting democracy).  

Rosen’s critique is spot on, but used in this context it unintentionally concedes that any poll could tell us anything worthwhile about next year’s election – or even about public opinion that will matter then. 

Today, I want to show that, regardless of the methodology, pollster, or publication, horse race polling is worse than useless.¹ This is especially true more than a year ahead of the election – but, as I’ll explain, it’s also true in the weeks and months before. I will also discuss why the problem with the media’s² reporting on polling isn’t just that it’s a silly waste of time, but that it violates the journalistic principles to which flagship media institutions hold their other reporting. Finally, I will plead with everyone who pays attention to polling to realize that when a race is deemed to be within the margin of error, that really means it is within the margin of effort – the work we must always put in to get enough of those who dread a MAGA future to turn out to vote. The only FDA-approved cure for Mad Poll Disease is to pay attention to what matters: the ongoing MAGA threat. 

Horse Race Polling Is Punditry in Disguise

Imagine that in September 2016 you were watching a panel with five pundits discussing the presidential race in Florida. The first says they think that Clinton is up by 4 points, the second thinks she is ahead by 3, another two think she’s only ahead by 1 point, and the fifth thinks Trump is ahead by 1 point. Because I used the words “pundit” and “think,” you have no trouble understanding that those were opinions. But since polling has a scientific veneer to it, many people don’t understand that polling is also dependent on the opinions of the pollster! To do a poll, pollsters have to make their best guess about who will eventually vote, and then weight the survey results they get according to what percentage of the electorate they think those respondents will comprise. 

The pundit panel I described above actually happened – with pollsters – thanks to one of the most useful pieces of political data journalism of the ages in the New York Times. The Times asked four respected pollsters to independently evaluate the same set of survey data to estimate the margin of victory for Clinton or Trump in Florida, and the answers, including the Times’ own estimate, ranged from Clinton +4 to Trump +1. The 5 point range had absolutely nothing to do with a statistical margin of error, and everything to do with the opinions those pollsters had about who was going to vote and how to weight the exact same 867 interviews according to that prediction. (Forty-nine days after that piece was published, Trump won Florida by 1 and a half points.) 

But that lesson didn’t stick in the New York Times’ own coverage. In October 2022, the Times blew up the then-conventional wisdom that Democrats were competitive when it released its survey showing that Republicans had a nearly 4-point lead in the House, when Democrats had been ahead by nearly 2 points just a month before. Based on that survey, a front-page headline also declared that Democrats were losing ground with Independents and women. What the paper didn’t disclose was this: Independent voters hadn’t changed their minds; the New York Times changed its mind about which Independents would vote. The story compared a September survey of registered voters (RV) to an October survey of likely voters (LV) – apples to oranges. Worse yet, the October “likely voter” results simply reflected the judgment of the New York Times about which of their 792 interviewees would vote. An apples-to-apples comparison of registered voters from that same October survey would have shown not a 9-point net drop for Democrats among Independents since September, but the opposite – a 5-point net gain for Democrats.  

In every other context, reporters go out of their way to attribute opinions to sources. But political journalists report their survey results as mathematical facts – which makes it more difficult, especially for general readers, to remember the caveats that should accompany all polling results. (Indeed, in modern polling, even caveats about “margin of error” are borderline pseudoscience; see this footnote for more on why.³) Any time you read a sentence like “38 percent of white voters support Biden,” it actually means “our survey found that 38 percent of white voters support Biden.” 

To be clear, those who know me (or are longtime readers) know that I am hardly uncomfortable with data. I’m not a data nihilist. On the contrary, I am very proud of the data-related infrastructure for progressive politics I’ve been fortunate enough to play a role in building. But I see data as a way to develop testable hypotheses and to attempt to validate strategic propositions. Data should make us humble because it allows us to test ourselves.

Unfortunately, the media is driving us into an epistemological cul de sac where what’s seen in surveys is presumptively more “true” than other evidence (such as administrative records, other sources of data, real-world observations, and more). To be sure, polling can offer important insights. But, to be useful, the results must always be placed in dialogue with other imperfect sources of signals about the electorate. Our confidence in any particular proposition should depend on the number and credibility of independent sources of evidence corroborating the proposition. By failing to meet this standard, the media has become a reckless super-spreader of Mad Poll Disease. 

Horse Race Polling Can’t Tell Us Anything We Don’t Already Know 

To begin to cure Mad Poll Disease, make this your mantra: Horse race polling can’t tell us anything we don’t already know before Election Day about who will win the Electoral College. All we know, or can know, is this: 

1. A popular vote landslide is very unlikely. 

America is a rigidly divided nation in which the last six presidential elections have been decided by an average of 3 points, and, since 1996 (other than 2008), none have been decided by as much as 5 points.⁴ 

2. The Electoral College is too close to call. 

The Electoral College will almost certainly be decided by which candidate wins at least Georgia or Pennsylvania, plus two out of three of the other battleground states: Arizona, Michigan, and Wisconsin.⁵ In both 2016 and 2020, the margin of victory in most of these five states was less than 1 point. Yet a few months ago, when the New York Times patted itself on the back for having been “deemed the best pollster in the country” by FiveThirtyEight, this was because the final Times/Siena polls differed from the actual results in 2022 by only 1.9 percentage points on average. In other words, in 2016 and 2020, first Trump and then Biden won the states they needed to win the Electoral College by margins too small for the “best” polling to detect in the weeks before the midterms, when tens of millions of people had already voted.

If you are worried that you will miss something crucial by ignoring the polls, consider that the FiveThirtyEight forecast for the governors’ races on Election Day last year favored the loser in two of the five states that will decide the Electoral College and, likely, control of the Senate,⁶ and their forecast for the Senate races on Election Day last year favored the loser in three of the five battleground Senate races.⁷ I’m not saying that FiveThirtyEight did a poor job; indeed, FiveThirtyEight has been essential in modeling best practices and data transparency, and serves as an important check on unscrupulous claims by outlier pollsters. 

I’m saying that, when elections are very close, it’s simply not possible for any polls or forecasting to tell us anything more specific than “the race will be close.” They’re just not accurate enough; it’s like trying to look for bacteria using a magnifying glass. 

If polls taken weeks or months before the election can’t tell us anything useful about close races (which, again, are the only races that matter in our current system), why on earth would we pay attention to polls taken more than a year out? Why should anyone be concerned when, for instance, the New York Times reports in August of 2023 that Biden and Trump are tied⁸ in a race that won’t take place for another 15 months, and won’t even be decided by the national vote? 

3. Whether the anti-MAGA vote turns out again in the battleground states will determine the winner. 

Since 2016, Arizona, Georgia, Michigan, Pennsylvania, and Wisconsin have been a continuous ground zero for referendums on MAGA candidates. When the stakes of electing a MAGA candidate are clear to voters in those states, MAGA consistently loses. That's why the Red Wave never happened in those states. And that's why, even though Trump won all five of those states in 2016, Republicans had trifectas in four of the states, as well as 6 of the 10 US Senators, MAGA/Trump has lost 23 of the 27 presidential, Senate, and governors' races in those states since 2016, and only Georgia is still a Republican trifecta.⁹ 

What we’ve seen in 2023 seems to indicate more of the same: whenever voters have a chance to be anti-MAGA, they are taking it. Democrats have been significantly overperforming their partisan index in special election after special election, and that’s not even counting Wisconsin Supreme Court Justice Janet Protasiewicz’ 11-point statewide victory in what was technically a non-partisan race, but was well understood by voters in Wisconsin as a MAGA versus anti-MAGA contest. Imagine the Mad Poll Disease we would all be dealing with if the MAGA candidate in Wisconsin had won by 11, instead of lost by 11.

Polling failed to anticipate the anti-MAGA dam that held back the Red Wave, partly because polling cannot tell us in advance who will turn out to vote. This matters because, as I’ve explained before, Trump’s shocking election activated new or infrequent voters. Biden and Democrats have been winning handily among those who did not vote in 2016 but have voted since.¹⁰ Unlike a voter’s choice between Biden and Trump – which hasn’t changed much in the last 1,052 days¹¹ and is very unlikely to change in the next 411 days – those who do not vote in every election are notoriously poor at forecasting their own behavior even a month before the election.¹² 

Do you still feel like ignoring the polling entirely is just a bit too extreme? Let’s look at FiveThirtyEight’s 2022 Senate forecast. Notice that it began five months before the election, or nine months closer to the election than we are today. In those five months, the odds of Republicans winning a Senate majority started at 60 percent, fell to 30 percent, and then rebounded to 59 percent the day of the election. Then, voters thwarted those expectations by increasing Democrats’ Senate majority, which was obviously even less probable than them simply holding their 50 seats. 

Furthermore, the same five Senate races in battleground states mentioned above that were actually close on Election Day were also considered competitive a year earlier, when the Cook Report with Amy Walter issued its November 2021 set of race ratings. There is nothing that horse race polling could have told you that you didn’t already know. 

The Media Needs a Hippocratic Oath

Famously, doctors swear an oath to “do no harm.” Media outlets should swear a similar oath when they decide how to allocate their resources. Remember that unlike the decisions editors routinely make about what stories to cover, the only reason “Trump and Biden are tied” is news is because the media outlets conduct polls to make it news. “Trump and Biden are tied” isn’t a thing happening in the world. And, to get a bit meta here, it’s likely something that a poll would show a majority of Americans aren’t interested in now. 

The problem isn’t just that horse race polling routinely gets the numbers wrong (which should be reason enough for the media to stop covering it so obsessively). It’s that the media’s focus on polling steals oxygen from coverage of why an election matters. And it saps our agency as voters by creating a false sense of inevitability about the final outcome. 

With Great Power Comes Great Responsibility

While the mainstream media cannot tell us what to think about this or that issue, it has a powerful influence on what we think about. 

In 1974, before the rise of the polling-industrial complex, the midterms were about Watergate – and even though Republicans had mostly abandoned Nixon, they still paid a steep electoral price, losing 49 seats. If the 2022 midterms had been covered in the same way, the central question would have properly been, “Will voters hold Republicans accountable for their efforts to overturn the election?” 

But, after January 6th, most political reporters – made savvy by academic research about how midterms are always thermostatic elections – didn’t even entertain the notion that the midterms could be about Trump/MAGA. They regularly told us that, according to their polls, voters only cared about rising prices and crime. Democrats' losses in the November 2021 elections were reported as confirmation. After the J6 hearings began, the media tried to sustain that narrative by insisting that the hearings were not substantially increasing the number of Amerians who thought Trump was guilty. They didn't consider asking whether those who thought Trump was guilty all along were becoming more convinced of the importance of keeping his MAGA fellow travelers out of office.  

The combined impact of the J6 hearings and then Dobbs was not fully accepted by the media until after the Kansas initiative made it impossible to sustain the “abortion doesn’t matter" midterm narrative. (Remember – between the Dobbs leak and Kansas, the media was telling us that their polling showed Dobbs wouldn’t matter because, as with J6, few voters were changing their minds about Roe. But it turned out that a hell of a lot of voters were changing their minds about how important Roe was.)

Then, with astonishingly little self-awareness, the media – which was once again emphasizing prices and crime in its coverage – found in their polls that more voters were thinking about prices and crime than about democracy or abortion rights. Thus, they created the narrative that J6 and Dobbs – and democracy writ large – were not voter priorities.

In the months before the midterms, when I asked reporters why they weren’t doing more coverage about the criminal conspiracy to overturn our elections, they often said that there weren’t any more January 6th hearings to cover. If Woodward and Bernstein had taken that attitude, there would never have been any Watergate hearings to cover in the first place. Democrats’ surprising strength in the midterms showed that both the January 6th hearings and the Dobbs decision had a lasting impact beyond when they were most prominently in the headlines. 

And yet, while voters can always defy a media narrative, they also can’t help but be influenced by it. Members of the media might think they are merely observing and documenting an objective, pre-existing reality. But their relationship with the electorate is really more like quantum physics – the act of observing a phenomenon changes it. In Research Collaborative focus groups of undecided voters in the summer of 2022, when the news was still dominated by the January 6th hearings, Dobbs, the classified documents at Mar-a-Lago, and more, most participants said that they could not vote for an election denier. Voters understood that election denial wasn't just a lie, but part of a plot to take away their freedom to vote by installing the losers into power. However, closer to the election, some participants were saying that election denial was just another lie that politicians tell. For those respondents, without the coverage of J6, “election denier” no longer meant someone so committed to taking away people’s freedoms that they would try to overturn a legitimate election. 

As I’ve explained before, Democrats turned out in higher numbers in 15 states, where the MAGA threat seemed more salient – but in the other 35 states, we saw the expected Red Wave. It’s not unreasonable to imagine that if House races in “safe” Blue states like California, New Jersey and New York had been covered the same way as Senate races in swing states – with a constant emphasis that control of the chamber was at stake – the results in the House could have been quite different, too. 

Crucially, I am not arguing that journalists have a responsibility to help Democrats get elected; I am arguing that journalists' most important, First-Amendment-justifying responsibility is to give voters the information they need to be democratic citizens. 

After each election, the fraternity of leading media pollsters (and it is, sadly, pretty much a fraternity) devote their autopsies to how close they were to their horse race estimates. But there has been no public soul searching about how to better understand what motivates voters even after this latest epic miss. 

Avoid the “But Her Emails” Trap 

Reporting on the “odds” also serves an ideological purpose – not in terms of left vs. right, but in terms of the press’s ideological commitment to appearing not to have any ideology at all. They are deeply committed to embodying what Rosen calls the “view from nowhere.” 

In 2016, for instance, the media spent so much time covering Hillary’s emails because they didn’t want to seem biased against Trump, who they had to cover because he did and said so many shockingly bad things. Many outlets still have not changed their strategy. Trump now faces 91 criminal indictments. As Biden presides over an objectively strong economy and can boast significant legislative and foreign policy achievements, the media has to resort to bad poll numbers to generate negative stories to seem balanced in comparison. In other words, to create the appearance of balance, the media has to report on polls showing that many people don’t think Biden is doing a good job – not even stories about Biden actually doing a bad job. 

All of these polls by flagship media outlets bolster Trump’s and his defenders’ otherwise baseless claim that he is being indicted because Biden and the Democrats know they cannot beat him in a fair fight. 

How to Do No Harm (And Do Better)

An easy way for the media to “do no harm” would be to refrain from covering horse race polls (and fielding them in the first place), especially a year or more out from the election. In the meantime, here is a hardly-exhaustive list of ideas for political journalists to cover, instead of polls, that would be actively helpful to voters’ understanding of our elections and the issues driving them. 

• MAGA Congressional Republican Wrongdoing 

MAGA congressional Republicans remain Trump’s unindicted co-conspirators in the January 6th plot – and all of Trump’s other alleged crimes that were only made possible by Republicans refusing to vote for his impeachment. Yet their complicity has been all but memory-holed in the mainstream press. (I wrote more about this here in “America on Trial.”) 

• Campaign Finance 

In the wake of the Watergate scandal (which also encompassed corruption that forced Spiro Agnew to resign as Vice President, and uncovered payoffs to the Nixon campaign to influence support prices for milk), Congress enacted contribution limits and strong disclosure requirements. FEC disclosures were supposed to help the public hold their representatives accountable, and the Center for Responsive Politics (Open Secrets) long ago set up an easy-to-use, interactive website to track not just campaign contributions but lobbying disclosures as well. But for the most part, the media uses this tool for one purpose only – to report on who is raising more money, and to decide how “strong” the candidates are based on their fundraising prowess. 

Inexplicably, as long as candidates fill out the proper forms, the media has little interest in exposing the regular flow of special interest money to candidates even though it’s never been easier to do so. We desperately need a more comprehensive and analytical approach. We should be told who has been funding MAGA candidates, and which industries are lining up behind which parties or candidates. And incredibly, campaign finance disclosures are much, much more likely to show up in stories in which the amount raised is used as an indicator of how viable a candidate is than to tell us who is back their campaign. 

• Federalist Society Agenda

All six “conservative” Supreme Court justices who voted to overturn Roe are current or former members of the Federalist Society, an enterprise sponsored by right wing billionaires and corporations bent on capturing the legal system to roll back civil rights and dismantle the New Deal order. No modern Republican president will nominate a Supreme Court justice without the Federalist Society's approval, and Trump secured the Roe-destroying majority with his three appointments. Yet for the most part, media coverage of SCOTUS continues to focus on the details of the individual cases on the docket as if those cases will be considered in good faith. In the context of our current crisis, however, doing this is like narrating each segment of a bullet’s trajectory without naming the assassin or his target. (More here: “To The Supreme Court, the 20th Century was Wrongly Decided.”)

• MAGA Republican Commitment to Elections (or Lack Thereof)

Election denial is more than just another lie politicians tell. Any time a MAGA Republican defends Trump’s lies and criminal behavior around the election, they are promising that they would have no compunction with doing the same thing themselves. It’s a promise to break democracy if voters give them a chance.  

• Deconstruction of the Administrative State

Just about every Republican presidential candidate – Trump and Ramaswamy in particular – has promised to at least “rein in” the “deep state,” which is usually code for the administrative state that lets our government fulfill its basic functions. Despite the fact that Trump began the process of dismantling the administrative state four years ago, where are the deep analyses or exposes about how the Trump kakistocracy has diminished our ability to respond to future emergencies? Nor have we seen exposes on the sweetheart deals showering CARES dollars on Trump supporters. Those kinds of investigations are not looking in the rear view mirror; they are the best way for Americans to understand exactly what it means when MAGA promises to deconstruct the administrative state. 

Coda: Any Election Inside the Margin of Error Is Inside the Margin of Effort

Over the summer, my wife and I spent a week in Berlin. Germans have a profound respect for the importance of not forgetting. We visited the Topography of Terror museum, the Reichstag, the Sachsenhausen concentration camp, and immersed ourselves in contemporary accounts of the Holocaust, as well as the body of literature¹³, art, and cinema¹⁴ from successive generations who tried to reckon with the atrocity. 

This month, Steve Levitsky and Daniel Ziblatt’s book Tyranny of the Minority, an essential follow-up to their 2018 book How Democracies Die, was published. They break new ground in recounting the circumstances that determine whether fascist wannabes succeed in other times and places. (If you haven’t already, you should read their essay in the New York Times last week, “Democracy’s Assassins Always Have Accomplices.”) 

The most terrifying theme running through their book and our visit is that to succeed, fascists require reluctant and overconfident elites who can’t bring themselves to sound the alarm until it’s too late, and who overlook the overwhelming and mounting evidence that the soon-to-be tyrants are exactly who they say they are, and they will do exactly what they say they will do, if they are elected. 

These are the stakes we face in 2024. To pretend otherwise is to avoid telling the truth – and to be complicit in obscuring this truth from the public. Unfortunately, our political press is doing just that.

Any election within the margin of error is also within the margin of effort. Campaign professionals know this; it is their job to put in the effort required to turn out enough supporters to win. But we should also talk about the effort the media should put in – not to swing the election one way or another, but to report on what is true and what matters. 

1. Horse race polls, more formally known as trial heat polls, ask respondents who they intend to vote for.

2. Obviously “the media” is not a monolith, and many individual reporters and publications don’t have this problem. When I refer to “the media” or “the mainstream media” as a whole, it’s shorthand for the preponderance of coverage at prominent outlets that influences broader narratives. 

3. The original meaning of the term “margin of error” applied to polling when random digit dialing and high response rates made it plausible to base our confidence in the results on the principles of probability. Think of a bag with a million marbles in which half are red and half are blue. If you pick 10 marbles at random, you obviously won’t get five red and five blue each time. If you happen to pick four blue marbles and six red marbles, the rules of probability determine that you can have 95 percent confidence that the proportion of blue marbles in the bag is between 10 percent and 70 percent. The margin of error is +/- 30 percent. The MOE drops with bigger samples – to 10 percent if you pick 100 marbles, and to 3 percent if you pick 1,000 marbles. The key assumption that makes this true with marbles is that each marble in the bag has an equal chance of being selected. 

That’s the way surveys used to work with interviews being the marbles. But miniscule response rates shred the cardinal assumption of probability based polling, which is that every voter in America has an equal chance of being a survey respondent. Now, with the use of online panels for example, pollsters try to simulate representativeness, and they calculate their best estimate of what the MOE would be if it had been a random sample. So the MOE’s you see attached to surveys now are opinions, not tied to the actual science of probability. 

4. Even the one exception, the 2008 election, in which the result was driven by backlash against the mismanagement leading to the worst economic downturn since the Great Depression, Iraq, and Katrina (among other things), Obama’s margin was smaller than it had been in 18 of the previous 24 presidential elections.

5. Nevada will be close as well. However, there are very few ways in which Biden could win three of the five states and lose the Electoral College because he lost Nevada as well, as long as all the other states line up the way they have in the last two elections.

6. Wisconsin and Arizona.

7. Battleground Senate states were Nevada, Arizona, Wisconsin, Pennsylvania, and Georgia. Despite the election day forecast favoring them, Laxalt, Oz, and Walker lost their Senate races. 

8. I’m using the Times as an example for sharper exposition, but all of this would be true of any of the other surveys as well. 

9

• Arizona -  1-5 (w/l) - won 2018 governor, lost 2020 president, 2018, 2020 and 2022 Senate, 2022 governor

• Georgia - 2-4 - won 2018 and 2022 governor; lost 2020 president, 2020 (x2) and 2022 Senate

• Michigan - 0-5: lost 2018 and 2020 Senate, 2020 President, 2018 and 2020 governor

• Pennsylvania - 0-5: lost 2018 and 2022 Senate, 2020 President, 2018 and 2022 governor

• Wisconsin - 1-4: won 2022 Senate, lost 2018 Senate, 2020 President, 2018 and 2022 governor

10. See The Emerging Anti-MAGA Majority, as well as Pew’s 2020 validated voter study, Catalist’s What Happened in 2020, and VoteCast’s and CNN Exits. I know of no sources providing evidence to the contrary.

11. i.e. since the 2020 election. According to the New York Times surveys showing Biden and Trump even, only 2 percent of those who had voted for either Biden or Trump in 2020 said they intended to vote for the other in 2024. Major panel back series confirm that very few people have been switching sides. As I show in All Politics Are Local; All Political Data Is National, most poll reporting makes it seem otherwise because changes in the composition of demographic groups from one poll to another are reported as if individuals in that demographic group had changed their vote. 

12. For example, “Vote Self-Prediction Hardly Predicts Who Will Vote, and Is (Misleadingly) Unbiased”

13. In the Garden of the Beasts, Orders of the Day, Berlin Noir, Eichman in Jerusalem, The Nazi Seizure of Power, A Village in the Third Reich, Laughter in the Dark, and The Wages of Destruction

14. Generation war, Deutschland 83, The Lives of Others, Cabaret, Downfall, Germany Year Zero, Berlin Alexanderplatz, The Whole Ribbon, Labyrinth of Lies, In the Fade, Babylon Berlin, Heimat and Judgment at Nuremberg.