Support The Bulwark and subscribe today.
  Join Now

Beware of the Polls

The polls in Iowa missed not only the finish, but the trend lines. The polls in New Hampshire might, too. And after that, polling the right sample only gets harder.
February 9, 2020
Beware of the Polls
(Hannah Yoest / Shutterstock)

As the Democratic race shifts into high gear and the fallout from the impeachment trial settles in, I have a respectful and heartfelt cautionary admonition about polling data: Caveat emptor.

Let’s begin with Iowa. Not one poll before the Iowa caucuses accurately captured what was happening. Most of the closing polling data showed Bernie Sanders starting to creep towards 30 percent, with Joe Biden closely behind in the low 20s and Pete Buttigieg somewhere between 15 percent and just under 20 percent. Elizabeth Warren looked to be around 15 percent and Amy Klobuchar somewhere between 8 percent and 11 percent.

When the final polling averages were released on February 3, this is how the RealClear poll of polls had the race:

Sanders 24.2 percent

Biden 20.2 percent

Buttigieg 16.4 percent

Warren 15.6 percent

Klobuchar 8.6 percent.

FiverThirtyEight had their own poll of polls, which used weights based upon quality of polling data. They had:

Sanders 22.1 percent

Biden 20.7 percent

Buttigieg 15.5 percent

Warren 14.7 percent

Klobuchar 10.1 percent.

These averages did not reflect what—we think—happened.

The Iowa caucuses seem to have ended in a dead heat between Sanders and Buttigieg (Sanders ahead in the initial vote, with Buttigieg picking up steam to be a smidge ahead in the formula for allocating delegates), with Warren finishing a solid third (with 18 percent) and Biden falling to fourth, just ahead of Klobuchar who finished at 12 percent.

The Sanders surge that was the media buzz pre-Iowa, did not fully materialize. Or rather, what materialized was what the polling showed was there—there was no forward momentum happening.

The polls missed the real surge, which was for Buttigieg—capturing neither his final number nor even the direction in which he was heading. And while the polls were close on Warren, they missed the Biden collapse entirely.

Now the pollsters if queried on this collective failure, would no doubt fall back on the margin of the margin of error in their data and point to the fact that most of them got Sanders share accurately and some got one or more of the other candidates within the margin of error.

But in reality, the polls missed the actual dynamics of what was really happening. Not one poll showed a snapshot of end result: Buttigieg and Sanders in a dead heat, Warren a strong third (which limited Sanders’ ability to earn a clear victory) with the underlying reality being that the poor showing of Biden and the incomplete rise of Klobuchar enabled Buttigieg to gain sharply on Sanders in the second vote.

And yet here we are, less than a week later, looking at the polls from New Hampshire as though they must be better.

Spoiler: Maybe.

Historically, New Hampshire is difficult to poll.

Yes, some candidates get a bounce from Iowa. But also, the voters in New Hampshire tend to reject the Iowa winner more often than they ratify Iowa’s choice.

Second, the real impact of Iowa on New Hampshire voters has historically been that it helps them focus upon discarding candidates from serious contention. Meaning that those who underperform in Iowa are often ignored by Granite State voters.

Third, Granite State voters have earned their reputation for independence bordering upon the quirky. Sometimes New Hampshire voters—regardless of party—ratify the Iowa choice (Carter, Kerry, Gore) and sometimes they reject that choice (Reagan over Bush in 1980; McCain over Bush in 2000; Sanders over Clinton in 2016).

And sometimes they reject Iowa’s choice even when the Iowa winner seems to skyrocketing in the late New Hampshire polls. After winning Iowa in 2008, Barack Obama rocketed to the top of the New Hampshire polls and led Hillary Clinton by a baker’s dozen in the final polling. Clinton won New Hampshire.

Fourth, compounding the accuracy challenge of primary polling in New Hampshire, is that independents can vote in the Democratic primary.

And all of this leaves aside the flinty sense of humor Granite State voters like to exhibit in creating decisive momentum at the very last minute when it can’t be tracked: plucking Gary Hart out of obscurity to victory over Walter Mondale in 1984; or in 1964 when Henry Cabot Lodge won the Republican primary as a write-in while serving as Johnson’s ambassador to Vietnam, never having set foot in the state during the campaign.

Having said that, the post-Iowa polling data from New Hampshire shows Sanders ahead with Buttigieg getting a spike and closing the gap.

If that trend holds, Buttigieg could pull off an upset. But the end result may well be driven by whether Warren can finish strong again (which would hurt Sanders) or whether Biden rebounds (which would hurt Buttigieg). Not to mention the combination of Tulsi Gabbard, Andrew Yang and Tom Steyer, whose support could impact the outcome—together they are polling at just under 20 percent—as their shares either wax or wane.

But again “if that trend holds” is a very, very, very big qualifier.

And the truth is, the polling only gets harder in the next phase of the campaign.

That’s because the upcoming states have large percentages of African-American and Hispanic voters and there is no bigger accuracy gap in public polling than the chronic problem with accurately measuring both the size and the break of minority voting cohorts. This is a theorem, not a postulate, which is proven by comparing pre-election polling to Democratic primaries and even general elections.

New Yorkers understand this problem well, so let me provide some context.

In 1982, polls presumed a low minority turnout allowing Ed Koch to beat Mario Cuomo in that year’s gubernatorial primary. Those polls showed Koch narrowly losing black voters and running even among Hispanics.

Instead, minority turnout exploded and Cuomo swept black voters and carried Hispanics comfortably, which proved to be the building blocks in his shocking upset victory.

In the 1989 New York City mayoral primary, polls showed a solid, but not a large, minority turnout in the Dinkins-Koch race which had two additional candidates (Dick Ravitch and Jay Golden).

This polling suggested that Koch would hold Dinkins under 40 percent, forcing a run-off than he might well win. Instead the minority turnout blew past expectations, making 1989 New York City’s first million-vote municipal primary. Dinkins not only topped 40 percent—he won an outright majority—because he won both black and Hispanic voters by landslide margins.

Finally, in 2009 the polls projected a Mike Bloomberg landslide, because the data suggested that he would carry Asian voters, split Hispanics evenly, and get a solid share of black voters. Instead, Bill Thompson won blacks by a huge margin and comfortably carried both Hispanics and Asian voters. (It was the first time Bloomberg lost the Asian vote in his three mayoral runs.) So instead of a landslide, Bloomberg won his final term by a narrow 5 percent margin.

This phenomenon is not peculiar to New York. In 1988 polls did not capture the surge in minority turnout or the margin by which Jesse Jackson would carry minority voters, especially black voters. Nor in 2016, did the polls grasp the breadth and depth of Hillary Clinton’s support from minority voters, especially black women.

The lesson here being that polls are likely to underestimate both the share of the primary vote cast by minority voters and the margin by which they support the candidate(s) of their choice.

Which brings us to the Gallup poll showing Donald Trump’s job approval leaping to 49 percent with only 50 percent disapproving in the wake of his acquittal at the Senate impeachment trial.

This poll (taken from January 16 to 29) struck me as odd as it represented an increase of 5 percent in job approval and a 3 percent decrease in job disapproval form Gallup’s January 2 to 15 survey. That’s a huge jump from two surveys separated only by a month. And the 49 percent job approval was the high water mark for Trump’s presidency.

The oddity of the result was heightened by the fact that the Reuters/Ipsos poll puts Trump’s job approval/disapproval at 42 percent to 55 percent, which is in line with what CBS News find (43 percent approve, 51 percent disapprove) and the IDB/TIPP poll (44 percent to 51 percent). All of these polls were contemporaneous (in the field between January 23 and February 4).

I do not dispute that when the economic data is looking positive Trump’s numbers tend to move up. That’s because independent voters are sensitive to (1) economic perceptions and (2) general chaos in government and the world at large.

But heretofore Trump has usually operated between bands of 40 percent to 45 percent approval and 51 percent to 57 percent disapproval. The Gallup poll became widely quoted, precisely because it was so different. Why was that?

It was because this Gallup sample had more Republicans than Democrats.

No other major poll has put more Republicans than Democrats into its sample. Last October, Pew Research put out a paper showing that their extensive polling found that among the general public there were 7 percent more Democrats than Republicans; among registered voters there were 4 percent more Democrats. Pew pointed out that this lead traditionally shrunk even further among likely voters in off-year elections, to the point that the parties often reach parity.

But the larger turnout in the 2018 election disrupted that trend. Democratic congressional candidates got 9 million more votes than Republicans in November of 2018, putting them close to their national 12 million voter registration advantage.

Here is what I believe happened in the Gallup poll: Gallup’s sample just came back with more Republicans this time. Maybe they reweighted, maybe they didn’t. Maybe this one survey had a rather extreme amount of bouncing in sampling shares along the critical fault lines of partisanship, gender, race, religion, age, or education levels.

When pollsters talk about “polarization” they mean something different than what laymen do: They mean the tendency of groups to break hard in opposite directions. This makes polling harder, because when the skew lines (what pollsters call elasticity) in terms of how voters break in support and opposition to candidates is high, then sampling becomes even more determinative of the results.

This is why, around Halloween of 2012, the Gallup poll projected that Mitt Romney was moving ahead among likely voters: The electorate was highly polarized and their sample undercounted minority voters and single women, who were strongly for Obama. Or why Midwestern polling in 2016 was off—these polls undercounted white men without a college education and over sampled those with a college education, thus misreading Trump’s chances of tearing down Clinton’s Blue Wall.

It’s too bad that Gallup only polls once a month rather than having a daily tracker, as they used to. The daily tracker smoothed down the rough edges of diverging sample shares. But it was expensive and the explosion of pollsters means more total data for everyone, while diminishing the return on investment for an organization like Gallup to do daily polling.

If the next Gallup poll comes back with a sample where the Democrats outnumber Republicans by 6 percent and that puts Trumps’ job approval down to 42 percent (or lower), it could still be a function of sampling differences, rather than a real shift in Trump’s standing with the public.

My goal here is not to pick on any one pollster, but to point out that while polling data is useful and an analytical tool, it is not the kind of stuff that Moses brought down from the Sinai.

Bruce Gyory

Bruce Gyory is a Democratic political strategist and an adjunct professor of political science at SUNY-Albany.