Support The Bulwark and subscribe today.
  Join Now

The Models Got COVID Right

If you paid attention to the models, then you understood what was happening in the real world. If you paid attention to Conservatism Inc., then you were trapped in fantasy land.
August 11, 2020
The Models Got COVID Right
TOPSHOT - US President Donald Trump holds up a graph showing COVID-19 tests during a meeting with the Governors of Colorado and North Dakota on May 13, 2020, in the Cabinet Room of the White House in Washington, DC. (Photo by Brendan Smialowski / AFP) (Photo by BRENDAN SMIALOWSKI/AFP via Getty Images)

1. Models

On Friday we talked about how the much-maligned COVID models have actually been pretty helpful in understanding the spread and death toll from the coronavirus.

Mark Panaggio, a big-brain math guy who works with data for a living, wrote in to put some meat on that bone.

Panaggio pulled the predictions for the big-12 COVID models and ran them against observed results to see how well they did.

He shared some charts from his observations and is generously letting me use them here. So let’s take a look.

Okay. So here’s what we’re looking at. On the x-axis we have the models and the y-axis we have the average percentage errors in their predictions for new deaths. The different color bars show you how well the predictions were four weeks out, three weeks out, two weeks out, and one week out.

What should immediately jump out at you is that 9 of the 12 models had error rates that were really low: mostly < 20 percent. Two of the models—from Johns Hopkins and Columbia—were not nearly as good. The University of Arizona was somewhere in the middle.

Panaggio helpfully put the data in a chart, too, in case you see numbers better that way:

The error rate for some of these models—the IHME, Los Alamos, UMass, U Texas, and Younyang Gu—are mostly in the low teens, which is wildly impressive.

The Ensemble model is like the RealClear Politics polling average: It’s the average of a whole bunch of models. And the Ensemble error rates were quite low, too.

So if you were just looking at that, you would have had a very clear picture of how the pandemic would progress.

I don’t mean to keep beating up on Richard Epstein, but he’s been unrepentantly stupid on this, so I will:

This idiot predicted that total U.S. deaths would be between 500 and 5,000. Which means that his error rate (so far) is between 3,200 percent and 32,000 percent.

There’s one other aspect of the models that’s worth looking at, which is the success of their confidence intervals. Models give a spread of predicted outcomes. This is called a confidence interval. In general, this spread is constructed by the model believing that 95 percent of the time, the actual result will fall within this range.

How did the models do with their confidence intervals?

Even in the midst of a pandemic with no modern precedent, five of them were bang-on almost 100 percent of the time. (UMass, take a bow.)

Now, the confidence interval tells you something different than the straight prediction. What it’s telling you is how confident the model is in itself. The UMass and Los Alamos models were giving themselves wider intervals than, say, U Texas and IHME, which tells you that even though their predictions were almost equivalent in terms of accuracy, the IHME and U Texas models were more aggressive in their certainty.

But again: Look at the Ensemble. If you had been looking at the averages of the various models and taking into account their confidence intervals, you would have had an almost perfect sense of what the next four weeks were going to look like at any point during the pandemic.

Keep all of this in mind the next time someone tries to tell you, “No one could have known” or “The models were all junk” or “Why should we listen to the supposed experts when they couldn’t even get their models right.”

None of those things are true. This isn’t magic. The people who were paying attention to the data had a clear sense of what was happening on the ground.

It was the people paying attention to Twitter or Conservatism Inc. or the president of the United States who had no idea what was coming.

2. No Recriminations

One of the recurring fantasies that people seem to have is that the Republicans and conservatives who have disgraced themselves over the last four years will pay some sort of price should Donald Trump lose this election.

The corollary to this fantasy is that Republicans and conservatives will find themselves hamstrung in the future by the various positions they have taken in support of Trump.

The most recent of these relates to Trump’s attempt to short-circuit the Constitution’s separation of powers by using an executive order to spend money, thus usurping the prerogatives of Congress.

Allow me to rain on this parade:

There will be no reckoning.

None of the Republicans and conservatives who are today cheering Trump’s executive orders will pause for a second before attacking the next Democratic president who misuses executive orders.

None of the Republicans and conservatives who have excused (or reveled) in Trump’s bad character will give the next Democratic president of bad character a pass.

The very next time Democrats propose an expansion of government, Republicans and conservatives will revert to opposition based on a love of limited government and they will not blush.

Et cetera ad infinitum.

Some of this is fine. Hypocrisy is baked into politics. Always has been. But the scale matters. A difference in degree can become a difference in kind.

And there is no reasonable way to look at the last four years and not understand that what we have witnessed has been so far outside the norm as to be closer to what one sees in a banana republic or failed state than to any previous American administration.

Which brings us to the point: It is precisely because this administration has been so corrupt that the Republican party and conservative movement will not be able to perform an autopsy on it.

In 2013, Republicans and conservatives were able to look at the 2012 election and ask themselves what went wrong. They were able to do this because no one had compromised themselves morally in supporting Mitt Romney. Everyone had behaved, by the standards of modern politics, more or less honorably.

Come 2021, nearly the entire Republican party and conservative movement will have been compromised. To ask what went wrong will be to admit to having behaved badly.

So they will not ask that question. And neither will Republican voters.

3. Shark vs. Orca

It’s Shark Week!

But never forget that sharks are an apex predator, but not the apex predator.

The undisputed champ of the seas is the orca, which will literally eat Great Whites for lunch:

In 2009, the team tagged 17 great whites, which spent months circling Southeast Farallon Island and picking off the local elephant seals. But this period of steady hunting ended on November 2 of that year, when two pods of killer whales (orcas) swam past the islands in the early afternoon. In the space of eight hours, all 17 great whites abruptly disappeared. They weren’t dead; their tags were eventually detected in distant waters. They had just fled from Farallon. And for at least a month, most of them didn’t return. . . .

“[Orcas] have a lot of social behaviors that sharks do not, which allows them to hunt effectively in groups, communicate among themselves, and teach their young.”

Combining both brains and brawn, orcas have been known to kill sharks in surprisingly complicated ways. Some will drive their prey to the surface and then karate chop them with overhead tail swipes. Others seem to have worked out that they can hold sharks upside-down to induce a paralytic state called tonic immobility. Orcas can kill the fastest species (makos) and the largest (whale sharks). And when they encounter great whites, a few recorded cases suggest that these encounters end very badly for the sharks.

In October 1997, a whale watch vessel near Southeast Farallon Island observed a young white shark swimming towards a pair of orcas that had earlier killed and partly eaten a sea lion. The whales killed the shark, and proceeded to eat its liver. More recently, after orcas passed by a South African beach, five great-white carcasses washed ashore. All were, suspiciously, missing their liver.

Read the whole thing. Especially if you want to learn how the shark livers are extracted.


Jonathan V. Last

Jonathan V. Last is editor of The Bulwark.