Vaccine Efficacy by the Numbers

Every week I do an Instagram Q&A, which is usually a good window into what people are worried about, especially around COVID. Last week, one of the first questions was “Since vaccines aren’t effective after six months, will we need boosters forever?” I started my answer with THIS IS WRONG. But it, along with many other questions I’ve been getting, makes it clear that there is a lot of confusion around vaccine efficacy.

So today I wanted to write a bit about how we measure and understand vaccine efficacy, why it’s complicated, and what we know so far. There are some interesting statistics in here too, and you know I like that.

The TL;DR is that vaccines are extremely protective (like, 90%+) against severe illness and death. But let’s get into it.

What Are Vaccines For?

I want to start by talking about the goal of vaccines. What can we expect from them? The most important goal of vaccines is to protect against serious illness and death, and that’s what they are best at. If you want to understand this in depth, read this amazing piece by Céline Gounder right now (I’ll wait!).

To briefly summarize, vaccines help protect you in two ways. First, you develop antibodies to the virus, which are immediately available to protect you if you’re exposed. Second, you produce memory cells, which remember how to produce antibodies if you need them later. The initial antibodies stick around for a while but not forever, but the memory cells remain.

If you’re exposed to the virus in the period shortly after vaccination, when you have active antibodies, your body is ready to go. You fight off the virus fast, and you probably won’t get sick at all. If you’re exposed later, though, it takes a few days for your memory cells to kick in. So you may get slightly sick, especially with a virus like SARS-CoV-2 that has a short incubation period. But, importantly, your immune system is ready. This means you get better faster, get less sick in the first place, and infect fewer people.

Side note: If you are vaccinated when pregnant, you pass the antibodies to the baby but not the memory cells, which is why the baby’s immunity wanes while yours does not.

What we should expect following vaccination is an initial period in which the vaccines are protective against serious illness, death, and milder illness. And then a period in which the vaccine shows waning immunity against mild illness but maintains protection against more serious illness. This continued protection against serious illness means the vaccines are working. As we will see below, that’s what is going on.

A lot of the problems and misconceptions seem to stem from having generated unrealistic expectations for the vaccines early on. I will admit to having made this mistake along with many other people; we should have been more circumspect about the longer term. And we should have been clearer about what we could expect from vaccines.

With this background, then, I think we can ask two follow-up questions. First: How have the vaccines stood up against hospitalization and death? Is their protection against those waning? And second, how much protection do they still provide against any disease?

Vaccine Efficacy: Hospitalization and Death

If you run a randomized trial, it’s easy to evaluate the protection afforded by vaccines against hospitalization and death. Half the sample is vaccinated and the other half is not, and you evaluate the percent reduction in risk. If there is a 1% chance of hospitalization in the unvaccinated group and a 0.05% chance in the vaccinated group, that’s 95% protection.

Evaluating vaccine efficacy in the real world is more complicated.

The parallel analysis to the randomized trial would involve, simply, comparing the hospitalization rate for vaccinated and unvaccinated individuals and doing a similar calculation. However, that can be misleading in a way the randomized trial is not, because in the real world, vaccination isn’t random. There are at least two issues. The first is that vaccinated people may behave differently than unvaccinated. If vaccinated people take more risks, this will mean real-world data understates vaccine efficacy (if they take fewer, it will overstate it).

Much more important in this case, though, is that in the real world, vaccination rates increase with increasing age. This is true because of the patterns of rollout and because of variation in desire for the vaccine. Whatever the reason, the result is that in the real world we may understate the efficacy of vaccines if we ignore age patterns.

We saw this very starkly in the analysis of some of the Israeli data (see a longer discussion here). If we just compare hospitalization rates, it looks like vaccine efficacy is down to 67%. But in fact, if we look within age groups, the efficacy ranges from 82% to 100%, with nearly all age groups having efficacy at or above 90%. This seems kind of weird, until you recognize that the 67% basically comes from the fact that the oldest groups have the highest vaccination rates but also are more likely to be hospitalized. (This idea — that the average efficacy could be lower than the efficacy in any group — is called Simpson’s paradox.)

Data out of New York, published a week or two ago, shows a similar 90% protective effect against hospitalization from May through late July (again, once adjusted for age). Much of this data is post-Delta. A study from the U.K. shows 90% protection against Delta from two-dose vaccination.

As we look at the data, it is important to remember that 90% isn’t 100%. The Financial Times had a good graphic illustrating the impact of vaccination on death risk by age. The writers’ point is that given the much higher baseline risk, a fully vaccinated 80-year-old, for example, has about the same mortality risk as an unvaccinated 50-year-old. Of note, a fully vaccinated older adult still has a much higher mortality risk than an unvaccinated child.

But 90% is still 90%, and the evidence doesn’t point to significantly waning protection against hospitalization and death. Vaccines remain very, very effective.

Vaccine Efficacy: Any Infection

You thought hospitalization was hard! Get ready for something harder.

Evaluating the protection afforded by the vaccines against any degree of infection could, in principle, be done in the same way as for hospitalizations. That is: look at the infection risk for vaccinated individuals versus unvaccinated. This runs into the same problem as with hospitalizations. In addition, it runs into the much more important problem of case detection. We do not detect all cases of mild or asymptomatic illness, and they may be less commonly detected in vaccinated individuals.

Based on the data from New York referenced above, the protection against any infection seemed to wane, from 90% to 80% over the course of May to July. This suggests a degree of waning immunity. Another recent CDC paper indicates vaccine efficacy dropping to 66%, though there are caveats. A similar number has come out of Israel.

Getting more precision on this number will take time, and careful study — basically, we need to be doing more comprehensive testing; we may see better data out of cohorts like universities and professional sports leagues with comprehensive testing programs and variation in vaccination. The data is pointing to efficacy against any infection declining. How much is unclear.

Don’t Make This Mistake!

There is one logical pitfall that I strongly advise you to avoid: the denominator mistake. In the discussion of the Provincetown outbreak, there was a lot of talk of the fact that 75% of infections were in vaccinated people. But the thing is that nearly everyone there at the time was vaccinated.

As more and more people get vaccinated, we can expect a larger share of infections to take place in vaccinated people. Don’t get hung up on this!

Bonus: Boosters, Natural Immunity, and Kids

Three bonus questions to address here.

First: With all this context, what is the deal with boosters?

A booster shot — literally, a third dose of the same vaccine — has two possible uses. First, for people who are immune-compromised (say, cancer patients), a third dose of vaccine may be necessary to prompt a strong immune response. There is widespread agreement for using boosters this way; one way to think about it is that this group should have been on a three-dose regimen initially.

For everyone else, who likely had a robust immune response initially, the value of a booster is less clear. The booster will prompt more antibody production, increasing immediate protection, but it isn’t clear that that’s necessary to improve protection against severe disease. Which is the point of the vaccines in the first place. This isn’t a straightforward trade, and reasonable people differ.

Second: What about natural immunity?

If you had COVID, how does that compare with being vaccinated? Is it more protective? Less? In practice, both vaccination and prior infection convey a lot of protection. Comparing the two is … you guessed it, difficult.

There is evidence from Israel, still in preprint form, that argues that infection is less likely after previous infection than after vaccination. That is to say, it finds high protection delivered by both natural immunity and full vaccination, but higher protection from natural immunity.

Notably, in that study the researchers found the highest protection from infection in individuals who had COVID-19 and also had at least one dose of the vaccine. A smaller study in Kentucky found that, among individuals who had COVID, those who were later vaccinated were less likely to be reinfected.

Putting this together, the data points to significant protection provided by natural immunity — likely comparable to vaccination — but that those who have had COVID before can be even better protected by also being vaccinated.

Third: Where the f**k are the vaccines for kids?

Trust me, I’m as frustrated as you are. The timeline seems totally unclear. Is it November? December? March? The chaos at the FDA isn’t helpful. I cannot think of what to do but continue to complain and wait.

However, I want to revisit what I said at the top about expectations. If vaccines for kids work as well as they do for adults, they’ll lower the risk of hospitalization. That risk is already low (perhaps 1 in 200), and a 90% reduction would lower it to 1 in 2,000. This would be a big percentage change, even if the baseline risk is already quite small. Still, it’s very good, which is why I want vaccines.

What we shouldn’t expect is for vaccines to eliminate the risk of kids getting asymptomatic or mild COVID. Yes, that risk will be lower. But it will not be zero. I’m not saying this to scare you, just to set realistic expectations. As much as we may wish it, zero COVID isn’t realistic, for kids or anyone.