Emily Oster

10 min Read Emily Oster

Emily Oster

Lots of Studies Are Bad

Emily Oster

10 min Read
One of my children’s favorite places in the world is the arcade at Ryan Family Amusements in Newport, R.I. I will confess to also loving it, and my go-to game is something we call “Ice Zombies,” where you shoot video zombies with a water gun. One of the main warnings the game gives you as you play is “Look out for zombies from both sides!

I was thinking about this phrase in considering two relatively new studies — one on the value of masking from the CDC, and one on the value of lockdowns out of Johns Hopkins University. These are on very different sides of the COVID divide. The first argues that masks lower the risk of COVID infection by between 56% and 83%. The second argues that lockdowns in general had almost no impact on mortality. The reality is that they are both flawed. There is such a strong tendency to see good in the studies we agree with and bad in the ones we do not. But there are bad studies — just like zombies — on both sides.

I’ll talk through both studies, and then reflect a bit on whether there is any way to see these flaws more clearly when studies are first released.

MMWR on masks

The latest CDC Morbidity and Mortality Weekly Report (MMWR), on masking, is here. The goal of this study was to compare COVID rates for people who masked and who did not, and the primary conclusion was that masking was extremely effective at preventing COVID. The agency made a little graphic, which was widely shared; I’ve included it below, albeit reluctantly given how misleading it is.

The method in this study is what’s called a “negative control.” The authors identified individuals in California who tested positive for COVID. They then identified a matched control (someone who was matched on age, sex, and region but who tested negative in the same time frame). They reached out for a survey to the positive and negative individuals and asked them about their characteristics. This included behaviors, masking, vaccination, demographics, etc. They then compared the behaviors for the two groups. The central result they present is the fact that people who tested negative were more likely to report wearing masks than people who tested positive.

There are an enormous number of issues with this study. They begin at recruitment and data collection. Only about 13.5% of the negative-test people contacted and 8.9% of the positive-test people contacted agreed to participate. These rates are very low overall, and we immediately worry that the selection in respondents is different in the two groups. The information about masking was collected based on self-reports from these individuals, which opens up the possibility that various biases could impact the results (for example, “social desirability bias,” where people tell interviewers what they think they want to hear).

Even putting aside the low response rates, there are huge issues with the selection of the two groups. Most notably, the groups are totally different in terms of their reasons for testing. In the group that tested positive, 78% of them tested due to symptoms. In the group that tested negative, only 16.7% tested due to symptoms. The most common reasons for testing in this group were that it was required for work or school (43.1%) or that it was required for a medical procedure (16.9%).

The groups also differed in their degree of vaccination (32.1% in the negative group, 17.6% in the positive group; these data started in February 2021, so vaccination is lower than current rates). The negative group was also higher-income.

Putting this all together, what this study is doing — at the core — is comparing the masking behavior among a select sample of wealthier individuals who are tested primarily for work, travel, or medical procedures with an even more select sample of lower-income individuals who are testing primarily due to symptoms. This comparison is virtually impossible to learn from. The authors claim they’ve done sensitivity analyses around some of these issues, but the populations are so fundamentally different as to render the entire exercise completely meaningless.

To be clear: I’m not claiming that this study shows that masks do not work. What I’m claiming is that this study shows nothing. At best, it perhaps shows that masking is more common in some demographic groups than others (interesting, yet already known). It tells us nothing at all about masking efficacy, because it’s just too poorly constructed.

Johns Hopkins on lockdowns

The other side of the zombie horde is this study, in the form of a working paper out of Johns Hopkins, on whether lockdowns prevented COVID mortality.

This paper is a review and meta-analysis of a large number of papers that have aimed to understand how various movement restrictions impact COVID-19 mortality. This includes both the impact of full “lockdown” measures and the impacts of more specific interventions like school or restaurant closures. Aggregating across papers, the authors argue that these measures had only a quite small impact on COVID-19 mortality.

The most basic issue with this paper is probably in the conception of the question. In general, a meta-analysis aims to combine many small studies of the same question together to estimate an average effect size. The key to that sentence is the phrase “of the same question.” Meta-analysis is well-suited to situations in which it is plausible to argue that many studies are doing the same thing.

An example would be something like a meta-analysis of the impact of having an epidural on the length of labor. In this case, the treatment is well-defined (having an epidural) and the outcome is also well-defined (length of labor).

Using meta-analysis for this COVID question meets a challenge at the outset: What is a lockdown? The paper is going to try to combine many studies evaluating “lockdowns” early on in COVID. But that’s not a well-defined concept. New Zealand completely eliminated travel into the country and met any outbreaks with intense restrictions on movement. This lockdown is different from what was implemented anywhere in the U.S. And even across jurisdictions in the U.S, how lockdown was defined varies.

Again, the aim of the meta-analysis is to combine papers that are looking at the same question. But the individual papers are not widely comparable, since “lockdowns” varied so much. This matters — at least I think it does — because this paper cannot hope to (for example) address the question of whether a lockdown like the one in New Zealand affected mortality.

The second issue is that the paper engages in some difficult-to-understand exclusion choices. In a review of this type, the name of the game is to try to identify all relevant papers and then edit them down to those that can be compared and answer the question. The authors initially identify about 18,000 papers, which are eventually whittled down to 34. Yes, you read that right.

The first part of this whittling makes sense — their broad search mostly finds papers that are not about this topic. But the last stage is a bit more complicated. They end up with 117 papers that broadly fit their criteria, and then get down to 34. The details are in the graphic below.

Some of these exclusions make sense — duplicates, or papers that did not actually evaluate mortality. But some of this is more confusing. The authors exclude student papers. They exclude papers with a “time series” approach. They exclude synthetic-control studies.

This last exclusion is an example of an odd one. Synthetic control is an approach frequently used in economics to generate a control group that is better matched to a treatment group (it’s a neat procedure that deserves its own newsletter at some point). It’s difficult to see why you would exclude papers which do this. The authors argue that they do so because one paper using it (which found that Sweden would have benefited from a lockdown) was criticized by another paper. But they don’t seem to use the same kind of criteria for all the papers they include, so I’m left a bit concerned that there is another motivation here.

The sheer volume of papers excluded makes it difficult to regard this as a true review.

Others commenters have raised more objections — pointing out that mortality may not be the only metric of interest, for example. For me, the first of my concerns is the most central. This paper is problematic because the question is ill-suited to a meta-analysis of this type.

That doesn’t mean lockdowns worked! Again, it’s not even clear what that would mean, since there is no one definition of lockdown. My point isn’t that this paper is wrong in its conclusions, just that it’s largely uninformative. The authors begin with an interesting graph showing a limited relationship between the stringency of COVID restrictions and mortality. That deserved more study, but this paper isn’t helping us understand it much.

Who should I trust? 

The reaction to these studies was enormously frustrating to me. Both studies are poor. And both were taken up seemingly uncritically by people whose priors they supported. People who oppose the relaxing of mask requirements pointed to the first as proving that masks work and that we should be reluctant to move away from them. People who oppose lockdowns picked up the second study as proving that lockdowns ruined lives with no benefits.

There may be merit to both of these positions. There may be evidence for both of them. But that evidence isn’t enhanced by these papers.

One question is, as a consumer, is there a way to actually identify which studies are problematic in these ways? Like, how would you know not to trust these? This can be hard to do when reading on your own, in part because some of the reasons to distrust them are kind of statistically complicated.

The simplest way I have found to try to impose discipline on your judgment is to consume some media or commentary outside of your echo chamber. This will make you mad! But hear me out. In the example of the mask study, reading the outcry from the political right identified a lot of the issues with the study, even if the tone was sometimes a bit much. A similar thing could be said for the outcry from the political left on the other.

For a while, I was trying to listen to a conservative podcast during half of my long runs and a liberal one on the other half, just for balance. But even if you do not want to do this regularly (indeed, I eventually had to save my sanity and switch to podcasts about sports), you can still take advantage of it in the moment. If you’re wondering whether to believe a study, start with what the skeptics say.

In these cases, though, there are zombies on all sides. Get out your water gun.

Covid-19 rapid antigen tests arranged in a pattern on a yellow background.

Feb 20 2023

12 min read

COVID-19: Where to Go from Here

A long-term view of the virus

Emily Oster
Covid-19 rapid antigen tests arranged in a pattern on a yellow background.

Oct 20 2022

9 min read

Should You Get the Bivalent Booster?

The latest on the risks and benefits of COVID vaccines boosters for older adults, pregnant people, and kids

Emily Oster
A line graph with pink, yellow, and blue dots representing life's ups and downs.

Aug 16 2022

3 min read

Wins, Woes, and Doing It Again

We have our first story from a dad! And it’s a good one. 10/10 —Girl Dad with Confidence Growing by Read more

Emily Oster
Covid-19 rapid antigen tests arranged in a pattern on a yellow background.

Aug 15 2022

8 min read

Updated CDC Guidelines for School and Child Care

NO QUARANTINES!!!

Emily Oster

Instagram

left right
Do you brand things a certain way to get your kid to accept it? Like calling carrots “rabbit popsicles”? Or telling them to put on their “super speed socks” in the morning? Share your rebrands in the comments below! You never know who you might be helping out 👇

#emilyoster #funnytweets #relatabletweets #parentingjokes #kidssaythedarndestthings

Do you brand things a certain way to get your kid to accept it? Like calling carrots “rabbit popsicles”? Or telling them to put on their “super speed socks” in the morning? Share your rebrands in the comments below! You never know who you might be helping out 👇

#emilyoster #funnytweets #relatabletweets #parentingjokes #kidssaythedarndestthings
...

Have you ever panic-googled a parenting question when everyone else is asleep? If so, you’re not alone. 

Today is the first episode of a new biweekly series on my podcast: Late-Night Panic Google. On these mini-episodes, you’ll hear from some familiar names about the questions keeping them up at night, and how data can help. First up: @claireholt!

Listen and subscribe to ParentData with Emily Oster in your favorite podcast app 🎧

#parentdata #emilyoster #claireholt #parentingstruggles #parentingtips #latenightpanicgoogle

Have you ever panic-googled a parenting question when everyone else is asleep? If so, you’re not alone.

Today is the first episode of a new biweekly series on my podcast: Late-Night Panic Google. On these mini-episodes, you’ll hear from some familiar names about the questions keeping them up at night, and how data can help. First up: @claireholt!

Listen and subscribe to ParentData with Emily Oster in your favorite podcast app 🎧

#parentdata #emilyoster #claireholt #parentingstruggles #parentingtips #latenightpanicgoogle
...

Sun safety is a must for all ages, especially babies! Here are my tips for keeping your littlest ones protected in the sunshine:
☀️ Most importantly, limit their time out in hot weather. (They get hotter than you do!)
☀️ Keep them in the shade as much as possible when you’re out.
☀️ Long-sleeve but lightweight clothing is your friend, especially on the beach, where even in the shade you can get sunlight reflecting off different surfaces.
☀️ If you want to add a little sunscreen on their hands and feet? Go for it! But be mindful as baby skin tends to more prone to irritation.

Comment “Link” for a DM to an article on the data around sun and heat exposure for babies.

#sunsafety #babysunscreen #babyhealth #parentdata #emilyoster

Sun safety is a must for all ages, especially babies! Here are my tips for keeping your littlest ones protected in the sunshine:
☀️ Most importantly, limit their time out in hot weather. (They get hotter than you do!)
☀️ Keep them in the shade as much as possible when you’re out.
☀️ Long-sleeve but lightweight clothing is your friend, especially on the beach, where even in the shade you can get sunlight reflecting off different surfaces.
☀️ If you want to add a little sunscreen on their hands and feet? Go for it! But be mindful as baby skin tends to more prone to irritation.

Comment “Link” for a DM to an article on the data around sun and heat exposure for babies.

#sunsafety #babysunscreen #babyhealth #parentdata #emilyoster
...

I’m calling on you today to share your story. I know that many of you have experienced complications during pregnancy, birth, or postpartum. It’s not something we want to talk about, but it’s important that we do. Not just for awareness, but to help people going through it feel a little less alone.

That’s why I’m asking you to post a story, photo, or reel this week with #MyUnexpectedStory and tag me. I’ll re-share as many as I can to amplify. Let’s fill our feeds with these important stories and lift each other up. Our voices can create change. And your story matters. 💙

#theunexpected #emilyoster #pregnancycomplications #pregnancystory

I’m calling on you today to share your story. I know that many of you have experienced complications during pregnancy, birth, or postpartum. It’s not something we want to talk about, but it’s important that we do. Not just for awareness, but to help people going through it feel a little less alone.

That’s why I’m asking you to post a story, photo, or reel this week with #MyUnexpectedStory and tag me. I’ll re-share as many as I can to amplify. Let’s fill our feeds with these important stories and lift each other up. Our voices can create change. And your story matters. 💙

#theunexpected #emilyoster #pregnancycomplications #pregnancystory
...

OUT NOW: My new book “The Unexpected: Navigating Pregnancy During and After Complications” is available on April 30th. All of my other books came out of my own experiences. I wrote them to answer questions I had, as a pregnant woman and then as a new parent. “The Unexpected” is a book not to answer my own questions but to answer yours. Specifically, to answer the thousands of questions I’ve gotten over the past decade from people whose pregnancies were more complicated than they had expected. This is for you. 💛 Order now at my link in bio!

OUT NOW: My new book “The Unexpected: Navigating Pregnancy During and After Complications” is available on April 30th. All of my other books came out of my own experiences. I wrote them to answer questions I had, as a pregnant woman and then as a new parent. “The Unexpected” is a book not to answer my own questions but to answer yours. Specifically, to answer the thousands of questions I’ve gotten over the past decade from people whose pregnancies were more complicated than they had expected. This is for you. 💛 Order now at my link in bio! ...

OUT NOW: My new book “The Unexpected: Navigating Pregnancy During and After Complications” is available on April 30th. All of my other books came out of my own experiences. I wrote them to answer questions I had, as a pregnant woman and then as a new parent. “The Unexpected” is a book not to answer my own questions but to answer yours. Specifically, to answer the thousands of questions I’ve gotten over the past decade from people whose pregnancies were more complicated than they had expected. This is for you. 💛 Order now at my link in bio!

OUT NOW: My new book “The Unexpected: Navigating Pregnancy During and After Complications” is available on April 30th. All of my other books came out of my own experiences. I wrote them to answer questions I had, as a pregnant woman and then as a new parent. “The Unexpected” is a book not to answer my own questions but to answer yours. Specifically, to answer the thousands of questions I’ve gotten over the past decade from people whose pregnancies were more complicated than they had expected. This is for you. 💛 Order now at my link in bio! ...

OUT NOW: My new book “The Unexpected: Navigating Pregnancy During and After Complications” is available on April 30th. All of my other books came out of my own experiences. I wrote them to answer questions I had, as a pregnant woman and then as a new parent. “The Unexpected” is a book not to answer my own questions but to answer yours. Specifically, to answer the thousands of questions I’ve gotten over the past decade from people whose pregnancies were more complicated than they had expected. This is for you. 💛 Order now at my link in bio!

OUT NOW: My new book “The Unexpected: Navigating Pregnancy During and After Complications” is available on April 30th. All of my other books came out of my own experiences. I wrote them to answer questions I had, as a pregnant woman and then as a new parent. “The Unexpected” is a book not to answer my own questions but to answer yours. Specifically, to answer the thousands of questions I’ve gotten over the past decade from people whose pregnancies were more complicated than they had expected. This is for you. 💛 Order now at my link in bio! ...

Is side sleeping important during pregnancy? Comment “Link” for a DM to an article on whether sleep position affects pregnancy outcomes.

Being pregnant makes you tired, and as time goes by, it gets increasingly hard to get comfortable. You were probably instructed to sleep on your side and not your back, but it turns out that advice is not based on very good data.

We now have much better data on this, and the bulk of the evidence seems to reject the link between sleep position and stillbirth or other negative outcomes. So go ahead and get some sleep however you are most comfortable. 💤

Sources:
📖 #ExpectingBetter pp. 160-163
📈 Robert M. Silver et al., “Prospective Evaluation of Maternal Sleep Position Through 30 Weeks of Gestation and Adverse Pregnancy Outcomes,” Obstetrics and Gynecology 134, no. 4 (2019): 667–76. 

#emilyoster #pregnancy #pregnancytips #sleepingposition #pregnantlife

Is side sleeping important during pregnancy? Comment “Link” for a DM to an article on whether sleep position affects pregnancy outcomes.

Being pregnant makes you tired, and as time goes by, it gets increasingly hard to get comfortable. You were probably instructed to sleep on your side and not your back, but it turns out that advice is not based on very good data.

We now have much better data on this, and the bulk of the evidence seems to reject the link between sleep position and stillbirth or other negative outcomes. So go ahead and get some sleep however you are most comfortable. 💤

Sources:
📖 #ExpectingBetter pp. 160-163
📈 Robert M. Silver et al., “Prospective Evaluation of Maternal Sleep Position Through 30 Weeks of Gestation and Adverse Pregnancy Outcomes,” Obstetrics and Gynecology 134, no. 4 (2019): 667–76.

#emilyoster #pregnancy #pregnancytips #sleepingposition #pregnantlife
...

My new book, “The Unexpected: Navigating Pregnancy During and After Complications” is available for preorder at the link in my bio!

I co-wrote #TheUnexpected with my friend and maternal fetal medicine specialist, Dr. Nathan Fox. The unfortunate reality is that about half of pregnancies include complications such as preeclampsia, miscarriage, preterm birth, and postpartum depression. Because these are things not talked about enough, it can not only be an isolating experience, but it can also make treatment harder to access.

The book lays out the data on recurrence and delves into treatment options shown to lower risk for these conditions in subsequent pregnancies. It also guides you through how to have productive conversations and make shared decisions with your doctor. I hope none of you need this book, but if you do, it’ll be here for you 💛

#pregnancy #pregnancycomplications #pregnancyjourney #preeclampsiaawareness #postpartumjourney #emilyoster

My new book, “The Unexpected: Navigating Pregnancy During and After Complications” is available for preorder at the link in my bio!

I co-wrote #TheUnexpected with my friend and maternal fetal medicine specialist, Dr. Nathan Fox. The unfortunate reality is that about half of pregnancies include complications such as preeclampsia, miscarriage, preterm birth, and postpartum depression. Because these are things not talked about enough, it can not only be an isolating experience, but it can also make treatment harder to access.

The book lays out the data on recurrence and delves into treatment options shown to lower risk for these conditions in subsequent pregnancies. It also guides you through how to have productive conversations and make shared decisions with your doctor. I hope none of you need this book, but if you do, it’ll be here for you 💛

#pregnancy #pregnancycomplications #pregnancyjourney #preeclampsiaawareness #postpartumjourney #emilyoster
...

We are better writers than influencers, I promise. Thanks to our kids for filming our unboxing videos. People make this look way too easy. 

Only two weeks until our book “The Unexpected” is here! Preorder at the link in my bio. 💙

We are better writers than influencers, I promise. Thanks to our kids for filming our unboxing videos. People make this look way too easy.

Only two weeks until our book “The Unexpected” is here! Preorder at the link in my bio. 💙
...

Exciting news! We have new, high-quality data that says it’s safe to take Tylenol during pregnancy and there is no link between Tylenol exposure and neurodevelopmental issues in kids. Comment “Link” for a DM to an article exploring this groundbreaking study.

While doctors have long said Tylenol was safe, confusing studies, panic headlines, and even a lawsuit have continually stoked fears in parents. As a result, many pregnant women have chosen not to take it, even if it would help them.

This is why good data is so important! When we can trust the data, we can trust our choices. And this study shows there is no blame to be placed on pregnant women here. So if you have a migraine or fever, please take your Tylenol.

#tylenol #pregnancy #pregnancyhealth #pregnancytips #parentdata #emilyoster

Exciting news! We have new, high-quality data that says it’s safe to take Tylenol during pregnancy and there is no link between Tylenol exposure and neurodevelopmental issues in kids. Comment “Link” for a DM to an article exploring this groundbreaking study.

While doctors have long said Tylenol was safe, confusing studies, panic headlines, and even a lawsuit have continually stoked fears in parents. As a result, many pregnant women have chosen not to take it, even if it would help them.

This is why good data is so important! When we can trust the data, we can trust our choices. And this study shows there is no blame to be placed on pregnant women here. So if you have a migraine or fever, please take your Tylenol.

#tylenol #pregnancy #pregnancyhealth #pregnancytips #parentdata #emilyoster
...

How many words should kids say — and when? Comment “Link” for a DM to an article about language development!

For this graph, researchers used a standardized measure of vocabulary size. Parents were given a survey and checked off all the words and sentences they have heard their child say.

They found that the average child—the 50th percentile line—at 24 months has about 300 words. A child at the 10th percentile—near the bottom of the distribution—has only about 50 words. On the other end, a child at the 90th percentile has close to 600 words. One main takeaway from these graphs is the explosion of language after fourteen or sixteen months. 

What’s valuable about this data is it can give us something beyond a general guideline about when to consider early intervention, and also provide reassurance that there is a significant range in this distribution at all young ages. 

#cribsheet #emilyoster #parentdata #languagedevelopment #firstwords

How many words should kids say — and when? Comment “Link” for a DM to an article about language development!

For this graph, researchers used a standardized measure of vocabulary size. Parents were given a survey and checked off all the words and sentences they have heard their child say.

They found that the average child—the 50th percentile line—at 24 months has about 300 words. A child at the 10th percentile—near the bottom of the distribution—has only about 50 words. On the other end, a child at the 90th percentile has close to 600 words. One main takeaway from these graphs is the explosion of language after fourteen or sixteen months.

What’s valuable about this data is it can give us something beyond a general guideline about when to consider early intervention, and also provide reassurance that there is a significant range in this distribution at all young ages.

#cribsheet #emilyoster #parentdata #languagedevelopment #firstwords
...