Sometimes a failure to replicate a study isn’t a failure at all

As anyone who has ever tried a diet knows, exerting willpower can be exhausting. After a whole day spent carefully avoiding the snack machine and attempting to take mindful joy in plain baked chicken and celery sticks, the siren call of cookies after dinner may be just too much to bear. This idea — that exercising self-control gets harder the more you have to do it — is called ego depletion, and it’s one of the most well-known concepts in social psychology. There are popular books on it. Most of us have probably have personal experience with it.

But what if a huge study of thousands of people found no evidence for ego depletion?  What if some cultures actually show reverse ego depletion — where exerting willpower actually makes exerting more willpower easier? What if I told you that ego depletion does exist — but only if you believe it does?

Do these recent studies mean that one of the most well-established phenomena in social psychology is headed for the dusty shelf of discredited theories? Does it mean that everything you thought you knew about willpower was wrong? Not at all. Ego depletion may still exist. It just may be limited to particular contexts and conditions, such as an ill-fated attempt at a crash diet.

Social psychology is not trying to troll you. But it does have to contend with a giant source of variability that other fields don’t. Gravity exists whether or not it had a happy childhood. An element has specific properties, and those properties don’t change because the element in question had a bad day.

But in social psychology, you’re dealing with something far more complicated than gravity or rocket science — the human mind. A replication attempt that fails doesn’t necessarily mean the original experiment was a poor one. And when a big idea in psychology is shown to be limited, or even entirely discredited, there’s often no reason to declare it dead and gone. Instead, the back-and-forth surrounding ego depletion is an opportunity to think about how psychology is done and how scientific ideas progress when we — unpredictable, biased, infuriating humans — are the variable.

To deplete or not to deplete?

Ego depletion has been “one of the most famously discussed phenomena in social psychology,” says Malte Friese, a social psychologist at Saarland University in Saarbrücken, Germany. The original 1998 paper describing the phenomenon has been cited more than 4,800 times. There have since been hundreds of studies showing that expending mental effort in one self-control task makes people slack off in the next. “It was regarded as very substantial and robust,” Friese says.

So much so that Miguel Vadillo, an experimental psychologist at the Autonomous University of Madrid in Spain, says he took for granted that ego depletion was a reliable effect. But when he came across a 2014 article in Frontiers in Psychology that questioned the strength of ego depletion evidence, he began to have second thoughts. “The published record of ego depletion was too good to be true,” he explains. The results were so good that Vadillo suspected that studies with less glowing results might be missing from the published record.

Vadillo and his colleagues decided to attempt their own validation of the ego depletion effect. They re-analyzed openly available data from the Many Labs 3 project, a huge set of studies that pooled data from students at 20 different universities and survey takers from Amazon’s Mechanical Turk. Participants did tests and exercises in an attempt to replicate previously published social psychology findings.

Ego depletion wasn’t on the list, but Vadillo realized that it could be. The project participants had performed two tasks used frequently in ego depletion research. One, the Stroop task, presents test takers with a series of words, all of which are colors, such as purple or green. The word green, however, is presented in orange text. The test takers are asked to press a button when they see the color orange, not the word orange. This task is, unsurprisingly, challenging and often frustrating. It’s frequently used to deplete people’s willpower.

The second task is an unsolvable anagram test. Students have to form a word from a collection of letters that are impossible to make words from. But the students had to try. Scientists measured how long participants persisted at these (again, challenging and very frustrating) anagram tasks.

Out of all the students in the Many Labs 3 study, 2,062 had performed those two tasks in the right order, and close enough in time to potentially influence each other. If ego depletion was a real phenomenon, Vadillo reasoned, students who had done the Stroop task before the anagram task should get frustrated and give up faster on the second task.

They didn’t.

While students who did the anagram task near the end of their series of tasks did give up faster, completing a Stroop test beforehand didn’t predict how well they’d do on the anagram task, Vadillo and his colleagues reported August 1 in Royal Society Open Science.

Does this mean that ego depletion doesn’t exist? “If a measurable effect existed, a study with this sample size should have been able to detect it,” Vadillo says. On the other hand, it’s possible that ego depletion as we understand it does exist, and, as the authors note, “persistence in anagram solving may not be an optimal measure to test it.”

Brian Nosek, a psychologist at the University of Virginia in Charlottesville who ran the Many Labs 3 Project, thought the new study was a nice reuse of existing data. “We did not design that study to test ego depletion, but the authors discovered there’s a manipulation that’s common for ego depletion,” Nosek says. “I thought it was a creative application of data re-analysis.”

But while the original studies on ego depletion did use a Stroop task and an anagram task, notes Greg Walton, a social psychologist at Stanford University, that doesn’t mean that ego depletion was the phenomenon that ended up being tested. “The assumption in the paper is that doing the Stroop test first would be depleting,” he explains. But people in the Many Labs 3 project were performing a whole suite of tasks, one after the other. The Stroop and the anagram task were only two among many, Walton says, “so it becomes an exercise in compliance, getting through a series of unrelated tasks.” The students, in other words, might not have been giving it their all. Perhaps the researchers didn’t see ego depletion, but that could be because, in this set of tests, they wouldn’t have been able to anyway. 

Your depletion is my power-up

Beyond Vadillo’s study, there are plenty of instances of variability in studies of willpower fatigue. For instance, in a study comparing almost 400 people in India to almost 450 people in the United States, social psychologist Veronika Job confirmed that people from India did not show ego depletion, while people from the United States did. And among participants from India, self-control on one task increased how long they persisted on the next, Job and her colleague Krishna Savani  reported in the Journal of Personality and Social Psychology in 2017.

“The daily exertion of self-control to become stronger is a part of the philosophical traditions in eastern Asian context,” notes Job, of the Technische Universität Dresden in Germany. “I definitely believe in the phenomenon,” of ego depletion, she says, but “it depends on the cultural effects and context.” 

Some of that context may just be the belief that it’s possible to exhaust your willpower at all. In an experiment reported in a 2010 paper in Psychological Science, Job, Walton and social psychologist Carol Dweck manipulated 46 college students’ ideas about ego depletion. In one study group, students filled in questionnaires that biased them toward believing that their willpower could become exhausted. In another, they were biased toward the idea that exercising willpower could energize them for the next tough task.

Then the students went on to do the rage-inducing busywork tasks of the ego depletion study. If the students were biased toward believing ego depletion was real, they showed ego depletion, making more mistakes as their willpower got lower. If they were biased toward believing that willpower begets more willpower, the effect went away. If they didn’t believe in ego depletion, the students made the same number of errors, whether or not they had been “ego depleted.”

“Do I think that when people believe that willpower is limited that they then become attuned to small cues that suggest they might run out of resources and fail to self-regulate? Yeah, I do believe that,” Walton says. “But do I think that’s because people are depleted and unable to regulate and it’s bottom up? No.”

One theory to rule them all?

People may still experience ego depletion, but not all the time, and not in every circumstance. And Roy Baumeister is the first to say so. The social and personality psychologist at the University of Queensland in St. Lucia, Australia, was the first to show (and name) ego depletion in the now-famous paper in the Journal of Personality and Social Psychology. “It’s clear lots of people [experience it], it’s clear some people don’t,” he says.

But, he explains, that’s due far more to the nature of people than to anything nefarious. “In psychology, nothing happens all the time. We find stuff that happens sometimes. That’s about as well as we can do.” People have bad days and good ones, sleepless nights and restful nights, good and bad childhoods. “I think the scientific question should be, ‘what are the conditions under which [ego depletion] does and does not happen?’” Baumeister says.

This idea — that sometimes it happens, sometimes it doesn’t — isn’t particularly satisfying. Most of us were taught that, in science, a scientist forms a hypothesis, tests it and then throws it out if it doesn’t work. That’s what the philosopher of science Karl Popper thought, says Janet Stemwedel, herself a philosopher of science at San Jose State University in California. In this view, scientists go out every day and “throw hypotheses in the deep end of the pool to see if they can swim.” By Popper’s standards, if ego depletion fails to replicate, it’s a failed hypothesis. It deserves to drown.

But there are other less hypothesis-drowning ideas of how science should work. Thomas Kuhn, for example, was a little more flexible. In Kuhn’s view, scientists don’t attempt to falsify a hypothesis. Instead, they work with it, seeing how many ways a hypothesis can explain things that happen in the world, Stemwedel explains. Once scientists can’t solve problems with that paradigm, it’s time to find a new one. In this view, not finding ego depletion in a particular context doesn’t mean it’s time to let it sink. It means it’s time to find new ways to look at the concept.

The way the replication efforts in social psychology are covered — replicated means good and not-replicated means bad — is too simplistic. A non-replicated effect like ego depletion doesn’t automatically mean the phenomenon doesn’t exist. The effect might have a limited scope — it could be a matter of belief or culture. It may mean that the concept of ego depletion needs to be reframed. But it doesn’t mean that the scientists who did the studies finding evidence of ego depletion were wrong. “In science in general I think there is almost never a point where we can say, yes, this is true,” Friese says. “We are probably pretty sure that gravity exists, but few things are as clear cut as gravity.”

Right now, social psychology lurches along with Popper’s ideas. One experiment forward, two failed replications back. To get more clear-cut results, Friese says, social psychology needs two things: Better methods and better theories. The better methods are coming: Social psychologists can preregister the experiments they do, and make their data open, so other scientists can see and follow up on their results.

Better theories? That’s more of a challenge. Conducting a set of experiments in one context — say, studying ego depletion on a college campus — is just the first step, Job explains. It’s a single experiment, a single set of observations. It’s not a way to fully understand what behavior is taking place, and — more importantly — why. An experiment can show an object falls when we drop it. But a single experiment is not the theory of gravity. “We need a broader theory of what happens,” Job says. Then social psychologists can place individual studies into that broader theory, rather than trying to construct a broad theory from the individual studies.

To develop those theories, social psychologists (and those who report on them) might want to think more in terms of Kuhn’s paradigms, and less in terms of Popper’s desire to drown hypotheses that don’t replicate. But embracing paradigms means that social psychologists need to let go of something very precious — their egos. “We get attached to our theoretical perspectives,” Nosek admits. “The [scientific] culture rewards advancing a perspective and being known for that perspective.”

Scientists will end up attached to their own ideas about whether ego depletion exists or not, because scientists are people. “It’s humans,” Nosek says. “If we could just give science to the machines we could avoid it, but we’re not there yet.” Of course, because the machines are also designed by humans, the machines will have their own biases.

Maybe in the end, social psychology will need to lean in to its very humanity. Because like humans themselves, scientific ideas have good times and bad ones. Sometimes, a hypothesis swims. Other days, it might not be so buoyant. But that doesn’t mean we need to let it drown.  

Follow me on Twitter: @BeeBrookshire