In science, a lack of replication shouldn’t kill your reputation

A single scientific study often represents years of sweat, tears and sometimes even blood. But it never stands alone. In science, success and validation of a finding depends not on the original work and person doing the experiments, but on the ability of other people to find the same thing.

What happens when they don’t?

“I was really confident our results would replicate,” says Ian Handley, a social psychologist at Montana State University in Bozeman. Handley was an author on a study that showed that people could be primed by words such as “action” to move more to fulfill an active goal or to perform better on cognitive tests. Words such as “still” would make them move less, and be relatively lazy in their cognitive tasks..  One experiment from that study was repeated in the Reproducibility Project, an initiative dedicated to duplicating findings in psychology. Despite Handley’s confidence, the replicators didn’t find any significant effect. Handley was philosophical. “That’s the breaks,” he says. “I’d be uncomfortable if I did something wrong. But we did an experiment. That’s what we found. We reported it.”

Handley doesn’t take the failed duplication of his work personally. But, he says, he did worry. “It was in the back of my mind: Will they think we’re frauds?”

A new study suggests that for Handley and others in his situation, it’s not as bad as they fear: Scientists overestimate how negatively other scientists will view them.

“The climate around replication as being a threat rather than a compliment has produced some unfortunate discussions that suggest a scientist’s reputation hinges on being right, rather than getting it right,” says Brian Nosek, a psychologist the University of Virginia in Charlottesville and executive director for the Center for Open Science, which runs the Reproducibility Project.

But replication is not an insult. It’s central to the scientific process. “There are always going to be things that are wrong in our science. No one should be blamed for having things be wrong or demonized for trying to correct them,” says Adam Fetterman, a social psychologist at the University of Essex in Colchester, England.  “It should be about the research and not about researcher.” But he says with social media promoting near-instant reactions, failed replications quickly get personal. “People are mocking failed replications and they’re equating a failed replication with questionable research practices, incompetence or cheating,” he explains.

Fetterman and his colleague Kai Sassenberg of the University Tübingen in Germany decided to find out if a failed duplication is really as bad for a scientist’s reputation as people fear. Via social media, they recruited 281 scientists who had published scientific studies.

In a survey, participants were asked to imagine a scientific result — either one of their own or a colleague’s. Imagine that someone has tried to replicate the result and failed. The scientist tweaks the methods, tries again, and fails again.

Given this cringe-inducing scenario, survey participants were asked to rate how much other researchers would trust their work afterward, if they might call other results into to question and if their future work might come under the microscope.

When it was their own work, the scientists assumed the impact of a failed replication would be larger than if the work was someone else’s. While the study can’t predict the effect of a replication failure on a scientist’s reputation in the real world, the results do suggest that scientists overestimate the negative effect on their reputations.

“It’s a lot like the spotlight effect,” Fetterman says. Just as when you think the world is watching when you slop coffee down your front, in reality, few notice, and those that do probably won’t judge you as hard as you are judging yourself.

If a repeat study proves the original wrong, is it better to stand by your findings? Or admit that the original results were wrong after all? When confronted with these options, the survey participants ranked those who admitted they were wrong more positively than those who defended themselves, Fetterman and Sassenberg report December 9 in PLOS ONE. Better to eat some humble pie in the face of a failed replication.

“Scientific reputations appear to be more grounded in the scientist’s process of investigation and discovery, not in whether they got it all right the first time,” says Nosek. “That’s a good thing, because we almost never get it right the first time!”

A single duplication failure shouldn’t doom an idea to the scientific dustbin, says Carina Sonnleitner, a psychologist at the University of Vienna in Austria. Sonnleitner was part of the team that tried to replicate Handley’s work. She explains that the failure to replicate is not a sign that Handley’s results were bad, or even wrong. The statistics involved in every scientific study are always about probability, she says. So there is always a chance, however small, that the first study — or its replication — was just a fluke. Many replications, and analysis of all the replications together, need to be done before drawing a conclusion.

The new study also noted that admitting you’re wrong in the face of a failed replication can make people view you more positively. But it’s not always right to jump straight to groveling, argues Richard Petty, a social psychologist at Ohio State University in Columbus. “When someone fails to replicate, it’s not [clear] who’s wrong,” he says. When it’s certain that you are wrong, admit it, he says. But context — and other replications — matter as well. Petty says that in the case of his own work, “15 other labs have shown the same thing. If one attempt failed to get it, I’d be crazy to jump and say I must be wrong.”

Reputations are important, but of course, the real question is, can Fetterman and Sassenberg’s finding be replicated? “Maybe,” says Fetterman. “There’s another set of researchers doing similar research. From the people I’ve talked to, they seem to have similar findings.” But the proof, as in the rest of science, will be in the eventual replication.