Clinical trial reanalyses may alter who should get treated

Fresh looks at previously published clinical trial data can change recommendations about which patients to treat. In fact, in a new study, roughly one-third of reanalyses drew conclusions opposing the original ones.

Clinical trials, the gold standard of medical research, are rarely reanalyzed. The new work identified only 37 statistical redos in all medical literature since 1966. Of those, 13 came to a conclusion different from the original paper’s, researchers report in the Sept. 10 JAMA.

In three articles, the alternate findings led to recommendations that different patients be treated. One reanalysis concluded that fewer patients should be treated, and nine recommended treating more patients. For example, a study found that an experimental drug for Alzheimer’s disease didn’t work, but when the reanalysis took age into account, younger patients seemed to benefit.

“This is pretty worrisome,” says study coauthor John Ioannidis, a clinical epidemiologist at Stanford University. Doctors rely on clinical trials to determine the best remedies for their patients’ ills. Erroneous conclusions may have costly or even deadly consequences.

Clinical trials should be so carefully designed and executed that reanalyzing the data should produce the same results every time, Ioannidis says. This is especially true when the second investigation is carried out by the same authors, as was the case for 32 of the reanalyses in the study. “Anything less than 100 percent and I start worrying,” he says.

Still, he points out that the findings don’t mean that a third of clinical trial results are wrong. Instead, the fraction is probably much lower. Researchers who reanalyze data and get the same answer as the original inquiry did would have a hard time publishing their results, Ioannidis says. “People will say, ‘This is a me-too exercise. Why should we publish that?’ ”

Another reason that so few clinical trials have undergone a second test, says Eric Peterson, director of the Duke Clinical Research Institute, is that most clinical trial data have been jealously guarded by researchers or companies. Databases either weren’t available or have been presented in a format that makes outsiders’ examination very difficult, he says.

The reasons researchers got different answers in reanalyses varied, Ioannidis and colleagues say. Four reanalyses corrected errors. Some of the redos used more sophisticated statistical methods or updated criteria for determining patient benefits.

And in a few cases, the original researchers or the reanalyzers may have biased the results, purposely or unconsciously. Ioannidis and colleagues cite one case in which the original study found that neither acupuncture nor a drug called amitriptyline eased nerve pain in HIV-infected patients. Proponents of alternative medicine reanalyzed the data using different criteria for success and concluded that, compared with drug treatment, acupuncture had lower dropout and mortality rates.

In an editorial in the same issue of JAMA, Peterson and Harlan Krumholz of the Yale School of Medicine call for more open sharing of data. Just as taxpayers take extra precautions to avoid audits, clinical researchers might be extra careful not to bias their data if they knew someone else could pore over it, Peterson says. “Most scientists are honest, but it would increase the honesty factor a little bit more if you know somebody could potentially replicate [the study].”

Many pharmaceutical companies have already started opening their treasure troves of clinical trial data. That’s both a savvy public relations move to demonstrate transparency and a bid to stave off more stringent government requirements, Peterson says.

Ioannidis says he would welcome greater openness. “The default should be that all data should be publically available.”