Vitamin E didn't protect the heart in men or women. Hormone therapy didn't protect the heart in women. Nitric oxide inhalation didn't help patients with respiratory distress syndrome.
Another finding turned out later to be exaggerated: Taking flavonoids reduces coronary artery disease risk only by 20%, not by 68% as originally reported.
The only finding of the six that stood the test of time was a small study that reported that a chemical called all-trans retinoic acid was effective in treating acute promyelocytic leukemia.
The studies that overturned each of these epidemiological findings, Ioannidis says, "caused major waves of surprise when they first appeared, because everybody had believed the observational studies. And then the randomized trials found something completely different."
To be fair, Ioannidis also tested whether the most frequently cited randomized studies held up. He found that these had a much better track record. Only nine of 39 oft-cited ones were later contradicted or turned out to be exaggerated when other randomized studies were done.
True, Ioannidis looked at only six studies. But Young says he sees the same trend in his own informal counts of epidemiological claims. "When, in multiple papers, 15 out of 16 claims don't replicate, there is a problem," he says.
Belief can be costly, Young adds. For example, one part of the large, randomized Women's Heath Initiative study tested the widely held belief -- based in large part on epidemiological studies -- that a low-fat diet decreases the risk of colorectal cancer, heart disease, or stroke.
The findings suggested that there was no effect. "$415 million later, none of the claims were supported," Young says.
Other scientists, while more cautious than epidemiology's most outspoken detractors, agree that there are many flawed studies. When Kramer first saw Ioannidis' number, "I said to myself, 'It can't be that bad,' " he says. "But I can't prove that it isn't. I know there are a lot of bad studies out there."
Ioannidis, Kramer says, is voicing what many know to be true.
--
Method in doubt
Why does this happen?
Young believes there's something fundamentally wrong with the method of observational studies -- something that goes way beyond that thorny little issue of confounding factors. It's about another habit of epidemiology some call data-mining.
