In fact, Stampfer says, a recent reanalysis of the Women's Health initiative data suggested a trend that hormone therapy may be less risky for younger than older women. The effect was not statistically significant, but, Stampfer says, it's further support for the idea that hormones have different effects depending on when women start taking them.
And the vitamin E studies? The 1993 observational studies followed people who didn't have heart disease. The randomized study looked at people with known heart disease who were on many other medications. All those meds could easily override the effect of vitamin E, says Dr. Walter Willett, a professor of epidemiology and nutrition at the Harvard School of Public Health, who was a coauthor on the hormone and the vitamin E epidemiological studies.
And finally, the low-fat trial from the Women's Health Initiative. It's not surprising, Willett and Stampfer say, that this gold-standard trial failed to find what epidemiology had -- that low-fat diets ward off heart disease, colorectal cancer and stroke. The women in these trials didn't stick to their diets.
"The compliance with the low-fat diet was definitely far lower than anticipated," Willett says, "and probably far worse than even acknowledged in the papers."
Such arguments do not sway epidemiology's detractors.
Each time a study doesn't replicate, "they make a specific argument why the studies are different," Young says. He concedes that epidemiology did uncover the truth about the risks of smoking -- but only because the effects are so strong."Even a blind hog occasionally finds an acorn," he says.
Yet epidemiologists warn that discarding results because of a correction for multiple testing may risk missing true and important effects -- especially in cases where there's a good biological reason suggesting an effect, such as in studies of drugs that have been shown to work in animal experiments. And setting the bar too high might sometimes be dangerous, says Sander Greenland, a professor of epidemiology and statistics at UCLA. "Do you want to screen for medical side-effects with the attitude, 'So what if we miss side effects?' " he asks. "That's deadly. That's ridiculous!"
The debate is unlikely to be resolved any time soon. "If you put five epidemiologists and five statisticians in a room and have this debate," Young says, "and try to get each one to convince the other side, at the end of the day it will still be five to five."
