Numbers can lie

Most epidemiological studies, according to Young, don't account for the fact that they often check many different things in one study. "They think it is fine to ask many questions of the same data set," Young says. And the more things you check, the more likely it becomes that you'll find something that's statistically significant -- just by chance, luck, nothing more.

It's like rolling a pair of dice many times without anyone looking until you get snake eyes and then claiming you'd only rolled it once. Often, epidemiological researchers ask dozens, maybe hundreds of questions in the questionnaires they send to the people they study. They ask so many questions that something, eventually, is bound to come out positive.

That's where the Canadian star sign study comes into play, Young says. It was only because the authors deliberately asked a lot of questions -- to prove a point -- that it was able to come up with significant results for something that couldn't be true.

The study's lead author, statistician Peter Austin of the Institute for Clinical Evaluative Sciences in Toronto, says that once he cleaned up his methodology (by adding a statistical correction for the large number of questions he asked) the association between Leos and internal bleeding and Sagittarians and leg-breaks disappeared.

--

On the defensive

Many epidemiologists do not agree with the critics' assertion that most epidemiological studies are wrong and that randomized studies are more reliable.

"Randomized studies often contradict one another, as do observational studies," says Harvard's Stampfer, who is an author on both the frequently cited vitamin E and hormone replacement studies that Ioannidis says were later refuted.

Instead, Stampfer says, the two types of studies often test different things. "It's not an issue here that observational studies got it wrong and randomized trials got it right," he says, referring to the hormone replacement studies. "My view is that [both] were right and they were addressing different questions."

For one thing, the randomized studies on hormone replacement and vitamin E that Ioannidis cited in his 2005 paper looked at different populations than the observational studies they refuted, says Stampfer, who takes vitamin E himself.

In the hormone replacement case, the observational studies looked at women around the age of menopause. The randomized trial looked at women who were mostly well past that age.