Observational Studies: The Good, Bad, and the Ugly

What do we mean when we say that a study is observational?

Magnifying glass

The Beating Edge raised a ruckus the other day with two words.  “Junk science”.  The phrase occurred in a quote from Steven Nissen, MD, chair of Cardiovascular Medicine at Cleveland Clinic. Dr. Nissen was evaluating an observational study that appeared to coffee that people who drank coffee lived longer lives. Dr. Nissen’s issue was how observational studies in general are reported as fact. Not coffee.

Advertising Policy

Cleveland Clinic is a non-profit academic medical center. Advertising on our site helps support our mission. We do not endorse non-Cleveland Clinic products or services. Policy

In any case, commenters jumped all over the statement, defending the validity of observational studies against the “junk science” label.  Since TBE has come up against this issue before, we thought we’d take a moment to clarify our position on observational studies, and acknowledge what about them is good, what’s bad, and what’s … well.

What do we mean when we say that a study is observational? An observational study looks at an existing or unfolding phenomenon and reports on it in relation to a certain outcome. The subject of an observation is something that already exists or would be going on anyway, whether or not the scientists were reporting on it.

The Good

Observational studies get us thinking. They generate ideas, make associations, and suggest what kind of things we ought to be looking at and what kind of questions we ought to be asking. Good science begins in observation. But it shouldn’t end there.

Advertising Policy

The Bad

Appearances can be deceiving. If you trusted your eyes alone, you might think that there were little people in your television set. Or think that because lightning and rain appear at the same time, lightning causes rain. Dr. Nissen and his Heart 411 co-author A. Marc Gillinov, MD, give the real-life example of hormone replacement therapy (HRT).  Multiple observational studies appeared to show that HRT gave women a lower risk of heart disease. This conclusion should seem to be supported by logic and common sense: Young women had more hormones, and they also had less heart disease. But when these observational conclusions were tested by valid means (more on these later), it turns out that HRT didn’t lower women’s risk of heart disease. In fact, HRT raised the risk of breast cancer – a totally unexpected finding.  So it’s good to make observational studies. But it’s generally not wise to act on them before testing them more rigorously.

The Ugly

The Media devours observational studies and regurgitates them as established fact. Especially when the studies appear to validate a popular habit or prejudice (“sex makes you smarter”, “taxes cause cancer”). When observational studies are published in reputable journals (as some deserve to be), people tend to glance at the source, and think “Oh, it’s in the New England Journal of Medicine. It must be an established fact”.  Things really get ugly when media reports of an observational study prompt people to stop using a drug or start using a therapy or supplement without consulting their doctors first. When that happens, things may go from ugly to deadly.

Crowd-sourced studies: Some of the hottest news in science is now being generated by something called crowd-sourced studies.  This type of observational study is made possible by the internet and mobile devices and their capabilities for transmitting massive amounts of data from enormous numbers of people – people who use a certain drug, for instance. If crowd-sourced studies provide information to inspire scientists to test the findings under more rigorous conditions – good!  If they give the impression that they are producing actionable conclusions – bad.  If they set off a media frenzy of wild generalizations and unsupported conclusions – ugly.

Advertising Policy

So are observational studies “junk science”?  As the first point on a gradient of increasingly more rigorous testing that leads to controlled double-blind experiments and falsifiable results – no, they are not junk science.  As generators of data you can risk your life and health on – yes: they’re landfill.

TBE is fortunate to have access to physicians and scientists of the Cleveland Clinic to help you evaluate incoming studies on cardiovascular health.  We urge you to read this previous post on how to separate fact from fiction in medical information.

Advertising Policy
Advertising Policy