Why I Doubt Evidence Based Medicine

A naive medical student who knows it all

When I was third year medical student and I was roaming the hospital checking off tasks given to me to complete by my senior resident, I came upon an old-timer doctor in the twilight of his career. He was dutifully writing an admission note for one of his patients. He moved kinda slow and didn’t teach the residents much. I was chatting with one of my fellow medical students about a lecture we heard that day about medical study design. This old timer interrupted our conversation and offered his opinion of how he thought “Evidence Based Medicine” was overrated. We could only stare back in disbelief; if he had stripped naked and started dancing around the unit, it would have been less surprising to us.

You have to realize in the academic halls of the University of Michigan Medical School, this statement was heresy punishable by metaphorical public flogging. At the time, I just presumed the old timer spoke so misguidedly because he was from a prehistoric time in medicine and wasn’t up to date on the concept of medical literature analysis; he didn’t understand medicine was practiced in the modern world and I felt sorry for his patients.

I have since changed my position.

Studies, studies and more studies

For the last two decades there has been a concerted effort to scientifically study “Complementary and Alternative Medicine” (CAM). Many of these studies have concluded, seemingly more frequently than not, that many if not most of what are considered alternative therapies and treatments are no better than a placebo. Now, I’m sure that when some of these therapies are examined under the microscope of academic rigor they are correctly revealed to be bogus. But, I would also propose that there may exist fundamental complexities inherent to some of these alternative treatments that cannot easily be examined within the confines of a typical study design used in modern day clinical research.

In other words, maybe this stuff can’t be studied the same way we study other medical treatments?… or maybe we are studying them wrongly?

Questioning how we conduct scientific studies in this day and age is a very slippery slope. In a world with Trumpian “alternative facts,” it can be a dangerous game to explore the limitations of the scientific method. Some fool on the TV news can quote it as evidence for any thought that comes to mind. It’s also a major cop out to say “Oh this treatment can’t be studied by science.” If somebody said that to me about a particular treatment, I would first ask… what are they selling and for how much? And then, if they were not looking to turn a profit, I would ask… what are they smoking and how much?

I’m now the old timer

So how did I come to transition from journal-happy medical student to an old timer who has has fundamental doubts about many aspects of Evidence Based Medicine, routinely questioning conclusions made by even the best studies in the most prestigious journals? My evolution came from years of experience working in an Intensive Care Unit (ICU) with extremely complicated patients where my experience didn’t always jibe with what the journals told me should be true. When things get more complicated, we discover treatment is never as black and white as detailed in the “discussion section” of a journal article.

Not all medical studies are treated equal.

Some are case reports of some weird clinical situation (mostly allowing a resident a paid vacation to a national conference to present her experience and then go out and get free dinners from drug reps).

Some are retrospective studies- examining data on events that have already occurred. These studies always end with a quote detailing “further study is warranted and no firm conclusions can be made from this data.” These are the studies that often make mainstream media headlines because they show potential associations or risk factors for certain diseases that seem interesting and could be important (like with more cell phone use, brain cancer has gone up), but practitioners pretty much ignore when it comes to actually treating patients because they are simply often straight up wrong (vaccines potentially causing autism- they do not).

The “Gold Standard” of clinical medical studies, the design used when a top notch researcher is trying to get as close to an conclusive answer to a question as possible, is the randomized controlled double blinded study. It is the most rigorous way to do research in clinical medicine. But they are not easy or cheap to conduct.

So what is a randomized controlled double blinded study? That’s a lot of words.

Here is the quick and simple explanation in five easy steps:

  1. Patients are assessed for certain inclusion criteria- in other words are they are eligible to join the study. Maybe the researchers only want people of certain age or gender, or people that have a certain disease, or to not have a confounding disease. If you fit the criteria, you can move to step 2- if not- well you can go home.
  2. These accepted patients are randomized in a “Control group” or an “Intervention group.” Think of it this way: a coin is flipped and the patient is randomly placed into one of the two groups. Half will get the intervention, half will not. At the beginning of the study, the researchers know they will need a certain number of people- could be 50, 500 or 5000. (They got ways to calculate how many people are needed to see a difference between groups if it exists.) At the end of randomization- half are in one group, half in other.
  3. The researchers and the patients are blinded to which group they are in. If they are getting a drug for instance- one group will get a real drug and the other will get a placebo (or inactive drug) and neither the patient nor the person administering it will know which is which. Both groups would could get a funny looking purple bag labeled “Study Drug”
  4. The results are collected (for example if the medication is supposed to cure baldness, detailed hair growth data will be obtained, along with a tabulation of many potential side effects and overall mortality). The people collecting the data don’t know if the patient got the real drug or the placebo. (They could probably guess though if one group had flowing Fabio inspired manes and the other group looked like a gaggle of men at a Mr Clean themed Halloween party)
  5. The data is analyzed and a statistical analysis is done to calculate the odds that the difference between the two groups occurred solely by chance alone (and not due to the intervention). Think about it- it is physically possible to flip a coin and come up heads 1 million times in a row, but the odds of that occurring by chance alone are infinitesimally low. It’s much more likely something is wrong with your coin (like a coin with heads on both sides). In science if the odds that a result happened solely by chance are less than 5%- the scientific world has decided, at that percentage, to call a study positive and the intervention is considered clinically important.

So why am talking about clinical medical study design?

Well, it may be useful to understand how clinical studies are performed so that when we read the newly touted journal article proclaiming that “a randomized controlled study shows that covering your head in magnetic mud does not cure baldness better than does placebo mud’ that we don’t jump to conclusions.

We don’t assume that this one study is the end all be all to the question. It’s not as if Moses has come down from high and etched it on his tablets as the eleventh commandment.

However, we can’t dismiss the results of our magnetic mud study either. Science is a process, a long tedious process. Often certain knowledge is considered fact, and then without warning the next day our knowledge base is expanded and what we once knew as fact is now history. (Anyone remember the planet Pluto?).

This constant flux does not demean science in any way. Our scientific knowledge is a repository of probabilities, and with each additional study, we either increase or decrease the probability that our paradigm of understanding is correct- especially in clinical medicine.

One other point…

If you noticed, in a randomized controlled study we tested the intervention against a placebo.

What exactly is the placebo effect?

It’s probably one of the most powerful tools in medicine and also the most underutilized. Placebos do not have any active substance that could cause a beneficial or negative effect, thus the effect seen is solely from the patients belief that it works. It is possible that an expectation of an effect, itself causes the effect. And the effect can be as real as if you treated it with a medication. It’s really really powerful.

Occasionally while administering anesthesia, I wake some one up and they are in pain from their operation (I guessed wrong on how much pain meds to give while they were asleep). Well if I’ve run out of all the opioid pain meds in my cart and to get more I have to go to a secure pharmacy are down the hallway in the recovery room, I have a little trick I use. I grab a syringe of saline, show the patient, and straight up lie to them: “I just gave you some more pain meds! It’s going to make you feel better.” You would be surprised how often that works. Am I lying? Well sort of, but I like to think I’m injecting them with a healthy dose of placebo- and it often does make them feel better.

If you think about the placebo effect for two seconds, it’s kind of mind boggling. It’s also a bit nuts that their aren’t countless podcasts and think tanks dedicated to exploring its use. I mean it’s the mind’s expectation causing a tangible physical effect. Makes you think about that kid in college who went to his first pot brownie party, ate one down, and then started acting all high and running around telling everyone how awesome weed is. Then his buddies revealed they were just normal brownies with no marijuana in them at all and he became an embarrassed laughing stock. Well maybe…, if you take into account the power of placebo- he was actually feeling high and it was awesome. It was weed placebo! Redemption my embarrassed college friend.

Could it be that the placebo effect is one of the reasons that make so many people strongly believe that alternative treatments work?

Or maybe more precisely, could the placebo effect be the reason why so many alternative treatments do work?