Skip to main content

Why most published research findings are false, and why you usually can't read them anyway--the pioneering work of Dr. John Ioannidis

In 2005 Dr. John Ioannidis, a Greek researcher and professor of health policy at Stanford University, best known for his critiques of the science of medicine, published a paper entitled "Why Most Published Research Findings are False." This was not from the point of view of a science denier--actually closer to the opposite. Ioannidis loves good science, but points out that the vast majority of scientific studies today are biased, often asking the wrong questions and making the wrong inferences. In the case of medicine, this often means that claims of the effectiveness of a treatment or diagnostic test are exaggerated and often just plain wrong. This stems partly from the fact that positive and exciting results lead to further funding for the researcher involved and that the sources of this funding are often entities such as drug companies that stand to benefit from a certain outcome.

Recently Dr. Ioannidis published a new article, much more accessible than the first, entitled "Evidence Based Medicine Has Been Hijacked: A Letter to Dr. David Sackett." The first was very much based on math and statistics. He observed that most studies, when repeated, came up with different results. This was particularly true of studies with smaller numbers of subjects and ones where the effect sizes were small. Such studies were more likely to come out of fields in which there was money to be made out of a positive result and ones in which the field of study was particularly hot and there fore several groups were competing to get results.

The second and most recent article is a conversation with one of Ioannidis' most important mentors, a man named David Sackett who was possibly the first person to introduce the concept of evidence-based medicine. By this he meant combining understanding of science and research with clinical judgment and experience. This idea was inspiring to John Ioannidis and his relationship with David Sackett (physician and founder of the Center for Evidence-Based medicine at Oxford University) was profoundly influential in his career. David Sackett died in May of 2015. He was apparently not only a wonderful clinical teacher but a great and appreciative listener. Dr. Ioannidis has been explaining his hopes and frustrations to the David Sackett who remains very much alive in his mind, and in this article Dr. Ioannidis shares with his internal Dr. Sackett his frustration with what has become of evidence based medicine. It is a delightful article and well worth a read. In it he laments the growing body of crappy and biased research upon which much of our advice to patients is now built.

This article is important for all practicing physicians to read and yet, when I tried to find it, the journal in which it was published asked that I part with around $32 to see it. This felt a bit ironic. The article by the man who champions truth and transparency was guarded by trolls who wanted $32 a pop. But then, when I checked it a few days later, it became free, and if you click on the link above, you will be able to read it. I'm not sure there is a moral to this part of the story, but I'm guessing that the irony was noted by Dr. Ioannidis who told the journal editors that they could do whatever they wanted with the rest of the content of their issues, but they could jolly well make his article available for free. Still, in addition to the bias present in medical studies, lack of free access to the original articles further dilutes any truth to be found in them. Any scientific study that is likely to be "click bait"--that is to say interesting enough to readers that they will click on a link to read more about it--is described in the secondary literature by a journalist who strips it of any actual detail and spins it in any way that will engender further clicking behaviors. I venture to say that the vast majority of learning about clinical research by practicing physicians is through articles written about articles. These are produced by companies such as Medpage Today whose entire mission is to make money through advertising based on the number of times we click on their headline news. Their articles on articles appear to us to be a vital service, though, because most research articles are not free to us in their entirety and keeping up on the breadth of medical knowledge by subscribing to a vast number of journals is neither efficient nor affordable.

These are fascinating things to think about. My present distilled words of wisdom are:
1. Read Ioannidis' article while it's still free, before the journal changes its mind.
2. Don't take what passes for science too terribly seriously, especially if the effect is small or it goes against common sense and what you know about human physiology.
3. Really don't base your practice off of news releases about articles you haven't read or thought about.
4. Agitate for free and open access to important scientific research so you can read it critically for yourself.

Comments

Popular posts from this blog

How to make your own ultrasound gel (which is also sterile and edible and environmentally friendly) **UPDATED--NEW RECIPE**

I have been doing lots of bedside ultrasound lately and realized how useful it would be in areas far off the beaten track like Haiti, for instance. With a bedside ultrasound (mine fits in my pocket) I could diagnose heart disease, kidney and gallbladder problems, various cancers as well as lung and intestinal diseases. Then I realized that I would have to take a whole bunch of ultrasound gel with me which would mean that I would have to check luggage, which is a real pain when traveling light to a place where luggage disappears. I heard that you can use water, or spit, in a pinch, or even lotion, though oil based coupling media apparently break down the surface of the transducer. Or, of course, you can just use ultrasound gel. Ultrasound requires an aqueous interface between the transducer and the skin or else all you see is black. Ultrasound gel is a clear goo, looks like hair gel or aloe vera, and is made by several companies out of various combinations of propylene glycol, glyce

Ivermectin for Covid--Does it work? We don't know.

  Lately there has been quite a heated controversy about whether to use ivermectin for Covid-19.  The FDA , a US federal agency responsible for providing unbiased information to protect people from harmful drugs, foods, even tobacco products, has said that there is not good evidence of ivermectin's safety and effectiveness in treating Covid 19, and that just about sums up what we truly know about ivermectin in the context of Covid. The CDC, Centers for Disease Control, a branch of the department of Health and Human Services, tasked with preventing and treating disease and injury, also recently warned  people not to use ivermectin to treat Covid outside of actual clinical trials. Certain highly qualified physicians, including ones who practice critical care medicine and manage many patients with severe Covid infections in the intensive care unit vocally support the use of ivermectin to treat Covid and have published dosing schedules and reviews of the literature supporting it for tr

Old Fangak, South Sudan--Bedside Ultrasound and other stuff

I just got back from a couple of weeks in Old Fangak, a community of people living by the Zaraf River in South Sudan. It's normally a small community, with an open market and people who live by raising cows, trading on the river, fishing and gardening. Now there are tens of thousands of people there, still displaced from their homes by the civil war which has gone on intermittently for decades. There are even more people now than there were last year. There is a hospital in Old Fangak, which is run by Jill Seaman, one of the founders of Sudan Medical relief and a fierce advocate for treatment of various horrible and neglected tropical diseases, along with some very skilled and committed local clinical officers and nurses and a contingent of doctors, nurses and support staff from Medecins Sans Frontieres (Doctors Without Borders, also known as MSF) who have been helping out for a little over a year. The hospital attempts to do a lot with a little, and treats all who present ther