The report of “a new study” in the media is often followed by a comment on social media pointing out that scientists can’t seem to agree on anything. One week, meat consumption should be reduced; the next, it’s perfectly fine. Why is it, the Internet is often asked, that scientists keep changing their minds? If they can’t agree on anything, maybe they shouldn’t be trusted on anything.
The good news is that trust in scientists is still very high overall (though there are topics where a disturbing gulf emerges between scientists and the lay public). But there are reasons why scientists appear to change their minds constantly on the important issues of our day. Let’s use the fictional example of whether or not eating bat wings will reduce your risk of developing cancer. Dr. Orlok is on the case.
1. The world is full of low-quality studies
Not all studies are created equal and the scientific literature is flooded with uninformative papers. If Dr. Orlok recruits ten people, puts them on a bat wing diet and only follows up on their cancer status for a year, he can still get published. Probably not in The New England Journal of Medicine, but given the existence of over 28,000 scientific journals, someone somewhere will publish this. How to recognize low-quality studies? They can lack a control group, they can have too few participants, their follow-up may be too short. These studies are easier to produce because they take less time to do and don’t cost too much. Sometimes, studies are given to Master’s and Ph.D. students who must graduate within a few years and must thus work on projects that are limited in time. But the conclusions that can be drawn from them are not reliable.
2. You can’t control for every variable
Dr. Orlok may follow people who eat bat wings and people who don’t for many years to see if more cancers arise in the latter group, and he may ensure that both groups have roughly the same number of smokers, as smoking is a known risk factor for cancer. But maybe he forgets to check on asbestos exposure in these groups, given that asbestos is also a risk factor for lung cancer. When you’re interested in finding out if eating bat wings specifically impacts your risk of developing cancer, you have to control for every other variable that influences cancer risk, and that can be quite a challenge. If you don’t, you may think you’ve found a scientific gem but better studies will prove you wrong, giving the illusion of flip-flopping on the part of scientists.
3. Scientists have freedoms that allow them to fudge rigour
Dr. Orlok doesn’t tell the scientific journal that will eventually publish his findings what he’s planning to do. This allows him to put his best foot forward. Maybe he keeps recruiting and testing participants until he gets the result he’s looking for and stops. Maybe he only puts the experiments that worked in his paper and leaves out the ones that didn’t or that gave ambiguous results, because they don’t tell a sexy enough story. Maybe he never publishes a particular project because the results were negative and scientific journals tend not to be interested in the publication of negative results. There are bad incentives in scientific research which can unfortunately nudge scientists away from diligence.
4. Exploratory research can be sold as solid knowledge
Maybe Dr. Orlok ran a study designed to look at the rates of colorectal cancer in bat-wing-eating individuals… but when the results were in, he spontaneously decided to also look at diabetes. Studies are designed around an outcome measure (e.g. death, a particular cancer rate, pulmonary function), but it’s possible for researchers to decide, once they have their data set, to mine it in search of something else that might shine. This is known as exploratory research, and while there’s nothing inherently wrong with it, it’s easy to present the results of exploratory research as definitive knowledge. Exploratory research can help us design future studies that will specifically tackle a potential association, to confirm whether or not it’s real, but they cannot be sold as conclusive (though they often are, leading to confusion).
5. Show me the money
Dr. Orlok’s bat-wing-eating study is funded by a major bat wing corporation that wants to sell more bat wing burgers to the public. Now it would be one thing for the corporation to give Dr. Orlok the money and go away, but its board members are involved with the study design and with the calculations done on the data. It doesn’t mean that the final paper is inaccurate, but there’s every reason to be skeptical. That being said, university researchers are being asked more and more to collaborate with industry partners, making financial independence from vested interests harder and harder to achieve. The “disclosure” section of a paper should reveal whether or not the funding source had a hand in analyzing the results. If it did, don’t be surprised if the findings agree with its business interests.
6. Some scientists commit fraud
Dr. Orlok is desperate for a good publication but the cancer data he gets in the lab is not very convincing. So he cheats and alters the images coming from his microscope. A potentially insipid story becomes a game-changer and he gets published in a major journal. Scientific misconduct—which includes plagiarism, ghostwriting, distorting findings or fabricating them from whole cloth—is rather rare but it can contribute to the apparent flip-flopping of science. One of the better known examples of it in medicine was Andrew Wakefield’s dishonest study into the measles-mumps-rubella vaccine and its false association with autism, in which Wakefield changed children’s diagnoses and the time of symptom onset to align his results with a particular narrative.
7. Valid findings get distorted by the hype machine
Maybe Dr. Orlok’s study did show that there was a slight decrease in rates of colorectal cancer among women who ate bat wings regularly. Since Dr. Orlok needs future funding for his research, he embellishes his conclusions in the paper. Then his university, which wants to show off on the international stage, further amplifies the strength and reach of his results in its press release. And finally a journalist picks up on the story and their coverage gets stuck with the headline: “Eating bat wings may open doors to curing cancer.” Hype is the inevitable outcome in a culture of click baits. This means that exaggeration is not rare. Also, many studies that make the news are done in cells or in rodents, which are good stepping scientific stones but rarely applicable in humans. These preclinical studies, which often can’t be replicated in humans, often shine in the media spotlight, while the mice themselves are left on the cutting room floor.
8. Scientific controversies can be manufactured…
Dr. Orlok may have the most rigorous data in the world showing that a bat-wing-rich diet will do nothing to protect you from cancer, but if “Dr.” Bathory, a naturopath, claims the opposite because “bat wings are natural”, the public may be confused. “Dr.” Bathory may even have done studies and published them in theJournal of Naturopathic Bat-Wingery, but she is not a serious scientist. She is starting from a conclusion she likes and cherry-picking the evidence that reinforces her belief. It is easy to blind the public with “science” and to make them believe there are two sides to an issue when in fact the science is clear. For example, the tobacco industry has acted like merchants of doubt, manufacturing a controversy (a “manufactroversy”, if you will) that cast doubt on reliable findings that smoking tobacco was an important risk factor for lung cancer. And that gives the appearance of scientific uncertainty.
9. … but there are genuine disagreements in science
When we remove all of the above, we are left with genuine disagreements in science that exist because science is hard and the universe is complex. Sometimes scientists find a weak effect and disagree over whether or not we should take it seriously. Other times, too few studies have been done to know for sure. Also, the tools we have to probe interesting questions about the universe around us may not be good enough to answer them properly, leading to debates and disagreements.
Research into what we should and shouldn’t eat certainly ticks all of the above boxes. Many studies are of poor quality, while others get selectively reported for maximum sexiness. Exploratory research is sold as near-divine revelations and hype predominates in the ensuing media cycle. And multiple interest groups get to distort the science and create manufactroversies.
In the end, science is a human enterprise that aspires to rigour. It gives us a set of tools we have been mastering and optimizing over many centuries to reduce our uncertainties about the world we live in and to make better and better predictions. The nauseating see-saw movement of science headlines may try our patience, but the answer is not to lose trust. It is to demand diligence and accept that its price will be nuance.
Take-home message:
- The idea that scientists keep changing their mind can sometimes be due to actual disagreements in a field of study
- Often though, it is the result of poorly done or selectively reported studies, of hype generated in the reporting of results, and sometimes of interest groups creating fake controversies to sow doubt
Want to comment on this article? Visit our fb page
