It has been a little over a year since the World Health Organization declared the COVID-19 outbreak a global pandemic. Medical scientists quickly started testing a barrage of potential therapies against the disease, which gave hope that new treatments might quickly be identified.
What may surprise many is that, for at least three trendy COVID-19 interventions—namely hydroxychloroquine, ivermectin and convalescent plasma—the quality of the evidence that has accumulated over 12 months of dedicated research is, for the most part, pretty low. If scientific studies of these interventions were fruits, we would be facing a large display of borderline rotten apples. There are some edible wonders in here, and COVID-19 research has produced a number of gems (for example, the vaccine clinical trials), but we have reasons to be, dare I say, disappointed.
So what is the quality of the evidence we now have for hydroxychloroquine, ivermectin, and convalescent plasma when it comes to COVID-19?
Hydroxychloroquine
Here is a drug that requires little introduction. Commonly known as Plaquenil and used against malaria and for autoimmune conditions like lupus, it shot to stardom when a French researcher became its very vocal cheerleader in March 2020. It then received a massive boost when Donald Trump called it a “game changer” a few days later. Here is what the best studies have actually shown.
The Cochrane Collaboration looked at the published evidence as of September 15, 2020, which comprised 12 clinical trials involving 8,569 participants. With high certainty, the authors of this review concluded that hydroxychloroquine did not decrease the number of deaths from COVID-19 when used as a treatment. With moderate certainty, they noted that it did not alter the odds of a patient requiring mechanical ventilation, but that it did lead to three times more risk of negative side effects. It’s important to point out that hydroxychloroquine is not innocuous: it can, in some people, impact the heart, the eyes and the central nervous system, and it is easy to take a toxic dose of it because its therapeutic window is so narrow.
The review goes on to state that no results had been reported on the use of hydroxychloroquine to prevent COVID-19 in people who had not been exposed to the virus. When looking at what little evidence there was for its use in preventing the onset of the disease in people who had been exposed to the virus, hydroxychloroquine did not seem to help. The authors concluded that no further trials of hydroxychloroquine for the treatment of COVID-19 should be done.
Many studies of hydroxychloroquine have been published, but as was pointed out over and over by scientists, a large percentage of them are not rigorous. They have too few patients; the way in which these patients were randomized to hydroxychloroquine or to a control arm is either unclear or poorly done; other treatments were administered which makes it hard to know what the effect of hydroxychloroquine really was; and sometimes, doctors deviated from protocol and gave hydroxychloroquine to patients who were not supposed to receive it or vice versa. These studies are not informative.
Despite loud political voices promoting hydroxychloroquine as our champion, the best evidence we have shows it is ineffective as a treatment for COVID-19, and while it can’t completely be excluded as protection for people exposed to the virus, it is unlikely to help there either.
Ivermectin
Ivermectin is a drug that has been used in humans since 1981 and which features on the World Health Organization’s list of essential medicines. It is naturally produced by a type of bacteria that grow in the soil. It was isolated from these bacteria by scientists working at Merck in the 1970s: they discovered that the family of compounds ivermectin belonged to could cure mice of worm infections, and these scientists would go on to receive the 2015 Nobel Prize in Physiology or Medicine for their discovery. Ivermectin is an effective medication against various parasites because at low concentrations it binds strongly to important components of nerves and muscles that worms have but that we, as vertebrates, don’t have.
The coronavirus does not have nerves and muscles that ivermectin can mess with, but scientists have discovered that the drug does possess some antiviral activity in the laboratory, meaning that it can interfere with a virus’ replication in cells. This was seen, for example, with dengue which, like COVID-19, is caused by a single-stranded RNA virus, so Australian scientists decided to test ivermectin against the novel coronavirus in the lab. Their paper, which reported that ivermectin could indeed interfere with the coronavirus’ replication, opened the floodgates for people to seek out the popular anti-parasitic drug as either a treatment or a preventative measure against COVID-19, but there was a big catch: not only had the Australian scientists not tested ivermectin in humans, but the doses they had used in the lab were orders of magnitude above what is typically given to patients. This led to concerns that giving these massive doses to patients might cause harm. It was theoretically possible that giving ivermectin with some other drug might boost ivermectin’s effectiveness at a lower dose; that it could be inhaled and reach a higher concentration in the lungs; or that a lower dose would somehow still benefit patients with COVID-19. This needed testing.
To this day, what followed were, with rare exceptions, studies so flawed that they simply muddy the waters instead of providing convincing evidence either for or against the use of ivermectin for COVID-19. A pilot study reported on 16 patients who received the drug, a sample too small to conclude anything. A different study looked at 13 patients. A Bangladeshi study had 24 patients who received the drug and another 24 who received it in combination with an antibiotic, but the small number of patients was compounded by the fact that they were evaluating virus clearance, which requires a PCR test to confirm, and the test was only administered roughly once a week, making it hard to see anything shy of a large difference. In India, healthcare workers at one institute who were tested for the coronavirus during a one-month period were divided into those who tested positive and those who tested negative. Among both groups, the authors looked to see who had voluntarily been taking ivermectin as a preventative to see if they could find more in the negative group. This type of study, where individuals choose to take a drug or not, is prone to bias.
In fact, when an independent, not-for-profit Canadian organization called CADTH published a review of the evidence for ivermectin and COVID-19 in February of this year, the bottom line was that all the studies they found were at a high risk of bias. “Very low quality of evidence,” the authors wrote, “precludes the ability to draw any strong conclusions.”
Since then, a double-blind, randomized controlled trial was published in JAMA, and while it is not perfect (it involved a single site and its participants were otherwise young and healthy and had a mild form of the disease), it provided good evidence that ivermectin did not significantly improve the time it took for symptoms to go away.
As of now, there is no good evidence that ivermectin works at regular doses to treat COVID-19 in humans or to prevent it. Even Merck, the company that developed the drug, stated as much recently. It is not completely implausible that it would work, but there are colossal hurdles to overcome to show that it is both safe and effective against the disease. There are positive studies (in fact the website ivmmeta.com claims to be “a real-time meta-analysis of 45 studies” showing the awesomeness of ivermectin against COVID-19 to the point where they twist the JAMA trial into a positive trial somehow!), but assessing studies is not merely about collecting them and gasping at how shiny they are. It’s about properly evaluating their worth. Otherwise, you end up with a lot of pyrite in your bucket of gold.
Convalescent plasma
Finally, there is convalescent plasma. This is when people with COVID-19 are given an immune booster in the form of the antibodies made by people who survived the disease in order to get better faster and avoid death, in theory. Well over a hundred studies of convalescent plasma for COVID-19 were begun in 2020, so you would be forgiven for thinking that we now know for a fact if the technique is worth it or not.
The Cochrane Collaboration publishes an ever-evolving review of the evidence for this potential treatment. The latest edition, released in October of last year, looked at two randomized clinical trials on this issue as well as more than a dozen non-randomized studies that had been completed by then. Unfortunately, the authors had to conclude that their confidence in these studies was low or very low. For example, they pointed out that many patients were receiving all sorts of other treatments alongside the plasma in these studies. This “everything but the kitchen sink” approach is understandable when trying to keep a patient alive in the absence of good evidence but it does not make for a great learning experience.
CADTH also keeps a living review of the evidence, and their more recent update published earlier this month looked at more studies of convalescent plasma. Their conclusion makes much use of the words “unclear” and “inconclusive,” as the studies tend to be of low or moderate quality, not always well reported, and not always comparable to one another. Some of the trials had to end prematurely due to a lack of participants, which means that it is harder to detect whether or not the plasma really helped.
A large Canadian trial on this issue was recently stopped when an interim analysis showed that it was unlikely to show benefit for the plasma even if more participants joined up. Meanwhile, the largest clinical trial of convalescent plasma for the treatment of an infection, RECOVERY, recently published its results on a pre-print server (they have yet to undergo formal review by other scientists). With nearly 6,000 COVID-19 patients receiving standard of care and another near-6,000 additionally receiving convalescent plasma containing high levels of coronavirus-fighting antibodies, the trial came out negative. No difference in deaths, no difference in time to hospital discharge.
The book has closed on hydroxychloroquine. The page is turning on convalescent plasma. For ivermectin, there are too many uncertainties to cleanly rule one way or the other, although knowing how infrequently in vitro benefits translate to a safe and effective drug in humans does make me pessimistic.
Not all studies are created equal
What can we learn, a year into the pandemic, from our clumsy repeated attempts at answering the question of “does this therapeutic intervention help with COVID-19 or not?” One lesson is that what we saw was simply a sped-up, widely publicized display of scientific research with its current baggage of warts, inefficiencies, and triumphs. Individual studies are almost always flawed. Sometimes it’s a microscopic crack and sometimes it’s enough to break the whole thing. Cracks can be due to inattention, corner-cutting, a lack of money, short-sightedness, a lack of expertise, the inherent difficulty of juggling a study with thousands of participants, and just plain reality gumming up the works of a scientist’s aspirations. In the case of COVID-19, the desire of medical doctors to save lives by trying anything did not always mesh well with science’s need for rigour. The academic literature thus ended up littered with unusable studies that could be picked up and unfurled by opinionated defenders.
When the next pandemic strikes and the biomedical world turns to the large bag of potential therapies we have accumulated, I hope we see better and bigger collaborations, more standardization in how studies are conducted and reported, and a more focused attention on rigour. Bad studies are not harmless; they create noise that can be interpreted to justify therapies that do more harm than good.
Take-home message:
- A year into the COVID-19 pandemic, there is reliable evidence that hydroxychloroquine does not help in the treatment of the disease
- Despite many poor studies of convalescent plasma, we are starting to see that the intervention probably does not work to treat COVID-19
- Most of the studies of ivermectin for COVID-19 are too poorly done to conclude anything, but based on a laboratory study that used extremely high doses to show an effect and on how infrequently experiments in Petri dishes lead to usable drugs in humans, ivermectin is unlikely to become an efficacious treatment for COVID-19
