Technology is imperfect. Anyone who’s ever used a computer knows this. You search for a file that you know to be on your hard drive and somehow the search function just cannot find it. If computers are flawed, is it any wonder that diagnostic and screening tests are too?
There is a lot of hope attached to COVID-19 serological tests. “Serological” means that it relates to the serum, the liquid component of our blood that has been purged of clotting factors but that contains proteins like antibodies: it is these antibodies that are being earmarked for our salvation. Antibodies are produced by a subset of our blood cells in response to an infection. If you have antibodies in your blood against the new coronavirus, that must mean you had the virus inside your body and managed to fight it off. The presence of these antibodies infers a protection against becoming sick a second time from this coronavirus (although it is not yet guaranteed, but small studies in chimps are encouraging), and being able to tell whose body is now “shielded” from COVID-19 from those who are immunologically naïve could prove critical to public health authorities.
However, tests are technological in nature, and technology is imperfect. Here’s how a COVID-19 antibody test result could be wrong.
A new gold rush
The first limitation of antibody tests is that they are dependent on our immune system. You can’t detect something that isn’t there, and our blood cells do not start making antibodies against the coronavirus as soon as we become infected. In fact, it takes between two and three weeks after the start of symptoms before enough antibodies are made that they can be detected (though in some cases it can be as short as four days). Someone in their first week of infection would most likely get a negative result from an antibody test: that’s why these particular tests, unlike the diagnostic tests that use a long Q-tip to collect the virus and detect its genetic material, are best used to identify people who have had the virus but don’t anymore. It’s also important to point out that not everyone who encounters the virus will develop antibodies against it or enough antibodies that can be detected by the test.
But another big problem is incompetence. The universal need for these antibody kits has ushered in a new gold rush, and just like an inexperienced buyer may be fooled into believing pyrite is gold, unqualified companies are developing tests of a dubious accuracy to cash in on the demand. Some of the COVID-19 antibody kits on the market were manufactured by companies with no experience producing medical equipment (one such company usually sells Chinese herbal remedies while another sells solar panels and tires), and independent testing revealed that their accuracy was no better than flipping a coin (and a coin is after all much cheaper). Health Canada has been very cautious, only authorizing two kits at the time of writing. The United States, however, opened the floodgates for companies wanting to sell their serological tests, telling them they did not need to go through an onerous approval process: simply let the FDA know what your kit can and cannot do and we will allow you to sell it. They have since corrected their course and voiced concerns about fraudulent kits. Pyrite, after all, can’t do much harm but a false sense of immunity in the middle of a pandemic can be deadly.
A final issue with these antibody kits--and with any diagnostic or screening test--is that they are never 100% sensitive and 100% specific. Using the analogy of a home pregnancy test, you want the test to tell you that you are not pregnant only if you are not pregnant, and to tell you that you are pregnant only if you are. You want all of the positive test results to be true positives and all of the negative test results to be true negatives. But you may at the same time ask for world peace because these ideals simply cannot be attained in the real world. For example, a kit could tell you that you have antibodies against the coronavirus but it turns out it mistakenly detected your antibodies against a different type of coronavirus, like one of the many that cause the common cold. Health Canada asks kit manufacturers to specifically test for this and make sure it doesn’t happen, but these pre-approval tests are limited in scope and technology is fallible so it remains a possibility.
Even if an antibody testing kit has a very high sensitivity (meaning it will detect antibodies if they are there) and a very high specificity (meaning it will yield a negative result when antibodies are not there), there is still one more issue to tackle and it involves some eye-opening mathematics.
One number changes everything
Home pregnancy tests are not unerring but they are pretty reliable, and in part it’s because the people who use them have a higher chance of being pregnant. They often use them because they have had sex and their period hasn’t happened, so they suspect they might be pregnant. But if you gave this test to every person capable of bearing a child on the face of the Earth, the accuracy of the test would decrease. And it boils down to an important concept: how common the thing you are trying to find is in the population you are testing.
Let’s take one of the two COVID-19 antibody kits approved by Health Canada, with a sensitivity of 97.4% (it will almost always tell you you have antibodies against the coronavirus when you do) and a specificity of 98.9% (it will almost always tell you you do not have antibodies when you don’t). Sounds like a near-perfect kit.
Imagine a scenario in which only 1% of the Canadian population has antibodies against the coronavirus. If every Canadian’s blood is tested for antibodies and you personally receive a positive result (meaning you are told you have antibodies and may now be immune), your result has a 47.2% chance of being accurate (a coin toss). But if 5% of the population has these antibodies and you test positive, your result has an 82.3% chance of being correct. As the percentage of the population that is exposed to the coronavirus and that develops antibodies against it grows, from 10% to 25% to 50%, the chances that your positive result is real similarly swell, from 90.8% to 96.7% to 98.9%. The more people have antibodies against the coronavirus, the likelier the odds that your positive result from this one test is accurate. A negative result’s accuracy is similarly influenced by this phenomenon.
Even with a robust kit, if the thing you are looking for is rare, the results you get will tend toward being unreliable. By that logic, being randomly tested with an antibody kit at the beginning of a pandemic will produce results that will not be as dependable as those generated toward the end of a pandemic. This is why the Centers for Disease Control in the U.S. have recently acknowledged that using antibody tests--even reliable ones--in populations where very few people are thought to have been infected with the coronavirus will yield inaccurate results.
This is also why mass screening for cancer is not as beneficial as it might sound: the more people without cancer you screen, the more false positive results you will generate, which will lead to anxiety and more invasive testing. As COVID-19 serological tests get deployed to screen the population, we need to keep in mind the many ways in which they may err. They have their place, but as imperfect technologies, they have their flaws, both in their construction and their application.
For those who want to delve deeper into “How to calculate the chances of a positive result from a test being accurate“, click here
This will be particularly useful to science and medical students who are curious about how a test’s sensitivity and specificity do not tell the whole story. How did I calculate the percentage chance of a positive result being accurate? That percentage is known as the “positive predictive value” (or PPV), but a big caveat: you need to know how common (or prevalent) the thing being tested is in the population in which the test is being used. The positive predictive value will not tell you the odds that your positive test result is a true positive in real life; what it can do is tell you how reliable the test would be in different scenarios and thus help you decide if using this test in a specific population would be wise.
The PPV is simply the number of true positives who test positive divided by the total number of individuals who test positive, times one hundred to convert it to a percentage.
To calculate the first variable (how many true positives test positive), you need to multiply the sensitivity of the test (a 90% sensitivity would be entered as 0.90) with the fraction of the population that is truly positive (or that is known to be positive by using a gold-standard test) and with the total number of people being tested. A similar number called the “negative predictive value” (or NPV) can be calculated to learn the odds of a negative test result being accurate in a particular scenario.
You can see an example of these calculations here, where the use of tables may clarify the math.
Take-home message:
- Antibody testing for COVID-19 is useful but not perfect
- Some people who are infected by the new coronavirus do not develop antibodies, and even those who do typically start producing them a few weeks into the illness
- Many antibody kit manufacturers have no experience making these types of tests and are using this medical “gold rush” to make a quick buck
-Every test will have a percentage of false positive and false negative results, but even when these percentages are tiny, a test’s positive results will be more and more accurate as the thing it detects is more and more common in the population
