AS A BLOSSOMING lockdown sceptic, I developed a big problem in the summer, which has tortured me since – this does not feel like an exaggeration. It must have been the 14th July I had my first False Positive Moment because I remember seeing 3 cases had been reported in Scotland on that particular day. I remembered it was around 5,000 tests had been reported – now I have checked, I see the precise number was 4908.
So on that day there was a positivity rate of 0.06 per cent.
I remember thinking, ‘That’s an astonishingly low false positive rate!’ and the thought almost came at me by surprise. I had an innate knowledge of false positives I had almost forgotten.
I remembered from my Physics and Medical Physics training about how all tests have a false positive rate. They also have a false negative rate. All false results have to be corrected for when looking at a body of results. ‘I wonder how they are correcting for that and whether those poor three people have to be put under self-isolation falsely too?’
I think back to those thoughts now astonished at my own naivety then.
During a time of likely almost-zero Covid in the summer, to have known the false positive rate would have been to expose how little Covid there was (obviously), and also taught us about the seasonality of this new virus. We could have learned much, and saved many people unnecessary isolation. Throughout our management of this pandemic, huge decisions have been made to isolate hospital staff and segregate patients into Covid wards and take whole classes of children out of school and so on, on the basis of a single test with an unknown false results rate. The problems only multiply as time progresses.
What is a false positive? It’s a result that comes back positive in error. Every test has a false positive rate. It’s just mathematically impossible for any test not to have false results returned because where humans and biology are involved, errors and unexpected events happen. Biology can be unpredictable. A mistake might be made at any and every stage of the testing process – and tests themselves are human-made and therefore subject to human limitations. It’s just Mathematical Law that there will always be an error rate in testing and the errors only multiply if tests are done on a large scale.
On the way up an infection curve in the beginning of an epidemic, the false negative rate is of significant concern because there are truly infected people being given a ‘not infected’ label. This has implications for treatment and infectious spread.
On the way out of an epidemic, as the infection curve falls, the false positive rate is the bigger concern because if not corrected, you will never escape epidemic mode, and also, as we will see later, the problem of false positives becomes greater as prevalence falls. In the case of Covid, you will be denying people their rights, liberty and human contact on the basis of an assumption that they are infected and infectious, when in fact neither is true.
We all know that false positives are a phenomenon in all medical tests. Your doctor will never diagnose anything serious (or not serious) on the basis of a single test result – not for cervical cancer, nor breast cancer, nor prostate cancer, nor bowel cancer, nor flu, nor any virus, nor any other condition infectious or otherwise – especially if you are without symptoms. And yet, we have been making hugely impactful decisions about people’s lives – and medical treatment– on the basis of a single positive Covid test result, very often in the absence of relevant, or indeed any, symptoms.
If you have suggestive symptoms and a positive Covid test, okay – you might have Covid. The problem is that Covid symptoms are shared with many other viruses – so you might not. If you are asymptomatic with a positive test – well, you are more likely to have a false positive. You’re being told – and treated as if – you have Covid, but you almost certainly do not have Covid. That’s it.
Despite the absolutely bizarre and highly successful efforts of a few scientists to make us believe there are no false positives, there just are, unless we somehow left the laws of Mathematics behind in 2020. Maybe I am indeed in a parallel Universe?
There are in fact many complex reasons why false results come out of a test, not the least of which is contamination, which can be a very important contributor for a test as sensitive as a PCR test. And the false positive rate may change in time and there will be statistical variations day-to-day too – ‘noise’ will always contribute to a data set. But I am not going to get in to that here. I’m just going to demonstrate Bayes Theorem to you, which is part of Probability Theory.
It describes the probability of an event based on prior knowledge.
There are some interesting lessons for life in Bayes Theorem too: if you have an unfounded belief, you will always occasionally stumble across evidence that supports it. If you understand Bayes Theorem, you understand the importance of the scientific method, which is that if you wish to prove a hypothesis, you should always first seek to disprove it.
Bayes Theorem also shows how two apparently contradictory things can be true. The Covid testing strategy and infrastructure implemented by the UK Government was brilliant, exemplary and inspiring at the start of the spring epidemic. The same testing strategy now is wildly inappropriate.
First, let’s take a look at two very simple diagrams, then I’ll take you through the theorem in the simplest way I can find.
The Office for National Statistics has stated that the prevalence of Covid in the week 23rd to 30th January in Scotland was 1 in 115. This is 0.87 per cent. Let’s make it a nice round 1 per cent.
We don’t have a known value stated for specificity of these tests. So let’s assume it’s a nice round 95 per cent for now.
What is ‘specificity’? It’s the chance that a person tested who is not infected will get a negative result. So 95 per cent specificity means that 95 per cent of those tested who are not infected will be correctly identified as being negative. It therefore means that 5 per cent of those tested who are not infected will test POSITIVE. This might seem insignificant to you – but you may well be surprised by the implications.
Let’s consider the circumstances in Scotland now. There is a viral prevalence of 1 per cent; 1 person in 100 is infected. Imagine 100 people in a room (impossible I know) – that red dot in the diagram below is the infected person.
Now let’s consider what the test makes the picture look like. There are still 99 uninfected people in that room. But a test of 95 per cent specificity will only correctly identify 95 per cent of them – let’s round 95 per cent of 99 people to: 95 people.
The test will (most likely) identify the infected person, but it will identify four other unlucky people as being infected who are not. In the post-2020 era, these people are labelled ‘asymptomatic’ and put under what is effectively house arrest or (worse) on to a Covid ward etc, when in fact a second test would be enough to confirm beyond reasonable doubt whether the person was truly infected. You’ll note the group of labelled people is 5 in number when 1 is in fact truly infected.
This is an over-simplification – because it does not take sensitivity into account nor prior probability of infection – see below. But it makes the point that it is not as simple as: a positive test = a case, especially in low prevalence. Imagine prevalence was 5 per cent. Then we would have more true positives but still around half of the positives identified would be false.
The implication of this basic arithmetic remains: that it is vitally important to establish false positive and negative rates – especially if you are testing people in the absence of any symptoms. And this is why mass screening of a population should always be conducted with the utmost caution and care – and confirmatory tests should be the standard protocol for contentious results, i.e. a positive without symptoms, or a negative with symptoms.
This is not about ‘accuracy’ or how good the test is. This is just arithmetic, which can be counter-intuitive, but is nevertheless the basic language describing the mathematical laws we all live under – and can never escape from. You may not attempt to argue them away. Mathematical law doesn’t work like that.
Let’s have a good wander through the numbers, just to show you I’m not ‘havering’! What follows is the proof of what I’m saying. I know it’s a lot of arithmetic – sorry. I’d also like to stress these examples are meant to be illustrative and to show the reader why mass testing can be so complex. I am not necessarily providing correct values for the variables in question.
Bayes Theorem explained……
What is the probability you have Covid and a positive test? Let’s call this ‘Chance your +ve test is right’! (+ve = ‘positive’)
I think, even if you’re extremely uncomfortable with arithmetic generally, this equation is fairly intuitive and believable – if you tilt your head and squint. Or just take my word for it because this is Bayes’ Theorem and you can look it up. Let’s define the terms.
So, now we can rewrite the above equation, having defined these concepts.
But what is the Chance of getting a positive?
This is slightly more complicated to define. Remember, there are two types of positives: true and false. You have a chance of getting a true positive test and a chance of getting a false positive – so the Chance of getting a +ve is the sum of those two chances.
So, now let’s take some values and put them in our equation.
In ‘Interpreting a Covid-19 test result’the PCR test was quoted as having a sensitivity of 71-98 per cent [REF: BMJ 2020;369:m1808 doi: 10.1136/bmj.m1808 (Published 12 May 2020)] .
I’m going to start with 98 per cent sensitivity, 95 per cent specificity and 1 per cent prevalence
This means 83 per cent of all positives are false.
Let’s make specificity really high and sensitivity a little lower.
This means 51 per cent of all positives are false – and that’s with a very high specificity rate!
Let’s use some different values again and make sensitivity a bit lower.
This means 86 per cent of all positives are false – and you can also see sensitivity (PCR’s strong point) doesn’t make a huge difference to the reliability of the results in low prevalence.
Now, you might be thinking that this seems completely hopeless then! How can there be so many false positives for a test with such good sensitivity and specificity? And you are not wrong to question it. The problem lies in low prevalence – and the laws of arithmetic. The test itself, no matter how close to perfect it might be, will always be limited by Probability Theory.
Let’s try making the prevalence higher and see what happens then.
What if we only test the symptomatic? Then the predictive value of a test improves hugely. So imagine we test only in a group with symptoms and the prevalence in this group therefore goes to 10 per cent.
Now…
This means 32 per cent of all positives are false.
This is still a significant chunk of results that are false, but it is a vast improvement. Let’s put the prevalence of the tested group up to 20 per cent…
This means 17 per cent of all positives are false – a very much lower false positives rate. This shows why, at the start of an epidemic in spring, when prevalence is high and testing is more targeted, a mass testing strategy is more worthwhile.
You still need to correct for those 17 per cent of positives which are false, you would think, by re-testing. If a positive comes back negative after a confirmatory test, it is very safe to assume the positive was false. The point is that in testing, the priority must always be either to ensure that the group you are testing has a high prevalence of disease, or that the possibility of low prevalence is corrected for.
It is, frankly, astonishing that for so long we have been testing asymptomatic people (in whom there is no reason to believe they are infected) and not re-testing positives, although there are encouraging signs that this practice is being rectified by using a lateral flow test confirmed by PCR in some cases.
I am also going to leave you with a final chilling calculation. What if we had zero Covid? What if it only existed in trace amounts in our population?
Let’s use some different values – and make specificity really high!
This means 99.8 per cent of all positives are false – and that’s with really high specificity.
Any test – no matter how amazingly good – will always produce some positives (and negatives) in error. If the Scottish Government is indeed pursuing zero Covid cases (as it hints frequently and openly states occasionally) that goal can never be realised unless we start to use a much more intelligent approach to mass testing for a disease with low prevalence. Indeed, as we approach truly zero Covid, our false results rate will only get worse, as demonstrated above.
We can correct for all of this quite easily by: confirming a positive with more tests and/or limiting testing to those with symptoms.
I’d like to stress again there are signs that the testing strategy is improving – which may partly or fully explain the recent dramatic fall in positive tests being returned. And the lateral flow test, which is being used more now, does seem to be an improvement on PCR for many reasons. Unfortunately we are not getting the results of the lateral flow test made publicly available to us for analysis. But the public should be made aware of the limitations of testing in the asymptomatic and be permitted to be more sceptical about the results emerging from mass testing for Covid, especially when the implications of a positive test on individual rights can be so severe. Implications for hospital patients can be very serious if a positive test is returned.
It is astonishing that such poor practice of accepting the result of a single positive in the asymptomatic – and denying those with a positive test all right to personal liberty – has ever been permitted for any time at all.
If you enjoyed this article please share and follow us on Twitter here – and like and comment on facebook here.
Christine Padgham was a health physicist who now is analysing Scottish health trends following the Coronavirus crisis with the help of many other professionals, scientists and activists on the website InformScotland.uk – where she posts on a daily basis.
Image by lukasmilan from Pixabay
