Can a Test be 100% Accurate? Examining the Evidence

[Merlin3189]

I read in this morning's newspaper, that we now have a test which was "found to be 100% accurate by experts".
My personal crap detector pinged. (If you were not around in the 60's, just say I was sceptical.)
Maybe they were not implying that the test IS 100% accurate, just that in their tests they found 0 failures. Even then, PCD is saying, then you'd better ask how many tests they did.

So, can one have a 100% accurate test? Or am I just a grumpy, old sceptic, sorry, denier?

 

[256bits]

I suppose the test is for the virus that is going around.
How do they know it is 100% accurate?
Do they mean accuracy on no false positive, no false negatives or both?

Someone wrote a statistical analysis on testing.
https://mrgilmartin.files.wordpress.com/2014/05/example-ia-medicine-testing.pdf

If you are a grumpy old denier then you are probably correct.

 

[Merlin3189]

My question was not so much about this test, but whether any sort of test can be 100%.

I suppose I'm thinking, first that biological experiments are often statistical, due to the large number of uncontrolable factors involved, and second that it is a measurement. Even in honest science like physics I take it as axiomatic that all measurements carry some error.

PS. I like the article. I went through this nearly half a century ago, but it is interesting to see it rehearsed again in a current context.

 

[256bits]

Say one becomes infected ( or not ).

(When) Can the test determine that the person is positive 5 minutes, an hour, a day, after the onset of infection?
Right there is a source of error I would think, as the viral load should be a major factor.

Problems arise, not in the test itself, but in the decisions made due to the test outcome.
Some people would be given the all clear, and experience joy for a while, until they develop symptoms, or are given another some time later test of verification that can determine the positive (negative ) outcome.

Computers logic chips do 100% tests all the time -ie the bit is on or it is off, so the 100% is/should be theoretically possible , even with more complicated systems I would think, maybe not, as you say because of the uncontrolable factors.

 

[Vanadium 50]

Computers logic chips do 100% tests all the time -ie the bit is on or it is off, so the 100% is/should be theoretically possible

Then why do they sell ECC memory?

 

[BillTre]

I read in this morning's newspaper, that we now have a test which was "found to be 100% accurate by experts".
My personal crap detector pinged. (If you were not around in the 60's, just say I was sceptical.)
Maybe they were not implying that the test IS 100% accurate, just that in their tests they found 0 failures. Even then, PCD is saying, then you'd better ask how many tests they did.

So, can one have a 100% accurate test? Or am I just a grumpy, old sceptic, sorry, denier?

I think you have to consider the source of the information in the newspaper.
Is it a quote or more like an opinion article?
Several "experts" that are currently being cited are not all that expert.
In addition, depending upon their employment (employed by companies that might profit in some way) and/or politically motivated motives (directed toward influencing public opinion), they may be "streaching" the truth.

As someone said in a thread around here somewhere, politicians like certainty in the advice they get from their "scientific experts". This makes any statements more useful in influencing public opinion.

This has resulted in a lot of, what at other times, might be comically stupid statements.

 

[256bits]

Then why do they sell ECC memory?

Yes. glitches do happen.
But the reference, not explained, and certainly not evident how I wrote it skimping detail, was to the threshold of activation of the electronic circuits, so while the circuits do 100% testing an error can pop up
EEC memory is self correcting, so f the "wrong" but is written and stored, the memory will still protect that wrong bit,

 

[collinsmark]

There's no real-world, physical test (as opposed to an idealized, hypothetical test) that is 100% accurate. I'm guessing the confusion might have come from the reporter rounding the accuracy up to 100, or maybe misinterpreting the accuracy due to limited sample size. A link to the study or even to the newspaper article might shed more light on what's missing.

-----

Regarding ECC memory: Even ECC memory can't correct just anything. The capabilities of how many bits an ECC memory module can correct/detect depends on the manufacturer's implementation, but whatever the case there are limits. ECC memory might be able to correct a single bit error, and maybe even detect if excactly two bits are in error (depending). But if several bits are in error the errors can sail through undetected.

Error correction schemes can and [usually] do reduce the probability of bit error, p_e (or bit error rate, BER), but don't eliminate it entirely.

 

[hutchphd]

My personal crap detector pinged

Golly, I hope you can turn that thing off! Otherwise you will be totally unable to function.

As a former designer of medical test instrumentation, I can state authoritatively: none of the CLIA standards require 0% error. (But we, in the USA, do have "the best testing" ...!)

 

[Jameson]

I work in statistical and machine learning modeling. Accuracy is a very bad metric to use because it can be so misleading so easily. Example: you have a company and fraud occurs very rarely but when it does happen it's bad and you want to stop it or try your best. So let's say historically it's happened 0.1% of all situations. In this case, if your "model" was to guess nothing is fraud is would be 99.9% accurate. Sounds great right?! No coding required. Just guess "no fraud" all the time.

The take away I have is if anyone claims 100% accuracy with anything and doesn't explain that this is unusual or likely impossible, they are not someone I would take seriously. Theoretically something that is truly deterministic could have this quality but we don't know that anything is or we can't measure it well enough to know all the pieces. Even the most fundamental seeming things have odd outcomes.

Most students cover these concepts in basic stats or similar courses when they cover false-positive, false-negative, etc. and how they tie into accuracy. The public at large does not want to hear these terms I think from my experience.

 

[sysprog]

Apparently Roche has not opened the specifics of the processes in question to peer review, as the allegedly include trade secrets.

It is reported, in some of the less non-rigorous popular venues, that Roche is, with some caveats and provisos, claiming 98.8% specificity for its test for the presence of antibodies specific to the SARS-CoV-2 virus that causes COVID-19, but is claiming no more than 70% accuracy, for its antibody test.

That means that Roche claims that it is finding no more than 0.2% false positives, and that it presumably conjectures up to 30% false negatives.

Professor Carl Heneghan, from the University of Oxford, said:

Without seeing the study methods and the data it's impossible to verify these claims of accuracy.​

 

[Merlin3189]

Thanks for that interesting info.
I believe prof Karl Sikora is also suggesting that there are other immune responses which do not involve antibodies. Whether that means people who have been exposed and successfuly rejected the virus will continue to be immune, is not clear. But this is completely outwith my areas of even slight knowledge.

Just to be clear, my question was more, am I being unreasonable to immediately react to any such claim (100% accurate test of anything) with deep scepticism.

 

[sysprog]

Just to be clear, my question was more, am I being unreasonable to immediately react to any such claim (100% accurate test of anything) with deep scepticism.

I think that such skepticism is apt to be warranted; however, in this instance, it is not clear exactly what is being claimed, and a cursory investigation suggests that someone rounded up a slightly lower originally claimed number. That can be misleading when the result thereby becomes '100%', because of the special relation that number has to absolute certainty. Such terms as 'invariably', 'always', 'never', 'impossible', 'cannot', etc., should, in my opinion and in that of many others, when encountered in empirical science, almost always be viewed with some degree of diffidence.