Test Agreement Is Not Diagnostic Accuracy
October 13, 2026 · 5 min read · Aqrab editorial team
A new test matches an established test 97% of the time. That sounds reassuring. But if the established test is not a suitable reference standard for determining the target condition, agreement tells us only how often the two tests coincide—not how often either one is correct.
What a 97% Match Can Hide
Imagine the following 1,000 paired results. The numbers are hypothetical and describe no real test. Neither test is assumed to establish the true condition status.
| New test | Comparator positive | Comparator negative |
|---|---|---|
| Positive | 90 | 10 |
| Negative | 20 | 880 |
The tests agree in 90 + 880 = 970 of 1,000 comparisons, or 97%. Most agreements are negative. Yet the table cannot tell whether both tests correctly ruled out the condition in those 880 people—or both missed some cases. It contains no independent truth column.
The practical takeaway
When a paper says “97% accurate,” look at the column headings. If they compare two non-reference tests, relabel the result as agreement and ask how the target condition was established.
Use the Right Names for the Right Denominators
Sensitivity asks how often the new test is positive among people who truly have the condition; specificity asks how often it is negative among people who truly do not. Both require a suitable reference standard for condition status. If the comparator is another non-reference test, the paired table instead supports positive percent agreement and negative percent agreement relative to that comparator. Those are not interchangeable with sensitivity and specificity.
Agreement can still be useful: it may show whether a new assay tends to reproduce an existing workflow. But reproducing a workflow is a different claim from detecting disease correctly or improving patient outcomes.
Four Questions for a Methods Review
- Comparator: Is it a defensible reference standard for the target condition, or merely the current test?
- Full table: Are all four paired-result cells reported, including disagreements and indeterminate results?
- Denominator: Are the reported percentages overall agreement, positive/negative percent agreement, or sensitivity/specificity against verified condition status?
- Intended use: Were participants and specimens drawn from the population in which the new test would be used?
Even a valid accuracy estimate does not, alone, establish the benefit of changing care based on the test.
Where Aqrab Fits
Use Aqrab to inspect a test-comparison paper’s comparator, denominators, missing results, and intended-use population. A methods critique should make the evidentiary gap visible; it cannot turn agreement into verified accuracy.
Source Anchors
The FDA diagnostic-test reporting guidance distinguishes comparison with a reference standard from comparison with a non-reference method, and recommends agreement terminology for the latter. STARD 2015 calls for the reference-standard rationale, result tables, and handling of indeterminate and missing results. Neither source validates the hypothetical counts above.
Keep reading
Don't stop at one method.
Good methods judgment comes from contrast. Read the neighboring guides, see where the assumptions diverge, and avoid treating every observational problem like it needs the same hammer.
Ninety Percent Sensitive Based on How Many Cases?
The same 90% sensitivity can rest on 20 or 200 condition-positive participants. A worked confidence-interval comparison shows why the denominator matters before a test is called reliable.
A Negative Test Is Not a Rule-Out: Calculate the Risk Left Behind
A worked negative likelihood-ratio example shows why the same negative test can leave very different residual risks—and what to check before calling a result reassuring.
The Trial Ended. Who Made It Into the Open-Label Extension?
A denominator-first audit of long-term extension findings: trace who enrolled, who stayed, and what an uncontrolled extension can actually establish.