The Consensus Read Was Unanimous. What Happened Before the Meeting?
November 5, 2026 · 5 min read · Aqrab editorial team
A diagnostic imaging paper says that two readers reviewed each image and resolved disagreements by consensus. The final dataset has one tidy label per image. That label may be useful for the study, but it hides a different result: how often the readers disagreed before they talked. A consensus process cannot be used as its own evidence of independent reader agreement.
Keep the First Reads Visible
Imagine 100 images read independently by two clinicians. Before discussion, they agree on 90 images and disagree on 10. They then discuss those 10 and assign a single final label to each. The fictional counts below illustrate the sequence; they are not findings from a real study.
| Reader A | Reader B positive | Reader B negative |
|---|---|---|
| Positive | 40 | 5 |
| Negative | 5 | 50 |
The first reads agree in 40 + 50 = 90 cases, or 90%. After discussion, all 100 cases have a final label, but “100% resolved” is not “100% reader agreement.” The ten difficult cases did not disappear; the study made a decision about them. Report the initial disagreement separately from the resolution outcome.
The practical takeaway
When a paper reports consensus labels, request the independent first-read table and the number of cases changed or resolved. Do not calculate reader agreement from post-discussion labels.
Five Questions for the Methods Review
- Were first reads independent? Ask whether readers could see each other’s interpretations or discuss cases before recording their own calls.
- What information was available? Note clinical history, other test results, and any reference-standard result available at each stage.
- Where are the initial counts? Ask for the full paired table, including disagreements and indeterminate reads, rather than only the final consensus labels.
- How was consensus reached? Was a third reader involved? Did a prespecified rule settle disputes, or did the group discuss until one label remained?
- What does the final label represent? A consensus interpretation may be the study’s operational label. It is not automatically an independent reference standard for true condition status.
Agreement and accuracy remain separate questions. The first-read table describes consistency between readers under the study conditions. If the paper also claims diagnostic accuracy, inspect how condition status was established and compare test calls with an appropriate reference standard.
Where Aqrab Fits
Use Aqrab to structure a methods critique around first reads, the consensus procedure, and the source of any claimed ground truth. It can expose an unreported denominator; it cannot reconstruct first reads that the study did not retain.
Source Anchors
The GRRAS reporting checklist calls for describing whether ratings were independent, the rating process, reader and subject characteristics, and agreement estimates with uncertainty. The STARD 2015 checklist asks for the reference standard, information available to index-test readers, and diagnostic-accuracy estimates. Neither checklist supplies the hypothetical counts above.
Keep reading
Don't stop at one method.
Good methods judgment comes from contrast. Read the neighboring guides, see where the assumptions diverge, and avoid treating every observational problem like it needs the same hammer.
Ninety Percent Sensitive Based on How Many Cases?
The same 90% sensitivity can rest on 20 or 200 condition-positive participants. A worked confidence-interval comparison shows why the denominator matters before a test is called reliable.
Test Agreement Is Not Diagnostic Accuracy
A worked 97% agreement example shows why matching another imperfect test cannot establish sensitivity or specificity, plus four questions for appraising the comparator and denominators.
A Negative Test Is Not a Rule-Out: Calculate the Risk Left Behind
A worked negative likelihood-ratio example shows why the same negative test can leave very different residual risks—and what to check before calling a result reassuring.
This is the newest guide so far.