Development of a reference standard to assign bacterial versus viral infection etiology using an all-inclusive methodology for comparison of novel diagnostic tool performance
Coburn Allen, J Kate Deanehan, Yaniv Dotan, Matthew A Eisenberg, Andrew M Fine, Jonathan Isenberg, Ann Kane, Dani Kirshner, Todd W Lyons, Yasmin Maor, Ami Neuberger, Daniel G Ostermayer, Sharona Paz, Oded Scheuerman, Shahaf Shiber, Victoria A Statler, Michal Stein, Renata Yakubov, Shirly Yanai, Roy Navon, Lior Kellerman, Tanya M Gottlieb, Eran EdenAbstract
Background
Diagnostic test evaluation requires a reference standard. We describe an approach for creating a reference standard for acute infection using unrestricted adjudication and apply it to compare biomarker tools.
Methods
Adults and children with suspected acute infection enrolled in three prospective studies at emergency departments and urgent cares were included. Adjudicators, blinded to C-reactive protein, procalcitonin, and MeMed BV (MMBV), labeled each case (bacterial/viral/non-infectious/indeterminate). Initial adjudication involved 3 adjudicators. Reference standard cohorts were defined: Microbiologically confirmed (3/3 adjudicators concur with high confidence and a concordant microbiological finding), unanimous (3/3 adjudicators concur with high confidence), suspected (3/3 adjudicators concur with high/moderate confidence or 2/3 adjudicators concur with high confidence) and all-inclusive (remaining unlabeled cases were reviewed by up to 7 additional adjudicators until reaching a leading label).
Results
Among 1016 patients, 156 difficult-to-diagnose cases required over 3 adjudicators. The area under the receiver operating characteristic curve in the microbiologically confirmed (n=427), unanimous (n=565), suspected (n=860) and all-inclusive (n=1016) cohorts for MMBV were 0.98 (95% confidence interval 0.94-1.00), 0.98 (0.95-1.00), 0.95 (0.92-0.98) and 0.90 (0.87-0.93), respectively, and for procalcitonin were 0.69 (0.57-0.81), 0.77 (0.68-0.86), 0.74 (0.68-0.80) and 0.70 (0.65-0.75), respectively. A delta in performance between MMBV and procalcitonin was maintained across the different cohorts.
Conclusion
Creating a reference standard that includes difficult-to-diagnose cases demands an approach to addressing diagnostic uncertainty in acute infections. Tool performance depends on the reference standard applied and decreases as the difficulty to diagnose increases, highlighting the importance of using the same reference standard when comparing tools.