Temporary Regulatory Deviations and the Coronavirus Disease 2019 (COVID-19) PCR Labeling Update Study Indicate What Laboratory-Developed Test Regulation by the US Food and Drug Administration (FDA) Could Look Like

When False-Positives Arise: Troubleshooting a SARS-Coronavirus-2 (SARS-CoV-2) Detection Assay on a Semi-Automated Platform

BACKGROUND

During the COVID-19 pandemic, many molecular diagnostic laboratories performed high-throughput SARS-CoV-2 testing often with implementation of automated workflows. In parallel, vaccination campaigns resulted increasingly in specimens from fully vaccinated patients, with resultant clinical inquiries regarding positive results in this patient population. This prompted a quality improvement initiative to investigate the semi-automated testing workflow for false-positive results. The troubleshooting workflow is described and procedural improvements are outlined that serve as a resource for other molecular diagnostic laboratories that need to overcome testing anomalies in a semi-automated environment.

METHODS

This workflow utilized the MagMax-96 Viral RNA kit and the CDC 2019-nCoV RT-qPCR Panel on the Agilent Bravo Liquid-Handler (Bravo). Screening of the environment, personnel, and the mechanical performance of instrumentation using low Ct checkerboard challenges was executed to identify sources of cross-contamination. Evaluation of the assay and reporting design was conducted.

RESULTS

Specimen contamination was observed during the viral extraction process on the Bravo. Changes to the program reduced plate contamination by 50% and importantly revealed consistent hallmarks of contaminated samples. We adjusted the reporting algorithm using these indicators of false positives. False positives that were identified made up 0.11% of the 45 000+ tests conducted over the following 8 months.

CONCLUSIONS

These adjustments provided confident and quality results while maintaining turnaround time for patients and pandemic-related public health initiatives. This corrected false-positive rate is concordant with previously published studies from diagnostic laboratories utilizing automated systems and may be considered a laboratory performance standard for this type of testing.

Diagnostic quality model (DQM): an integrated framework for the assessment of diagnostic quality when using AI/ML

BACKGROUND

Laboratory medicine has reached the era where promises of artificial intelligence and machine learning (AI/ML) seem palpable. Currently, the primary responsibility for risk-benefit assessment in clinical practice resides with the medical director. Unfortunately, there is no tool or concept that enables diagnostic quality assessment for the various potential AI/ML applications. Specifically, we noted that an operational definition of laboratory diagnostic quality - for the specific purpose of assessing AI/ML improvements - is currently missing.

METHODS

A session at the 3rd Strategic Conference of the European Federation of Laboratory Medicine in 2022 on "AI in the Laboratory of the Future" prompted an expert roundtable discussion. Here we present a conceptual diagnostic quality framework for the specific purpose of assessing AI/ML implementations.

RESULTS

The presented framework is termed diagnostic quality model (DQM) and distinguishes AI/ML improvements at the test, procedure, laboratory, or healthcare ecosystem level. The operational definition illustrates the nested relationship among these levels. The model can help to define relevant objectives for implementation and how levels come together to form coherent diagnostics. The affected levels are referred to as scope and we provide a rubric to quantify AI/ML improvements while complying with existing, mandated regulatory standards. We present 4 relevant clinical scenarios including multi-modal diagnostics and compare the model to existing quality management systems.

CONCLUSIONS

A diagnostic quality model is essential to navigate the complexities of clinical AI/ML implementations. The presented diagnostic quality framework can help to specify and communicate the key implications of AI/ML solutions in laboratory diagnostics.