Using machine learning to develop smart reflex testing protocols

OBJECTIVE

Reflex testing protocols allow clinical laboratories to perform second line diagnostic tests on existing specimens based on the results of initially ordered tests. Reflex testing can support optimal clinical laboratory test ordering and diagnosis. In current clinical practice, reflex testing typically relies on simple "if-then" rules; however, this limits the opportunities for reflex testing since most test ordering decisions involve more complexity than traditional rule-based approaches would allow. Here, using the analyte ferritin as an example, we propose an alternative machine learning-based approach to "smart" reflex testing.

METHODS

Using deidentified patient data, we developed a machine learning model to predict whether a patient getting CBC testing will also have ferritin testing ordered. We evaluate applications of this model to reflex testing by assessing its performance in comparison to possible rule-based approaches.

RESULTS

Our underlying machine learning models performed moderately well in predicting ferritin test ordering (AUC=0.731 in reference to actual ordering) and demonstrated promising potential to underlie key clinical applications. In contrast, none of the many traditionally framed, rule-based, hypothetical reflex protocols we evaluated offered sufficient agreement with actual ordering to be clinically feasible. Using chart review, we further demonstrated that the strategic deployment of our model could avoid important ferritin test ordering errors.

CONCLUSIONS

Machine learning may provide a foundation for new types of reflex testing with enhanced benefits for clinical diagnosis.

Using Machine Learning to Develop Smart Reflex Testing Protocols

Objective

Reflex testing protocols allow clinical laboratories to perform second line diagnostic tests on existing specimens based on the results of initially ordered tests. Reflex testing can support optimal clinical laboratory test ordering and diagnosis. In current clinical practice, reflex testing typically relies on simple "if-then" rules; however, this limits their scope since most test ordering decisions involve more complexity than a simple rule will allow. Here, using the analyte ferritin as an example, we propose an alternative machine learning-based approach to "smart" reflex testing with a wider scope and greater impact than traditional rule-based approaches.

Methods

Using patient data, we developed a machine learning model to predict whether a patient getting CBC testing will also have ferritin testing ordered, consider applications of this model to "smart" reflex testing, and evaluate the model by comparing its performance to possible rule-based approaches.

Results

Our underlying machine learning models performed moderately well in predicting ferritin test ordering and demonstrated greater suitability to reflex testing than rule-based approaches. Using chart review, we demonstrate that our model may improve ferritin test ordering. Finally, as a secondary goal, we demonstrate that ferritin test results are missing not at random (MNAR), a finding with implications for unbiased imputation of missing test results.

Conclusions

Machine learning may provide a foundation for new types of reflex testing with enhanced benefits for clinical diagnosis and laboratory utilization management.

Structural basis for continued antibody evasion by the SARS-CoV-2 receptor binding domain

Many studies have examined the impact of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) variants on neutralizing antibody activity after they have become dominant strains. Here, we evaluate the consequences of further viral evolution. We demonstrate mechanisms through which the SARS-CoV-2 receptor binding domain (RBD) can tolerate large numbers of simultaneous antibody escape mutations and show that pseudotypes containing up to seven mutations, as opposed to the one to three found in previously studied variants of concern, are more resistant to neutralization by therapeutic antibodies and serum from vaccine recipients. We identify an antibody that binds the RBD core to neutralize pseudotypes for all tested variants but show that the RBD can acquire an N-linked glycan to escape neutralization. Our findings portend continued emergence of escape variants as SARS-CoV-2 adapts to humans.

Presymptomatic Transmission of Severe Acute Respiratory Syndrome Coronavirus 2 Among Residents and Staff at a Skilled Nursing Facility: Results of Real-time Polymerase Chain Reaction and Serologic Testing

High rates of asymptomatic infection suggest benefits to routine testing in congregate care settings. SARS-CoV-2 screening was undertaken in a single nursing facility without a known case of COVID-19, demonstrating an 85% prevalence among residents and 37% among staff. Serology was not helpful in identifying infections.

COVID-19-neutralizing antibodies predict disease severity and survival

Coronavirus disease 2019 (COVID-19) exhibits variable symptom severity ranging from asymptomatic to life-threatening, yet the relationship between severity and the humoral immune response is poorly understood. We examined antibody responses in 113 COVID-19 patients and found that severe cases resulting in intubation or death exhibited increased inflammatory markers, lymphopenia, pro-inflammatory cytokines, and high anti-receptor binding domain (RBD) antibody levels. Although anti-RBD immunoglobulin G (IgG) levels generally correlated with neutralization titer, quantitation of neutralization potency revealed that high potency was a predictor of survival. In addition to neutralization of wild-type SARS-CoV-2, patient sera were also able to neutralize the recently emerged SARS-CoV-2 mutant D614G, suggesting cross-protection from reinfection by either strain. However, SARS-CoV-2 sera generally lacked cross-neutralization to a highly homologous pre-emergent bat coronavirus, WIV1-CoV, which has not yet crossed the species barrier. These results highlight the importance of neutralizing humoral immunity on disease progression and the need to develop broadly protective interventions to prevent future coronavirus pandemics.

COVID-19 neutralizing antibodies predict disease severity and survival

COVID-19 exhibits variable symptom severity ranging from asymptomatic to life-threatening, yet the relationship between severity and the humoral immune response is poorly understood. We examined antibody responses in 113 COVID-19 patients and found that severe cases resulting in intubation or death exhibited increased inflammatory markers, lymphopenia, and high anti-RBD antibody levels. While anti-RBD IgG levels generally correlated with neutralization titer, quantitation of neutralization potency revealed that high potency was a predictor of survival. In addition to neutralization of wild-type SARS-CoV-2, patient sera were also able to neutralize the recently emerged SARS-CoV-2 mutant D614G, suggesting protection from reinfection by this strain. However, SARS-CoV-2 sera was unable to cross-neutralize a highly-homologous pre-emergent bat coronavirus, WIV1-CoV, that has not yet crossed the species barrier. These results highlight the importance of neutralizing humoral immunity on disease progression and the need to develop broadly protective interventions to prevent future coronavirus pandemics.

Clinical sensitivity and interpretation of PCR and serological COVID-19 diagnostics for patients presenting to the hospital

The diagnosis of COVID-19 requires integration of clinical and laboratory data. Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) diagnostic assays play a central role in diagnosis and have fixed technical performance metrics. Interpretation becomes challenging because the clinical sensitivity changes as the virus clears and the immune response emerges. Our goal was to examine the clinical sensitivity of two most common SARS-CoV-2 diagnostic test modalities, polymerase chain reaction (PCR) and serology, over the disease course to provide insight into their clinical interpretation in patients presenting to the hospital. We conducted a single-center, retrospective study. To derive clinical sensitivity of PCR, we identified 209 PCR-positive SARS-CoV-2 patients with multiple PCR test results (624 total PCR tests) and calculated daily sensitivity from date of symptom onset or first positive test. Clinical sensitivity of PCR decreased with days post symptom onset with >90% clinical sensitivity during the first 5 days after symptom onset, 70%-71% from Days 9 to 11, and 30% at Day 21. To calculate daily clinical sensitivity by serology, we utilized 157 PCR-positive patients with a total of 197 specimens tested by enzyme-linked immunosorbent assay for IgM, IgG, and IgA anti-SARS-CoV-2 antibodies. In contrast to PCR, serological sensitivity increased with days post symptom onset with >50% of patients seropositive by at least one antibody isotype after Day 7, >80% after Day 12, and 100% by Day 21. Taken together, PCR and serology are complimentary modalities that require time-dependent interpretation. Superimposition of sensitivities over time indicate that serology can function as a reliable diagnostic aid indicating recent or prior infection.

Early Prediction of Acute Kidney Injury in Critical Care Setting Using Clinical Notes and Structured Multivariate Physiological Measurements

The onset of acute kidney injury (AKI) during an intensive care unit (ICU) admission is associated with increased morbidity and mortality. Developing novel methods to identify early AKI onset is of critical importance in preventing or reducing AKI complications. We built and applied multiple machine learning models to integrate clinical notes and structured physiological measurements and estimate the risk of new AKI onset using the MIMIC-III database. From the clinical notes, we generated clinically meaningful word representations and embeddings. Four supervised learning classifiers and mixed-feature deep learning architecture were used to construct prediction models. The best configurations consistently utilized both structured and unstructured clinical features and yielded competitive AUCs above 0.83. Our work suggests that integrating structured and unstructured clinical features can be effectively applied to assist clinicians in identifying the risk of incident AKI onset in critically-ill patients upon admission to the ICU.

"Pre-pre" and "post-post" analytical error: high-incidence patient safety hazards involving the clinical laboratory

Data from recent studies suggest that the highest incidence of laboratory-related errors occurs in the pre-analytical phase of laboratory testing. However, few studies have examined the frequency of errors in laboratory test selection and interpretation. A survey of physicians who use our clinical laboratory demonstrated that the largest number of test ordering errors appear to involve physicians simply ordering the wrong test. Diagnostic algorithms providing guidance for test selection in specific disorders are also used as the basis for the establishment of reflex protocols in the clinical laboratory. The provision of an expert-driven interpretation by laboratory professionals resulted in improvements both in the time to and the accuracy of diagnosis. A survey of our physician staff has shown that in the absence of such an interpretation, for patients being assessed for a coagulation disorder, approximately 75% of the cases would have involved some level of test result misinterpretation.

Visionary leadership and the future of VA health system

As the U.S. Department of Veterans Affairs (VA) makes the change over to Veterans Integrated Service Network (VISNs) the need for new and better leadership is warranted if VA wants to not only survive, but thrive in the emerging twenty-first century healthcare system. VA can prepare for the future and meet the challenges facing them by adopting a system of visionary leadership. The use of scenarios and vision techniques are explained as they relate to VA's efforts to move toward their new system of VISNs. The four scenarios provide snapshots of possible futures for the U.S. healthcare system as well as the possible future role and mission of VA--from VA disappearing to its becoming a premier virtual organization.