Benefit-risk Evaluation for Diagnostics: A Framework (BED-FRAME)

DxGoals: A Software Tool for Determining and Analyzing Clinically Meaningful Classification Accuracy Goals for Diagnostic Tests

BACKGROUND

To evaluate diagnostic tests for low prevalence conditions, classification accuracy metrics such as sensitivity, specificity, and positive likelihood ratio (PLR) and negative likelihood ratio (NLR) are advantageous because they are prevalence-independent and thus estimable in studies enriched for the condition. However, classification accuracy goals are often chosen without a clear understanding of whether they are clinically meaningful. Pennello (2021) proposed a risk stratification framework for determining classification accuracy goals. A software application is needed to determine the goals and provide data analysis.

METHODS

We introduce DxGoals, a freely available, R-Shiny software application for determining, visualizing, and analyzing classification accuracy goals for diagnostic tests. Given prevalence p for the target condition and specification that a test's positive and negative predictive values PPVand NPV=1-cNPV should satisfy PPV>PPV* and cNPV

RESULTS

We illustrate DxGoals on tests for penicillin allergy, ovarian cancer, and cervical cancer. The inputs cNPV*,p, and PPV* were informed by clinical management guidelines.

CONCLUSIONS

DxGoals facilitates determination, visualization, and analysis of clinically meaningful standalone and comparative classification accuracy goals. It is a potentially useful tool for diagnostic test evaluation.

Collapse

Key Words Collapse

MESH Headings

• Humans
• Software
• Sensitivity and Specificity
• Bayes Theorem
• Diagnostic Tests, Routine/standards
• Diagnostic Tests, Routine/methods
• Predictive Value of Tests

Collapse

Grants Collapse

Affiliation(s)

Ngoc-Ty Nguyen U.S. Food and Drug Administration, Center for Biologics Evaluation and Research, Silver Spring, MD, United States
Gene A Pennello U.S. Food and Drug Administration, Center for Devices and Radiological Health, Silver Spring, MD, United States

Collapse

Clinical scoring model for predicting cefotaxime-resistance in Klebsiella pneumoniae bacteremia: development and validation based on portal of entry

We developed a prediction model for cefotaxime resistance in patients with K. pneumoniae bacteremia. Adult patients with K. pneumoniae bacteremia were grouped into derivation (from March 2018 to December 2019) and validation (from January 2020 to August 2020) cohorts. The prediction scoring system was based on factors associated with cefotaxime resistance identified by the logistic regression model. A total of 358 patients were enrolled (256 for derivation, 102 for validation). In the multivariable analysis, age ≥65 years, hospital-acquired infection, prior antimicrobial use, and an updated Charlson comorbidity index ≥3 points were associated with cefotaxime resistance in the derivation cohort. When each variable was counted as 1 point, the values of the area under the curve were 0.761 in the derivation and 0.781 in the validation cohorts. The best cutoff value using the Youden index was ≥2 with 73.6% sensitivity and 67.5% specificity. Our simple scoring system favorably predicted cefotaxime resistance.

Duke Infective Endocarditis Criteria 3.0 for the Clinician: Defining What Is Possible

This commentary summarizes the results and clinical implications of validation studies evaluating the performance of the 2023 Duke-ISCID criteria for infective endocarditis.

Comparison of three rapid diagnostic tests for bloodstream infections using Benefit-risk Evaluation Framework (BED-FRAME)

Rapid diagnostic tests (RDTs) for bloodstream infections have the potential to reduce time to appropriate antimicrobial therapy and improve patient outcomes. Previously, an in-house, lipid-based, matrix-assisted laser desorption/ionization-time of flight mass spectrometry (MALDI-TOF MS) method, Fast Lipid Analysis Technique (FLAT MS), has shown promise as a rapid pathogen identification method. In this study, FLAT MS for direct from blood culture identification was evaluated and compared to FDA-cleared identification methods using the Benefit-risk Evaluation Framework (BED-FRAME) analysis. FLAT MS was evaluated and compared to Bruker Sepsityper and bioMérieux BioFire FilmArray BCID2 using results from a previous study. For this study, 301 positive blood cultures were collected from the University of Maryland Medical Center. The RDTs were compared by their sensitivities, time-to-results, hands-on time, and BED-FRAME analysis. The overall sensitivity of all platforms compared to culture results from monomicrobial-positive blood cultures was 88.3%. However, the three RDTs differed in their accuracy for identifying Gram-positive bacteria, Gram-negative bacteria, and yeast. Time-to-results for FLAT MS, Sepsityper, and BioFire BCID2 were all approximately one hour. Hands-on times for FLAT MS, Sepsityper, and BioFire BCID2 were 10 (±1.3), 40 (±2.8), and 5 (±0.25) minutes, respectively. BED-FRAME demonstrated that each RDT had utility at different pathogen prevalence and relative importance. BED-FRAME is a useful tool that can used to determine which RDT is best for a healthcare center.

The Future Ain't What It Used to Be…Out With the Old…In With the Better: Antibacterial Resistance Leadership Group Innovations

Clinical research networks conduct important studies that would not otherwise be performed by other entities. In the case of the Antibacterial Resistance Leadership Group (ARLG), such studies include diagnostic studies using master protocols, controlled phage intervention trials, and studies that evaluate treatment strategies or dynamic interventions, such as sequences of empiric and definitive therapies. However, the value of a clinical research network lies not only in the results from these important studies but in the creation of new approaches derived from collaborative thinking, carefully examining and defining the most important research questions for clinical practice, recognizing and addressing common but suboptimal approaches, and anticipating that the standard approaches of today may be insufficient for tomorrow. This results in the development and implementation of new methodologies and tools for the design, conduct, analyses, and reporting of research studies. These new methodologies directly impact the studies conducted within the network and have a broad and long-lasting impact on the field, enhancing the scientific value and efficiency of generations of research studies. This article describes innovations from the ARLG in diagnostic studies, observational studies, and clinical trials evaluating interventions for the prevention and treatment of antibiotic-resistant bacterial infections.

The Antibacterial Resistance Leadership Group: Scientific Advancements and Future Directions

In this overview, we describe important contributions from the Antibacterial Resistance Leadership Group (ARLG) to patient care, clinical trials design, and mentorship while outlining future priorities. The ARLG research agenda is focused on 3 key areas: gram-positive infections, gram-negative infections, and diagnostics. The ARLG has developed an innovative approach to clinical trials design, the desirability of outcome ranking (DOOR), which uses an ordinal measure of global outcome to assess both benefits and harms. DOOR was initially applied to observational studies to determine optimal dosing of vancomycin for methicillin-resistant Staphylcococcus aureus bacteremia and the efficacy of ceftazidime-avibactam versus colistin for the treatment of carbapenem-resistant Enterobacterales infection. DOOR is being successfully applied to the analysis of interventional trials and, in collaboration with the US Food and Drug Administration (FDA), for use in registrational trials. In the area of diagnostics, the ARLG developed Master Protocol for Evaluating Multiple Infection Diagnostics (MASTERMIND), an innovative design that allows simultaneous testing of multiple diagnostic platforms in a single study. This approach will be used to compare molecular assays for the identification of fluoroquinolone-resistant Neisseria gonorrhoeae (MASTER GC) and to compare rapid diagnostic tests for bloodstream infections. The ARLG has initiated a first-in-kind randomized, double-blind, placebo-controlled trial in participants with cystic fibrosis who are chronically colonized with Pseudomonas aeruginosa to assess the pharmacokinetics and antimicrobial activity of bacteriophage therapy. Finally, an engaged and highly trained workforce is critical for continued and future success against antimicrobial drug resistance. Thus, the ARLG has developed a robust mentoring program targeted to each stage of research training to attract and retain investigators in the field of antimicrobial resistance research.

Under the Hood: The Scientific Leadership, Clinical Operations, Statistical and Data Management, and Laboratory Centers of the Antibacterial Resistance Leadership Group

Developing and implementing the scientific agenda of the Antibacterial Resistance Leadership Group (ARLG) by soliciting input and proposals, transforming concepts into clinical trials, conducting those trials, and translating trial data analyses into actionable information for infectious disease clinical practice is the collective role of the Scientific Leadership Center, Clinical Operations Center, Statistical and Data Management Center, and Laboratory Center of the ARLG. These activities include shepherding concept proposal applications through peer review; identifying, qualifying, training, and overseeing clinical trials sites; recommending, developing, performing, and evaluating laboratory assays in support of clinical trials; and designing and performing data collection and statistical analyses. This article describes key components involved in realizing the ARLG scientific agenda through the activities of the ARLG centers.

Net benefit of diagnostic tests for multistate diseases: an indicator variables approach

A limitation of the common measures of diagnostic test performance, such as sensitivity and specificity, is that they do not consider the relative importance of false negative and false positive test results, which are likely to have different clinical consequences. Therefore, the use of classification or prediction measures alone to compare diagnostic tests or biomarkers can be inconclusive for clinicians. Comparing tests on net benefit can be more conclusive because clinical consequences of misdiagnoses are considered. The literature suggested evaluating the binary diagnostic tests based on net benefit, but did not consider diagnostic tests that classify more than two disease states, e.g., stroke subtype (large-artery atherosclerosis, cardioembolism, small-vessel occlusion, stroke of other determined etiology, stroke of undetermined etiology), skin lesion subtype, breast cancer subtypes (benign, mass, calcification, architectural distortion, etc.), METAVIR liver fibrosis state (F0- F4), histopathological classification of cervical intraepithelial neoplasia (CIN), prostate Gleason grade, brain injury (intracranial hemorrhage, mass effect, midline shift, cranial fracture) . Other diseases have more than two stages, such as Alzheimer's disease (dementia due to Alzheimer's disease, mild cognitive disability (MCI) due to Alzheimer's disease, and preclinical presymptomatics due to Alzheimer's disease). In diseases with more than two states, the benefits and risks may vary between states. This paper extends the net-benefit approach of evaluating binary diagnostic tests to multi-state clinical conditions to rule-in or rule-out a clinical condition based on adverse consequences of work-up delay (due to false negative test result) and unnecessary workup (due to false positive test result). We demonstrate our approach with numerical examples and real data.

From bottle to bedside: Implementation considerations and antimicrobial stewardship considerations for bloodstream infection rapid diagnostic testing

Antimicrobial stewardship (AMS) programs have been quick to adopt novel molecular rapid diagnostic technologies (mRDTs) for bloodstream infections (BSIs) to improve antimicrobial management. As such, most of the literature demonstrating the clinical and economic benefits of mRDTs for BSI is in the presence of active AMS intervention. Leveraging mRDTs to improve antimicrobial therapy for BSI is increasingly integral to AMS program activities. This narrative review discusses available and future mRDTs, the relationship between the clinical microbiology laboratory and AMS programs, and practical considerations for optimizing the use of these tools within a health system. Antimicrobial stewardship programs must work closely with their clinical microbiology laboratories to ensure that mRDTs are used to their fullest benefit while remaining cognizant of their limitations. As more mRDT instruments and panels become available and AMS programs continue to expand, future efforts must consider the expansion beyond traditional settings of large academic medical centers and how combinations of tools can further improve patient care.

A Statistical Review: Why Average Weighted Accuracy, not Accuracy or AUC?

Sensitivity and specificity are key aspects in evaluating the performance of diagnostic tests. Accuracy and AUC are commonly used composite measures that incorporate sensitivity and specificity. Average Weighted Accuracy (AWA) is motivated by the need for a statistical measure of diagnostic yield that can be used to compare diagnostic tests from the medical costs and clinical impact point of view, while incorporating the relevant prevalence range of the disease as well as the relative importance of false positive versus false negative cases. We derive the variance/covariance estimators and testing procedures in four different scenarios comparing diagnostic tests: (i) one diagnostic test vs. the best random test, (ii) two diagnostic tests from two independent samples, (iii) two diagnostic tests from the same sample, and (iv) more than two diagnostic tests from different or the same samples. The impacts of sample size, prevalence, and relative importance on power and average medical costs/clinical loss are examined through simulation studies. Accuracy has the highest power while AWA provides a consistent criterion in selecting the optimal threshold and better diagnostic tests with direct clinical interpretations. The use of AWA is illustrated on a three-arm clinical trial evaluating three different assays in detecting Neisseria gonorrhoeae (NG) and Chlamydia trachomatis (CT) in the rectum and pharynx.

Desirability of Outcome Ranking for the Management of Antimicrobial Therapy (DOOR MAT): A Framework for Assessing Antibiotic Selection Strategies in the Presence of Drug Resistance

The complexities of antibiotic resistance mean that successful stewardship must consider both the effectiveness of a given antibiotic and the spectrum of that therapy to minimize imposing further selective pressure. To meet this challenge, we propose the Desirability of Outcome Ranking approach for the Management of Antimicrobial Therapy (DOOR MAT), a flexible quantitative framework that evaluates the desirability of antibiotic selection. Herein, we describe the steps required to implement DOOR MAT and present examples to illustrate how the desirability of treatment selection can be evaluated using resistance information. While treatments and the scoring of treatment selections must be adapted to specific clinical settings, the principle of DOOR MAT remains constant: The most desirable antibiotic choice effectively treats the patient while exerting minimal pressure on future resistance.

Antibacterial Resistance Leadership Group 2.0 - Back to Business

In December 2019, the Antibacterial Resistance Leadership Group (ARLG) was awarded funding for another seven-year cycle to support a clinical research network on antibacterial resistance. ARLG 2.0 has three overarching research priorities: (1) infections caused by antibiotic resistant (AR) Gram-negative bacteria; (2) infections caused by AR Gram-positive bacteria, and (3) diagnostic tests to optimize use of antibiotics. To support the next generation of AR researchers, the ARLG offers three mentoring opportunities: the ARLG Fellowship, Early Stage Investigator Seed Grants, and the Trialists in Training Program. The purpose of this article is to update the scientific community on the progress made in the original funding period and to encourage submission of clinical research that addresses one or more of the research priority areas of ARLG 2.0.

Rapid Molecular Diagnostics to Inform Empiric Use of Ceftazidime/Avibactam and Ceftolozane/Tazobactam Against Pseudomonas aeruginosa: PRIMERS IV

BACKGROUND

Overcoming β-lactam resistance in pathogens such as Pseudomonas aeruginosa is a major clinical challenge. Rapid molecular diagnostics (RMDs) have the potential to inform selection of empiric therapy in patients infected by P. aeruginosa.

METHODS

In this study, we used a heterogeneous collection of 197 P. aeruginosa that included multidrug-resistant isolates to determine whether 2 representative RMDs (Acuitas Resistome test and VERIGENE gram-negative blood culture test) could identify susceptibility to 2 newer β-lactam/β-lactamase inhibitor (BL-BLI) combinations, ceftazidime/avibactam (CZA) and ceftolozane/tazobactam (TOL/TAZO).

RESULTS

We found that the studied RMD platforms were able to correctly identify BL-BLI susceptibility (susceptibility sensitivity, 100%; 95% confidence interval [CI], 97%, 100%) for both BLs-BLIs. However, their ability to detect resistance to these BLs-BLIs was lower (resistance sensitivity, 66%; 95% CI, 52%, 78% for TOL/TAZO and 33%; 95% CI, 20%, 49% for CZA).

CONCLUSIONS

The diagnostic platforms studied showed the most potential in scenarios where a resistance gene was detected or in scenarios where a resistance gene was not detected and the prevalence of resistance to TOL/TAZO or CZA is known to be low. Clinicians need to be mindful of the benefits and risks that result from empiric treatment decisions that are based on resistance gene detection in P. aeruginosa, acknowledging that such decisions are impacted by the prevalence of resistance, which varies temporally and geographically.

Average Weighted Accuracy: Pragmatic Analysis for a Rapid Diagnostics in Categorizing Acute Lung Infections (RADICAL) Study

Patient management relies on diagnostic information to identify appropriate treatment. Standard evaluations of diagnostic tests consist of estimating sensitivity, specificity, positive/negative predictive values, likelihood ratios, and accuracy. Although useful, these metrics do not convey the tests' clinical value, which is critical to informing decision-making. Full appreciation of the clinical impact of a diagnostic test requires analyses that integrate sensitivity and specificity, account for the disease prevalence within the population of test application, and account for the relative importance of specificity vs sensitivity by considering the clinical implications of false-positive and false-negative results. We developed average weighted accuracy (AWA), representing a pragmatic metric of diagnostic yield or global utility of a diagnostic test. AWA can be used to compare test alternatives, even across different studies. We apply the AWA methodology to evaluate a new diagnostic test developed in the Rapid Diagnostics in Categorizing Acute Lung Infections (RADICAL) study.

Benefit-risk assessment for binary diagnostic tests

In diagnostic device evaluation, it is important to have an integrated benefit-risk (BR) assessment for safety and effectiveness, which is not same as the assessment for drugs and therapeutic devices. Correct diagnosis does not lead to direct clinical outcome such as longer survival, release of symptoms, tumor shrinkage, etc.; but leads to the proper treatment in time while incorrect diagnosis may result in serious consequences of unnecessary tests and wrong treatments. Some common measures used in evaluating the accuracy of a diagnostic device include sensitivity, specificity, positive predictive value and negative predictive value. Here, we propose a BR measure by incorporating information about true-positive and true-negative cases (correct diagnosis) and false-positive and false-negative cases (incorrect diagnosis) for facilitating the necessary decision-making. Three decision rules are discussed depending on the purpose of the clinical study. Different statistical models are developed for estimating the BR measure for data obtained from different sampling schemes (cross-sectional and case-control sampling). The construction of confidence intervals (CIs) for the proposed BR measure is based on (i) the asymptotic normality of the maximum likelihood estimators (MLEs), and (ii) parametric bootstrap re-sampling technique. The performance of these CIs is evaluated by intensive Monte-Carlo simulations which reveal that both CIs perform reasonably well. Finally, the proposed methodology is applied to two clinical trial datasets.

Diagnostic Performance of a Molecular Test versus Clinician Assessment of Vaginitis

Vaginitis is a common complaint, diagnosed either empirically or using Amsel's criteria and wet mount microscopy. This study sought to determine characteristics of an investigational test (a molecular test for vaginitis), compared to reference, for detection of bacterial vaginosis, Candida spp., and Trichomonas vaginalis Vaginal specimens from a cross-sectional study were obtained from 1,740 women (≥18 years old), with vaginitis symptoms, during routine clinic visits (across 10 sites in the United States). Specimens were analyzed using a commercial PCR/fluorogenic probe-based investigational test that detects bacterial vaginosis, Candida spp., and Trichomonas vaginalis Clinician diagnosis and in-clinic testing (Amsel's test, potassium hydroxide preparation, and wet mount) were also employed to detect the three vaginitis causes. All testing methods were compared to the respective reference methods (Nugent Gram stain for bacterial vaginosis, detection of the Candida gene its2, and Trichomonas vaginalis culture). The investigational test, clinician diagnosis, and in-clinic testing were compared to reference methods for bacterial vaginosis, Candida spp., and Trichomonas vaginalis The investigational test resulted in significantly higher sensitivity and negative predictive value than clinician diagnosis or in-clinic testing. In addition, the investigational test showed a statistically higher overall percent agreement with each of the three reference methods than did clinician diagnosis or in-clinic testing. The investigational test showed significantly higher sensitivity for detecting vaginitis, involving more than one cause, than did clinician diagnosis. Taken together, these results suggest that a molecular investigational test can facilitate accurate detection of vaginitis.

Advancing Diagnostics to Address Antibacterial Resistance: The Diagnostics and Devices Committee of the Antibacterial Resistance Leadership Group

Diagnostics are a cornerstone of the practice of infectious diseases. However, various limitations frequently lead to unmet clinical needs. In most other domains, diagnostics focus on narrowly defined questions, provide readily interpretable answers, and use true gold standards for development. In contrast, infectious diseases diagnostics must contend with scores of potential pathogens, dozens of clinical syndromes, emerging pathogens, rapid evolution of existing pathogens and their associated resistance mechanisms, and the absence of gold standards in many situations. In spite of these challenges, the importance and value of diagnostics cannot be underestimated. Therefore, the Antibacterial Resistance Leadership Group has identified diagnostics as 1 of 4 major areas of emphasis. Herein, we provide an overview of that development, highlighting several examples where innovation in study design, content, and execution is advancing the field of infectious diseases diagnostics.

Fundamentals and Catalytic Innovation: The Statistical and Data Management Center of the Antibacterial Resistance Leadership Group

The Statistical and Data Management Center (SDMC) provides the Antibacterial Resistance Leadership Group (ARLG) with statistical and data management expertise to advance the ARLG research agenda. The SDMC is active at all stages of a study, including design; data collection and monitoring; data analyses and archival; and publication of study results. The SDMC enhances the scientific integrity of ARLG studies through the development and implementation of innovative and practical statistical methodologies and by educating research colleagues regarding the application of clinical trial fundamentals. This article summarizes the challenges and roles, as well as the innovative contributions in the design, monitoring, and analyses of clinical trials and diagnostic studies, of the ARLG SDMC.

Win Ratio -An Intuitive and Easy-To-Interpret Composite Outcome in Medical Studies

In medical studies with multiple outcomes, researchers always need to make choices as to whether to use a composite outcome (after combining multiple outcomes) as their primary outcome. In this paper we review a new measurement of the treatment effect - win ratio, which can be easily used in studies with prioritized multiple outcomes. We also propose some research topics to be done in this area.

MASTERMIND: Bringing Microbial Diagnostics to the Clinic

New diagnostics are urgently needed to address emerging antimicrobial resistance. The Antibacterial Resistance Leadership Group proposes a strategy called MASTERMIND (Master Protocol for Evaluating Multiple Infection Diagnostics) for advancement of infectious diseases diagnostics. The goal of this strategy is to generate the data necessary to support US Food and Drug Administration clearance of new diagnostic tests by promoting research that might not have otherwise been feasible with conventional trial designs. MASTERMIND uses a single subject's sample(s) to evaluate multiple diagnostic tests at the same time, providing efficiencies of specimen collection and characterization. MASTERMIND also offers central trial organization, standardization of methods and definitions, and common comparators.

Comparing diagnostic tests on benefit-risk

Comparing diagnostic tests on accuracy alone can be inconclusive. For example, a test may have better sensitivity than another test yet worse specificity. Comparing tests on benefit risk may be more conclusive because clinical consequences of diagnostic error are considered. For benefit-risk evaluation, we propose diagnostic yield, the expected distribution of subjects with true positive, false positive, true negative, and false negative test results in a hypothetical population. We construct a table of diagnostic yield that includes the number of false positive subjects experiencing adverse consequences from unnecessary work-up. We then develop a decision theory for evaluating tests. The theory provides additional interpretation to quantities in the diagnostic yield table. It also indicates that the expected utility of a test relative to a perfect test is a weighted accuracy measure, the average of sensitivity and specificity weighted for prevalence and relative importance of false positive and false negative testing errors, also interpretable as the cost-benefit ratio of treating non-diseased and diseased subjects. We propose plots of diagnostic yield, weighted accuracy, and relative net benefit of tests as functions of prevalence or cost-benefit ratio. Concepts are illustrated with hypothetical screening tests for colorectal cancer with test positive subjects being referred to colonoscopy.