A novel approach to the determination of clinical decision thresholds

Piloting a new method to estimate action thresholds in medicine through intuitive weighing

OBJECTIVES

In clinical decision-making, physicians take actions such as prescribing treatment only when the probability of disease is sufficiently high. The lowest probability at which the action will be considered, is the action threshold. Such thresholds play an important role whenever decisions have to be taken under uncertainty. However, while several methods to estimate action thresholds exist, few methods give satisfactory results or have been adopted in clinical practice. We piloted the adapted nominal group technique (aNGT), a new prescriptive method based on a formal consensus technique adapted for use in clinical decision-making.

DESIGN, SETTING AND PARTICIPANTS

We applied this method in groups of postgraduate students using three scenarios: treat for rifampicin-resistant tuberculosis (RR-TB), switch to second-line HIV treatment and isolate for SARS-CoV-2 infection.

INTERVENTIONS

The participants first summarise all harms of wrongly taking action when none is required and wrongly not taking action when it would have been useful. Then they rate the statements on these harms, discuss their importance in the decision-making process, and finally weigh the statements against each other.

MAIN OUTCOME MEASURES

The resulting consensus threshold is estimated as the relative weights of the harms of the false positives divided by the total harm, and averaged out over participants. In some applications, the thresholds are compared with an existing method based on clinical vignettes.

RESULTS

The resulting action thresholds were just over 50% for RR-TB treatment, between 20% and 50% for switching HIV treatment and 43% for COVID-19 isolation. These results were considered acceptable to all participants. Between sessions variation was low for RR-TB and moderate for HIV. Threshold estimates were moderately lower with the method based on clinical vignettes.

CONCLUSIONS

The aNGT gives sensible results in our pilot and has the potential to estimate action thresholds, in an efficient manner, while involving all relevant stakeholders. Further research is needed to study the value of the method in clinical decision-making and its ability to generate acceptable thresholds that stakeholders can agree on.

Clinician Testing and Treatment Thresholds for Management of Urinary Tract Infection

Greater understanding of clinical decision thresholds may improve inappropriate testing and treatment of urinary tract infection (UTI). We used a survey of clinicians to examine UTI decision thresholds. Although overestimates of UTI occurred, testing and treatment thresholds were generally rational, were lower than previously reported, and differed by type of clinician.

Targeting repetitive laboratory testing with electronic health records-embedded predictive decision support: A pre-implementation study

INTRODUCTION

Unnecessary laboratory testing contributes to patient morbidity and healthcare waste. Despite prior attempts at curbing such overutilization, there remains opportunity for improvement using novel data-driven approaches. This study presents the development and early evaluation of a clinical decision support tool that uses a predictive model to help providers reduce low-yield, repetitive laboratory testing in hospitalized patients.

METHODS

We developed an EHR-embedded SMART on FHIR application that utilizes a laboratory test result prediction model based on historical laboratory data. A combination of semi-structured physician interviews, usability testing, and quantitative analysis on retrospective laboratory data were used to inform the tool's development and evaluate its acceptability and potential clinical impact.

KEY RESULTS

Physicians identified culture and lack of awareness of repeat orders as key drivers for overuse of inpatient blood testing. Users expressed an openness to a lab prediction model and 13/15 physicians believed the tool would alter their ordering practices. The application received a median System Usability Scale score of 75, corresponding to the 75th percentile of software tools. On average, physicians desired a prediction certainty of 85% before discontinuing a routine recurring laboratory order and a higher certainty of 90% before being alerted. Simulation on historical lab data indicates that filtering based on accepted thresholds could have reduced ∼22% of repeat chemistry panels.

CONCLUSIONS

The use of a predictive algorithm as a means to calculate the utility of a diagnostic test is a promising paradigm for curbing laboratory test overutilization. An EHR-embedded clinical decision support tool employing such a model is a novel and acceptable intervention with the potential to reduce low-yield, repetitive laboratory testing.

Derivation and validation of clinical prediction rules for diagnosis of infectious mononucleosis: a prospective cohort study

OBJECTIVES

Infectious mononucleosis (IM) is a clinical syndrome that is characterised by lymphadenopathy, fever and sore throat. Although generally not considered a serious illness, IM can lead to significant loss of time from school or work due to profound fatigue, or the development of chronic illness. This study aimed to derive and externally validate clinical prediction rules (CPRs) for IM caused by Epstein-Barr virus (EBV).

DESIGN

Prospective cohort study.

SETTING AND PARTICIPANTS

328 participants were recruited prospectively for the derivation cohort, from seven university-affiliated student health centres in Ireland. Participants were young adults (17-39 years old, mean age 20.6 years) with sore throat and one other additional symptom suggestive of IM. The validation cohort was a retrospective cohort of 1498 participants from a student health centre at the University of Georgia, USA.

MAIN OUTCOME MEASURES

Regression analyses were used to develop four CPR models, internally validated in the derivation cohort. External validation was carried out in the geographically separate validation cohort.

RESULTS

In the derivation cohort, there were 328 participants, of whom 42 (12.8%) had a positive EBV serology test result. Of 1498 participants in the validation cohort, 243 (16.2%) had positive heterophile antibody tests for IM. Four alternative CPR models were developed and compared. There was moderate discrimination and good calibration for all models. The sparsest CPR included presence of enlarged/tender posterior cervical lymph nodes and presence of exudate on the pharynx. This model had moderate discrimination (area under the receiver operating characteristic curve (AUC): 0.70; 95% CI: 0.62-0.79) and good calibration. On external validation, this model demonstrated reasonable discrimination (AUC: 0.69; 95% CI: 0.67-0.72) and good calibration.

CONCLUSIONS

The alternative CPRs proposed can provide quantitative probability estimates of IM. Used in conjunction with serological testing for atypical lymphocytosis and immunoglobulin testing for viral capsid antigen, CPRs can enhance diagnostic decision-making for IM in community settings.

The impact of a rapid home test on telehealth decision-making for influenza: a clinical vignette study

Background

Home testing for influenza has the potential to aid triage and management decisions for patients with influenza-like illness. As yet, little is known about the effect of the home influenza testing on clinical decision-making via telehealth. The goal of this study was to determine the clinicians’ decision thresholds for influenza and whether the availability of a home influenza test affects clinical decisions.

Methods

We identified primary care physicians at 4 different sites in the US, largely via in-person continuing education meetings. Clinicians were asked for each vignette whether to treat empirically (“rule in”), ask the patient come to the clinic for further evaluation (“test”), or neither test nor treat (“rule out”). They were then given the results of a home influenza test, and were again asked to select from these three options. We measured the agreement of physician estimates of the likelihood of influenza with the probability based on a clinical prediction model. The test and treatment thresholds of influenza were determined based on mixed-effect logistic regressions.

Results

In total, 202 clinicians made 570 sets of clinical decisions. Agreement between estimated and actual probability of influenza was fair. The test and treatment thresholds were 24% (95% CI: 22% to 25%) and 63% (95% CI: 58% to 65%) before revealing the actual likelihood of influenza. After providing the results of a home flu test the thresholds were similar, 26% (95% CI: 24% to 29%) and 59% (95% CI: 56% to 62%). However, approximately half of clinicians changed their cliical management decision after being given the home influenza test result, largely by categorizing more patients in the “rule out” and “rule in” groups, and reducing the need for in-person evaluation from 41% of patients to only 20%.

Conclusion

In the context of a telehealth visit for a patient with influenza-like illness, we identified a test threshold of approximately 25% and a treatment threshold of approximately 60%. Adding the home influenza test results reduced uncertainty and significantly decreased the need for in-person visits.

Clinically Significant Risk Thresholds in the Management of Primary Cutaneous Melanoma: A Survey of Melanoma Experts

BACKGROUND

Risk-based thresholds to guide management are undefined in the treatment of primary cutaneous melanoma but are essential to advance the field from traditional stage-based treatment to more individualized care.

METHODS

To estimate treatment risk thresholds, hypothetical clinical melanoma scenarios were developed and a stratified random sample was distributed to expert melanoma clinicians via an anonymous web-based survey. Scenarios provided a defined 5-year risk of recurrence and asked for recommendations regarding clinical follow-up, imaging, and adjuvant therapy. Marginal probability of response across the spectrum of 5-year recurrence risk was estimated. The risk at which 50% of respondents recommended a treatment was defined as the risk threshold.

RESULTS

The overall response rate was 56% (89/159). Three separate multivariable models were constructed to estimate the recommendations for clinical follow-up more than twice/year, for surveillance cross-sectional imaging at least once/year, and for adjuvant therapy. A 36% 5-year risk of recurrence was identified as the threshold for recommending clinical follow-up more than twice/year. The thresholds for recommending cross-sectional imaging and adjuvant therapy were 30 and 59%, respectively. Thresholds varied with the age of the hypothetical patient: at younger ages they were constant but increased rapidly at ages 60 years and above.

CONCLUSIONS

To our knowledge, these data provide the first estimates of clinically significant treatment thresholds for patients with cutaneous melanoma based on risk of recurrence. Future refinement and adoption of thresholds would permit assessment of the clinical utility of novel prognostic tools and represents an early step toward individualizing treatment recommendations.

Learning decision thresholds for risk stratification models from aggregate clinician behavior

Using a risk stratification model to guide clinical practice often requires the choice of a cutoff—called the decision threshold—on the model’s output to trigger a subsequent action such as an electronic alert. Choosing this cutoff is not always straightforward. We propose a flexible approach that leverages the collective information in treatment decisions made in real life to learn reference decision thresholds from physician practice. Using the example of prescribing a statin for primary prevention of cardiovascular disease based on 10-year risk calculated by the 2013 pooled cohort equations, we demonstrate the feasibility of using real-world data to learn the implicit decision threshold that reflects existing physician behavior. Learning a decision threshold in this manner allows for evaluation of a proposed operating point against the threshold reflective of the community standard of care. Furthermore, this approach can be used to monitor and audit model-guided clinical decision making following model deployment.

Identifying neonatal early-onset sepsis test and treatment decision thresholds

OBJECTIVE

To derive testing and treatment thresholds for early-onset neonatal sepsis and compare them to thresholds used in the Kaiser-Permanente (KP) Sepsis Calculator.

METHODS

Using surveys distributed in the United States, Brazil and Italy, decision thresholds were derived via self-identified thresholds selected from structured lists (Method 1), and based on clinical vignette responses for testing and treatment with or without inclusion of associated relative risk (Methods 2 and 3).

RESULTS

Using Method 1, both testing and treatment thresholds were higher than the KP calculator thresholds. Test thresholds were lower (Method 2) or equivalent (Method 3) to KP using clinical vignettes. No vignette reached the 50% cutoff necessary to define a treatment threshold.

CONCLUSION

The test threshold used by the KP calculator is the same as the threshold chosen by clinicians given a vignette and risk estimate. The KP treatment threshold is lower than that derived using all 3 methods.

Case Report: Therapeutic Threshold for Rifampicin-Resistant Tuberculosis in a Patient from Maputo, Mozambique

We present a case of a patient in Mozambique, who initiated treatment for rifampicin-resistant tuberculosis (RR-TB) without proof of resistance. For this patient, we estimated the probability of RR-TB using likelihood ratios of clinical arguments. The probability of RR-TB in Mozambique, positive HIV status, and treatment failure after a first treatment and after retreatment were included as confirming arguments, and a rapid molecular test showing rifampicin susceptibility as excluding argument. The therapeutic threshold to start treatment for RR-TB is unknown, but probably lower than 47% and should be calculated to guide clinical decisions.

Accuracy of signs, symptoms and blood tests for diagnosing acute bacterial rhinosinusitis and CT-confirmed acute rhinosinusitis in adults: protocol of an individual patient data meta-analysis

INTRODUCTION

This protocol outlines a diagnostic individual patient data (IPD) meta-analysis aimed at developing simple prediction models based on readily available signs, symptoms and blood tests to accurately predict acute bacterial rhinosinusitis and CT-confirmed (fluid level or total opacification in any sinus) acute rhinosinusitis (ARS) in adults presenting to primary care with clinically diagnosed ARS, target conditions associated with antibiotic benefit.

METHODS AND ANALYSIS

The systematic searches of PubMed and Embase of a review on the accuracy of signs and symptoms for diagnosing ARS in ambulatory care will be updated to April 2020 to identify relevant studies. Authors of eligible studies will be contacted and invited to provide IPD. Methodological quality of the studies will be assessed using the Quality Assessment of Diagnostic Accuracy Studies-2 tool. Candidate predictor selection will be based on knowledge from existing literature, clinical reasoning and availability. Multivariable logistic regression analyses will be used to develop prediction models aimed at calculating absolute risk estimates. Large unexplained between-study heterogeneity in predictive accuracy of the models will be explored and may lead to either model adjustment or derivation of separate context-specific models. Calibration and discrimination will be evaluated to assess the models' performance. Bootstrap resampling techniques will be used to assess internal validation and to inform on possible adjustment for overfitting. In addition, we aim to perform internal-external cross-validation procedures.

ETHICS AND DISSEMINATION

In this IPD meta-analysis, no identifiable patient data will be used. As such, the Medical Research Involving Humans Subject Act does not apply, and official ethical approval is not required. Findings will be published in international peer-reviewed journals and presented at scientific conferences.

PROSPERO REGISTRATION NUMBER

PROSPERO CRD42020175659.

Correlation of clinical decision-making with probability of disease: A web-based study among general practitioners

BACKGROUND

Medical decision-making relies partly on the probability of disease. Current recommendations for the management of common diseases are based increasingly on scores that use arbitrary probability thresholds.

OBJECTIVE

To assess decision-making in pharyngitis and appendicitis using a set of clinical vignettes, and the extent to which management is congruent with the true probability of having the disease.

DESIGN

We developed twenty-four clinical vignettes with clinical presentations corresponding to specific probabilities of having disease defined by McIsaac (pharyngitis) or Alvarado (appendicitis) scores. Each participant answered four randomly selected web-based vignettes.

PARTICIPANTS

General practitioners (GP) working in primary care structures in Switzerland and the USA.

MAIN MEASURES

A comparison between the GP's management decision according to the true probability of having the disease and to the GP's estimated probability, investigating the GP's ability to estimate probability of disease.

KEY RESULTS

The mean age of the GPs was 48 years (SD 12) and 66% were men. The correlation between the GP's clinical management decision based on the vignette and the recommendations was stronger for appendicitis than pharyngitis (kw = 0.74, 95% CI 0.70-0.78 vs. kw = 0.66, 95% CI 0.62-0.71). On the other hand, the association between the clinical management decision and the probability of disease estimated by GPs was more congruent with recommendations for pharyngitis than appendicitis (kw = 0.70, 95% CI 0.66-0.73 vs. 0.61, 95% CI 0.56-0.66). Only a minority of GPs correctly estimated the probability of disease (29% for appendicitis and 39% for pharyngitis).

CONCLUSIONS

Despite the fact that general practitioners often misestimate the probability of disease, their management decisions are usually in line with recommendations. This means that they use other approaches, perhaps more subjective, to make decisions, such as clinical judgment or reasoning that integrate factors other than just the risk of the disease.

How to use clinical signs and symptoms to estimate the probability of limb ischaemia in patients with a diabetic foot ulcer

Assessment of ischaemia and interventions to improve perfusion are key elements in the management of the diabetic foot ulcer to achieve wound healing. This article will review, analyse and interpret the value of clinical information in determining the likelihood of limb ischaemia and thereby the need for referral to a vascular specialist. I conducted an historical review of the genesis of several currently recommended clinical signs and their diagnostic properties in the assessment of limb ischaemia. Changes in limb ischaemia probability based on such results were calculated using Bayes theorem, and the use of such probabilities is discussed in the context of the threshold approach to clinical decision making. Some clinical signs have negligible value in altering probability of limb ischaemia, possibly because they were advocated by Buerger for the diagnosis of thromboangiitis obliterans, an occlusive vascular disease of a different pathophysiology than atherosclerosis. Pedal pulse palpation, the most widely studied clinical sign, will marginally change a pre-test probability of ischaemia of 50% to 76% if positive (abnormal pulses) and to 36% if negative (normal pulses). Higher or lower pre-test probabilities of ischaemia will change the post-test probabilities such that these are too low or high to rule in or rule out ischaemia, respectively. Individual clinical signs convey little information regarding the presence of limb ischaemia but may have some value in combination or with pre-test probability near 50% in the assessment of ischaemia and the decision to refer for a vascular consultation.

Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature

Clinical prediction rules (CPRs) that predict the absolute risk of a clinical condition or future outcome for individual patients are abundant in the medical literature; however, systematic reviews have demonstrated shortcomings in the methodological quality and reporting of prediction studies. To maximise the potential and clinical usefulness of CPRs, they must be rigorously developed and validated, and their impact on clinical practice and patient outcomes must be evaluated. This review aims to present a comprehensive overview of the stages involved in the development, validation and evaluation of CPRs, and to describe in detail the methodological standards required at each stage, illustrated with examples where appropriate. Important features of the study design, statistical analysis, modelling strategy, data collection, performance assessment, CPR presentation and reporting are discussed, in addition to other, often overlooked aspects such as the acceptability, cost-effectiveness and longer-term implementation of CPRs, and their comparison with clinical judgement. Although the development and evaluation of a robust, clinically useful CPR is anything but straightforward, adherence to the plethora of methodological standards, recommendations and frameworks at each stage will assist in the development of a rigorous CPR that has the potential to contribute usefully to clinical practice and decision-making and have a positive impact on patient care.

Study to determine clinical decision thresholds in small animal veterinary practice

This study aimed to determine clinical decision thresholds for six common conditions in small animal veterinary practice. Participants were provided with an online survey. Five questions described scenarios of canine patients with suspected panosteitis, hypothyroidism, urinary tract infection (UTI), mechanical gastrointestinal obstruction (GIO) and idiopathic epilepsy, and one question described a feline patient with suspected chronic kidney disease. A range of probabilities was applied to each scenario. Test and treatment threshold levels were computed for each scenario from 297 usable responses. The test and treatment thresholds were determined for UTI (test=12.8 per cent; 95 per centCI=1.1 to 20.7; treatment=82.0per cent; 95 per centCI=66.3 to 100) and GIO (test=3.2 per cent; 95 per cent CI=0 to 10.4; treatment=87.3 per cent; 95 per centCI=82.6 to 93.5). All other scenarios did not provide data that allowed interpretable test and treatment thresholds. This pilot study has used a new approach in determining clinical thresholds in small animal medicine. Thresholds were successfully determined for two common conditions-canine mechanical GIO and canine UTI. Future research should broaden investigation of methods to determine group clinical threshold levels among veterinarians, which may be used as the basis for clinical decision rules.

Diagnosis and treatment of community-acquired pneumonia in patients with acute cough: a quantitative study of decision thresholds in primary care

BACKGROUND

Test and treatment thresholds have not yet been described for decision-making regarding the likelihood of pneumonia in patients with acute cough.

AIM

To determine decision thresholds in the management of patients with acute cough.

DESIGN AND SETTING

Set among primary care physicians attending meetings in the US and Switzerland, using data from a prospective cohort of primary care patients.

METHOD

Clinical vignettes were used to study the clinical decisions of physicians regarding eight patients with cough that varied by six signs and symptoms. The probability of community-acquired pneumonia (CAP) was determined for each vignette based on a multivariate model. A previously published approach based on logistic regression was used to determine test and treatment thresholds.

RESULTS

In total, 256 physicians made 764 clinical decisions. Initial physician estimates systematically overestimated the likelihood of CAP; 75% estimating a higher probability than that predicted by the multivariate model. Given the probability of CAP from a multivariate model, 16.7% (125 of 749) changed their decision from 'treat' to 'test' or 'test' to 'rule out', whereas only 3.5% (26/749) changed their decision from 'rule out' to 'test' or 'test' to 'treat'. Test and treatment thresholds were 9.5% (95% confidence interval (CI) = 8.7 to 10.5) and 43.1% (95% CI = 40.1 to 46.4) and were updated to 12.7% (95% CI = 11.7 to 13.8) and 51.3% (95% CI = 48.3 to 54.9) once the true probability of CAP was given. Test thresholds were consistent between subgroups. Treatment thresholds were higher if radiography was available, for Swiss physicians, and for non-primary care physicians.

CONCLUSION

Test and treatment thresholds for CAP in patients with acute cough were 9.5% and 43.1%, respectively. Physicians tended to overestimate the likelihood of CAP, and providing information from a clinical decision rule (CDR) changed about 1 in 6 clinical decisions.

Determining clinical decision thresholds for HIV-positive patients suspected of having tuberculosis

Clinical decision thresholds may aid the evaluation of diagnostic tests but have rarely been determined for tuberculosis (TB). We presented clinicians with six web-based clinical scenarios, describing patients with HIV and possible TB at various sites and with a range of clinical stability. The probability of disease was varied randomly and clinicians asked to make treatment decisions; threshold curves and therapeutic thresholds were calculated. Test and treatment thresholds were calculated using Bayes theorem and the diagnostic accuracy of Xpert MTB/RIF. We received 165 replies to our survey. Therapeutic thresholds vary depending on the clinical stability and site of suspected disease. For inpatients, it ranges from 3.4% in unstable to 79.6% in stable patients. For TB meningitis, it ranges from 0% in unstable to 51.4% in stable patients and for pulmonary TB in outpatients it ranges from 29.1% in unstable to 74.5% in the stable patients. Test and treatment thresholds vary in a similar way with test thresholds ranging from 0 in unstable patients with suspected meningitis to 8.2% for stable inpatients. Treatment thresholds vary from 0 for unstable patients with suspected meningitis to 97% for stable inpatients. Therapeutic thresholds for TB can be determined by presenting clinicians with patient scenarios with random probabilities of disease and can be used to calculate test and treatment thresholds using Bayes theorem. Thresholds are lower when patients are more clinically unstable and when the implications of inappropriately withholding therapy are more serious. These results can be used to improve use and evaluation of diagnostic tests.

Classification and Regression Tree (CART) analysis to predict influenza in primary care patients

BACKGROUND

The use of neuraminidase-inhibiting anti-viral medication to treat influenza is relatively infrequent. Rapid, cost-effective methods for diagnosing influenza are needed to enable appropriate prescribing. Multi-viral respiratory panels using reverse transcription polymerase chain reaction (PCR) assays to diagnose influenza are accurate but expensive and more time-consuming than low sensitivity rapid influenza tests. Influenza clinical decision algorithms are both rapid and inexpensive, but most are based on regression analyses that do not account for higher order interactions. This study used classification and regression trees (CART) modeling to estimate probabilities of influenza.

METHODS

Eligible enrollees ≥ 5 years old (n = 4,173) who presented at ambulatory centers for treatment of acute respiratory illness (≤7 days) with cough or fever in 2011-2012, provided nasal and pharyngeal swabs for PCR testing for influenza, information on demographics, symptoms, personal characteristics and self-reported influenza vaccination status.

RESULTS

Antiviral medication was prescribed for just 15 % of those with PCR-confirmed influenza. An algorithm that included fever, cough, and fatigue had sensitivity of 84 %, specificity of 48 %, positive predictive value (PPV) of 23 % and negative predictive value (NPV) of 94 % for the development sample.

CONCLUSIONS

The CART algorithm has good sensitivity and high NPV, but low PPV for identifying influenza among outpatients ≥5 years. Thus, it is good at identifying a group who do not need testing or antivirals and had fair to good predictive performance for influenza. Further testing of the algorithm in other influenza seasons would help to optimize decisions for lab testing or treatment.