A method to describe physician decision thresholds and its application in examining the diagnosis of coronary artery disease based on exercise treadmill testing

Artificial intelligence suppression as a strategy to mitigate artificial intelligence automation bias

BACKGROUND

Incorporating artificial intelligence (AI) into clinics brings the risk of automation bias, which potentially misleads the clinician's decision-making. The purpose of this study was to propose a potential strategy to mitigate automation bias.

METHODS

This was a laboratory study with a randomized cross-over design. The diagnosis of anterior cruciate ligament (ACL) rupture, a common injury, on magnetic resonance imaging (MRI) was used as an example. Forty clinicians were invited to diagnose 200 ACLs with and without AI assistance. The AI's correcting and misleading (automation bias) effects on the clinicians' decision-making processes were analyzed. An ordinal logistic regression model was employed to predict the correcting and misleading probabilities of the AI. We further proposed an AI suppression strategy that retracted AI diagnoses with a higher misleading probability and provided AI diagnoses with a higher correcting probability.

RESULTS

The AI significantly increased clinicians' accuracy from 87.2%±13.1% to 96.4%±1.9% (P < .001). However, the clinicians' errors in the AI-assisted round were associated with automation bias, accounting for 45.5% of the total mistakes. The automation bias was found to affect clinicians of all levels of expertise. Using a logistic regression model, we identified an AI output zone with higher probability to generate misleading diagnoses. The proposed AI suppression strategy was estimated to decrease clinicians' automation bias by 41.7%.

CONCLUSION

Although AI improved clinicians' diagnostic performance, automation bias was a serious problem that should be addressed in clinical practice. The proposed AI suppression strategy is a practical method for decreasing automation bias.

A Quantitative Study of Decision Thresholds for Initiation of Antibiotics in Suspected Sepsis

BACKGROUND

Clinicians' decision thresholds for initiating antibiotics in patients with suspected sepsis have not been quantified. We aimed to define an average threshold of infection likelihood at which clinicians initiate antibiotics when treating a patient with suspected infection and to evaluate the influence of severity of illness and clinician-related factors on the threshold.

DESIGN

This was a prospective survey of 153 clinicians responding to 8 clinical vignettes constructed from real-world data from 3 health care systems in the United States. We treated each hour in the vignette as a decision to treat or not treat with antibiotics and assigned an infection probability to each hour using a previously developed infection prediction model. We then estimated decision thresholds using regression models based on the timing of antibiotic initiation. We compared thresholds across categories of severity of illness and clinician-related factors.

RESULTS

Overall, the treatment threshold occurred at a 69% probability of infection, but the threshold varied significantly across severity of illness categories-when patients had high severity of illness, the treatment threshold occurred at a 55% probability of infection; when patients had intermediate severity, the threshold for antibiotic initiation occurred at an infection probability of 69%, and the threshold was 84% when patients had low severity of illness (P < 0.001 for group differences). Thresholds differed significantly across specialty, highest among infectious disease and lowest among emergency medicine clinicians and across years of experience, decreasing with increasing years of experience.

CONCLUSIONS

The threshold infection probability above which physicians choose to initiate antibiotics in suspected sepsis depends on illness severity as well as clinician factors.

IMPLICATIONS

Incorporating these context-dependent thresholds into discriminating and well-calibrated models will inform the development of future sepsis clinical decision support systems. Clinician-related differences in treatment thresholds suggests potential unwarranted variation and opportunities for performance improvement.

HIGHLIGHTS

Decision making about antibiotic initiation in suspected sepsis occurs under uncertainty, and little is known about clinicians' thresholds for treatment.In this prospective study, 153 clinicians from 3 health care systems reviewed 8 real-world clinical vignettes representing patients with sepsis and indicated the time that they would initiate antibiotics.Using a model-based approach, we estimated decision thresholds and found that thresholds differed significantly across illness severity categories and by clinician specialty and years of experience.

Learning decision thresholds for risk stratification models from aggregate clinician behavior

Using a risk stratification model to guide clinical practice often requires the choice of a cutoff—called the decision threshold—on the model’s output to trigger a subsequent action such as an electronic alert. Choosing this cutoff is not always straightforward. We propose a flexible approach that leverages the collective information in treatment decisions made in real life to learn reference decision thresholds from physician practice. Using the example of prescribing a statin for primary prevention of cardiovascular disease based on 10-year risk calculated by the 2013 pooled cohort equations, we demonstrate the feasibility of using real-world data to learn the implicit decision threshold that reflects existing physician behavior. Learning a decision threshold in this manner allows for evaluation of a proposed operating point against the threshold reflective of the community standard of care. Furthermore, this approach can be used to monitor and audit model-guided clinical decision making following model deployment.

Study to determine clinical decision thresholds in small animal veterinary practice

This study aimed to determine clinical decision thresholds for six common conditions in small animal veterinary practice. Participants were provided with an online survey. Five questions described scenarios of canine patients with suspected panosteitis, hypothyroidism, urinary tract infection (UTI), mechanical gastrointestinal obstruction (GIO) and idiopathic epilepsy, and one question described a feline patient with suspected chronic kidney disease. A range of probabilities was applied to each scenario. Test and treatment threshold levels were computed for each scenario from 297 usable responses. The test and treatment thresholds were determined for UTI (test=12.8 per cent; 95 per centCI=1.1 to 20.7; treatment=82.0per cent; 95 per centCI=66.3 to 100) and GIO (test=3.2 per cent; 95 per cent CI=0 to 10.4; treatment=87.3 per cent; 95 per centCI=82.6 to 93.5). All other scenarios did not provide data that allowed interpretable test and treatment thresholds. This pilot study has used a new approach in determining clinical thresholds in small animal medicine. Thresholds were successfully determined for two common conditions-canine mechanical GIO and canine UTI. Future research should broaden investigation of methods to determine group clinical threshold levels among veterinarians, which may be used as the basis for clinical decision rules.

A novel approach to the determination of clinical decision thresholds

Our objective was to determine the test and treatment thresholds for common acute primary care conditions. We presented 200 clinicians with a series of web-based clinical vignettes, describing patients with possible influenza, acute coronary syndrome (ACS), pneumonia, deep vein thrombosis (DVT) and urinary tract infection (UTI). We randomly varied the probability of disease and asked whether the clinician wanted to rule out disease, order tests or rule in disease. By randomly varying the probability, we obtained clinical decisions across a broad range of disease probabilities that we used to create threshold curves. For influenza, the test (4.5% vs 32%, p<0.001) and treatment (55% vs 68%, p=0.11) thresholds were lower for US compared with Swiss physicians. US physicians had somewhat higher test (3.8% vs 0.7%, p=0.107) and treatment (76% vs 58%, p=0.005) thresholds for ACS than Swiss physicians. For both groups, the range between test and treatment thresholds was greater for ACS than for influenza (which is sensible, given the consequences of incorrect diagnosis). For pneumonia, US physicians had a trend towards higher test thresholds and lower treatment thresholds (48% vs 64%, p=0.076) than Swiss physicians. The DVT and UTI scenarios did not provide easily interpretable data, perhaps due to poor wording of the vignettes. We have developed a novel approach for determining decision thresholds. We found important differences in thresholds for US and Swiss physicians that may be a function of differences in healthcare systems. Our results can also guide development of clinical decision rules and guidelines.

Comparison of an effect-model-law-based method versus traditional clinical practice guidelines for optimal treatment decision-making: application to statin treatment in the French population

Healthcare authorities make difficult decisions about how to spend limited budgets for interventions that guarantee the best cost-efficacy ratio. We propose a novel approach for treatment decision-making, OMES-in French: Objectif thérapeutique Modèle Effet Seuil (in English: Therapeutic Objective-Threshold-Effect Model; TOTEM). This approach takes into consideration results from clinical trials, adjusted for the patients' characteristics in treatment decision-making. We compared OMES with the French clinical practice guidelines (CPGs) for the management of dyslipidemia with statin in a computer-generated realistic virtual population, representing the adult French population, in terms of the number of all-cause deaths avoided (number of avoided events: NAEs) under treatment and the individual absolute benefit. The total budget was fixed at the annual amount reimbursed by the French social security for statins. With the CPGs, the NAEs was 292 for an annual cost of 122.54 M€ compared with 443 with OMES. For a fixed NAEs, OMES reduced costs by 50% (60.53 M€ yr(-1)). The results demonstrate that OMES is at least as good as, and even better than, the standard CPGs when applied to the same population. Hence the OMES approach is a practical, useful alternative which will help to overcome the limitations of treatment decision-making based uniquely on CPGs.

Observational research--opportunities and limitations

Medical research continues to progress in its ability to identify treatments and characteristics associated with benefits and adverse outcomes. The principal engine for the evaluation of treatment efficacy is the randomized controlled trial (RCT). Due to the cost and other considerations, RCTs cannot address all clinically important decisions. Observational research often is used to address issues not addressed or not addressable by RCTs. This article provides an overview of the benefits and limitations of observational research to serve as a guide to the interpretation of this category of research designs in diabetes investigations. The potential for bias is higher in observational research but there are design and analysis features that can address these concerns although not completely eliminate them. Pharmacoepidemiologic research may provide important information regarding relative safety and effectiveness of diabetes pharmaceuticals. Such research must effectively address the important issue of confounding by indication in order to produce clinically meaningful results. Other methods such as instrumental variable analysis are being employed to enable stronger causal inference but these methods also require fulfillment of several key assumptions that may or may not be realistic. Nearly all clinical decisions involve probabilistic reasoning and confronting uncertainly, so a realistic goal for observational research may not be the high standard set by RCTs but instead the level of certainty needed to influence a diagnostic or treatment decision.

A new method for determining physician decision thresholds using empiric, uncertain recommendations

Background

The concept of risk thresholds has been studied in medical decision making for over 30 years. During that time, physicians have been shown to be poor at estimating the probabilities required to use this method. To better assess physician risk thresholds and to more closely model medical decision making, we set out to design and test a method that derives thresholds from actual physician treatment recommendations. Such an approach would avoid the need to ask physicians for estimates of patient risk when trying to determine individual thresholds for treatment. Assessments of physician decision making are increasingly relevant as new data are generated from clinical research. For example, recommendations made in the setting of ocular hypertension are of interest as a large clinical trial has identified new risk factors that should be considered by physicians. Precisely how physicians use this new information when making treatment recommendations has not yet been determined.

Results

We derived a new method for estimating treatment thresholds using ordinal logistic regression and tested it by asking ophthalmologists to review cases of ocular hypertension before expressing how likely they would be to recommend treatment. Fifty-eight physicians were recruited from the American Glaucoma Society. Demographic information was collected from the participating physicians and the treatment threshold for each physician was estimated. The method was validated by showing that while treatment thresholds varied over a wide range, the most common values were consistent with the 10-15% 5-year risk of glaucoma suggested by expert opinion and decision analysis.

Conclusions

This method has advantages over prior means of assessing treatment thresholds. It does not require physicians to explicitly estimate patient risk and it allows for uncertainty in the recommendations. These advantages will make it possible to use this method when assessing interventions intended to alter clinical decision making.

MedView: an instrument for clinical research and education in oral medicine

The etiology for many of the mucosal lesions we encounter in clinical practice is frequently uncertain or unknown and there is reason to believe that multicausality plays an important role. To detect multicausal relationships, the analysis must include multiple variables and large amounts of data. A traditional retrospective analysis is often based on a limited number of variables and frequently entails methodological errors where vital information may be missing. Prospective studies may be hampered by the fact that the prevalences of many conditions are relatively low. The search for new knowledge in oral medicine should therefore be facilitated by prospective use of formalized information gathered in multicenter studies. MedView is a computer program that is based on formalized input and registration of all clinical information. The output applications are focused on visualization and statistical analysis. MedView is aimed at clinical research and is well suited for multicenter studies. It also contains applications for education and distant consultations.

Emerging paradigms of cognition in medical decision-making

The limitations of the classical or traditional paradigm of decision research are increasingly apparent, even though there has been a substantial body of empirical research on medical decision-making over the past 40 years. As decision-support technology continues to proliferate in medical settings, it is imperative that "basic science" decision research develop a broader-based and more valid foundation for the study of medical decision-making as it occurs in the natural setting. This paper critically reviews both traditional and recent approaches to medical decision making, considering the integration of problem-solving and decision-making research paradigms, the role of conceptual knowledge in decision-making, and the emerging paradigm of naturalistic decision-making. We also provide an examination of technology-mediated decision-making. Expanding the scope of decision research will better enable us to understand optimal decision processes, suitable coping mechanisms under suboptimal conditions, the development of expertise in decision-making, and ways in which decision-support technology can successfully mediate decision processes.

Conceptual knowledge and decision strategies in relation to hypercholesterolemia and coronary heart disease

This paper reports on a study that examines physicians' understanding of concepts and decision making in problems pertaining to hypercholesterolemia and coronary heart disease (CHD). The study was carried out in two phases: (1) a simulated clinical interview in which two clinical problems were presented and (2) a session in which subjects responded to a series of questions. The questions were related to the analysis of risk factors, diagnostic criteria (DC) for determining elevated lipid values, and differential diagnosis for lipid disorders. The subjects included 12 family practitioners who were randomly selected from a continuing medical education program at McGill University. The results indicate that all subjects exhibited gaps in their understanding of domain concepts. In particular, most physicians demonstrated a lack of knowledge concerning the primary genetic disorders that contribute to CHD, as well as deficiencies in understanding the secondary causes of hypercholesterolemia. The majority of subjects tended to overestimate the lipid value intervals for determining patients at high risk. Physicians had no difficulty diagnosing the first patient problem of familial hypercholesterolemia, but failed to identify the problem of elevated lipids secondary to hypothyroidism. We observed a dissociation between subjects' conceptual understanding and their application of knowledge in solving patient problems. The implications of this work are discussed in terms of the cognitive dimensions of technologies for supporting learning and evidence-based decision making.

Application of treatment thresholds to diagnostic-test evaluation: an alternative to the comparison of areas under receiver operating characteristic curves

Diagnostic tests are often evaluated by comparison of the areas under receiver operating characteristic (ROC) curves. In this study the authors compared this approach with a more direct method that takes into account consequences of a diagnosis. Data from a prospective study of diagnosis of pulmonary embolism were used for a motivating example. Using multivariable logistic regression analysis, three diagnostic models were built and compared based on their ROC curves. Although model 1 (0.706) and model 2 (0.702) had the same ROC-curve area, they performed differently when risks and benefits of subsequent decisions were considered by applying the treatment probability threshold. Models 1 and 3 (0.611) had substantially different ROC-curve areas but performed similarly taking into account the therapeutic consequences. This demonstrates that comparison of diagnostic tests using the areas under the ROC curves may lead to erroneous conclusions about therapeutic usefulness. To correspond to daily practice, it would be more appropriate to also consider the clinical implications in evaluating diagnostic tests. This is made feasible by explicit definition and application of a treatment threshold.