Prognostic prediction tools and clinician communication: a qualitative study of the effect of the STUMBL tool on clinical practice

Algorithm Versus Expert: Machine Learning Versus Surgeon-Predicted Symptom Improvement After Carpal Tunnel Release

BACKGROUND AND OBJECTIVES

Surgeons rely on clinical experience when making predictions about treatment effects. Incorporating algorithm-based predictions of symptom improvement after carpal tunnel release (CTR) could support medical decision-making. However, these algorithm-based predictions need to outperform predictions made by surgeons to add value. We compared predictions of a validated prediction model for symptom improvement after CTR with predictions made by surgeons.

METHODS

This cohort study included 97 patients scheduled for CTR. Preoperatively, surgeons estimated each patient's probability of improvement 6 months after surgery, defined as reaching the minimally clinically important difference on the Boston Carpal Tunnel Syndrome Symptom Severity Score. We assessed model and surgeon performance using calibration (calibration belts), discrimination (area under the curve [AUC]), sensitivity, and specificity. In addition, we assessed the net benefit of decision-making based on the prediction model's estimates vs the surgeon's judgement.

RESULTS

The surgeon predictions had poor calibration and suboptimal discrimination (AUC 0.62, 95%-CI 0.49-0.74), while the prediction model showed good calibration and appropriate discrimination (AUC 0.77, 95%-CI 0.66-0.89, P = .05). The accuracy of surgeon predictions was 0.65 (95%-CI 0.37-0.78) vs 0.78 (95%-CI 0.67-0.89) for the prediction model ( P = .03). The sensitivity of surgeon predictions and the prediction model was 0.72 (95%-CI 0.15-0.96) and 0.85 (95%-CI 0.62-0.97), respectively ( P = .04). The specificity of the surgeon predictions was similar to the model's specificity ( P = .25). The net benefit analysis showed better decision-making based on the prediction model compared with the surgeons' decision-making (ie, more correctly predicted improvements and/or fewer incorrectly predicted improvements).

CONCLUSION

The prediction model outperformed surgeon predictions of improvement after CTR in terms of calibration, accuracy, and sensitivity. Furthermore, the net benefit analysis indicated that using the prediction model instead of relying solely on surgeon decision-making increases the number of patients who will improve after CTR, without increasing the number of unnecessary surgeries.

Predicting serious complication risks after bariatric surgery: external validation of the Michigan Bariatric Surgery Collaborative risk prediction model using the Dutch Audit for Treatment of Obesity

BACKGROUND

Risk-prediction tools can support doctor-patient (shared) decision making in clinical practice by providing information on complication risks for different types of bariatric surgery. However, external validation is imperative to ensure the generalizability of predictions in a new patient population.

OBJECTIVE

To perform an external validation of the risk-prediction model for serious complications from the Michigan Bariatric Surgery Collaborative (MBSC) for Dutch bariatric patients using the nationwide Dutch Audit for Treatment of Obesity (DATO).

SETTING

Population-based study, including all 18 hospitals performing bariatric surgery in the Netherlands.

METHODS

All patients registered in the DATO undergoing bariatric surgery between 2015 and 2020 were included as the validation cohort. Serious complications included, among others, abdominal abscess, bowel obstruction, leak, and bleeding. Three risk-prediction models were validated: (1) the original MBSC model from 2011, (2) the original MBSC model including the same variables but updated to more recent patients (2015-2020), and (3) the current MBSC model. The following predictors from the MBSC model were available in the DATO: age, sex, procedure type, cardiovascular disease, and pulmonary disease. Model performance was determined using the area under the curve (AUC) to assess discrimination (i.e., the ability to distinguish patients with events from those without events) and a graphical plot to assess calibration (i.e., whether the predicted absolute risk for patients was similar to the observed prevalence of the outcome).

RESULTS

The DATO validation cohort included 51,291 patients. Overall, 986 patients (1.92%) experienced serious complications. The original MBSC model, which was extended with the predictors "GERD (yes/no)," "OSAS (yes/no)," "hypertension (yes/no)," and "renal disease (yes/no)," showed the best validation results. This model had a good calibration and an AUC of .602 compared with an AUC of .65 and moderate to good calibration in the Michigan model.

CONCLUSION

The DATO prediction model has good calibration but moderate discrimination. To be used in clinical practice, good calibration is essential to accurately predict individual risks in a real-world setting. Therefore, this model could provide valuable information for bariatric surgeons as part of shared decision making in daily practice.

Building consensus on inpatient discharge pathway components in the management of blunt thoracic injuries: An e-Delphi study amongst an international professional expert panel

INTRODUCTION

Access to a standardised and evidence informed approach to blunt thoracic injury (BTI) management remains challenging across organised trauma systems globally. It remains important to optimise recovery through pathway-based interventions. The aim of this study was to identify components of care that are important in the effective discharge process for patients with BTI and pinpoint core and optional components for a patient pathway-based intervention.

METHODS

Components of care within the hospital discharge process after BTI were identified using existing literature and expert opinion. These initial data were entered into a three-round e-Delphi consensus method where round one involved further integrating and categorising components of discharge care from the expert panel. The panel comprised of an international interdisciplinary group of healthcare professionals with experience in the management of BTI. All questionnaires were completed anonymously using an online survey and involved rating care components using Likert scales (Range: 1-6). The final consensus threshold for pathway components were defined as a group rating of greater than 70% scoring in either the moderate importance (3-4) or high importance category (5-6) and less than 15% of the panel scoring within the low importance category (1-2).

RESULTS

Of 88 recruited participants, 67 (76%) participated in round one. Statements were categorised into nine themes: (i) Discharge criteria; (ii) Physical function and Self-care; (iii) Pain management components; (iv) Respiratory function components; (v) General care components; (vi) Follow-up; (vii) Psychological care components; (viii) Patient, family and communication; (ix) 'Red Flag' signs and symptoms. Overall, 70 statements were introduced into the consensus building exercise in round two. In round three, 40 statements from across these categorises achieved consensus amongst the expert panel, forming a framework of core and optional care components within the discharge process after BTI.

CONCLUSIONS

These data will be used to build a toolkit containing guidance on developing discharge pathways for patients with BTI and for the development of audit benchmarks for analysing healthcare provision in this area. It is important that interventions developed using this framework are validated locally and evaluated for efficacy using appropriate research methodology.

Implementation of a calculator to predict cesarean delivery during labor induction: a qualitative evaluation of the clinician perspective

BACKGROUND

We previously conducted a prospective cohort study (n=1610) demonstrating that the implementation of a validated calculator to predict likelihood of cesarean delivery during labor induction was associated with reduced maternal morbidity, reduced cesarean delivery rate, and improved birth satisfaction.

OBJECTIVE

To optimize future implementation, we used qualitative interviews to understand the clinician perspective on: (1) the cesarean delivery risk calculator implementation and (2) the mechanisms by which the use of the calculator resulted in the observed improved outcomes.

STUDY DESIGN

After completion of the prospective study (June 30, 2019), 20 trainees and attending clinicians (including nurse-midwives, obstetrical physicians, and family medicine physicians) at the study site participated in a single, brief semistructured interview from March 1, 2020, to June 30, 2020. Transcriptions were coded using a systematic approach.

RESULTS

Overall, clinicians had favorable perspectives regarding the cesarean delivery risk calculator. Clinicians described the calculator as offering "objective data" and a "standardized snapshot of the labor trajectory." Concerns were raised regarding "overreliance" on calculator output. Barriers to use included time for patient counseling and "awkwardness" around the interactions and perceived patient misunderstanding of the calculator result. Although most senior clinicians (n=8) reported that the calculator did not impact patient management, trainee clinicians (n=12) more often felt that the calculator influenced care at the extremes of cesarean delivery risk. Furthermore, more senior clinicians felt "neutral" regarding any impact of counseling patients on cesarean delivery risk compared with trainee clinicians, who felt that the counseling "built [patient-clinician] trust."

CONCLUSION

This qualitative evaluation characterized the generally positive clinician perspective around the cesarean delivery risk calculator, while identifying specific facilitators and barriers to implementation. In addition, we elucidated potential mechanisms by which the calculator may have been related to clinician decision making and patient-clinician interactions, leading to reduced maternal morbidity and improved patient birth satisfaction. This information is important as widespread implementation of the cesarean delivery risk calculator begins.