A novel approach selected small sets of diagnosis codes with high prediction performance in large healthcare datasets

An Approach for Combining Clinical Judgment with Machine Learning to Inform Medical Decision Making: Analysis of Nonemergency Surgery Strategies for Acute Appendicitis in Patients with Multiple Long-Term Conditions

BACKGROUND

Machine learning (ML) methods can identify complex patterns of treatment effect heterogeneity. However, before ML can help to personalize decision making, transparent approaches must be developed that draw on clinical judgment. We develop an approach that combines clinical judgment with ML to generate appropriate comparative effectiveness evidence for informing decision making.

METHODS

We motivate this approach in evaluating the effectiveness of nonemergency surgery (NES) strategies, such as antibiotic therapy, for people with acute appendicitis who have multiple long-term conditions (MLTCs) compared with emergency surgery (ES). Our 4-stage approach 1) draws on clinical judgment about which patient characteristics and morbidities modify the relative effectiveness of NES; 2) selects additional covariates from a high-dimensional covariate space (P > 500) by applying an ML approach, least absolute shrinkage and selection operator (LASSO), to large-scale administrative data (N = 24,312); 3) generates estimates of comparative effectiveness for relevant subgroups; and 4) presents evidence in a suitable form for decision making.

RESULTS

This approach provides useful evidence for clinically relevant subgroups. We found that overall NES strategies led to increases in the mean number of days alive and out-of-hospital compared with ES, but estimates differed across subgroups, ranging from 21.2 (95% confidence interval: 1.8 to 40.5) for patients with chronic heart failure and chronic kidney disease to -10.4 (-29.8 to 9.1) for patients with cancer and hypertension. Our interactive tool for visualizing ML output allows for findings to be customized according to the specific needs of the clinical decision maker.

CONCLUSIONS

This principled approach of combining clinical judgment with an ML approach can improve trust, relevance, and usefulness of the evidence generated for clinical decision making.

HIGHLIGHTS

Machine learning (ML) methods have many potential applications in medical decision making, but the lack of model interpretability and usability constitutes an important barrier for the wider adoption of ML evidence in practice.We develop a 4-stage approach for integrating clinical judgment into the way an ML approach is used to estimate and report comparative effectiveness.We illustrate the approach in undertaking an evaluation of nonemergency surgery (NES) strategies for acute appendicitis in patients with multiple long-term conditions and find that NES strategies lead to better outcomes compared with emergency surgery and that the effects differ across subgroups.We develop an interactive tool for visualizing the results of this study that allows findings to be customized according to the user's preferences.

The future of cancer care in the UK-time for a radical and sustainable National Cancer Plan

Cancer affects one in two people in the UK and the incidence is set to increase. The UK National Health Service is facing major workforce deficits and cancer services have struggled to recover after the COVID-19 pandemic, with waiting times for cancer care becoming the worst on record. There are severe and widening disparities across the country and survival rates remain unacceptably poor for many cancers. This is at a time when cancer care has become increasingly complex, specialised, and expensive. The current crisis has deep historic roots, and to be reversed, the scale of the challenge must be acknowledged and a fundamental reset is required. The loss of a dedicated National Cancer Control Plan in England and Wales, poor operationalisation of plans elsewhere in the UK, and the closure of the National Cancer Research Institute have all added to a sense of strategic misdirection. The UK finds itself at a crossroads, where the political decisions of governments, the cancer community, and research funders will determine whether we can, together, achieve equitable, affordable, and high-quality cancer care for patients that is commensurate with our wealth, and position our outcomes among the best in the world. In this Policy Review, we describe the challenges and opportunities that are needed to develop radical, yet sustainable plans, which are comprehensive, evidence-based, integrated, patient-outcome focused, and deliver value for money.

Case-Ascertainment Models to Identify Adults with Obstructive Sleep Apnea Using Health Administrative Data: Internal and External Validation

Background

There is limited evidence on whether obstructive sleep apnea (OSA) can be accurately identified using health administrative data.

Study Design and Methods

We derived and validated a case-ascertainment model to identify OSA using linked provincial health administrative and clinical data from all consecutive adults who underwent a diagnostic sleep study (index date) at two large academic centers (Ontario, Canada) from 2007 to 2017. The presence of moderate/severe OSA (an apnea–hypopnea index≥15) was defined using clinical data. Of 39 candidate health administrative variables considered, 32 were tested. We used classification and regression tree (CART) methods to identify the most parsimonious models via cost-complexity pruning. Identified variables were also used to create parsimonious logistic regression models. All individuals with an estimated probability of 0.5 or greater using the predictive models were classified as having OSA.

Results

The case-ascertainment models were derived and validated internally through bootstrapping on 5099 individuals from one center (33% moderate/severe OSA) and validated externally on 13,486 adults from the other (45% moderate/severe OSA). On the external cohort, parsimonious models demonstrated c-statistics of 0.75–0.81, sensitivities of 59–60%, specificities of 87–88%, positive predictive values of 79%, negative predictive values of 73%, positive likelihood ratios (+LRs) of 4.5–5.0 and –LRs of 0.5. Logistic models performed better than CART models (mean integrated calibration indices of 0.02–0.03 and 0.06–0.12, respectively). The best model included: sex, age, and hypertension at the index date, as well as an outpatient specialty physician visit for OSA, a repeated sleep study, and a positive airway pressure treatment claim within 1 year since the index date.

Interpretation

Among adults who underwent a sleep study, case-ascertainment models for identifying moderate/severe OSA using health administrative data had relatively low sensitivity but high specificity and good discriminative ability. These findings could help study trends and outcomes of OSA individuals using routinely collected health care data.

Logistic regression and machine learning predicted patient mortality from large sets of diagnosis codes comparably

OBJECTIVE

The objective of the study was to compare the performance of logistic regression and boosted trees for predicting patient mortality from large sets of diagnosis codes in electronic healthcare records.

STUDY DESIGN AND SETTING

We analyzed national hospital records and official death records for patients with myocardial infarction (n = 200,119), hip fracture (n = 169,646), or colorectal cancer surgery (n = 56,515) in England in 2015-2017. One-year mortality was predicted from patient age, sex, and socioeconomic status, and 202 to 257 International Classification of Diseases 10th Revision codes recorded in the preceding year or not (binary predictors). Performance measures included the c-statistic, scaled Brier score, and several measures of calibration.

RESULTS

One-year mortality was 17.2% (34,520) after myocardial infarction, 27.2% (46,115) after hip fracture, and 9.3% (5,273) after colorectal surgery. Optimism-adjusted c-statistics for the logistic regression models were 0.884 (95% confidence interval [CI]: 0.882, 0.886), 0.798 (0.796, 0.800), and 0.811 (0.805, 0.817). The equivalent c-statistics for the boosted tree models were 0.891 (95% CI: 0.889, 0.892), 0.804 (0.802, 0.806), and 0.803 (0.797, 0.809). Model performance was also similar when measured using scaled Brier scores. All models were well calibrated overall.

CONCLUSION

In large datasets of electronic healthcare records, logistic regression and boosted tree models of numerous diagnosis codes predicted patient mortality comparably.