Published models that predict hospital readmission: a critical appraisal

Risk-prediction models for intravenous immunoglobulin resistance in Kawasaki disease: Risk-of-Bias Assessment using PROBAST

BACKGROUND

The prediction model of intravenous immunoglobulin (IVIG) resistance in Kawasaki disease can calculate the probability of IVIG resistance and provide a basis for clinical decision-making. We aim to assess the quality of these models developed in the children with Kawasaki disease.

METHODS

Studies of prediction models for IVIG-resistant Kawasaki disease were identified through searches in the PubMed, Web of Science, and Embase databases. Two investigators independently performed literature screening, data extraction, quality evaluation, and discrepancies were settled by a statistician. The checklist for critical appraisal and data extraction for systematic reviews of prediction modeling studies (CHARMS) was used for data extraction, and the prediction models were evaluated using the Prediction Model Risk of Bias Assessment Tool (PROBAST).

RESULTS

Seventeen studies meeting the selection criteria were included in the qualitative analysis. The top three predictors were neutrophil measurements (peripheral neutrophil count and neutrophil %), serum albumin level, and C-reactive protein (CRP) level. The reported area under the curve (AUC) values for the developed models ranged from 0.672 (95% confidence interval [CI]: 0.631-0.712) to 0.891 (95% CI: 0.837-0.945); The studies showed a high risk of bias (ROB) for modeling techniques, yielding a high overall ROB.

CONCLUSION

IVIG resistance models for Kawasaki disease showed high ROB. An emphasis on improving their quality can provide high-quality evidence for clinical practice.

IMPACT STATEMENT

This study systematically evaluated the risk of bias (ROB) of existing prediction models for intravenous immunoglobulin (IVIG) resistance in Kawasaki disease to provide guidance for future model development meeting clinical expectations. This is the first study to systematically evaluate the ROB of IVIG resistance in Kawasaki disease by using PROBAST. ROB may reduce model performance in different populations. Future prediction models should account for this problem, and PROBAST can help improve the methodological quality and applicability of prediction model development.

Frailty Is Superior to Age for Predicting Readmission, Prolonged Length of Stay, and Wound Infection in Elective Otology Procedures

OBJECTIVE

To determine the predictive ability of the 5-point modified frailty index relative to age in elective otology patients.

STUDY DESIGN

Retrospective database analysis.

SETTING

Multicenter, national database of surgical patients.

PATIENTS

We selected all elective surgical patients who received tympanoplasty, tympanomastoidectomy, mastoidectomy, revision mastoidectomy, and cochlear implant procedures from 2016 to 2019 from the National Surgical Quality Improvement database.

INTERVENTIONS

Therapeutic.

MAIN OUTCOME MEASURES

Readmission rates, discharge disposition, reoperation rates, and extended length of hospital stay.

RESULTS

Utilizing receiver operating characteristics with area under the curve (AUC) analysis, nonrobust status was determined to be a superior predictor relative to age of readmission (AUC = 0.628 [p < 0.001] versus AUC = 0.567 [p = 0.047], respectively) and open wound infection relative to age (AUC = 0.636 [p = 0.024] versus AUC = 0.619 [p = 0.048], respectively). Nonrobust otology patients were more likely to have dyspnea at rest and an American Society of Anesthesiology score higher than 2 before surgery (odds ratios, 13.304 [95% confidence interval, 2.947-60.056; p < 0.001] and 7.841 [95% confidence interval, 7.064-8.704; p < 0.001], respectively).

CONCLUSION

Nonrobust status was found to be a useful predictor of readmission and prolonged length of stay in patients undergoing elective otology procedures, which generally have low complication rate. Given the aging population and corresponding increase in otology disease, it is important to use age-independent risk stratification measures. Frailty may provide a useful risk stratification tool to select surgical candidates within the aging population.

Impact of social determinants of health on improving the LACE index for 30-day unplanned readmission prediction

Objective

Early and accurate prediction of patients at risk of readmission is key to reducing costs and improving outcomes. LACE is a widely used score to predict 30-day readmissions. We examine whether adding social determinants of health (SDOH) to LACE can improve its predictive performance.

Methods

This is a retrospective study that included all inpatient encounters in the state of Maryland in 2019. We constructed predictive models by fitting Logistic Regression (LR) on LACE and different sets of SDOH predictors. We used the area under the curve (AUC) to evaluate discrimination and SHapley Additive exPlanations values to assess feature importance.

Results

Our study population included 316 558 patients of whom 35 431 (11.19%) patients were readmitted after 30 days. Readmitted patients had more challenges with individual-level SDOH and were more likely to reside in communities with poor SDOH conditions. Adding a combination of individual and community-level SDOH improved LACE performance from AUC = 0.698 (95% CI [0.695–0.7]; ref) to AUC = 0.708 (95% CI [0.705–0.71]; P < .001). The increase in AUC was highest in black patients (+1.6), patients aged 65 years or older (+1.4), and male patients (+1.4).

Discussion

We demonstrated the value of SDOH in improving the LACE index. Further, the additional predictive value of SDOH on readmission risk varies by subpopulations. Vulnerable populations like black patients and the elderly are likely to benefit more from the inclusion of SDOH in readmission prediction.

Conclusion

These findings provide potential SDOH factors that health systems and policymakers can target to reduce overall readmissions.

Applicability of predictive models for 30-day unplanned hospital readmission risk in paediatrics: a systematic review

OBJECTIVES

To summarise multivariable predictive models for 30-day unplanned hospital readmissions (UHRs) in paediatrics, describe their performance and completeness in reporting, and determine their potential for application in practice.

DESIGN

Systematic review.

DATA SOURCE

CINAHL, Embase and PubMed up to 7 October 2021.

ELIGIBILITY CRITERIA

English or German language studies aiming to develop or validate a multivariable predictive model for 30-day paediatric UHRs related to all-cause, surgical conditions or general medical conditions were included.

DATA EXTRACTION AND SYNTHESIS

Study characteristics, risk factors significant for predicting readmissions and information about performance measures (eg, c-statistic) were extracted. Reporting quality was addressed by the 'Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis' (TRIPOD) adherence form. The study quality was assessed by applying six domains of potential biases. Due to expected heterogeneity among the studies, the data were qualitatively synthesised.

RESULTS

Based on 28 studies, 37 predictive models were identified, which could potentially be used for determining individual 30-day UHR risk in paediatrics. The number of study participants ranged from 190 children to 1.4 million encounters. The two most common significant risk factors were comorbidity and (postoperative) length of stay. 23 models showed a c-statistic above 0.7 and are primarily applicable at discharge. The median TRIPOD adherence of the models was 59% (P₂₅-P₇₅, 55%-69%), ranging from a minimum of 33% to a maximum of 81%. Overall, the quality of many studies was moderate to low in all six domains.

CONCLUSION

Predictive models may be useful in identifying paediatric patients at increased risk of readmission. To support the application of predictive models, more attention should be placed on completeness in reporting, particularly for those items that may be relevant for implementation in practice.

Trust in AI: why we should be designing for APPROPRIATE reliance

Use of artificial intelligence in healthcare, such as machine learning-based predictive algorithms, holds promise for advancing outcomes, but few systems are used in routine clinical practice. Trust has been cited as an important challenge to meaningful use of artificial intelligence in clinical practice. Artificial intelligence systems often involve automating cognitively challenging tasks. Therefore, previous literature on trust in automation may hold important lessons for artificial intelligence applications in healthcare. In this perspective, we argue that informatics should take lessons from literature on trust in automation such that the goal should be to foster appropriate trust in artificial intelligence based on the purpose of the tool, its process for making recommendations, and its performance in the given context. We adapt a conceptual model to support this argument and present recommendations for future work.