Screening for data clustering in multicenter studies: the residual intraclass correlation

How to develop, validate, and update clinical prediction models using multinomial logistic regression

OBJECTIVES

Multicategory prediction models (MPMs) can be used in health care when the primary outcome of interest has more than two categories. The application of MPMs is scarce, possibly due to added methodological complexities compared to binary outcome models. We provide a guide of how to develop, validate, and update clinical prediction models based on multinomial logistic regression.

STUDY DESIGN AND SETTING

We present guidance and recommendations based on recent methodological literature, illustrated by a previously developed and validated MPM for treatment outcomes in rheumatoid arthritis. Prediction models using multinomial logistic regression can be developed for nominal outcomes, but also for ordinal outcomes. This article is intended to supplement existing general guidance on prediction model research.

RESULTS

This guide is split into three parts: 1) outcome definition and variable selection, 2) model development, and 3) model evaluation (including performance assessment, internal and external validation, and model recalibration). We outline how to evaluate and interpret the predictive performance of MPMs. R code is provided.

CONCLUSION

We recommend the application of MPMs in clinical settings where the prediction of a multicategory outcome is of interest. Future methodological research could focus on MPM-specific considerations for variable selection and sample size criteria for external validation.

Adverse Childhood Experiences Among U.S. Adults: National and State Estimates by Adversity Type, 2019-2020

INTRODUCTION

Although adverse childhood experiences (ACEs) are associated with lifelong health harms, current surveillance data on exposures to childhood adversity among adults are either unavailable or incomplete for many states. In this study, recent data from a nationally representative survey were used to obtain the current and complete estimates of ACEs at the national and state levels.

METHODS

Current, complete, by-state estimates of adverse childhood experiences were obtained by applying small area estimation technique to individual-level data on adults aged ≥18 years from 2019-2020 Behavioral Risk Factor Surveillance System survey. The standardized questions about childhood adversity included in the 2019-2020 survey allowed for obtaining estimates of ACE consistent across states. All missing responses to childhood adversity questions (states did not offer such questions or offered them to only some respondents; respondents skipped questions) were predicted through multilevel mixed-effects logistic small area estimation regressions. The analyses were conducted between October 2022 and May 2023.

RESULTS

An estimated 62.8% of U.S. adults had past exposure to ACEs (range: 54.9% in Connecticut; 72.5% in Maine). Emotional abuse (34.5%) was the most common; household member incarceration (10.6%) was the least common. Sexual abuse varied markedly between females (22.2%) and males (5.4%). Exposure to most types of adverse childhood experiences was lowest for adults who were non-Hispanic White, had the highest level of education (college degree) or income (annual income ≥$50,000), or had access to a personal healthcare provider.

CONCLUSIONS

Current complete estimates of ACEs demonstrate high countrywide exposures and stark sociodemographic inequalities in the burden, highlighting opportunities to prevent adverse childhood experiences by focusing social, educational, medical, and public health interventions on populations disproportionately impacted.

Quality of care in patients with hypertension: a retrospective cohort study of primary care routine data in Germany

BACKGROUND

Hypertension is a leading cause of morbidity and mortality if not properly managed. Primary care has a major impact on these outcomes if its strengths, such as continuity of care, are deployed wisely. The analysis aimed to evaluate the quality of care for newly diagnosed hypertension in routine primary care data.

METHODS

In the retrospective cohort study, routine data (from 2016 to 2022) from eight primary care practices in Germany were exported in anonymized form directly from the electronic health record (EHR) systems and processed for this analysis. The analysis focused on five established quality indicators for the care of patients who have been recently diagnosed with hypertension.

RESULTS

A total of 30,691 patients were treated in the participating practices, 2,507 of whom have recently been diagnosed with hypertension. Prior to the pandemic outbreak, 19% of hypertensive patients had blood pressure above 140/90 mmHg and 68% received drug therapy (n = 1,372). After the pandemic outbreak, the proportion of patients with measured blood pressure increased from 63 to 87%, while the other four indicators remained relatively stable. Up to 80% of the total variation of the quality indicators could be explained by individual practices.

CONCLUSION

For the majority of patients, diagnostic procedures are not used to the extent recommended by guidelines. The analysis showed that quality indicators for outpatient care could be mapped onto the basis of routine data. The results could easily be reported to the practices in order to optimize the quality of care.

Clinical Validity of a Machine Learning Decision Support System for Early Detection of Hepatitis B Virus: A Binational External Validation Study

HepB LiveTest is a machine learning decision support system developed for the early detection of hepatitis B virus (HBV). However, there is a lack of evidence on its generalisability. In this study, we aimed to externally assess the clinical validity and portability of HepB LiveTest in predicting HBV infection among independent patient cohorts from Nigeria and Australia. The performance of HepB LiveTest was evaluated by constructing receiver operating characteristic curves and estimating the area under the curve. Delong's method was used to estimate the 95% confidence interval (CI) of the area under the receiver-operating characteristic curve (AUROC). Compared to the Australian cohort, patients in the derivation cohort of HepB LiveTest and the hospital-based Nigerian cohort were younger (mean age, 45.5 years vs. 38.8 years vs. 40.8 years, respectively; p < 0.001) and had a higher incidence of HBV infection (1.9% vs. 69.4% vs. 57.3%). In the hospital-based Nigerian cohort, HepB LiveTest performed optimally with an AUROC of 0.94 (95% CI, 0.91-0.97). The model provided tailored predictions that ensured most cases of HBV infection did not go undetected. However, its discriminatory measure dropped to 0.60 (95% CI, 0.56-0.64) in the Australian cohort. These findings indicate that HepB LiveTest exhibits adequate cross-site transportability and clinical validity in the hospital-based Nigerian patient cohort but shows limited performance in the Australian cohort. Whilst HepB LiveTest holds promise for reducing HBV prevalence in underserved populations, caution is warranted when implementing the model in older populations, particularly in regions with low incidence of HBV infection.

Eating disorders in weight-related therapy (EDIT): Protocol for a systematic review with individual participant data meta-analysis of eating disorder risk in behavioural weight management

The Eating Disorders In weight-related Therapy (EDIT) Collaboration brings together data from randomised controlled trials of behavioural weight management interventions to identify individual participant risk factors and intervention strategies that contribute to eating disorder risk. We present a protocol for a systematic review and individual participant data (IPD) meta-analysis which aims to identify participants at risk of developing eating disorders, or related symptoms, during or after weight management interventions conducted in adolescents or adults with overweight or obesity. We systematically searched four databases up to March 2022 and clinical trials registries to May 2022 to identify randomised controlled trials of weight management interventions conducted in adolescents or adults with overweight or obesity that measured eating disorder risk at pre- and post-intervention or follow-up. Authors from eligible trials have been invited to share their deidentified IPD. Two IPD meta-analyses will be conducted. The first IPD meta-analysis aims to examine participant level factors associated with a change in eating disorder scores during and following a weight management intervention. To do this we will examine baseline variables that predict change in eating disorder risk within intervention arms. The second IPD meta-analysis aims to assess whether there are participant level factors that predict whether participation in an intervention is more or less likely than no intervention to lead to a change in eating disorder risk. To do this, we will examine if there are differences in predictors of eating disorder risk between intervention and no-treatment control arms. The primary outcome will be a standardised mean difference in global eating disorder score from baseline to immediately post-intervention and at 6- and 12- months follow-up. Identifying participant level risk factors predicting eating disorder risk will inform screening and monitoring protocols to allow early identification and intervention for those at risk.

There is no such thing as a validated prediction model

BACKGROUND

Clinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context?

MAIN BODY

We argue to the contrary because (1) patient populations vary, (2) measurement procedures vary, and (3) populations and measurements change over time. Hence, we have to expect heterogeneity in model performance between locations and settings, and across time. It follows that prediction models are never truly validated. This does not imply that validation is not important. Rather, the current focus on developing new models should shift to a focus on more extensive, well-conducted, and well-reported validation studies of promising models.

CONCLUSION

Principled validation strategies are needed to understand and quantify heterogeneity, monitor performance over time, and update prediction models when appropriate. Such strategies will help to ensure that prediction models stay up-to-date and safe to support clinical decision-making.

Hospital effects drive variation in access to inpatient rehabilitation after trauma

BACKGROUND

Postacute care rehabilitation is critically important to recover after trauma, but many patients do not have access. A better understanding of the drivers behind inpatient rehabilitation facility (IRF) use has the potential for major cost-savings as well as higher-quality and more equitable patient care. We sought to quantify the variation in hospital rates of trauma patient discharge to inpatient rehabilitation and understand which factors (patient vs. injury vs. hospital level) contribute the most.

METHODS

We performed a retrospective cohort study of 668,305 adult trauma patients admitted to 900 levels I to IV trauma centers between 2011 and 2015 using the National Trauma Data Bank. Participants were included if they met the following criteria: age >18 years, Injury Severity Score of ≥9, identifiable injury type, and who had one of the Centers for Medicare & Medicaid Services preferred diagnoses for inpatient rehabilitation under the "60% rule."

RESULTS

The overall risk- and reliability-adjusted hospital rates of discharge to IRF averaged 18.8% in the nonelderly adult cohort (18-64 years old) and 23.4% in the older adult cohort (65 years or older). Despite controlling for all patient-, injury-, and hospital-level factors, hospital discharge of patients to IRF varied substantially between hospital quintiles and ranged from 9% to 30% in the nonelderly adult cohort and from 7% to 46% in the older adult cohort. Proportions of total variance ranged from 2.4% (patient insurance) to 12.1% (injury-level factors) in the nonelderly adult cohort and from 0.3% (patient-level factors) to 26.0% (unmeasured hospital-level factors) in the older adult cohort.

CONCLUSION

Among a cohort of injured patients with diagnoses that are associated with significant rehabilitation needs, the hospital at which a patient receives their care may drive a patient's likelihood of recovering at an IRF just as much, if not more, than their clinical attributes.

LEVEL OF EVIDENCE

Care management, level IV.

Ultrasound-based risk model for preoperative prediction of lymph-node metastases in women with endometrial cancer: model-development study

OBJECTIVE

To develop a preoperative risk model, using endometrial biopsy results and clinical and ultrasound variables, to predict the individual risk of lymph-node metastases in women with endometrial cancer.

METHODS

A mixed-effects logistic regression model for prediction of lymph-node metastases was developed in 1501 prospectively included women with endometrial cancer undergoing transvaginal ultrasound examination before surgery, from 16 European centers. Missing data, including missing lymph-node status, were imputed. Discrimination, calibration and clinical utility of the model were evaluated using leave-center-out cross validation. The predictive performance of the model was compared with that of risk classification from endometrial biopsy alone (high-risk defined as endometrioid cancer Grade 3/non-endometrioid cancer) or combined endometrial biopsy and ultrasound (high-risk defined as endometrioid cancer Grade 3/non-endometrioid cancer/deep myometrial invasion/cervical stromal invasion/extrauterine spread).

RESULTS

Lymphadenectomy was performed in 691 women, of whom 127 had lymph-node metastases. The model for prediction of lymph-node metastases included the predictors age, duration of abnormal bleeding, endometrial biopsy result, tumor extension and tumor size according to ultrasound and undefined tumor with an unmeasurable endometrium. The model's area under the curve was 0.73 (95% CI, 0.68-0.78), the calibration slope was 1.06 (95% CI, 0.79-1.34) and the calibration intercept was 0.06 (95% CI, -0.15 to 0.27). Using a risk threshold for lymph-node metastases of 5% compared with 20%, the model had, respectively, a sensitivity of 98% vs 48% and specificity of 11% vs 80%. The model had higher sensitivity and specificity than did classification as high-risk, according to endometrial biopsy alone (50% vs 35% and 80% vs 77%, respectively) or combined endometrial biopsy and ultrasound (80% vs 75% and 53% vs 52%, respectively). The model's clinical utility was higher than that of endometrial biopsy alone or combined endometrial biopsy and ultrasound at any given risk threshold.

CONCLUSIONS

Based on endometrial biopsy results and clinical and ultrasound characteristics, the individual risk of lymph-node metastases in women with endometrial cancer can be estimated reliably before surgery. The model is superior to risk classification by endometrial biopsy alone or in combination with ultrasound. Copyright © 2019 ISUOG. Published by John Wiley & Sons Ltd.

Calibration: the Achilles heel of predictive analytics

BACKGROUND

The assessment of calibration performance of risk prediction models based on regression or more flexible machine learning algorithms receives little attention.

MAIN TEXT

Herein, we argue that this needs to change immediately because poorly calibrated algorithms can be misleading and potentially harmful for clinical decision-making. We summarize how to avoid poor calibration at algorithm development and how to assess calibration at algorithm validation, emphasizing balance between model complexity and the available sample size. At external validation, calibration curves require sufficiently large samples. Algorithm updating should be considered for appropriate support of clinical practice.

CONCLUSION

Efforts are required to avoid poor calibration when developing prediction models, to evaluate calibration when validating models, and to update models when indicated. The ultimate aim is to optimize the utility of predictive analytics for shared decision-making and patient counseling.

Impact of predictor measurement heterogeneity across settings on the performance of prediction models: A measurement error perspective

It is widely acknowledged that the predictive performance of clinical prediction models should be studied in patients that were not part of the data in which the model was derived. Out‐of‐sample performance can be hampered when predictors are measured differently at derivation and external validation. This may occur, for instance, when predictors are measured using different measurement protocols or when tests are produced by different manufacturers. Although such heterogeneity in predictor measurement between derivation and validation data is common, the impact on the out‐of‐sample performance is not well studied. Using analytical and simulation approaches, we examined out‐of‐sample performance of prediction models under various scenarios of heterogeneous predictor measurement. These scenarios were defined and clarified using an established taxonomy of measurement error models. The results of our simulations indicate that predictor measurement heterogeneity can induce miscalibration of prediction and affects discrimination and overall predictive accuracy, to extents that the prediction model may no longer be considered clinically useful. The measurement error taxonomy was found to be helpful in identifying and predicting effects of heterogeneous predictor measurements between settings of prediction model derivation and validation. Our work indicates that homogeneity of measurement strategies across settings is of paramount importance in prediction research.

Rethinking Thirty-Day Hospital Readmissions: Shorter Intervals Might Be Better Indicators Of Quality Of Care

Public reporting and payment programs in the United States have embraced thirty-day readmissions as an indicator of between-hospital variation in the quality of care, despite limited evidence supporting this interval. We examined risk-standardized thirty-day risk of unplanned inpatient readmission at the hospital level for Medicare patients ages sixty-five and older in four states and for three conditions: acute myocardial infarction, heart failure, and pneumonia. The hospital-level quality signal captured in readmission risk was highest on the first day after discharge and declined rapidly until it reached a nadir at seven days, as indicated by a decreasing intracluster correlation coefficient. Similar patterns were seen across states and diagnoses. The rapid decay in the quality signal suggests that most readmissions after the seventh day postdischarge were explained by community- and household-level factors beyond hospitals' control. Shorter intervals of seven or fewer days might improve the accuracy and equity of readmissions as a measure of hospital quality for public accountability.

Familiality and SNP heritability of age at onset and episodicity in major depressive disorder

BACKGROUND

Strategies to dissect phenotypic and genetic heterogeneity of major depressive disorder (MDD) have mainly relied on subphenotypes, such as age at onset (AAO) and recurrence/episodicity. Yet, evidence on whether these subphenotypes are familial or heritable is scarce. The aims of this study are to investigate the familiality of AAO and episode frequency in MDD and to assess the proportion of their variance explained by common single nucleotide polymorphisms (SNP heritability).

METHOD

For investigating familiality, we used 691 families with 2-5 full siblings with recurrent MDD from the DeNt study. We fitted (square root) AAO and episode count in a linear and a negative binomial mixed model, respectively, with family as random effect and adjusting for sex, age and center. The strength of familiality was assessed with intraclass correlation coefficients (ICC). For estimating SNP heritabilities, we used 3468 unrelated MDD cases from the RADIANT and GSK Munich studies. After similarly adjusting for covariates, derived residuals were used with the GREML method in GCTA (genome-wide complex trait analysis) software.

RESULTS

Significant familial clustering was found for both AAO (ICC = 0.28) and episodicity (ICC = 0.07). We calculated from respective ICC estimates the maximal additive heritability of AAO (0.56) and episodicity (0.15). SNP heritability of AAO was 0.17 (p = 0.04); analysis was underpowered for calculating SNP heritability of episodicity.

CONCLUSIONS

AAO and episodicity aggregate in families to a moderate and small degree, respectively. AAO is under stronger additive genetic control than episodicity. Larger samples are needed to calculate the SNP heritability of episodicity. The described statistical framework could be useful in future analyses.

Practical guidance for applying the ADNEX model from the IOTA group to discriminate between different subtypes of adnexal tumors

All gynecologists are faced with ovarian tumors on a regular basis, and the accurate preoperative diagnosis of these masses is important because appropriate management depends on the type of tumor. Recently, the International Ovarian Tumor Analysis (IOTA) consortium published the Assessment of Different NEoplasias in the adneXa (ADNEX) model, the first risk model that differentiates between benign and four types of malignant ovarian tumors: borderline, stage I cancer, stage II-IV cancer, and secondary metastatic cancer. This approach is novel compared to existing tools that only differentiate between benign and malignant tumors, and therefore questions may arise on how ADNEX can be used in clinical practice. In the present paper, we first provide an in-depth discussion about the predictors used in ADNEX and the ability for risk prediction with different tumor histologies. Furthermore, we formulate suggestions about the selection and interpretation of risk cut-offs for patient stratification and choice of appropriate clinical management. This is illustrated with a few example patients. We cannot propose a generally applicable algorithm with fixed cut-offs, because (as with any risk model) this depends on the specific clinical setting in which the model will be used. Nevertheless, this paper provides a guidance on how the ADNEX model may be adopted into clinical practice.

Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study

OBJECTIVES

To develop a risk prediction model to preoperatively discriminate between benign, borderline, stage I invasive, stage II-IV invasive, and secondary metastatic ovarian tumours.

DESIGN

Observational diagnostic study using prospectively collected clinical and ultrasound data.

SETTING

24 ultrasound centres in 10 countries.

PARTICIPANTS

Women with an ovarian (including para-ovarian and tubal) mass and who underwent a standardised ultrasound examination before surgery. The model was developed on 3506 patients recruited between 1999 and 2007, temporally validated on 2403 patients recruited between 2009 and 2012, and then updated on all 5909 patients.

MAIN OUTCOME MEASURES

Histological classification and surgical staging of the mass.

RESULTS

The Assessment of Different NEoplasias in the adneXa (ADNEX) model contains three clinical and six ultrasound predictors: age, serum CA-125 level, type of centre (oncology centres v other hospitals), maximum diameter of lesion, proportion of solid tissue, more than 10 cyst locules, number of papillary projections, acoustic shadows, and ascites. The area under the receiver operating characteristic curve (AUC) for the classic discrimination between benign and malignant tumours was 0.94 (0.93 to 0.95) on temporal validation. The AUC was 0.85 for benign versus borderline, 0.92 for benign versus stage I cancer, 0.99 for benign versus stage II-IV cancer, and 0.95 for benign versus secondary metastatic. AUCs between malignant subtypes varied between 0.71 and 0.95, with an AUC of 0.75 for borderline versus stage I cancer and 0.82 for stage II-IV versus secondary metastatic. Calibration curves showed that the estimated risks were accurate.

CONCLUSIONS

The ADNEX model discriminates well between benign and malignant tumours and offers fair to excellent discrimination between four types of ovarian malignancy. The use of ADNEX has the potential to improve triage and management decisions and so reduce morbidity and mortality associated with adnexal pathology.