External Validation of a Prediction Model for Falls in Older People Based on Electronic Health Records in Primary Care

OBJECTIVE

Early identification of older people at risk of falling is the cornerstone of fall prevention. Many fall prediction tools exist but their external validity is lacking. External validation is a prerequisite before application in clinical practice. Models developed with electronic health record (EHR) data are especially challenging because of the uncontrolled nature of routinely collected data. We aimed to externally validate our previously developed and published prediction model for falls, using a large cohort of community-dwelling older people derived from primary care EHR data.

DESIGN

Retrospective analysis of a prospective cohort drawn from EHR data.

SETTING AND PARTICIPANTS

Pseudonymized EHR data were collected from individuals aged ≥65 years, who were enlisted in any of the participating 59 general practices between 2015 and 2020 in the Netherlands.

METHODS

Ten predictors were defined and obtained using the same methods as in the development study. The outcome was 1-year fall and was obtained from free text. Both reproducibility and transportability were evaluated. Model performance was assessed in terms of discrimination using the area under the receiver operating characteristic curve (ROC-AUC), and in terms of calibration, using calibration-in-the-large, calibration slope and calibration plots.

RESULTS

Among 39,342 older people, 5124 (13.4%) fell in the 1-year follow-up. The characteristics of the validation and the development cohorts were similar. ROC-AUCs of the validation and development cohort were 0.690 and 0.705, respectively. Calibration-in-the-large and calibration slope were 0.012 and 0.878, respectively. Calibration plots revealed overprediction for high-risk groups in a small number of individuals.

CONCLUSIONS AND IMPLICATIONS

Our previously developed prediction model for falls demonstrated good external validity by reproducing its predictive performance in the validation cohort. The implementation of this model in the primary care setting could be considered after impact assessment.

Can National Registries Contribute to Predict the Risk of Cancer? The Cancer Risk Assessment Model (CRAM)

Simple Summary

Early identification of individuals with an increased risk of cancer is an important challenge. Danish administrative registers may be useful in this respect because they cover the entire population and include comprehensive and consistently coded long-term data. We aimed to develop a predictive model based on Danish administrative registers to facilitate the automated identification of individuals at risk of any type of cancer. In addition to age, almost all the included factors contributed statistically significantly, but also only marginally, to the prediction models, which means that we have not overlooked obvious information available in the register. Future prediction studies should focus on specific cancer types where more precise risk estimations might be expected. It is our ultimate ambition that an effective model can be used at the point of care, integrated into electronic patient record systems to alert physicians of patients at a high risk of cancer.

Abstract

Purpose: To develop a predictive model based on Danish administrative registers to facilitate automated identification of individuals at risk of any type of cancer. Methods: A nationwide register-based cohort study covering all individuals in Denmark aged +20 years. The outcome was all-type cancer during 2017 excluding nonmelanoma skin cancer. Diagnoses, medication, and contact with general practitioners in the exposure period (2007–2016) were considered for the predictive model. We applied backward selection to all variables by logistic regression to develop a risk model for cancer. We applied the models to the validation cohort, calculated the receiver operating characteristic curves, and estimated the corresponding areas under the curve (AUC). Results: The study population consisted of 4.2 million persons; 32,447 (0.76%) were diagnosed with cancer in 2017. We identified 39 predictive risk factors in women and 42 in men, with age above 30 as the strongest predictor for cancer. Testing the model for cancer risk showed modest accuracy, with an AUC of 0.82 (95% CI 0.81–0.82) for men and 0.75 (95% CI 0.74–0.75) for women. Conclusion: We have developed and tested a model for identifying the individual risk of cancer through the use of administrative data. The models need to be further investigated before being applied to clinical practice.

Gait Characteristics and Cognitive Function in Middle-Aged Adults with and without Type 2 Diabetes Mellitus: Data from ENBIND

Type 2 Diabetes Mellitus (T2DM) in midlife is associated with a greater risk of dementia in later life. Both gait speed and spatiotemporal gait characteristics have been associated with later cognitive decline in community-dwelling older adults. Thus, the assessment of gait characteristics in uncomplicated midlife T2DM may be important in selecting-out those with T2DM at greatest risk of later cognitive decline. We assessed the relationship between Inertial Motion Unit (IMUs)-derived gait characteristics and cognitive function assessed via Montreal Cognitive Assessment (MoCA)/detailed neuropsychological assessment battery (CANTAB) in middle-aged adults with and without uncomplicated T2DM using both multivariate linear regression and a neural network approach. Gait was assessed under (i) normal walking, (ii) fast (maximal) walking and (iii) cognitive dual-task walking (reciting alternate letters of the alphabet) conditions. Overall, 138 individuals were recruited (n = 94 with T2DM; 53% female, 52.8 ± 8.3 years; n = 44 healthy controls, 43% female, 51.9 ± 8.1 years). Midlife T2DM was associated with significantly slower gait velocity on both slow and fast walks (both p < 0.01) in addition to a longer stride time and greater gait complexity during normal walk (both p < 0.05). Findings persisted following covariate adjustment. In analyzing cognitive performance, the strongest association was observed between gait velocity and global cognitive function (MoCA). Significant associations were also observed between immediate/delayed memory performance and gait velocity. Analysis using a neural network approach did not outperform multivariate linear regression in predicting cognitive function (MoCA) from gait velocity. Our study demonstrates the impact of uncomplicated T2DM on gait speed and gait characteristics in midlife, in addition to the striking relationship between gait characteristics and global cognitive function/memory performance in midlife. Further studies are needed to evaluate the longitudinal relationship between midlife gait characteristics and later cognitive decline, which may aid in selecting-out those with T2DM at greatest-risk for preventative interventions.

Validation of two IgA nephropathy risk-prediction tools using a cohort with a long follow-up

BACKGROUND

Recently, two immunoglobulin A nephropathy prediction tools were developed that combine clinical and histopathological parameters. The International IgAN Prediction Tool predicts the risk for 50% declines in the estimated glomerular filtration rate or end-stage renal disease up to 80 months after diagnosis. The IgA Nephropathy Clinical Decision Support System uses artificial neural networks to estimate the risk for end-stage renal disease. We aimed to externally validate both prediction tools using a Norwegian cohort with a long-term follow-up.

METHODS

We included 306 patients with biopsy-proven primary immunoglobulin A nephropathy in this study. Histopathologic samples were retrieved from the Norwegian Kidney Biopsy Registry and reclassified according to the Oxford classification. We used discrimination and calibration as principles for externally validating the prognostic models.

RESULTS

The median patient follow-up was 17.1 years. A cumulative dynamic time-dependent receiver operating characteristic analysis showed area under the curve values of ranging from 0.90 at 5 years to 0.83 at 20 years for the International IgAN Prediction Tool, while time-naive analysis showed an area under the curve value at 0.83 for the IgA Nephropathy Clinical Decision Support System. The International IgAN Prediction Tool was well calibrated, while the IgA Nephropathy Clinical Decision Support System tends to underestimate risk for patients with higher risk, and overestimates risk in the lower risk categories.

CONCLUSIONS

We have externally validated two prediction tools for IgA nephropathy. The International IgAN Prediction Tool performed well, while the IgA Nephropathy Clinical Decision Support System has some limitations.

Machine Learning-Based Models Enhance the Prediction of Prostate Cancer

Purpose

PSA is currently the most commonly used screening indicator for prostate cancer. However, it has limited specificity for the diagnosis of prostate cancer. We aim to construct machine learning-based models and enhance the prediction of prostate cancer.

Methods

The data of 551 patients who underwent prostate biopsy were retrospectively retrieved and divided into training and test datasets in a 3:1 ratio. We constructed five PCa prediction models with four supervised machine learning algorithms, including tPSA univariate logistic regression (LR), multivariate LR, decision tree (DT), random forest (RF), and support vector machine (SVM). The five prediction models were compared based on model performance metrics, such as the area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, specificity, calibration curve, and clinical decision curve analysis (DCA).

Results

All five models had good calibration in the training dataset. In the training dataset, the RF, DT, and multivariate LR models showed better discrimination, with AUCs of 1.0, 0.922 and 0.91, respectively, than the tPSA univariate LR and SVM models. In the test dataset, the multivariate LR model exhibited the best discrimination (AUC=0.918). The multivariate LR model and SVM model had better extrapolation and generalizability, with little change in performance between the training and test datasets. Compared with the DCA curves of the tPSA LR model, the other four models exhibited better net clinical benefits.

Conclusion

The results of the current retrospective study suggest that machine learning techniques can predict prostate cancer with significantly better AUC, accuracy, and net clinical benefits.

Artificial intelligence in healthcare: a primer for medical education in radiomics

The application of artificial intelligence (AI) to healthcare has garnered significant enthusiasm in recent years. Despite the adoption of new analytic approaches, medical education on AI is lacking. We aim to create a usable AI primer for medical education. We discuss how to generate a clinical question involving AI, what data are suitable for AI research, how to prepare a dataset for training and how to determine if the output has clinical utility. To illustrate this process, we focused on an example of how medical imaging is employed in designing a machine learning model. Our proposed medical education curriculum addresses AI's potential and limitations for enhancing clinicians' skills in research, applied statistics and care delivery.

ABV-CoViD: An Ensemble Forecasting Model to Predict Availability of Beds and Ventilators for COVID-19 Like Pandemics

Recently, healthcare stakeholders have orchestrated steps to strengthen and curb the COVID-19 wave. There has been a surge in vaccinations to curb the virus wave, but it is crucial to strengthen our healthcare resources to fight COVID-19 and like pandemics. Recent researchers have suggested effective forecasting models for COVID-19 transmission rate, spread, and the number of positive cases, but the focus on healthcare resources to meet the current spread is not discussed. Motivated from the gap, in this paper, we propose a scheme, ABV-CoViD (Availibility of Beds and Ventilators for COVID-19 patients), that forms an ensemble forecasting model to predict the availability of beds and ventilators (ABV) for the COVID-19 patients. The scheme considers a region-wise demarcation for the allotment of beds and ventilators (BV), termed resources, based on region-wise ABV and COVID-19 positive patients (inside the hospitals occupying the BV resource). We consider an integration of artificial neural network (ANN) and auto-regressive integrated neural network (ARIMA) model to address both the linear and non-linear dependencies. We also consider the effective wave spread of COVID-19 on external patients (not occupying the BV resources) through a [Formula: see text]- ARNN model, which gives us long-term complex dependencies of BV resources in the future time window. We have considered the COVID-19 healthcare dataset on 3 USA regions (Illinois, Michigan, and Indiana) for testing our ensemble forecasting scheme from January 2021 to May 2022. We evaluated our scheme in terms of statistical performance metrics and validated that ensemble methods have higher accuracy. In simulation, for linear modelling, we considered the [Formula: see text] model, and [Formula: see text] model for ANN modelling. We considered the [Formula: see text](12,6) forecasting. On a population of 2,93,90,897, the obtained mean absolute error (MAE) on average for 3 regions is 170.5514. The average root means square error (RMSE) of [Formula: see text]-ARNN is 333.18, with an accuracy of 98.876%, which shows the scheme's efficacy in ABV measurement over conventional and manual resource allocation schemes.

[Effect Factors and Model Prediction of Soil Heavy Metal Bioaccessibility]

The soil environmental pollution situation has been severe in recent years, but studies on evaluating with bioavailability testing and prediction models are lacking, which makes it difficult to accurately assess the ecological risks of contaminated soil. As an important indicator of bioavailability, the bioaccessibility of cadmium (Cd), arsenic (As), copper (Cu), zinc (Zn), and lead (Pb) in the soil was analyzed in this study. The bioaccessibility content and their corresponding soil property data were screened and systematically analyzed to explore the relationship between bioaccessibility content and soil properties. Furthermore, some testing methods for bioaccessibility were summarized to analyze the relationship between bioaccessibility content, test methods, and bioavailability content. Additionally, the bioaccessibility content prediction models were established. The results showed that there was a strong correlation between the bioaccessibility content and the total content of heavy metals (P<0.01) and a significant (P<0.05) correlation with the soil pH. Different test methods had obvious effects on bioavailability. The proportion of bioaccessibility content determined via various test methods was as follows:in vitro gastrointestinal tract simulation>chemical reagent extraction. The proportions of bioaccessibility content of Cd and Pb in natural soil were relatively high, with mean values of 42.12% and 37.33%, respectively, indicating that Cd and Pb had higher risks of being absorbed by soil organisms. Moreover, 30 bioaccessibility prediction models for five heavy metals were constructed, which involved the soil properties and test methods. The results of this study can provide scientific information and bioaccessibility prediction models that can help in accurately assessing the ecological risks of contaminated soil.

Shear Strength of Ultra-High-Performance Concrete (UHPC) Beams without Transverse Reinforcement: Prediction Models and Test Data

The use of Ultra-High-Performance Concrete (UHPC) in beams has been growing rapidly in the past two decades due to its superior mechanical and durability properties compared to conventional concrete. One of the areas of interest to designers is the elimination of transverse reinforcement as it simplifies beam fabrication/construction and could result in smaller and lighter beams. UHPC has a relatively high post-cracking tensile strength due to the presence of steel fibers, which enhance its shear strength and eliminate the need for transverse reinforcement. In this paper, UHPC shear test data were collected from the literature to study the effect of the following parameters on the shear strength of UHPC beams without transverse reinforcement: compressive strength, tensile strength, level of prestressing, longitudinal reinforcement ratio, and fiber volume fraction. Statistical analysis of test data indicated that level of prestressing and tensile strength are the most significant parameters for prestressed UHPC beams, whereas longitudinal reinforcement ratio and tensile strength are the most significant parameters for non-prestressed UHPC beams. Additionally, shear strength of the tested UHPC beams was predicted using five models: RILEM TC 162-TDF, 2003, fib Model Code, 2010, French Standard NF P 18-710, 2016, PCI-UHPC Structures Design Guide, 2021, and Draft of AASHTO Guide Specification for Structural Design with UHPC, 2021. Comparing measured against predicted shear strength indicated that the French Standard model provides the closest prediction with the least scatter, where the average measured-to-predicted strength was 1.1 with a standard deviation of 0.38. The Draft of AASHTO provided the second closest prediction where the average measured-to-predicted strength was 1.3 with a standard deviation of 0.64. The other three models underestimated the shear strength.

Machine Learning-Derived Prenatal Predictive Risk Model to Guide Intervention and Prevent the Progression of Gestational Diabetes Mellitus to Type 2 Diabetes: Prediction Model Development Study

BACKGROUND

The increasing prevalence of gestational diabetes mellitus (GDM) is concerning as women with GDM are at high risk of type 2 diabetes (T2D) later in life. The magnitude of this risk highlights the importance of early intervention to prevent the progression of GDM to T2D. Rates of postpartum screening are suboptimal, often as low as 13% in Asian countries. The lack of preventive care through structured postpartum screening in several health care systems and low public awareness are key barriers to postpartum diabetes screening.

OBJECTIVE

In this study, we developed a machine learning model for early prediction of postpartum T2D following routine antenatal GDM screening. The early prediction of postpartum T2D during prenatal care would enable the implementation of effective strategies for diabetes prevention interventions. To our best knowledge, this is the first study that uses machine learning for postpartum T2D risk assessment in antenatal populations of Asian origin.

METHODS

Prospective multiethnic data (Chinese, Malay, and Indian ethnicities) from 561 pregnancies in Singapore's most deeply phenotyped mother-offspring cohort study-Growing Up in Singapore Towards healthy Outcomes-were used for predictive modeling. The feature variables included were demographics, medical or obstetric history, physical measures, lifestyle information, and GDM diagnosis. Shapley values were combined with CatBoost tree ensembles to perform feature selection. Our game theoretical approach for predictive analytics enables population subtyping and pattern discovery for data-driven precision care. The predictive models were trained using 4 machine learning algorithms: logistic regression, support vector machine, CatBoost gradient boosting, and artificial neural network. We used 5-fold stratified cross-validation to preserve the same proportion of T2D cases in each fold. Grid search pipelines were built to evaluate the best performing hyperparameters.

RESULTS

A high performance prediction model for postpartum T2D comprising of 2 midgestation features-midpregnancy BMI after gestational weight gain and diagnosis of GDM-was developed (BMI_GDM CatBoost model: AUC=0.86, 95% CI 0.72-0.99). Prepregnancy BMI alone was inadequate in predicting postpartum T2D risk (ppBMI CatBoost model: AUC=0.62, 95% CI 0.39-0.86). A 2-hour postprandial glucose test (BMI_2hour CatBoost model: AUC=0.86, 95% CI 0.76-0.96) showed a stronger postpartum T2D risk prediction effect compared to fasting glucose test (BMI_Fasting CatBoost model: AUC=0.76, 95% CI 0.61-0.91). The BMI_GDM model was also robust when using a modified 2-point International Association of the Diabetes and Pregnancy Study Groups (IADPSG) 2018 criteria for GDM diagnosis (BMI_GDM2 CatBoost model: AUC=0.84, 95% CI 0.72-0.97). Total gestational weight gain was inversely associated with postpartum T2D outcome, independent of prepregnancy BMI and diagnosis of GDM (P=.02; OR 0.88, 95% CI 0.79-0.98).

CONCLUSIONS

Midgestation weight gain effects, combined with the metabolic derangements underlying GDM during pregnancy, signal future T2D risk in Singaporean women. Further studies will be required to examine the influence of metabolic adaptations in pregnancy on postpartum maternal metabolic health outcomes. The state-of-the-art machine learning model can be leveraged as a rapid risk stratification tool during prenatal care.

TRIAL REGISTRATION

ClinicalTrials.gov NCT01174875; https://clinicaltrials.gov/ct2/show/NCT01174875.

Progress in cancer epidemiology: avoided deaths in Europe over the last three decades

Progress in cancer epidemiology and prevention has been a key determinant of the fall in cancer mortality in Europe. Using mortality and population figures from the WHO and Eurostat databases, we estimated the number of averted cancer deaths in the EU27 over the period 1989-2021, for both sexes, for all cancers, and nine major cancer sites. We also computed the avoided deaths for all cancers in five major European countries and the UK. We estimated a total of 4 958 000 (3 339 000 men and 1 619 000 women) avoided deaths for all neoplasms over the period 1989-2021 and 348 000 (246 000 men and 102 000 women) in 2021 alone in the EU27. For both sexes, we estimated 1 679 000 avoided deaths for stomach cancer, 747 000 for colorectum, 227 000 for bladder, 102 000 for leukemias. Avoided deaths for lung cancer accounted for 1 156 000 in men, while no reduction was estimated for women. For breast and uterine cancer, avoided deaths were about 300 000, for ovary 105 000 and for prostate 352 000. In the UK, a total of 1 061 000 (721 000 men and 340 000 women) deaths was avoided. Elimination of tobacco may avoid a further 20% of cancer mortality by 2050. Control of alcohol, overweight and obesity, and occupational and environmental carcinogens may avoid an additional 10% of cancer deaths. A similar reduction may be due to optimal adoption of cervical, colorectal, breast, and probably, lung and prostate cancer screening. Thus, primary and secondary cancer prevention can avoid an additional third of cancer deaths in Europe up to 2050.

Using the Prediction Model Risk of Bias Assessment Tool (PROBAST) to Evaluate Melanoma Prediction Studies

Simple Summary

The rising incidence of cutaneous melanoma over recent decades, combined with a general interest in cancer risk prediction, has led to a high number of published melanoma risk prediction models. The aim of our work was to assess the validity of these models in order to discuss the current state of knowledge about how to predict incident cutaneous melanoma. To assess the risk of bias, we used a standardized procedure based on PROBAST (Prediction model Risk Of Bias ASsessment Tool). Only one of the 42 studies identified was rated as having a low risk of bias. However, it was encouraging to observe a recent reduction of problematic statistical methods used in the analyses. Nevertheless, the evidence base of high-quality studies that can be used to draw conclusions on the prediction of incident cutaneous melanoma is currently much weaker than the high number of studies on this topic would suggest.

Abstract

Rising incidences of cutaneous melanoma have fueled the development of statistical models that predict individual melanoma risk. Our aim was to assess the validity of published prediction models for incident cutaneous melanoma using a standardized procedure based on PROBAST (Prediction model Risk Of Bias ASsessment Tool). We included studies that were identified by a recent systematic review and updated the literature search to ensure that our PROBAST rating included all relevant studies. Six reviewers assessed the risk of bias (ROB) for each study using the published “PROBAST Assessment Form” that consists of four domains and an overall ROB rating. We further examined a temporal effect regarding changes in overall and domain-specific ROB rating distributions. Altogether, 42 studies were assessed, of which the vast majority (n = 34; 81%) was rated as having high ROB. Only one study was judged as having low ROB. The main reasons for high ROB ratings were the use of hospital controls in case-control studies and the omission of any validation of prediction models. However, our temporal analysis results showed a significant reduction in the number of studies with high ROB for the domain “analysis”. Nevertheless, the evidence base of high-quality studies that can be used to draw conclusions on the prediction of incident cutaneous melanoma is currently much weaker than the high number of studies on this topic would suggest.

Prediction Models for Brain Distribution of Drugs Based on Biomimetic Chromatographic Data

The development of high-throughput approaches for the valid estimation of brain disposition is of great importance in the early drug screening of drug candidates. However, the complexity of brain tissue, which is protected by a unique vasculature formation called the blood−brain barrier (BBB), complicates the development of robust in silico models. In addition, most computational approaches focus only on brain permeability data without considering the crucial factors of plasma and tissue binding. In the present study, we combined experimental data obtained by HPLC using three biomimetic columns, i.e., immobilized artificial membranes, human serum albumin, and α1-acid glycoprotein, with molecular descriptors to model brain disposition of drugs. Kp,uu,brain, as the ratio between the unbound drug concentration in the brain interstitial fluid to the corresponding plasma concentration, brain permeability, the unbound fraction in the brain, and the brain unbound volume of distribution, was collected from literature. Given the complexity of the investigated biological processes, the extracted models displayed high statistical quality (R2 > 0.6), while in the case of the brain fraction unbound, the models showed excellent performance (R2 > 0.9). All models were thoroughly validated, and their applicability domain was estimated. Our approach highlighted the importance of phospholipid, as well as tissue and protein, binding in balance with BBB permeability in brain disposition and suggests biomimetic chromatography as a rapid and simple technique to construct models with experimental evidence for the early evaluation of CNS drug candidates.

Identifying the Presence, Activity, and Status of Extraintestinal Manifestations of Inflammatory Bowel Disease Using Natural Language Processing of Clinical Notes

BACKGROUND

Extraintestinal manifestations (EIMs) occur commonly in inflammatory bowel disease (IBD), but population-level understanding of EIM behavior is difficult. We present a natural language processing (NLP) system designed to identify both the presence and status of EIMs using clinical notes from patients with IBD.

METHODS

In a single-center retrospective study, clinical outpatient electronic documents were collected in patients with IBD. An NLP EIM detection pipeline was designed to determine general and specific symptomatic EIM activity status descriptions using Python 3.6. Accuracy, sensitivity, and specificity, and agreement using Cohen's kappa coefficient were used to compare NLP-inferred EIM status to human documentation labels.

RESULTS

The 1240 individuals identified as having at least 1 EIM consisted of 54.4% arthritis, 17.2% ocular, and 17.0% psoriasiform EIMs. Agreement between reviewers on EIM status was very good across all EIMs (κ = 0.74; 95% confidence interval [CI], 0.70-0.78). The automated NLP pipeline determining general EIM activity status had an accuracy, sensitivity, specificity, and agreement of 94.1%, 0.92, 0.95, and κ = 0.76 (95% CI, 0.74-0.79), respectively. Comparatively, prediction of EIM status using administrative codes had a poor sensitivity, specificity, and agreement with human reviewers of 0.32, 0.83, and κ = 0.26 (95% CI, 0.20-0.32), respectively.

CONCLUSIONS

NLP methods can both detect and infer the activity status of EIMs using the medical document an information source. Though source document variation and ambiguity present challenges, NLP offers exciting possibilities for population-based research and decision support in IBD.

Harnessing Natural Language Processing to Support Decisions Around Workplace-Based Assessment: Machine Learning Study of Competency-Based Medical Education

BACKGROUND

Residents receive a numeric performance rating (eg, 1-7 scoring scale) along with a narrative (ie, qualitative) feedback based on their performance in each workplace-based assessment (WBA). Aggregated qualitative data from WBA can be overwhelming to process and fairly adjudicate as part of a global decision about learner competence. Current approaches with qualitative data require a human rater to maintain attention and appropriately weigh various data inputs within the constraints of working memory before rendering a global judgment of performance.

OBJECTIVE

This study explores natural language processing (NLP) and machine learning (ML) applications for identifying trainees at risk using a large WBA narrative comment data set associated with numerical ratings.

METHODS

NLP was performed retrospectively on a complete data set of narrative comments (ie, text-based feedback to residents based on their performance on a task) derived from WBAs completed by faculty members from multiple hospitals associated with a single, large, residency program at McMaster University, Canada. Narrative comments were vectorized to quantitative ratings using the bag-of-n-grams technique with 3 input types: unigram, bigrams, and trigrams. Supervised ML models using linear regression were trained with the quantitative ratings, performed binary classification, and output a prediction of whether a resident fell into the category of at risk or not at risk. Sensitivity, specificity, and accuracy metrics are reported.

RESULTS

The database comprised 7199 unique direct observation assessments, containing both narrative comments and a rating between 3 and 7 in imbalanced distribution (scores 3-5: 726 ratings; and scores 6-7: 4871 ratings). A total of 141 unique raters from 5 different hospitals and 45 unique residents participated over the course of 5 academic years. When comparing the 3 different input types for diagnosing if a trainee would be rated low (ie, 1-5) or high (ie, 6 or 7), our accuracy for trigrams was 87%, bigrams 86%, and unigrams 82%. We also found that all 3 input types had better prediction accuracy when using a bimodal cut (eg, lower or higher) compared with predicting performance along the full 7-point rating scale (50%-52%).

CONCLUSIONS

The ML models can accurately identify underperforming residents via narrative comments provided for WBAs. The words generated in WBAs can be a worthy data set to augment human decisions for educators tasked with processing large volumes of narrative assessments.

Compressive Strength Estimation of Geopolymer Composites through Novel Computational Approaches

The application of artificial intelligence approaches like machine learning (ML) to forecast material properties is an effective strategy to reduce multiple trials during experimentation. This study performed ML modeling on 481 mixes of geopolymer concrete with nine input variables, including curing time, curing temperature, specimen age, alkali/fly ash ratio, Na₂SiO₃/NaOH ratio, NaOH molarity, aggregate volume, superplasticizer, and water, with CS as the output variable. Four types of ML models were employed to anticipate the compressive strength of geopolymer concrete, and their performance was compared to find out the most accurate ML model. Two individual ML techniques, support vector machine and multi-layer perceptron neural network, and two ensembled ML methods, AdaBoost regressor and random forest, were employed to achieve the study’s aims. The performance of all models was confirmed using statistical analysis, k-fold evaluation, and correlation coefficient (R²). Moreover, the divergence of the estimated outcomes from those of the experimental results was noted to check the accuracy of the models. It was discovered that ensembled ML models estimated the compressive strength of the geopolymer concrete with higher precision than individual ML models, with random forest having the highest accuracy. Using these computational strategies will accelerate the application of construction materials by decreasing the experimental efforts.

Predictors of same-day discharge following benign minimally invasive hysterectomy

BACKGROUND

Same-day discharge following minimally invasive hysterectomy has been shown to be safe and feasible in select populations, but many nonclinical factors influencing same-day discharge remain unexplored.

OBJECTIVE

To develop prediction models for same-day discharge following minimally invasive hysterectomy using both clinical and nonclinical attributes and to compare model concordance of individual attribute groups.

STUDY DESIGN

We performed a retrospective study of patients who underwent elective minimally invasive hysterectomy for benign gynecologic indications at 69 hospitals in a statewide quality improvement collaborative between 2012 and 2019. Potential predictors of same-day discharge were determined a priori and placed into 1 of 7 attribute groupings: intraoperative, surgeon, hospital, surgical timing, patient clinical, patient socioeconomic, and patient geographic attributes. To account for clustering of same-day discharge practices among surgeons and within hospitals, hierarchical multivariable logistic regression models were fitted using predictors from each attribute group individually and all predictors in a composite model. Receiver operator characteristic curves were generated for each model. The Hanley-McNeil test was used for comparisons, 95% confidence intervals for the areas under the receiver operator characteristic curve were calculated, and a P value of <.05 was considered significant.

RESULTS

Of the 23,513 patients in our study, 5062 (21.5%) had same-day discharge. The composite model had an area under the receiver operator characteristic curve of 0.770 (95% confidence interval, 0.763-0.777). Among models using factors from individual attribute groups, the model using intraoperative attributes had the highest concordance for same-day discharge (area under the receiver operator characteristic curve, 0.720; 95% confidence interval, 0.712-0.727). The models using surgeon and hospital attributes were the second and third most concordant, respectively (area under the receiver operator characteristic curve, 0.678; 95% confidence interval, 0.670-0.685; area under the receiver operator characteristic curve, 0.655; 95% confidence interval, 0.656-0.664). Models using surgical timing and patient clinical, socioeconomic, and geographic attributes had poor predictive ability (all areas under the receiver operator characteristic curve <0.6).

CONCLUSION

Clinical and nonclinical attributes contributed to a composite prediction model with good discrimination in predicting same-day discharge following minimally invasive hysterectomy. Factors related to intraoperative, hospital, and surgeon attributes produced models with the strongest predictive ability. Focusing on these attributes may aid efforts to improve utilization of same-day discharge following minimally invasive hysterectomy.

Emergency Medical Services Calls Analysis for Trend Prediction during Epidemic Outbreaks: Interrupted Time Series Analysis on 2020-2021 COVID-19 Epidemic in Lazio, Italy

(1) Background: During the COVID-19 outbreak in the Lazio region, a surge in emergency medical service (EMS) calls has been observed. The objective of present study is to investigate if there is any correlation between the variation in numbers of daily EMS calls, and the short-term evolution of the epidemic wave. (2) Methods: Data from the COVID-19 outbreak has been retrieved in order to draw the epidemic curve in the Lazio region. Data from EMS calls has been used in order to determine Excess of Calls (ExCa) in the 2020−2021 years, compared to the year 2019 (baseline). Multiple linear regression models have been run between ExCa and the first-order derivative (D’) of the epidemic wave in time, each regression model anticipating the epidemic progression (up to 14 days), in order to probe a correlation between the variables. (3) Results: EMS calls variation from baseline is correlated with the slope of the curve of ICU admissions, with the most fitting value found at 7 days (R2 0.33, p < 0.001). (4) Conclusions: EMS calls deviation from baseline allows public health services to predict short-term epidemic trends in COVID-19 outbreaks, and can be used as validation of current data, or as an independent estimator of future trends.

Artificial intelligence-based early detection of acute kidney injury after cardiac surgery

OBJECTIVES

This study aims to improve early detection of cardiac surgery-associated acute kidney injury using artificial intelligence-based algorithms.

METHODS

Data from consecutive patients undergoing cardiac surgery between 2008 and 2018 in our institution served as the source for artificial intelligence-based modeling. Cardiac surgery-associated acute kidney injury was defined according to the Kidney Disease Improving Global Outcomes criteria. Different machine learning algorithms were trained and validated to detect cardiac surgery-associated acute kidney injury within 12 hours after surgery. Demographic characteristics, comorbidities, preoperative cardiac status, intra- and postoperative variables including creatinine and hemoglobin values were retrieved for analysis.

RESULTS

From 7507 patients analyzed, 1699 patients (22.6%) developed cardiac surgery-associated acute kidney injury. The ultimate detection model, 'Detect-A(K)I', recognizes cardiac surgery-associated acute kidney injury within 12 hours with an area under the curve of 88.0%, sensitivity of 78.0%, specificity of 78.9%, and accuracy of 82.1%. The optimal parameter set includes serial changes of creatinine and hemoglobin, operative emergency, bleeding-associated variables, cardiac ischaemic time and cardiac function-associated variables, age, diuretics and active infection, chronic obstructive lung and peripheral vascular disease.

CONCLUSIONS

The 'Detect-A(K)I' model successfully detects cardiac surgery-associated acute kidney injury within 12 hours after surgery with the best discriminatory characteristics reported so far.

Using machine learning to predict internal rotation after anatomic and reverse total shoulder arthroplasty

BACKGROUND

Improvement in internal rotation (IR) after anatomic (aTSA) and reverse (rTSA) total shoulder arthroplasty is difficult to predict, with rTSA patients experiencing greater variability and more limited IR improvements than aTSA patients. The purpose of this study is to quantify and compare the IR score for aTSA and rTSA patients and create supervised machine learning that predicts IR after aTSA and rTSA at multiple postoperative time points.

METHODS

Clinical data from 2270 aTSA and 4198 rTSA patients were analyzed using 3 supervised machine learning techniques to create predictive models for internal rotation as measured by the IR score at 6 postoperative time points. Predictions were performed using the full input feature set and 2 minimal input feature sets. The mean absolute error (MAE) quantified the difference between actual and predicted IR scores for each model at each time point. The predictive accuracy of the XGBoost algorithm was also quantified by its ability to distinguish which patients would achieve clinical improvement greater than the minimal clinically important difference (MCID) and substantial clinical benefit (SCB) patient satisfaction thresholds for IR score at 2-3 years after surgery.

RESULTS

rTSA patients had significantly lower mean IR scores and significantly less mean IR score improvement than aTSA patients at each postoperative time point. Both aTSA and rTSA patients experienced significant improvements in their ability to perform activities of daily living (ADLs); however, aTSA patients were significantly more likely to perform these ADLs. Using a minimal feature set of preoperative inputs, our machine learning algorithms had equivalent accuracy when predicting IR score for both aTSA (0.92-1.18 MAE) and rTSA (1.03-1.25 MAE) from 3 months to >5 years after surgery. Furthermore, these predictive algorithms identified with 90% accuracy for aTSA and 85% accuracy for rTSA which patients will achieve MCID IR score improvement and predicted with 85% accuracy for aTSA patients and 77% accuracy for rTSA which patients will achieve SCB IR score improvement at 2-3 years after surgery.

DISCUSSION

Our machine learning study demonstrates that active internal rotation can be accurately predicted after aTSA and rTSA at multiple postoperative time points using a minimal feature set of preoperative inputs. These predictive algorithms accurately identified which patients will, and will not, achieve clinical improvement in IR score that exceeds the MCID and SCB patient satisfaction thresholds.

A Comparison of Methods to Detect Changes in Prediction Models

BACKGROUND

Prediction models inform decisions in many areas of medicine. Most models are fitted once and then applied to new (future) patients, despite the fact that model coefficients can vary over time due to changes in patients' clinical characteristics and disease risk. However, the optimal method to detect changes in model parameters has not been rigorously assessed.

METHODS

We simulated data, informed by post-lung transplant mortality data and tested the following two approaches for detecting model change: (1) the "Direct Approach," it compares coefficients of the model refit on recent data to those at baseline; and (2) "Calibration Regression," it fits a logistic regression model of the log-odds of the observed outcomes versus the linear predictor from the baseline model (i.e., the log-odds of the predicted probabilities obtained from the baseline model) and tests whether the intercept and slope differ from 0 and 1, respectively. Four scenarios were simulated using logistic regression for binary outcomes as follows: (1) we fixed all model parameters, (2) we varied the outcome prevalence between 0.1 and 0.2, (3) we varied the coefficient of one of the ten predictors between 0.2 and 0.4, and (4) we varied the outcome prevalence and coefficient of one predictor simultaneously.

RESULTS

Calibration regression tended to detect changes sooner than the Direct Approach, with better performance (e.g., larger proportion of true claims). When the sample size was large, both methods performed well. When two parameters changed simultaneously, neither method performed well.

CONCLUSION

Neither change detection method examined here proved optimal under all circumstances. However, our results suggest that if one is interested in detecting a change in overall incidence of an outcome (e.g., intercept), the Calibration Regression method may be superior to the Direct Approach. Conversely, if one is interested in detecting a change in other model covariates (e.g., slope), the Direct Approach may be superior.

How to model and interpret cross-lagged effects in psychotherapy mechanisms of change research: A comparison of multilevel and structural equation models

OBJECTIVE

Modeling cross-lagged effects in psychotherapy mechanisms of change studies is complex and requires careful attention to model selection and interpretation. However, there is a lack of field-specific guidelines. We aimed to (a) describe the estimation and interpretation of cross lagged effects using multilevel models (MLM) and random-intercept cross lagged panel model (RI-CLPM); (b) compare these models' performance and risk of bias using simulations and an applied research example to formulate recommendations for practice.

METHOD

Part 1 is a tutorial focused on introducing/describing dynamic effects in the form of autoregression and bidirectionality. In Part 2, we compare the estimation of cross-lagged effects in RI-CLPM, which takes dynamic effects into account, with three commonly used MLMs that cannot accommodate dynamics. In Part 3, we describe a Monte Carlo simulation study testing model performance of RI-CLPM and MLM under realistic conditions for psychotherapy mechanisms of change studies.

RESULTS

Our findings suggested that all three MLMs resulted in severely biased estimates of cross-lagged effects when dynamic effects were present in the data, with some experimental conditions generating statistically significant estimates in the wrong direction. MLMs performed comparably well only in conditions which are conceptually unrealistic for psychotherapy mechanisms of change research (i.e., no inertia in variables and no bidirectional effects).

DISCUSSION

Based on conceptual fit and our simulation results, we strongly recommend using fully dynamic structural equation modeling models, such as the RI-CLPM, rather than static, unidirectional regression models (e.g., MLM) to study cross-lagged effects in mechanisms of change research. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Using Natural Language Processing and Machine Learning to Preoperatively Predict Lymph Node Metastasis for Non-Small Cell Lung Cancer With Electronic Medical Records: Development and Validation Study

Background

Lymph node metastasis (LNM) is critical for treatment decision making of patients with resectable non–small cell lung cancer, but it is difficult to precisely diagnose preoperatively. Electronic medical records (EMRs) contain a large volume of valuable information about LNM, but some key information is recorded in free text, which hinders its secondary use.

Objective

This study aims to develop LNM prediction models based on EMRs using natural language processing (NLP) and machine learning algorithms.

Methods

We developed a multiturn question answering NLP model to extract features about the primary tumor and lymph nodes from computed tomography (CT) reports. We then combined these features with other structured clinical characteristics to develop LNM prediction models using machine learning algorithms. We conducted extensive experiments to explore the effectiveness of the predictive models and compared them with size criteria based on CT image findings (the maximum short axis diameter of lymph node >10 mm was regarded as a metastatic node) and clinician’s evaluation. Since the NLP model may extract features with mistakes, we also calculated the concordance correlation between the predicted probabilities of models using NLP-extracted features and gold standard features to explore the influence of NLP-driven automatic extraction.

Results

Experimental results show that the random forest models achieved the best performances with 0.792 area under the receiver operating characteristic curve (AUC) value and 0.456 average precision (AP) value for pN2 LNM prediction and 0.768 AUC value and 0.524 AP value for pN1&N2 LNM prediction. And all machine learning models outperformed the size criteria and clinician’s evaluation. The concordance correlation between the random forest models using NLP-extracted features and gold standard features is 0.950 and improved to 0.984 when the top 5 important NLP-extracted features were replaced with gold standard features.

Conclusions

The LNM models developed can achieve competitive performance using only limited EMR data such as CT reports and tumor markers in comparison with the clinician’s evaluation. The multiturn question answering NLP model can extract features effectively to support the development of LNM prediction models, which may facilitate the clinical application of predictive models.

Nitrogen excretion from beef cattle fed a wide range of diets compiled in an intercontinental dataset: a meta-analysis

Manure N from cattle contributes to nitrate leaching, nitrous oxide and ammonia emissions. Measurement of manure N outputs on commercial beef cattle operations is laborious, expensive, and impractical; therefore, models are needed to predict N excreted in urine and feces. Building robust prediction models requires extensive data from animals under different management systems worldwide. Thus, the study objectives were to (1) collate an international dataset of N excretion in feces and urine based on individual observations from beef cattle; (2) determine the suitability of key variables for predicting fecal, urinary, and total manure N excretion; (3) develop robust and reliable N excretion prediction models based on individual observation from beef cattle consuming various diets. A meta-analysis based on individual beef data from different experiments was carried from a raw dataset including 1,004 observations from 33 experiments collected from 5 research institutes in Europe (n = 3), North America (n = 1), and South America (n = 1). A sequential approach was taken in developing models of increasing complexity by incrementally adding significant variables that effected fecal, urinary, or total manure N excretion. Nitrogen excretion was predicted by fitting linear mixed models with experiment as a random effect. Simple models including dry matter intake (DMI) were better at predicting fecal N excretion, than those using only dietary nutrient composition or BW. Simple models based on N intake performed better for urinary and total manure N excretion than those based on DMI. A model including DMI and dietary component concentrations led to the most robust prediction of fecal and urinary N excretion, generating root mean square prediction errors as a percentage of the observed mean values of 25.0% for feces and 25.6% for urine. Complex total manure N excretion models based on BW and dietary component concentrations led to the lowest prediction errors of about 14.6%. In conclusion, several models to predict N excretion already exist, but the ones developed in this study are based on individual observations encompassing larger variability than the previous developed models. In addition, models that include information on DMI or N intake are required for accurate prediction of fecal, urinary and total manure N excretion. In the absence of intake data, equations have poor performance as compared to equations based on intake and dietary component concentrations.

Estimation of Daily Ground-Received Global Solar Radiation Using Air Pollutant Data

Ground-received solar radiation is affected by several meteorological and air pollution factors. Previous studies have mainly focused on the effects of meteorological factors on solar radiation, but research on the influence of air pollutants is limited. Therefore, this study aimed to analyse the effects of air pollution characteristics on solar radiation. Meteorological data, air quality index (AQI) data, and data on the concentrations of six air pollutants (O₃, CO, SO₂, PM₁₀, PM_2.5, and NO₂) in nine cities in China were considered for analysis. A city model (model-C) based on the data of each city and a unified model (model-U) based on national data were established, and the key pollutants under these conditions were identified. Correlation analysis was performed between each pollutant and the daily global solar radiation. The correlation between O₃ and daily global solar radiation was the highest (r = 0.575), while that between SO₂ and daily global solar radiation was the lowest. Further, AQI and solar radiation were negatively correlated, while some pollution components (e.g., O₃) were positively correlated with the daily global solar radiation. Different key pollutants affected the solar radiation in each city. In Shenyang and Guangzhou, the driving effect of particles on the daily global solar radiation was stronger than that of pollutants. However, there were no key pollutants that affect solar radiation in Shanghai. Furthermore, the prediction performance of model-U was not as good as that of model-C. The model-U showed a good performance for Urumqi (R² = 0.803), while the difference between the two models was not particularly significant in other areas. This study provides significant insights to improve the accuracy of regional solar radiation prediction and fill the gap regarding the absence of long-term solar radiation monitoring data in some areas.