Intervention in prediction measure: a new approach to assessing variable importance for random forests

Impacts of socioeconomic and environmental factors on neoplasms incidence rates using machine learning and GIS: a cross-sectional study in Iran

Neoplasm is an umbrella term used to describe either benign or malignant conditions. The correlations between socioeconomic and environmental factors and the occurrence of new-onset of neoplasms have already been demonstrated in a body of research. Nevertheless, few studies have specifically dealt with the nature of relationship, significance of risk factors, and geographic variation of them, particularly in low- and middle-income communities. This study, thus, set out to (1) analyze spatiotemporal variations of the age-adjusted incidence rate (AAIR) of neoplasms in Iran throughout five time periods, (2) investigate relationships between a collection of environmental and socioeconomic indicators and the AAIR of neoplasms all over the country, and (3) evaluate geographical alterations in their relative importance. Our cross-sectional study design was based on county-level data from 2010 to 2020. AAIR of neoplasms data was acquired from the Institute for Health Metrics and Evaluation (IHME). HotSpot analyses and Anselin Local Moran's I indices were deployed to precisely identify AAIR of neoplasms high- and low-risk clusters. Multi-scale geographically weight regression (MGWR) analysis was worked out to evaluate the association between each explanatory variable and the AAIR of neoplasms. Utilizing random forests (RF), we also examined the relationships between environmental (e.g., UV index and PM2.5 concentration) and socioeconomic (e.g., Gini coefficient and literacy rate) factors and AAIR of neoplasms. AAIR of neoplasms displayed a significant increasing trend over the study period. According to the MGWR, the only factor that significantly varied spatially and was associated with the AAIR of neoplasms in Iran was the UV index. A good accuracy RF model was confirmed for both training and testing data with correlation coefficients R2 greater than 0.91 and 0.92, respectively. UV index and Gini coefficient ranked the highest variables in the prediction of AAIR of neoplasms, based on the relative influence of each variable. More research using machine learning approaches taking the advantages of considering all possible determinants is required to assess health strategies outcomes and properly formulate policy planning.

Teardrop Alignment Changes After Volar Locking Plate Fixation of Distal Radius Fractures With Volar Ulnar Fragments

BACKGROUND

We assessed factors associated with change in radiographic teardrop angle following volar locking plate (VLP) fixation of volarly displaced intra-articular distal radius fractures with volar ulnar fragments (VUF) within the ICUC database. The primary outcome was change in radiographic alignment on follow-up imaging, defined as a change in teardrop angle from intra-operative fluoroscopy greater than 5°.

METHODS

Patients with distal radius fractures treated with a VLP within the ICUC database, an international collaborative and publicly available dataset, were identified. The primary outcome was volar rim loss of reduction on follow-up imaging, defined as a change in radiographic alignment from intra-operative fluoroscopy, teardrop angle less than 50°, or loss of normal radiocarpal alignment. Secondary outcomes were final range of motion (ROM) of the affected extremity. Radiographic Soong classification was used to grade plate position. Descriptive statistics were used to assess variables' distributions. A Random Forest supervised machine learning algorithm was used to classify variable importance for predicting the primary outcome. Traditional descriptive statistics were used to compare patient, fracture, and treatment characteristics with volar rim loss of reduction. Volar rim loss of reduction and final ROM in degrees and as compared with contralateral unaffected limb were also assessed.

RESULTS

Fifty patients with volarly displaced, intra-articular distal radius fractures treated with a VLP were identified. Six patients were observed to have a volar rim loss of reduction, but none required reoperation. Volar ulnar fragment size, Soong grade 0, and postfixation axial plate position in relation to the sigmoid notch were significantly associated (P < .05) with volar rim loss of reduction. All cases of volar rim loss of reduction occurred when VUF was 10.8 mm or less.

CONCLUSIONS

The size of the VUF was the most important variable for predicting volar rim loss of reduction followed by postfixation plate position in an axial position to the sigmoid notch and the number of volar fragments in the Random Forest machine learning algorithm. There were no significant differences in ROM between patients with volar ulnar escape and those without.

Exploring the variable importance in random forests under correlations: a general concept applied to donor organ quality in post-transplant survival

Random Forests are a powerful and frequently applied Machine Learning tool. The permutation variable importance (VIMP) has been proposed to improve the explainability of such a pure prediction model. It describes the expected increase in prediction error after randomly permuting a variable and disturbing its association with the outcome. However, VIMPs measure a variable's marginal influence only, that can make its interpretation difficult or even misleading. In the present work we address the general need for improving the explainability of prediction models by exploring VIMPs in the presence of correlated variables. In particular, we propose to use a variable's residual information for investigating if its permutation importance partially or totally originates from correlated predictors. Hypotheses tests are derived by a resampling algorithm that can further support results by providing test decisions and p-values. In simulation studies we show that the proposed test controls type I error rates. When applying the methods to a Random Forest analysis of post-transplant survival after kidney transplantation, the importance of kidney donor quality for predicting post-transplant survival is shown to be high. However, the transplant allocation policy introduces correlations with other well-known predictors, which raises the concern that the importance of kidney donor quality may simply originate from these predictors. By using the proposed method, this concern is addressed and it is demonstrated that kidney donor quality plays an important role in post-transplant survival, regardless of correlations with other predictors.

A neural network model to screen feature genes for pancreatic cancer

All the time, pancreatic cancer is a problem worldwide because of its high degree of malignancy and increased mortality. Neural network model analysis is an efficient and accurate machine learning method that can quickly and accurately predict disease feature genes. The aim of our research was to build a neural network model that would help screen out feature genes for pancreatic cancer diagnosis and prediction of prognosis. Our study confirmed that the neural network model is a reliable way to predict feature genes of pancreatic cancer, and immune cells infiltrating play an essential role in the development of pancreatic cancer, especially neutrophils. ANO1, AHNAK2, and ADAM9 were eventually identified as feature genes of pancreatic cancer, helping to diagnose and predict prognosis. Neural network model analysis provides us with a new idea for finding new intervention targets for pancreatic cancer.

Identifying Chinese social media users' need for affect from their online behaviors

The need for affect (NFA), which refers to the motivation to approach or avoid emotion-inducing situations, is a valuable indicator of mental health monitoring and intervention, as well as many other applications. Traditionally, NFA has been measured using self-reports, which is not applicable in today's online scenarios due to its shortcomings in fast, large-scale assessments. This study proposed an automatic and non-invasive method for recognizing NFA based on social media behavioral data. The NFA questionnaire scores of 934 participants and their social media data were acquired. Then we run machine learning algorithms to train predictive models, which can be used to automatically identify NFA degrees of online users. The results showed that Extreme Gradient Boosting (XGB) performed best among several algorithms. The Pearson correlation coefficients between predicted scores and NFA questionnaire scores achieved 0.25 (NFA avoidance), 0.31 (NFA approach) and 0.34 (NFA total), and the split-half reliabilities were 0.66-0.70. Our research demonstrated that adolescents' NFA can be identified based on their social media behaviors, and opened a novel way of non-intrusively perceiving users' NFA which can be used for mental health monitoring and other situations that require large-scale NFA measurements.

Identification of Feature Genes of a Novel Neural Network Model for Bladder Cancer

Background: The combination of deep learning methods and oncogenomics can provide an effective diagnostic method for malignant tumors; thus, we attempted to construct a reliable artificial neural network model as a novel diagnostic tool for Bladder cancer (BLCA). Methods: Three expression profiling datasets (GSE61615, GSE65635, and GSE100926) were downloaded from the Gene Expression Omnibus (GEO) database. GSE61615 and GSE65635 were taken as the train group, while GSE100926 was set as the test group. Differentially expressed genes (DEGs) were filtered out based on the logFC and FDR values. We also performed Gene Ontology (GO) and Kyoto Encyclopedia of Genes and Genomes (KEGG) analyses to explore the biological functions of the DEGs. Consequently, we utilized a random forest algorithm to identify feature genes and further constructed a neural network model. The test group was given the same procedures to validate the reliability of the model. We also explored immune cells' infiltration degree and correlation coefficients through the CiberSort algorithm and corrplot R package. The qRT-PCR assay was implemented to examine the expression level of the feature genes in vitro. Results: A total of 265 DEGs were filtered out and significantly enriched in muscle system processes, collagen-containing and focal adhesion signaling pathways. Based on the random forest algorithm, we selected 14 feature genes to construct the neural network model. The area under the curve (AUC) of the training group was 0.950 (95% CI: 0.850-1.000), and the AUC of the test group was 0.667 (95% CI: 0.333-1.000). Besides, we observed significant differences in the content of immune infiltrating cells and the expression levels of the feature genes. Conclusion: After repeated verification, our neural network model had clinical feasibility to identify bladder cancer patients and provided a potential target to improve the management of BLCA.

Understanding Artificial Intelligence and Predictive Analytics: A Clinically Focused Review of Machine Learning Techniques

»

Machine learning and artificial intelligence have seen tremendous growth in recent years and have been applied in numerous studies in the field of orthopaedics.

»

Machine learning will soon become critical in the day-to-day operations of orthopaedic practice; therefore, it is imperative that providers become accustomed to and familiar with not only the terminology but also the fundamental techniques behind the technology.

»

A foundation of knowledge regarding machine learning is critical for physicians so they can begin to understand the details in the algorithms that are being developed, which provide improved accuracy compared with clinicians, decreased time required, and a heightened ability to triage patients.

A risk score based on baseline risk factors for predicting mortality in COVID-19 patients

BACKGROUND

To develop a sensitive and clinically applicable risk assessment tool identifying coronavirus disease 2019 (COVID-19) patients with a high risk of mortality at hospital admission. This model would assist frontline clinicians in optimizing medical treatment with limited resources.

METHODS

6415 patients from seven hospitals in Wuhan city were assigned to the training and testing cohorts. A total of 6351 patients from another three hospitals in Wuhan, 2169 patients from outside of Wuhan, and 553 patients from Milan, Italy were assigned to three independent validation cohorts. A total of 64 candidate clinical variables at hospital admission were analyzed by random forest and least absolute shrinkage and selection operator (LASSO) analyses.

RESULTS

Eight factors, namely, Oxygen saturation, blood Urea nitrogen, Respiratory rate, admission before the date the national Maximum number of daily new cases was reached, Age, Procalcitonin, C-reactive protein (CRP), and absolute Neutrophil counts, were identified as having significant associations with mortality in COVID-19 patients. A composite score based on these eight risk factors, termed the OURMAPCN-score, predicted the risk of mortality among the COVID-19 patients, with a C-statistic of 0.92 (95% confidence interval [CI] 0.90-0.93). The hazard ratio for all-cause mortality between patients with OURMAPCN-score >11 compared with those with scores ≤ 11 was 18.18 (95% CI 13.93-23.71; p < .0001). The predictive performance, specificity, and sensitivity of the score were validated in three independent cohorts.

CONCLUSIONS

The OURMAPCN score is a risk assessment tool to determine the mortality rate in COVID-19 patients based on a limited number of baseline parameters. This tool can assist physicians in optimizing the clinical management of COVID-19 patients with limited hospital resources.

Ranking the Predictive Power of Clinical and Biological Features Associated With Disease Progression in Huntington's Disease

Huntington's disease (HD) is characterised by a triad of cognitive, behavioural, and motor symptoms which lead to functional decline and loss of independence. With potential disease-modifying therapies in development, there is interest in accurately measuring HD progression and characterising prognostic variables to improve efficiency of clinical trials. Using the large, prospective Enroll-HD cohort, we investigated the relative contribution and ranking of potential prognostic variables in patients with manifest HD. A random forest regression model was trained to predict change of clinical outcomes based on the variables, which were ranked based on their contribution to the prediction. The highest-ranked variables included novel predictors of progression—being accompanied at clinical visit, cognitive impairment, age at diagnosis and tetrabenazine or antipsychotics use—in addition to established predictors, cytosine adenine guanine (CAG) repeat length and CAG-age product. The novel prognostic variables improved the ability of the model to predict clinical outcomes and may be candidates for statistical control in HD clinical studies.

Investigating Predictors of Preserved Cognitive Function in Older Women Using Machine Learning: Women's Health Initiative Memory Study

BACKGROUND

Identification of factors that may help to preserve cognitive function in late life could elucidate mechanisms and facilitate interventions to improve the lives of millions of people. However, the large number of potential factors associated with cognitive function poses an analytical challenge.

OBJECTIVE

We used data from the longitudinal Women's Health Initiative Memory Study (WHIMS) and machine learning to investigate 50 demographic, biomedical, behavioral, social, and psychological predictors of preserved cognitive function in later life.

METHODS

Participants in WHIMS and two consecutive follow up studies who were at least 80 years old and had at least one cognitive assessment following their 80th birthday were classified as cognitively preserved. Preserved cognitive function was defined as having a score ≥39 on the most recent administration of the modified Telephone Interview for Cognitive Status (TICSm) and a mean score across all assessments ≥39. Cognitively impaired participants were those adjudicated by experts to have probable dementia or at least two adjudications of mild cognitive impairment within the 14 years of follow-up and a last TICSm score < 31. Random Forests was used to rank the predictors of preserved cognitive function.

RESULTS

Discrimination between groups based on area under the curve was 0.80 (95%-CI-0.76-0.85). Women with preserved cognitive function were younger, better educated, and less forgetful, less depressed, and more optimistic at study enrollment. They also reported better physical function and less sleep disturbance, and had lower systolic blood pressure, hemoglobin, and blood glucose levels.

CONCLUSION

The predictors of preserved cognitive function include demographic, psychological, physical, metabolic, and vascular factors suggesting a complex mix of potential contributors.

Conditional permutation importance revisited

BACKGROUND

Random forest based variable importance measures have become popular tools for assessing the contributions of the predictor variables in a fitted random forest. In this article we reconsider a frequently used variable importance measure, the Conditional Permutation Importance (CPI). We argue and illustrate that the CPI corresponds to a more partial quantification of variable importance and suggest several improvements in its methodology and implementation that enhance its practical value. In addition, we introduce the threshold value in the CPI algorithm as a parameter that can make the CPI more partial or more marginal.

RESULTS

By means of extensive simulations, where the original version of the CPI is used as the reference, we examine the impact of the proposed methodological improvements. The simulation results show how the improved CPI methodology increases the interpretability and stability of the computations. In addition, the newly proposed implementation decreases the computation times drastically and is more widely applicable. The improved CPI algorithm is made freely available as an add-on package to the open-source software R.

CONCLUSION

The proposed methodology and implementation of the CPI is computationally faster and leads to more stable results. It has a beneficial impact on practical research by making random forest analyses more interpretable.

Exploring Relationships Among Peripheral Amyloid Beta, Tau, Cytokines, Cognitive Function, and Psychosomatic Symptoms in Breast Cancer Survivors

OBJECTIVE

Accelerated brain aging has been proposed to explain cancer-related cognitive impairment, but empirical evidence for this relationship is lacking. The purpose of this study was to evaluate amyloid beta (Aβ) and tau, biomarkers of neurodegeneration, in relation to cognition in breast cancer survivors (BCSs). We explored relationships among peripheral concentrations of Aβ42, Aβ-40, tau, and cytokines; cognitive function; and psychosomatic symptoms in a cohort of BCSs post-chemotherapy.

METHODS

This secondary analysis of a cross-sectional study was conducted with 65 BCSs. Serum total Aβ-42, Aβ-40, and tau levels were measured with single molecule array technology. Cytokines (interleukin [IL]-6, tumor necrosis factor [TNF]-α, granulocyte-macrophage colony-stimulating factor [GM-CSF], interferon [IFN]-g, IL-10, IL-12, IL-13, IL1-b, IL-2, IL-4, IL-5, IL-7, and IL-8) were simultaneously measured in serum using multiplex assays. Cognitive function was measured with five standardized neuropsychological tests and psychosomatic symptoms (stress, loneliness, anxiety, depressive symptoms, fatigue, sleep quality, and daytime sleepiness) with self-report questionnaires. Data analyses included correlations and random forest regression (RFR).

RESULTS

Significant correlations were identified among hip-to-waste ratio, number of treatment modalities, Aβ-42, Aβ-40, and tau levels (rs = .27-.35, ps < .05). RFR modeling including Aβ-42, Aβ-40, tau, and cytokines as features explained significant variance in cognitive function (R2 = .71, F = 9.01, p < .0001) and psychosomatic symptoms (R2 = .74, F = 10.22, p < .0001).

CONCLUSIONS

This study suggests that neurodegenerative biomarkers interact with cytokines to influence cognitive functioning and psychosomatic symptoms in BCSs following chemotherapy, but additional research is needed.

A new approach for interpreting Random Forest models and its application to the biology of ageing

Motivation

This work uses the Random Forest (RF) classification algorithm to predict if a gene is over-expressed, under-expressed or has no change in expression with age in the brain. RFs have high predictive power, and RF models can be interpreted using a feature (variable) importance measure. However, current feature importance measures evaluate a feature as a whole (all feature values). We show that, for a popular type of biological data (Gene Ontology-based), usually only one value of a feature is particularly important for classification and the interpretation of the RF model. Hence, we propose a new algorithm for identifying the most important and most informative feature values in an RF model.

Results

The new feature importance measure identified highly relevant Gene Ontology terms for the aforementioned gene classification task, producing a feature ranking that is much more informative to biologists than an alternative, state-of-the-art feature importance measure.

Availability and implementation

The dataset and source codes used in this paper are available as 'Supplementary Material' and the description of the data can be found at: https://fabiofabris.github.io/bioinfo2018/web/.

Supplementary information

Supplementary data are available at Bioinformatics online.

Importance of details in food descriptions in estimating population nutrient intake distributions

Background

National food consumption surveys are important policy instruments that could monitor food consumption of a certain population. To be used for multiple purposes, this type of survey usually collects comprehensive food information using dietary assessment methods like 24-h dietary recalls (24HRs). However, the collection and handling of such detailed information require tremendous efforts. We aimed to improve the efficiency of data collection and handling in 24HRs, by identifying less important characteristics of food descriptions (facets) and assessing the impact of disregarding them on energy and nutrient intake distributions.

Methods

In the Dutch National Food Consumption Survey 2007–2010, food consumption data were collected through interviewer-administered 24HRs using GloboDiet software in 3819 persons. Interviewers asked participants about the characteristics of each food item according to applicable facets. Food consumption data were subsequently linked to the food composition database. The importance of facets for predicting energy and each of the 33 nutrients was estimated using the random forest algorithm. Then a simulation study was performed to determine the influence of deleting less important facets on population nutrient intake distributions.

Results

We identified 35% facets as unimportant and deleted them from the total food consumption database. The majority (79.4%) of the percent difference between percentile estimates of the population nutrient intake distributions before and after facet deletion ranged from 0 to 1%, while 20% cases ranged from 1 to 5% and 0.6% cases more than 10%.

Conclusion

We concluded that our procedure was successful in identifying less important food descriptions in estimating population nutrient intake distributions. The reduction in food descriptions has the potential to reduce the time needed for conducting interviews and data handling while maintaining the data quality of the survey.

Electronic supplementary material

The online version of this article (10.1186/s12937-019-0443-5) contains supplementary material, which is available to authorized users.

Bias in the intervention in prediction measure in random forests: illustrations and recommendations

Supplementary information

Supplementary data are available at Bioinformatics online.

Improved high-dimensional prediction with Random Forests by the use of co-data

Background

Prediction in high dimensional settings is difficult due to the large number of variables relative to the sample size. We demonstrate how auxiliary ‘co-data’ can be used to improve the performance of a Random Forest in such a setting.

Results

Co-data are incorporated in the Random Forest by replacing the uniform sampling probabilities that are used to draw candidate variables by co-data moderated sampling probabilities. Co-data here are defined as any type information that is available on the variables of the primary data, but does not use its response labels. These moderated sampling probabilities are, inspired by empirical Bayes, learned from the data at hand. We demonstrate the co-data moderated Random Forest (CoRF) with two examples. In the first example we aim to predict the presence of a lymph node metastasis with gene expression data. We demonstrate how a set of external p-values, a gene signature, and the correlation between gene expression and DNA copy number can improve the predictive performance. In the second example we demonstrate how the prediction of cervical (pre-)cancer with methylation data can be improved by including the location of the probe relative to the known CpG islands, the number of CpG sites targeted by a probe, and a set of p-values from a related study.

Conclusion

The proposed method is able to utilize auxiliary co-data to improve the performance of a Random Forest.

Electronic supplementary material

The online version of this article (doi:10.1186/s12859-017-1993-1) contains supplementary material, which is available to authorized users.