Estimating misclassification error: a closer look at cross-validation based methods

A nutritional biomarker score of the Mediterranean diet and incident type 2 diabetes: Integrated analysis of data from the MedLey randomised controlled trial and the EPIC-InterAct case-cohort study

BACKGROUND

Self-reported adherence to the Mediterranean diet has been modestly inversely associated with incidence of type 2 diabetes (T2D) in cohort studies. There is uncertainty about the validity and magnitude of this association due to subjective reporting of diet. The association has not been evaluated using an objectively measured biomarker of the Mediterranean diet.

METHODS AND FINDINGS

We derived a biomarker score based on 5 circulating carotenoids and 24 fatty acids that discriminated between the Mediterranean or habitual diet arms of a parallel design, 6-month partial-feeding randomised controlled trial (RCT) conducted between 2013 and 2014, the MedLey trial (128 participants out of 166 randomised). We applied this biomarker score in an observational study, the European Prospective Investigation into Cancer and Nutrition (EPIC)-InterAct case-cohort study, to assess the association of the score with T2D incidence over an average of 9.7 years of follow-up since the baseline (1991 to 1998). We included 22,202 participants, of whom 9,453 were T2D cases, with relevant biomarkers from an original case-cohort of 27,779 participants sampled from a cohort of 340,234 people. As a secondary measure of the Mediterranean diet, we used a score estimated from dietary-self report. Within the trial, the biomarker score discriminated well between the 2 arms; the cross-validated C-statistic was 0.88 (95% confidence interval (CI) 0.82 to 0.94). The score was inversely associated with incident T2D in EPIC-InterAct: the hazard ratio (HR) per standard deviation of the score was 0.71 (95% CI: 0.65 to 0.77) following adjustment for sociodemographic, lifestyle and medical factors, and adiposity. In comparison, the HR per standard deviation of the self-reported Mediterranean diet was 0.90 (95% CI: 0.86 to 0.95). Assuming the score was causally associated with T2D, higher adherence to the Mediterranean diet in Western European adults by 10 percentiles of the score was estimated to reduce the incidence of T2D by 11% (95% CI: 7% to 14%). The study limitations included potential measurement error in nutritional biomarkers, unclear specificity of the biomarker score to the Mediterranean diet, and possible residual confounding.

CONCLUSIONS

These findings suggest that objectively assessed adherence to the Mediterranean diet is associated with lower risk of T2D and that even modestly higher adherence may have the potential to reduce the population burden of T2D meaningfully.

TRIAL REGISTRATION

Australian New Zealand Clinical Trials Registry (ANZCTR) ACTRN12613000602729 https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=363860.

Single classifier vs. ensemble machine learning approaches for mental health prediction

Early prediction of mental health issues among individuals is paramount for early diagnosis and treatment by mental health professionals. One of the promising approaches to achieving fully automated computer-based approaches for predicting mental health problems is via machine learning. As such, this study aims to empirically evaluate several popular machine learning algorithms in classifying and predicting mental health problems based on a given data set, both from a single classifier approach as well as an ensemble machine learning approach. The data set contains responses to a survey questionnaire that was conducted by Open Sourcing Mental Illness (OSMI). Machine learning algorithms investigated in this study include Logistic Regression, Gradient Boosting, Neural Networks, K-Nearest Neighbours, and Support Vector Machine, as well as an ensemble approach using these algorithms. Comparisons were also made against more recent machine learning approaches, namely Extreme Gradient Boosting and Deep Neural Networks. Overall, Gradient Boosting achieved the highest overall accuracy of 88.80% followed by Neural Networks with 88.00%. This was followed by Extreme Gradient Boosting and Deep Neural Networks at 87.20% and 86.40%, respectively. The ensemble classifier achieved 85.60% while the remaining classifiers achieved between 82.40 and 84.00%. The findings indicate that Gradient Boosting provided the highest classification accuracy for this particular mental health bi-classification prediction task. In general, it was also demonstrated that the prediction results produced by all of the machine learning approaches studied here were able to achieve more than 80% accuracy, thereby indicating a highly promising approach for mental health professionals toward automated clinical diagnosis.

Challenging oppression: A social identity model of stigma resistance in higher-weight individuals

Many higher-weight individuals have internalised societal weight stigma, devaluing themselves because of their weight. Rejecting and challenging societal devaluation is generally associated with superior outcomes compared with stigma internalisation or inaction; however, stigma resistance has not been studied in higher-weight individuals, despite ubiquitous weight stigma in daily life. Applying a social identity framework, we utilised decision tree analysis to explore predictors of responses to weight stigma in 931 self-classified higher-weight individuals. While ingroup identification with the group 'Fat' was the major predictor of stigma resistance (versus internalisation), perceived illegitimacy of societal weight stigma defined a subgroup of resisters even in the absence of group identity. Interventions focusing on the illegitimacy of unequal social status and treatment may be effective at reducing internalisation and fostering resistance in a population with characteristically low ingroup identity.

Automated Prediction of Employee Attrition Using Ensemble Model Based on Machine Learning Algorithms

Competent employees are a rare commodity for great companies. The problem of maintaining good employees with experience threatens the owners of companies. The issue of employee attrition can cost employers a lot as it takes a lot to compensate for their expertise and efficiency. For this reason, in this research, we present an automated model that can predict employee attrition based on different predictive analytical techniques. These techniques have been applied with different pipeline architectures to select the best champion model. Also, an autotuning approach has been implemented to calculate the best combination of hyper parameters to build the champion model. Finally, we propose an ensemble model for selecting the most efficient model subject to different assessments measures. The results of the proposed model show that no model up until now could be considered ideal and perfect for each case of business context. Yet, our chosen model was pretty much optimal as per our requirements and adequately satisfied the intended goal.

Predicting hERG channel blockers with directed message passing neural networks

Compounds with human ether-à-go-go related gene (hERG) blockade activity may cause severe cardiotoxicity. Assessing the hERG liability in the early stages of the drug discovery process is important, and the in silico methods for predicting hERG channel blockers are actively pursued. In the present study, the directed message passing neural network (D-MPNN) was applied to construct classification models for identifying hERG blockers based on diverse datasets. Several descriptors and fingerprints were tested along with the D-MPNN model. Among all these combinations, D-MPNN with the moe206 descriptors generated from MOE (D-MPNN + moe206) showed significantly improved performances. The AUC-ROC values of the D-MPNN + moe206 model reached 0.956 ± 0.005 under random split and 0.922 ± 0.015 under scaffold split on Cai's hERG dataset, respectively. Moreover, the comparisons between our models and several recently reported machine learning models were made based on various datasets. Our results indicated that the D-MPNN + moe206 model is among the best classification models. Overall, the excellent performance of the DMPNN + moe206 model achieved in this study highlights its potential application in the discovery of novel and effective hERG blockers.

Compounds with human ether-à-go-go related gene (hERG) blockade activity may cause severe cardiotoxicity.

Predicting Future Geographic Hotspots of Potentially Preventable Hospitalisations Using All Subset Model Selection and Repeated K-Fold Cross-Validation

Long-term future prediction of geographic areas with high rates of potentially preventable hospitalisations (PPHs) among residents, or "hotspots", is critical to ensure the effective location of place-based health service interventions. This is because such interventions are typically expensive and take time to develop, implement, and take effect, and hotspots often regress to the mean. Using spatially aggregated, longitudinal administrative health data, we introduce a method to make such predictions. The proposed method combines all subset model selection with a novel formulation of repeated k-fold cross-validation in developing optimal models. We illustrate its application predicting three-year future hotspots for four PPHs in an Australian context: type II diabetes mellitus, heart failure, chronic obstructive pulmonary disease, and "high risk foot". In these examples, optimal models are selected through maximising positive predictive value while maintaining sensitivity above a user-specified minimum threshold. We compare the model's performance to that of two alternative methods commonly used in practice, i.e., prediction of future hotspots based on either: (i) current hotspots, or (ii) past persistent hotspots. In doing so, we demonstrate favourable performance of our method, including with respect to its ability to flexibly optimise various different metrics. Accordingly, we suggest that our method might effectively be used to assist health planners predict excess future demand of health services and prioritise placement of interventions. Furthermore, it could be used to predict future hotspots of non-health events, e.g., in criminology.

Prediction of Preeclampsia and Intrauterine Growth Restriction: Development of Machine Learning Models on a Prospective Cohort

BACKGROUND

Preeclampsia and intrauterine growth restriction are placental dysfunction-related disorders (PDDs) that require a referral decision be made within a certain time period. An appropriate prediction model should be developed for these diseases. However, previous models did not demonstrate robust performances and/or they were developed from datasets with highly imbalanced classes.

OBJECTIVE

In this study, we developed a predictive model of PDDs by machine learning that uses features at 24-37 weeks' gestation, including maternal characteristics, uterine artery (UtA) Doppler measures, soluble fms-like tyrosine kinase receptor-1 (sFlt-1), and placental growth factor (PlGF).

METHODS

A public dataset was taken from a prospective cohort study that included pregnant women with PDDs (66/95, 69%) and a control group (29/95, 31%). Preliminary selection of features was based on a statistical analysis using SAS 9.4 (SAS Institute). We used Weka (Waikato Environment for Knowledge Analysis) 3.8.3 (The University of Waikato, Hamilton, NZ) to automatically select the best model using its optimization algorithm. We also manually selected the best of 23 white-box models. Models, including those from recent studies, were also compared by interval estimation of evaluation metrics. We used the Matthew correlation coefficient (MCC) as the main metric. It is not overoptimistic to evaluate the performance of a prediction model developed from a dataset with a class imbalance. Repeated 10-fold cross-validation was applied.

RESULTS

The classification via regression model was chosen as the best model. Our model had a robust MCC (.93, 95% CI .87-1.00, vs .64, 95% CI .57-.71) and specificity (100%, 95% CI 100-100, vs 90%, 95% CI 90-90) compared to each metric of the best models from recent studies. The sensitivity of this model was not inferior (95%, 95% CI 91-100, vs 100%, 95% CI 92-100). The area under the receiver operating characteristic curve was also competitive (0.970, 95% CI 0.966-0.974, vs 0.987, 95% CI 0.980-0.994). Features in the best model were maternal weight, BMI, pulsatility index of the UtA, sFlt-1, and PlGF. The most important feature was the sFlt-1/PlGF ratio. This model used an M5P algorithm consisting of a decision tree and four linear models with different thresholds. Our study was also better than the best ones among recent studies in terms of the class balance and the size of the case class (66/95, 69%, vs 27/239, 11.3%).

CONCLUSIONS

Our model had a robust predictive performance. It was also developed to deal with the problem of a class imbalance. In the context of clinical management, this model may improve maternal mortality and neonatal morbidity and reduce health care costs.

Filtering maxRatio results with machine learning models increases quantitative PCR accuracy over the fit point method

With qPCR reaching thousands of reactions per run, assay validation needs automation. We applied support vector machine to qPCR analysis and we could identify reactions with 100% accuracy, dispensing them from further validation. We achieved a greatly reduced workload that could improve high-throughput qPCR analysis.

Analysis of TaqMan Array Cards Data by an Assumption-Free Improvement of the maxRatio Algorithm Is More Accurate than the Cycle-Threshold Method

Quantitative PCR diagnostic platforms are moving towards increased sample throughput, with instruments capable of carrying out thousands of reactions at once already in use. The need for a computational tool to reliably assist in the validation of the results is therefore compelling. In the present study, 328 residual clinical samples provided by the Public Health England at Addenbrooke's Hospital (Cambridge, UK) were processed by TaqMan Array Card assay, generating 15 744 reactions from 54 targets. The amplification data were analysed by the conventional cycle-threshold (CT) method and an improvement of the maxRatio (MR) algorithm developed to filter out the reactions with irregular amplification profiles. The reactions were also independently validated by three raters and a consensus was generated from their classification. The inter-rater agreement by Fleiss' kappa was 0.885; the agreement between either CT or MR with the raters gave Fleiss' kappa 0.884 and 0.902, respectively. Based on the consensus classification, the CT and MR methods achieved an assay accuracy of 0.979 and 0.987, respectively. These results suggested that the assumption-free MR algorithm was more reliable than the CT method, with clear advantages for the diagnostic settings.

Machine learning-, rule- and pharmacophore-based classification on the inhibition of P-glycoprotein and NorA

The efflux pumps P-glycoprotein (P-gp) in humans and NorA in Staphylococcus aureus are of great interest for medicinal chemists because of their important roles in multidrug resistance (MDR). The high polyspecificity as well as the unavailability of high-resolution X-ray crystal structures of these transmembrane proteins lead us to combining ligand-based approaches, which in the case of this study were machine learning, perceptual mapping and pharmacophore modelling. For P-gp inhibitory activity, individual models were developed using different machine learning algorithms and subsequently combined into an ensemble model which showed a good discrimination between inhibitors and noninhibitors (acctrain-diverse = 84%; accinternal-test = 92% and accexternal-test = 100%). For ligand promiscuity between P-gp and NorA, perceptual maps and pharmacophore models were generated for the detection of rules and features. Based on these in silico tools, hit compounds for reversing MDR were discovered from the in-house and DrugBank databases through virtual screening in an attempt to restore drug sensitivity in cancer cells and bacteria.

Predicting changes in cortical electrophysiological function after in vitro traumatic brain injury

Finite element (FE) models of traumatic brain injury (TBI) are capable of predicting injury-induced brain tissue deformation. However, current FE models are not equipped to predict the biological consequences of tissue deformation, which requires the determination of tolerance criteria relating the effects of mechanical stimuli to biologically relevant functional responses. To address this deficiency, we present functional tolerance criteria for the cortex for alterations in neuronal network electrophysiological function in response to controlled mechanical stimuli. Organotypic cortical slice cultures were mechanically injured via equibiaxial stretch with a well-characterized in vitro model of TBI at tissue strains and strain rates relevant to TBI (up to Lagrangian strain of 0.59 and strain rates up to 29 [Formula: see text]. At 4-6 days post-injury, electrophysiological function was assessed simultaneously throughout the cortex using microelectrode arrays. Electrophysiological parameters associated with unstimulated spontaneous network activity (neural event rate, duration, and magnitude), stimulated evoked responses (the maximum response [Formula: see text], the stimulus current necessary for a half-maximal response [Formula: see text], and the electrophysiological parameter [Formula: see text] that is representative of firing uniformity), and evoked paired-pulse ratios at varying interstimulus intervals were quantified for each cortical slice culture. Nonlinear regression was performed between mechanical injury parameters as independent variables (tissue strain and strain rate) and each electrophysiological parameter as output. Parsimonious best-fit equations were determined from a large pool of candidate equations with tenfold cross-validation. Changes in electrophysiological parameters were dependent on strain and strain rate in a complex manner. Compared to the hippocampus, the cortex was less spontaneously active, less excitable, and less prone to significant changes in electrophysiological function in response to controlled deformation (strain or strain rate). Our study provides functional data that can be incorporated into FE models to improve their predictive capabilities of the in vivo consequences of TBI.

Functional tolerance to mechanical deformation developed from organotypic hippocampal slice cultures

In this study, we measured changes in electrophysiological activity after mechanical deformation of living organotypic hippocampal brain slice cultures at tissue strains and strain rates relevant to traumatic brain injury (TBI). Electrophysiological activity was measured throughout the hippocampus with a 60-electrode microelectrode array. Electrophysiological parameters associated with unstimulated spontaneous activity (neural event firing rate, duration, and magnitude), stimulated evoked responses (the maximum response [Formula: see text], the stimulus current necessary for a half-maximal response [Formula: see text], and the electrophysiological parameter m that is representative of firing uniformity), and paired-pulse responses (paired-pulse ratio at varying interstimulus intervals) were quantified for each hippocampal region (CA1, CA3, and DG). We present functional tolerance criteria for the hippocampus in the form of mathematical relationships between the input tissue-level injury parameters (strain and strain rate) and altered neuronal network function. Most changes in electrophysiology were dependent on strain and strain rate in a complex fashion, independent of hippocampal anatomy, with the notable exception of [Formula: see text]. Until it becomes possible to directly measure brain tissue deformation in vivo, finite element (FE) models will be necessary to simulate and predict the in vivo consequences of TBI. One application of our study is to provide functional relationships that can be incorporated into these FE models to enhance their biofidelity of accident and collision reconstructions by predicting biological outcomes in addition to mechanical responses.