A comparative study of variable selection methods in the context of developing psychiatric screening instruments

The Association Between Psychotropic Medications and Cognitive Functioning in a Real-World Cohort of 869 Individuals with Schizophrenia

BACKGROUND

Cognitive deficits are a central feature of schizophrenia for which there are not any established pharmacological treatments. Antipsychotics are the mainstay of schizophrenia therapy but the effects of these and other psychotropic medications on the cognitive functioning of people schizophrenia have not been extensively studied in routine real-world settings.

STUDY DESIGN

A total of 869 people with schizophrenia receiving community-based care were assessed on a cognitive battery, the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS). The machine-learning tool of partialing-out least absolute shrinkage and selection operator (LASSO) regression were used to examine the independent association between the RBANS total and index scores and receipt of individual psychotropic medications considering relevant demographic, clinical, and environmental covariates. In this cross-sectional study, we also examined the effects of medication dosage and some medication combinations.

STUDY RESULTS

We found that 4 medications, clozapine, quetiapine, benztropine, and oral haloperidol were each independently associated with significantly reduced cognitive scores compared with people not receiving these medications. Three of the 4 medications, clozapine, haloperidol, and benztropine, showed a significant dose-related relationship with total cognitive score. We also documented further reductions in cognitive functioning in people receiving some pair-wise combinations of these medications. Reduced memory was the domain most associated with these medications, especially among people receiving clozapine.

CONCLUSIONS

Prescribers may consider minimizing doses and limiting the administration of combinations of the identified medications. Interventions should be further developed for people with schizophrenia to improve their cognitive functioning and quality of life.

An introduction to Sequential Monte Carlo for Bayesian inference and model comparison-with examples for psychology and behavioral science

Bayesian inference is becoming an increasingly popular framework for statistics in the behavioral sciences. However, its application is hampered by its computational intractability - almost all Bayesian analyses require a form of approximation. While some of these approximate inference algorithms, such as Markov chain Monte Carlo (MCMC), have become well known throughout the literature, other approaches exist that are not as widespread. Here, we provide an introduction to another family of approximate inference techniques known as Sequential Monte Carlo (SMC). We show that SMC brings a number of benefits, which we illustrate in three different examples: linear regression and variable selection for depression, growth curve mixture modeling of grade point averages, and in computational modeling of the Iowa Gambling Task. These use cases demonstrate that SMC is efficient in exploring posterior distributions, reaching similar predictive performance as state-of-the-art MCMC approaches in less wall-clock time. Moreover, they show that SMC is effective in dealing with multi-modal distributions, and that SMC not only approximates the posterior distribution but simultaneously provides a useful estimate of the marginal likelihood, which is the essential quantity in Bayesian model comparison. All of this comes at no additional effort from the end user.

Fast, smart, and adaptive: using machine learning to optimize mental health assessment and monitor change over time

In mental health, accurate symptom assessment and precise measurement of patient conditions are crucial for clinical decision-making and effective treatment planning. Traditional assessment methods can be burdensome, especially for vulnerable populations, leading to decreased motivation and potentially unreliable results. Computerized Adaptive Testing (CAT) has emerged as a solution, offering efficient and personalized assessments. In particular, Machine Learning-based CAT (MT-based CATs) enables adaptive, rapid, and accurate evaluations that are more easily implementable than traditional methods. This approach bypasses typical item selection processes and the associated computational costs while avoiding the rigid assumptions of traditional CAT approaches. This study investigates the effectiveness of Machine Learning-Model Tree-based CAT (ML-MT-based CAT) in detecting changes in mental health measures collected at four time points (6-month intervals between February 2018 and December 2019). Three CATs measuring generalized anxiety, depression, and social anxiety were developed and tested on a dataset with responses from 564 participants. A cross-validation approach based on real data simulations was used. Results showed that ML-MT-based CATs produced estimates of trait levels comparable to full-length tests while reducing the number of items administered by 50% or more. In addition, ML-MT-based CATs captured changes in trait levels consistent with full-length tests, outperforming short static measures.

Cognitive difficulties following adversity are not related to mental health: Findings from the ABCD study

Early life adversity is associated with differences in cognition and mental health that can impact on daily functioning. This study uses a hybrid machine-learning approach that combines random forest classification with hierarchical clustering to clarify whether there are cognitive differences between individuals who have experienced moderate-to-severe adversity relative to those have not experienced adversity, to explore whether different forms of adversity are associated with distinct cognitive alterations and whether these such alterations are related to mental health using data from the ABCD study (n = 5,955). Cognitive measures spanning language, reasoning, memory, risk-taking, affective control, and reward processing predicted whether a child had a history of adversity with reasonable accuracy (67%), and with good specificity and sensitivity (>70%). Two subgroups were identified within the adversity group and two within the no-adversity group that were distinguished by cognitive ability (low vs high). There was no evidence for specific associations between the type of adverse exposure and cognitive profile. Worse cognition predicted lower levels of mental health in unexposed children. However, while children who experience adversity had elevated mental health difficulties, their mental health did not differ as a function of cognitive ability, thus providing novel insight into the heterogeneity of psychiatric risk.

An abbreviated Chinese dyslexia screening behavior checklist for primary school students using a machine learning approach

To increase early identification and intervention of dyslexia, a prescreening instrument is critical to identifying children at risk. The present work sought to shorten and validate the 30-item Mandarin Dyslexia Screening Behavior Checklist for Primary School Students (the full checklist; Fan et al., , 19, 521-527, 2021). Our participants were 15,522 Mandarin-Chinese-speaking students and their parents, sampled from classrooms in grades 2-6 across regions in mainland China. A machine learning approach (lasso regression) was applied to shorten the full checklist (Fan et al., , 19, 521-527, 2021), constructing grade-specific brief checklists first, followed by a compilation of the common brief checklist based on the similarity across grade-specific checklists. All checklists (the full, grade-specific brief, and common brief versions) were validated and compared with data in our sample and an external sample (N = 114; Fan et al., , 19, 521-527, 2021). The results indicated that the six-item common brief checklist showed consistently high reliability (αs > .82) and reasonable classification performance (about 60% prediction accuracy and 70% sensitivity), comparable to that of the full checklist and all grade-specific brief checklists across our current sample and the external sample from Fan et al., , 19, 521-527, (2021). Our analysis showed that 2.42 (out of 5) was the cutoff score that helped classify children's reading status (children who scored higher than 2.42 might be considered at risk for dyslexia). Our final product is a valid, accessible, common brief checklist for prescreening primary school children at risk for Chinese dyslexia, which can be used across grades and regions in mainland China.

Estimating classification consistency of machine learning models for screening measures

This article illustrates novel quantitative methods to estimate classification consistency in machine learning models used for screening measures. Screening measures are used in psychology and medicine to classify individuals into diagnostic classifications. In addition to achieving high accuracy, it is ideal for the screening process to have high classification consistency, which means that respondents would be classified into the same group every time if the assessment was repeated. Although machine learning models are increasingly being used to predict a screening classification based on individual item responses, methods to describe the classification consistency of machine learning models have not yet been developed. This article addresses this gap by describing methods to estimate classification inconsistency in machine learning models arising from two different sources: sampling error during model fitting and measurement error in the item responses. These methods use data resampling techniques such as the bootstrap and Monte Carlo sampling. These methods are illustrated using three empirical examples predicting a health condition/diagnosis from item responses. R code is provided to facilitate the implementation of the methods. This article highlights the importance of considering classification consistency alongside accuracy when studying screening measures and provides the tools and guidance necessary for applied researchers to obtain classification consistency indices in their machine learning research on diagnostic assessments. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Clinical predictors of antipsychotic treatment resistance: Development and internal validation of a prognostic prediction model by the STRATA-G consortium

INTRODUCTION

Our aim was to, firstly, identify characteristics at first-episode of psychosis that are associated with later antipsychotic treatment resistance (TR) and, secondly, to develop a parsimonious prediction model for TR.

METHODS

We combined data from ten prospective, first-episode psychosis cohorts from across Europe and categorised patients as TR or non-treatment resistant (NTR) after a mean follow up of 4.18 years (s.d. = 3.20) for secondary data analysis. We identified a list of potential predictors from clinical and demographic data recorded at first-episode. These potential predictors were entered in two models: a multivariable logistic regression to identify which were independently associated with TR and a penalised logistic regression, which performed variable selection, to produce a parsimonious prediction model. This model was internally validated using a 5-fold, 50-repeat cross-validation optimism-correction.

RESULTS

Our sample consisted of N = 2216 participants of which 385 (17 %) developed TR. Younger age of psychosis onset and fewer years in education were independently associated with increased odds of developing TR. The prediction model selected 7 out of 17 variables that, when combined, could quantify the risk of being TR better than chance. These included age of onset, years in education, gender, BMI, relationship status, alcohol use, and positive symptoms. The optimism-corrected area under the curve was 0.59 (accuracy = 64 %, sensitivity = 48 %, and specificity = 76 %).

IMPLICATIONS

Our findings show that treatment resistance can be predicted, at first-episode of psychosis. Pending a model update and external validation, we demonstrate the potential value of prediction models for TR.

TCMI: a non-parametric mutual-dependence estimator for multivariate continuous distributions

The identification of relevant features, i.e., the driving variables that determine a process or the properties of a system, is an essential part of the analysis of data sets with a large number of variables. A mathematical rigorous approach to quantifying the relevance of these features is mutual information. Mutual information determines the relevance of features in terms of their joint mutual dependence to the property of interest. However, mutual information requires as input probability distributions, which cannot be reliably estimated from continuous distributions such as physical quantities like lengths or energies. Here, we introduce total cumulative mutual information (TCMI), a measure of the relevance of mutual dependences that extends mutual information to random variables of continuous distribution based on cumulative probability distributions. TCMI is a non-parametric, robust, and deterministic measure that facilitates comparisons and rankings between feature sets with different cardinality. The ranking induced by TCMI allows for feature selection, i.e., the identification of variable sets that are nonlinear statistically related to a property of interest, taking into account the number of data samples as well as the cardinality of the set of variables. We evaluate the performance of our measure with simulated data, compare its performance with similar multivariate-dependence measures, and demonstrate the effectiveness of our feature-selection method on a set of standard data sets and a typical scenario in materials science.

Psychometric and Machine Learning Approaches to Reduce the Length of Scales

Brief measures are important in psychology research because they reduce participant burden. Researchers can select items from longer measures either to build a short-form or to administer items conditional on a participant's previous responses. Researchers who carry out these item selection strategies either focus on estimating a precise score on the measure (typically carried out in a psychometric approach) or on predicting the score on the measure (possibly taking a machine learning approach). However, it is unclear how scores from the psychometric and machine learning approaches compare to each other. In this paper, the following four statistical approaches to select items are reviewed and illustrated: item response theory to build static short-forms, computerized adaptive testing, the genetic algorithm, and regression trees. Theoretical strengths and weaknesses between these four statistical approaches are discussed, and the overlap between the areas of psychometrics and machine learning is considered.

Enhancing data pipelines for forecasting student performance: integrating feature selection with cross-validation

Educators seek to harness knowledge from educational corpora to improve student performance outcomes. Although prior studies have compared the efficacy of data mining methods (DMMs) in pipelines for forecasting student success, less work has focused on identifying a set of relevant features prior to model development and quantifying the stability of feature selection techniques. Pinpointing a subset of pertinent features can (1) reduce the number of variables that need to be managed by stakeholders, (2) make "black-box" algorithms more interpretable, and (3) provide greater guidance for faculty to implement targeted interventions. To that end, we introduce a methodology integrating feature selection with cross-validation and rank each feature on subsets of the training corpus. This modified pipeline was applied to forecast the performance of 3225 students in a baccalaureate science course using a set of 57 features, four DMMs, and four filter feature selection techniques. Correlation Attribute Evaluation (CAE) and Fisher's Scoring Algorithm (FSA) achieved significantly higher Area Under the Curve (AUC) values for logistic regression (LR) and elastic net regression (GLMNET), compared to when this pipeline step was omitted. Relief Attribute Evaluation (RAE) was highly unstable and produced models with the poorest prediction performance. Borda's method identified grade point average, number of credits taken, and performance on concept inventory assessments as the primary factors impacting predictions of student performance. We discuss the benefits of this approach when developing data pipelines for predictive modeling in undergraduate settings that are more interpretable and actionable for faculty and stakeholders.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1186/s41239-021-00279-6.

Improving the Efficiency of Clinical Trial Recruitment Using an Ensemble Machine Learning to Assist With Eligibility Screening

Objective

Efficiently identifying eligible patients is a crucial first step for a successful clinical trial. The objective of this study was to test whether an approach using electronic health record (EHR) data and an ensemble machine learning algorithm incorporating billing codes and data from clinical notes processed by natural language processing (NLP) can improve the efficiency of eligibility screening.

Methods

We studied patients screened for a clinical trial of rheumatoid arthritis (RA) with one or more International Classification of Diseases (ICD) code for RA and age greater than 35 years, from a tertiary care center and a community hospital. The following three groups of EHR features were considered for the algorithm: 1) structured features, 2) the counts of NLP concepts from notes, 3) health care utilization. All features were linked to dates. We applied random forest and logistic regression with least absolute shrinkage and selection operator penalty against the following two standard approaches: 1) one or more RA ICD code and no ICD codes related to exclusion criteria (Screen_RAICD1+EX) and 2) two or more RA ICD codes (Screen_RAICD2). To test the portability, we trained the algorithm at one institution and tested it at the other.

Results

In total, 3359 patients at Brigham and Women’s Hospital (BWH) and 642 patients at Faulkner Hospital (FH) were studied, with 461 (13.7%) eligible patients at BWH and 84 (13.4%) at FH. The application of the algorithm reduced ineligible patients from chart review by 40.5% at the tertiary care center and by 57.0% at the community hospital. In contrast, Screen_RAICD2 reduced patients for chart review by 2.7% to 11.3%; Screen_RAICD1+EX reduced patients for chart review by 63% to 65% but excluded 22% to 27% of eligible patients.

Conclusion

The ensemble machine learning algorithm incorporating billing codes and NLP data increased the efficiency of eligibility screening by reducing the number of patients requiring chart review while not excluding eligible patients. Moreover, this approach can be trained at one institution and applied at another for multicenter clinical trials.

Three machine learning algorithms and their utility in exploring risk factors associated with primary cesarean section in low-risk women: A methods paper

Machine learning, a branch of artificial intelligence, is increasingly used in health research, including nursing and maternal outcomes research. Machine learning algorithms are complex and involve statistics and terminology that are not common in health research. The purpose of this methods paper is to describe three machine learning algorithms in detail and provide an example of their use in maternal outcomes research. The three algorithms, classification and regression trees, least absolute shrinkage and selection operator, and random forest, may be used to understand risk groups, select variables for a model, and rank variables' contribution to an outcome, respectively. While machine learning has plenty to contribute to health research, it also has some drawbacks, and these are discussed as well. To provide an example of the different algorithms' function, they were used on a completed cross-sectional study examining the association of oxytocin total dose exposure with primary cesarean section. The results of the algorithms are compared to what was done or found using more traditional methods.

Multiplexed quantitative proteomics provides mechanistic cues for malaria severity and complexity

Management of severe malaria remains a critical global challenge. In this study, using a multiplexed quantitative proteomics pipeline we systematically investigated the plasma proteome alterations in non-severe and severe malaria patients. We identified a few parasite proteins in severe malaria patients, which could be promising from a diagnostic perspective. Further, from host proteome analysis we observed substantial modulations in many crucial physiological pathways, including lipid metabolism, cytokine signaling, complement, and coagulation cascades in severe malaria. We propose that severe manifestations of malaria are possibly underpinned by modulations of the host physiology and defense machinery, which is evidently reflected in the plasma proteome alterations. Importantly, we identified multiple blood markers that can effectively define different complications of severe falciparum malaria, including cerebral syndromes and severe anemia. The ability of our identified blood markers to distinguish different severe complications of malaria may aid in developing new clinical tests for monitoring malaria severity.

Evidence-based statistical analysis and methods in biomedical research (SAMBR) checklists according to design features

BACKGROUND

Statistical analysis according to design features and objectives is essential to ensure the validity and reliability of the study findings and conclusions in biomedical research. Heterogeneity in reporting study design elements and conducting statistical analyses is often observed for the same study design and study objective in medical literatures. Sometimes, researchers face a lot of predicaments using appropriate statistical approaches highlighted by methodologists for a specific study design either due to lack of accessibility or understanding of statistical methods or unavailability of checklists related to design and analysis in a concise format. The purpose of this review is to provide the checklist of statistical analysis and methods in biomedical research (SAMBR) to applied researchers.

RECENT FINDINGS

We initially identified the important steps of reporting design features that may influence the choice of statistical analysis in biomedical research and essential steps of data analysis of common studies. We subsequently searched for statistical approaches employed for each study design/study objective available in publications and other resources. Compilation of these steps produced SAMBR guidance document, which includes three parts. Applied researchers can use part (A) and part (B) of SAMBR to describe or evaluate research design features and quality of statistical analysis, respectively, in reviewing studies or designing protocols. Part (C) of SAMBR can be used to perform essential and preferred evidence-based data analysis specific to study design and objective.

CONCLUSIONS

We believe that the statistical methods checklists may improve reporting of research design, standardize methodological practices, and promote consistent application of statistical approaches, thus improving the quality of research studies. The checklists do not enforce the use of suggested statistical methods but rather highlight and encourage to conduct the best statistical practices. There is a need to develop an interactive web-based application of the checklists for users for its wide applications.

Serum cytokines associated with behavior: A cross-sectional study in 5-year-old children

Nearly 10% of 5-year-old children experience social, emotional or behavioral problems and are at increased risk of developing mental disorders later in life. While animal and human studies have demonstrated that cytokines can regulate brain functions, it is unclear whether individual cytokines are associated with specific behavioral dimensions in population-based pediatric samples. Here, we used data and biological samples from 786 mother-child pairs participating to the French national mother-child cohort EDEN. At the age of 5, children were assessed for behavioral difficulties using the Strengths and Difficulties Questionnaire (SDQ) and had their serum collected. Serum samples were analyzed for levels of well-characterized effector or regulatory cytokines. We then used a penalized logistic regression method (Elastic Net), to investigate associations between serum levels of cytokines and each of the five SDQ-assessed behavioral dimensions after adjustment for relevant covariates and confounders, including psychosocial variables. We found that interleukin (IL)-6, IL-7, and IL-15 were associated with increased odds of problems in prosocial behavior, emotions, and peer relationships, respectively. In contrast, eight cytokines were associated with decreased odds of problems in one dimension: IL-8, IL-10, and IL-17A with emotional problems, Tumor Necrosis Factor (TNF)-α with conduct problems, C-C motif chemokine Ligand (CCL)2 with hyperactivity/inattention, C-X-C motif chemokine Ligand (CXCL)10 with peer problems, and CCL3 and IL-16 with abnormal prosocial behavior. Without implying causation, these associations support the notion that cytokines regulate brain functions and behavior and provide a rationale for launching longitudinal studies.

Transfer of motor skill between virtual reality viewed using a head-mounted display and conventional screen environments

BACKGROUND

Virtual reality viewed using a head-mounted display (HMD-VR) has the potential to be a useful tool for motor learning and rehabilitation. However, when developing tools for these purposes, it is important to design applications that will effectively transfer to the real world. Therefore, it is essential to understand whether motor skills transfer between HMD-VR and conventional screen-based environments and what factors predict transfer.

METHODS

We randomized 70 healthy participants into two groups. Both groups trained on a well-established measure of motor skill acquisition, the Sequential Visual Isometric Pinch Task (SVIPT), either in HMD-VR or in a conventional environment (i.e., computer screen). We then tested whether the motor skills transferred from HMD-VR to the computer screen, and vice versa. After the completion of the experiment, participants responded to questions relating to their presence in their respective training environment, age, gender, video game use, and previous HMD-VR experience. Using multivariate and univariate linear regression, we then examined whether any personal factors from the questionnaires predicted individual differences in motor skill transfer between environments.

RESULTS

Our results suggest that motor skill acquisition of this task occurs at the same rate in both HMD-VR and conventional screen environments. However, the motor skills acquired in HMD-VR did not transfer to the screen environment. While this decrease in motor skill performance when moving to the screen environment was not significantly predicted by self-reported factors, there were trends for correlations with presence and previous HMD-VR experience. Conversely, motor skills acquired in a conventional screen environment not only transferred but improved in HMD-VR, and this increase in motor skill performance could be predicted by self-reported factors of presence, gender, age and video game use.

CONCLUSIONS

These findings suggest that personal factors may predict who is likely to have better transfer of motor skill to and from HMD-VR. Future work should examine whether these and other predictors (i.e., additional personal factors such as immersive tendencies and task-specific factors such as fidelity or feedback) also apply to motor skill transfer from HMD-VR to more dynamic physical environments.

Laboratory Biomarkers, Cerebral Blood Flow Velocity, and Intellectual Function in Children with Sickle Cell Disease

Objective

The aim of this preliminary study was to describe putative markers of cerebral vasculopathy and investigate relationships among these markers, demographic factors, and cognitive function in a young sample of neurologically normal children with SCD. Study Design. Thirty-eight children with homozygous HbS, aged 4-11 years, were included. Estimated IQ and markers of coagulation and endothelial activation, hemolysis, and inflammation, as well as transcranial Doppler velocities, hydroxyurea use, and demographic information were obtained.

Results

Using multiple regression analyses, there were few significant independent associations between biomarkers or blood flow velocity and estimated IQ. Lactic dehydrogenase (LDH) independently predicted cognitive function, but blood flow velocity did not mediate this relationship. Maternal education, patient age, and hydroxyurea status were independent predictors of cognition. Given the small sample size, a LASSO statistical model was employed to further identify potential predictors of IQ, which identified LDH, absolute neutrophil count (ANC), platelet count, thrombin-antithrombin (TAT), tissue factor (TF), maternal education, age, and hydroxyurea as potential predictors of cognition.

Conclusions

In addition to effects of age and maternal education, some vasculopathic markers are associated with cognitive function in young children with SCD, and these relationships do not appear to be mediated through blood flow velocity. Although the lack of association among certain variables was not as predicted, results provide support for further research regarding the influence of vasculopathic markers on cognitive function in children with SCD without stroke, especially intravascular hemolysis and coagulation/endothelial activation, and a possible role for HU treatment in preventing or reversing cognitive decline.

Classification and Regression Tree and Computer Adaptive Testing in Cardiac Rehabilitation: Instrument Validation Study

Background

There is a need for shorter-length assessments that capture patient questionnaire data while attaining high data quality without an undue response burden on patients. Computerized adaptive testing (CAT) and classification and regression tree (CART) methods have the potential to meet these needs and can offer attractive options to shorten questionnaire lengths.

Objective

The objective of this study was to test whether CAT or CART was best suited to reduce the number of questionnaire items in multiple domains (eg, anxiety, depression, quality of life, and social support) used for a needs assessment procedure (NAP) within the field of cardiac rehabilitation (CR) without the loss of data quality.

Methods

NAP data of 2837 CR patients from a multicenter Cardiac Rehabilitation Decision Support System (CARDSS) Web-based program was used. Patients used a Web-based portal, MyCARDSS, to provide their data. CAT and CART were assessed based on their performances in shortening the NAP procedure and in terms of sensitivity and specificity.

Results

With CAT and CART, an overall reduction of 36% and 72% of NAP questionnaire length, respectively, was achieved, with a mean sensitivity and specificity of 0.765 and 0.817 for CAT, 0.777 and 0.877 for classification trees, and 0.743 and 0.40 for regression trees, respectively.

Conclusions

Both CAT and CART can be used to shorten the questionnaires of the NAP used within the field of CR. CART, however, showed the best performance, with a twice as large overall decrease in the number of questionnaire items of the NAP compared to CAT and the highest sensitivity and specificity. To our knowledge, our study is the first to assess the differences in performance between CAT and CART for shortening questionnaire lengths. Future research should consider administering varied assessments of patients over time to monitor their progress in multiple domains. For CR professionals, CART integrated with MyCARDSS would provide a feedback loop that informs the rehabilitation progress of their patients by providing real-time patient measurements.

Identifying cognitive deficits in cocaine dependence using standard tests and machine learning

There is a growing need to address the variability in detecting cognitive deficits with standard tests in cocaine dependence (CD). The aim of the current study was to identify cognitive deficits by means of Machine Learning (ML) algorithms: Generalized Linear Model (Glm), Random forest (Rf) and Elastic Net (GlmNet), to allow more effective categorization of CD and Non-dependent controls (NDC and to address common methodological problems. For our validation, we used two independent datasets, the first consisted of 87 participants (53 CD and 34 NDC) and the second of 40 participants (20 CD and 20 NDC). All participants were evaluated with neuropsychological tests that included 40 variables assessing cognitive domains. Using results from the cognitive evaluation, the three ML algorithms were trained in the first dataset and tested on the second to classify participants into CD and NDC. While the three algorithms had a receiver operating curve (ROC) performance over 50%, the GlmNet was superior in both the training (ROC = 0.71) and testing datasets (ROC = 0.85) compared to Rf and Glm. Furthermore, GlmNet was capable of identifying the eight main predictors of group assignment (CD or NCD) from all the cognitive domains assessed. Specific variables from each cognitive test resulted in robust predictors for accurate classification of new cases, such as those from cognitive flexibility and inhibition domains. These findings provide evidence of the effectiveness of ML as an approach to highlight relevant sections of standard cognitive tests in CD, and for the identification of generalizable cognitive markers.

Principal variable selection to explain grain yield variation in winter wheat from features extracted from UAV imagery

BACKGROUND

Automated phenotyping technologies are continually advancing the breeding process. However, collecting various secondary traits throughout the growing season and processing massive amounts of data still take great efforts and time. Selecting a minimum number of secondary traits that have the maximum predictive power has the potential to reduce phenotyping efforts. The objective of this study was to select principal features extracted from UAV imagery and critical growth stages that contributed the most in explaining winter wheat grain yield. Five dates of multispectral images and seven dates of RGB images were collected by a UAV system during the spring growing season in 2018. Two classes of features (variables), totaling to 172 variables, were extracted for each plot from the vegetation index and plant height maps, including pixel statistics and dynamic growth rates. A parametric algorithm, LASSO regression (the least angle and shrinkage selection operator), and a non-parametric algorithm, random forest, were applied for variable selection. The regression coefficients estimated by LASSO and the permutation importance scores provided by random forest were used to determine the ten most important variables influencing grain yield from each algorithm.

RESULTS

Both selection algorithms assigned the highest importance score to the variables related with plant height around the grain filling stage. Some vegetation indices related variables were also selected by the algorithms mainly at earlier to mid growth stages and during the senescence. Compared with the yield prediction using all 172 variables derived from measured phenotypes, using the selected variables performed comparable or even better. We also noticed that the prediction accuracy on the adapted NE lines (r = 0.58-0.81) was higher than the other lines (r = 0.21-0.59) included in this study with different genetic backgrounds.

CONCLUSIONS

With the ultra-high resolution plot imagery obtained by the UAS-based phenotyping we are now able to derive more features, such as the variation of plant height or vegetation indices within a plot other than just an averaged number, that are potentially very useful for the breeding purpose. However, too many features or variables can be derived in this way. The promising results from this study suggests that the selected set from those variables can have comparable prediction accuracies on the grain yield prediction than the full set of them but possibly resulting in a better allocation of efforts and resources on phenotypic data collection and processing.

Identifying incident dementia by applying machine learning to a very large administrative claims dataset

Alzheimer's disease and related dementias (ADRD) are highly prevalent conditions, and prior efforts to develop predictive models have relied on demographic and clinical risk factors using traditional logistical regression methods. We hypothesized that machine-learning algorithms using administrative claims data may represent a novel approach to predicting ADRD. Using a national de-identified dataset of more than 125 million patients including over 10,000 clinical, pharmaceutical, and demographic variables, we developed a cohort to train a machine learning model to predict ADRD 4-5 years in advance. The Lasso algorithm selected a 50-variable model with an area under the curve (AUC) of 0.693. Top diagnosis codes in the model were memory loss (780.93), Parkinson's disease (332.0), mild cognitive impairment (331.83) and bipolar disorder (296.80), and top pharmacy codes were psychoactive drugs. Machine learning algorithms can rapidly develop predictive models for ADRD with massive datasets, without requiring hypothesis-driven feature engineering.

A predictive model for conversion to psychosis in clinical high-risk patients

BACKGROUND

The authors developed a practical and clinically useful model to predict the risk of psychosis that utilizes clinical characteristics empirically demonstrated to be strong predictors of conversion to psychosis in clinical high-risk (CHR) individuals. The model is based upon the Structured Interview for Psychosis Risk Syndromes (SIPS) and accompanying clinical interview, and yields scores indicating one's risk of conversion.

METHODS

Baseline data, including demographic and clinical characteristics measured by the SIPS, were obtained on 199 CHR individuals seeking evaluation in the early detection and intervention for mental disorders program at the New York State Psychiatric Institute at Columbia University Medical Center. Each patient was followed for up to 2 years or until they developed a syndromal DSM-4 disorder. A LASSO logistic fitting procedure was used to construct a model for conversion specifically to a psychotic disorder.

RESULTS

At 2 years, 64 patients (32.2%) converted to a psychotic disorder. The top five variables with relatively large standardized effect sizes included SIPS subscales of visual perceptual abnormalities, dysphoric mood, unusual thought content, disorganized communication, and violent ideation. The concordance index (c-index) was 0.73, indicating a moderately strong ability to discriminate between converters and non-converters.

CONCLUSIONS

The prediction model performed well in classifying converters and non-converters and revealed SIPS measures that are relatively strong predictors of conversion, comparable with the risk calculator published by NAPLS (c-index = 0.71), but requiring only a structured clinical interview. Future work will seek to externally validate the model and enhance its performance with the incorporation of relevant biomarkers.

Variable selection in omics data: A practical evaluation of small sample sizes

In omics experiments, variable selection involves a large number of metabolites/ genes and a small number of samples (the n < p problem). The ultimate goal is often the identification of one, or a few features that are different among conditions- a biomarker. Complicating biomarker identification, the p variables often contain a correlation structure due to the biology of the experiment making identifying causal compounds from correlated compounds difficult. Additionally, there may be elements in the experimental design (blocks, batches) that introduce structure in the data. While this problem has been discussed in the literature and various strategies proposed, the over fitting problems concomitant with such approaches are rarely acknowledged. Instead of viewing a single omics experiment as a definitive test for a biomarker, an unrealistic analytical goal, we propose to view such studies as screening studies where the goal of the study is to reduce the number of features present in the second round of testing, and to limit the Type II error. Using this perspective, the performance of LASSO, ridge regression and Elastic Net was compared with the performance of an ANOVA via a simulation study and two real data comparisons. Interestingly, a dramatic increase in the number of features had no effect on Type I error for the ANOVA approach. ANOVA, even without multiple test correction, has a low false positive rates in the scenarios tested. The Elastic Net has an inflated Type I error (from 10 to 50%) for small numbers of features which increases with sample size. The Type II error rate for the ANOVA is comparable or lower than that for the Elastic Net leading us to conclude that an ANOVA is an effective analytical tool for the initial screening of features in omics experiments.

Clinical diagnosis of partial or complete anterior cruciate ligament tears using patients' history elements and physical examination tests

Objective

To assess the diagnostic validity of clusters combining history elements and physical examination tests to diagnose partial or complete anterior cruciate ligament (ACL) tears.

Design

Prospective diagnostic study.

Settings

Orthopaedic clinics (n = 2), family medicine clinics (n = 2) and community-dwelling.

Participants

Consecutive patients with a knee complaint (n = 279) and consulting one of the participating orthopaedic surgeons (n = 3) or sport medicine physicians (n = 2).

Interventions

Not applicable.

Main outcome measures

History elements and physical examination tests performed independently were compared to the reference standard: an expert physicians’ composite diagnosis including history elements, physical tests and confirmatory magnetic resonance imaging. Penalized logistic regression (LASSO) was used to identify history elements and physical examination tests associated with the diagnosis of ACL tear and recursive partitioning was used to develop diagnostic clusters. Diagnostic accuracy measures including sensitivity (Se), specificity (Sp), predictive values and positive and negative likelihood ratios (LR+/-) with associated 95% confidence intervals (CI) were calculated.

Results

Forty-three individuals received a diagnosis of partial or complete ACL tear (15.4% of total cohort). The Lachman test alone was able to diagnose partial or complete ACL tears (LR+: 38.4; 95%CI: 16.0–92.5). Combining a history of trauma during a pivot with a “popping” sensation also reached a high diagnostic validity for partial or complete tears (LR+: 9.8; 95%CI: 5.6–17.3). Combining a history of trauma during a pivot, immediate effusion after trauma and a positive Lachman test was able to identify individuals with a complete ACL tear (LR+: 17.5; 95%CI: 9.8–31.5). Finally, combining a negative history of pivot or a negative popping sensation during trauma with a negative Lachman or pivot shift test was able to exclude both partial or complete ACL tears (LR-: 0.08; 95%CI: 0.03–0.24).

Conclusion

Diagnostic clusters combining history elements and physical examination tests can support the differential diagnosis of ACL tears compared to various knee disorders.

Initial derivation of diagnostic clusters combining history elements and physical examination tests for symptomatic knee osteoarthritis

INTRODUCTION

The aim of the present study was to assess the validity of clusters combining history elements and physical examination tests to diagnose symptomatic knee osteoarthritis (SOA) compared with other knee disorders.

METHODS

This was a prospective diagnostic accuracy study, in which 279 consecutive patients consulting for a knee complaint were assessed. History elements and standardized physical examination tests were obtained independently by a physiotherapist and compared with an expert physician's composite diagnosis, including clinical examination and imaging. Recursive partitioning was used to develop diagnostic clusters for SOA. Diagnostic accuracy measures were calculated, including sensitivity, specificity, and positive and negative likelihood ratios (LR+/-), with associated 95% confidence intervals (CIs).

RESULTS

A total of 129 patients had a diagnosis of SOA (46.2%). Most cases (76%) had combined tibiofemoral and patellofemoral knee OA and 63% had radiological Kellgren-Lawrence grades of 2 or 3. Different combinations of history elements and physical examination tests were used in clusters accurately to discriminate SOA from other knee disorders. These included age of patients, body mass index, presence of valgus/varus knee misalignment, palpable knee crepitus and limited passive knee extension. Two clusters to rule in SOA reached an LR+ of 13.6 (95% CI 6.5 to 28.4) and three clusters to rule out SOA reached an LR- of 0.11 (95% CI 0.06 to 0.20).

DISCUSSION

Diagnostic clusters combining history elements and physical examination tests were able to support the differential diagnosis of SOA compared with various knee disorders without relying systematically on imaging. This could support primary care clinicians' role in the efficient management of these patients.

Development of an Algorithm to Identify Patients with Physician-Documented Insomnia

We developed an insomnia classification algorithm by interrogating an electronic medical records (EMR) database of 314,292 patients. The patients received care at Massachusetts General Hospital (MGH), Brigham and Women’s Hospital (BWH), or both, between 1992 and 2010. Our algorithm combined structured variables (such as International Classification of Diseases 9th Revision [ICD-9] codes, prescriptions, laboratory observations) and unstructured variables (such as text mentions of sleep and psychiatric disorders in clinical narrative notes). The highest classification performance of our algorithm was achieved when it included a combination of structured variables (billing codes for insomnia, common psychiatric conditions, and joint disorders) and unstructured variables (sleep disorders and psychiatric disorders). Our algorithm had superior performance in identifying insomnia patients compared to billing codes alone (area under the receiver operating characteristic curve [AUROC] = 0.83 vs. 0.55 with 95% confidence intervals [CI] of 0.76–0.90 and 0.51–0.58, respectively). When applied to the 314,292-patient population, our algorithm classified 36,810 of the patients with insomnia, of which less than 17% had a billing code for insomnia. In conclusion, an insomnia classification algorithm that incorporates clinical notes is superior to one based solely on billing codes. Compared to traditional methods, our study demonstrates that a classification algorithm that incorporates physician notes can more accurately, comprehensively, and quickly identify large cohorts of insomnia patients.

Validity of Combining History Elements and Physical Examination Tests to Diagnose Patellofemoral Pain

OBJECTIVE

To assess the validity of diagnostic clusters combining history elements and physical examination tests to diagnose or exclude patellofemoral pain (PFP).

DESIGN

Prospective diagnostic study.

SETTINGS

Orthopedic outpatient clinics, family medicine clinics, and community-dwelling.

PARTICIPANTS

Consecutive patients (N=279) consulting one of the participating orthopedic surgeons (n=3) or sport medicine physicians (n=2) for any knee complaint.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

History elements and physical examination tests were obtained by a trained physiotherapist blinded to the reference standard: a composite diagnosis including both physical examination tests and imaging results interpretation performed by an expert physician. Penalized logistic regression (least absolute shrinkage and selection operator) was used to identify history elements and physical examination tests associated with the diagnosis of PFP, and recursive partitioning was used to develop diagnostic clusters. Diagnostic accuracy measures including sensitivity, specificity, positive and negative predictive values, and positive and negative likelihood ratios with associated 95% confidence intervals (CIs) were calculated.

RESULTS

Two hundred seventy-nine participants were evaluated, and 75 had a diagnosis of PFP (26.9%). Different combinations of history elements and physical examination tests including the age of participants, knee pain location, difficulty descending stairs, patellar facet palpation, and passive knee extension range of motion were associated with a diagnosis of PFP and used in clusters to accurately discriminate between individuals with PFP and individuals without PFP. Two diagnostic clusters developed to confirm the presence of PFP yielded a positive likelihood ratio of 8.7 (95% CI, 5.2-14.6) and 3 clusters to exclude PFP yielded a negative likelihood ratio of .12 (95% CI, .06-.27).

CONCLUSIONS

Diagnostic clusters combining common history elements and physical examination tests that can accurately diagnose or exclude PFP compared to various knee disorders were developed. External validation is required before clinical use.

Diagnostic Validity of Combining History Elements and Physical Examination Tests for Traumatic and Degenerative Symptomatic Meniscal Tears

BACKGROUND

The current approach to the clinical diagnosis of traumatic and degenerative symptomatic meniscal tears (SMTs) proposes combining history elements and physical examination tests without systematic prescription of imaging investigations, yet the evidence to support this diagnostic approach is scarce.

OBJECTIVE

To assess the validity of diagnostic clusters combining history elements and physical examination tests to diagnose or exclude traumatic and degenerative SMT compared with other knee disorders.

DESIGN

Prospective diagnostic accuracy study.

SETTINGS

Patients were recruited from 2 orthopedic clinics, 2 family medicine clinics, and from a university community.

PATIENTS

A total of 279 consecutive patients who underwent consultation for a new knee complaint.

METHODS

Each patient was assessed independently by 2 evaluators. History elements and standardized physical examination tests performed by a physiotherapist were compared with the reference standard: an expert physicians' composite diagnosis including a clinical examination and confirmatory magnetic resonance imaging. Participating expert physicians were orthopedic surgeons (n = 3) or sport medicine physicians (n = 2). Penalized logistic regression (least absolute shrinkage and selection operator) was used to identify history elements and physical examination tests associated with the diagnosis of SMT and recursive partitioning was used to develop diagnostic clusters.

MAIN OUTCOME MEASURES

Diagnostic accuracy measures were calculated including sensitivity, specificity, positive and negative predictive values, and positive and negative likelihood ratios (LR+/-) with associated 95% confidence intervals (CIs).

RESULTS

Eighty patients had a diagnosis of SMT (28.7%), including 35 traumatic tears and 45 degenerative tears. The combination a history of trauma during a pivot, medial knee pain location, and a positive medial joint line tenderness test was able to diagnose (LR+ = 8.9; 95% CI 6.1-13.1) or exclude (LR- = 0.10; 95% CI 0.03-0.28) a traumatic SMT. Combining a history of progressive onset of pain, medial knee pain location, pain while pivoting, absence of valgus or varus knee misalignment, or full passive knee flexion was able to moderately diagnose (LR+ = 6.4; 95% CI 4.0-10.4) or exclude (LR- = 0.10; 95% CI 0.03-0.31) a degenerative SMT. Internal validation estimates were slightly lower for all clusters but demonstrated positive LR superior to 5 and negative LR inferior to 0.2 indicating moderate shift in posttest probability.

CONCLUSION

Diagnostic clusters combining history elements and physical examination tests can support the differential diagnosis of SMT. These results represent the initial derivation of the clusters and external validation is mandatory.

LEVEL OF EVIDENCE

I.

A new framework for prediction and variable selection for uncommon events in a large prospective cohort study

When prediction is a goal, validation utilizing data outside of the prediction effort is desirable. Typically, data is split into two parts: one for a development and one for validation. But this approach becomes less attractive when predicting uncommon events, as it substantially reduces power. When predicting uncommon events within a large prospective cohort study, we propose the use of a nested case-control design, which is an alternative to the full cohort analysis. By including all cases but only a subset of the non-cases, this design is expected to produce a result similar to the full cohort analysis. In our framework, variable selection is conducted and a prediction model is fit on those selected variables in the case-control cohort. Then, the fraction of true negative predictions (specificity) of the fitted prediction model in the case-control cohort is compared to that in the rest of the cohort (non-cases) for validation. In addition, we propose an iterative variable selection using random forest for missing data imputation, as well as a strategy for a valid classification. Our framework is illustrated with an application featuring high-dimensional variable selection in a large prospective cohort study.

Psychometric properties of the Brazilian version of the Personal Experience Screening Questionnaire

Introduction:

The Personal Experience Screening Questionnaire (PESQ) is an instrument devised for assessing the magnitude of drug misuse among adolescents. However, its psychometric properties have not been evaluated in adolescent samples outside the United States.

Objective:

To assess the internal reliability and validity of the Brazilian version of the PESQ.

Method:

A cross-sectional study was carried out with 84 adolescents from a clinical sample and a community-based sample. All of them answered the PESQ.

Results:

Among adolescents from the community, the PESQ problem severity index, which can vary from 18 to 72, was 26.48±9.28, whereas the clinical sample scored 42.89±10.02 (p<0.001). Cronbach's alpha was 0.91. Factor analysis resulted in a four-factor solution. Furthermore, both samples also had different mean scores for the other distinct content areas measured by the instrument.

Conclusion:

Evidence to support the reliability and validity of the Brazilian version of the Personal Experience Questionnaire was found.

Recursive Random Forests Enable Better Predictive Performance and Model Interpretation than Variable Selection by LASSO

Variable selection is of crucial significance in QSAR modeling since it increases the model predictive ability and reduces noise. The selection of the right variables is far more complicated than the development of predictive models. In this study, eight continuous and categorical data sets were employed to explore the applicability of two distinct variable selection methods random forests (RF) and least absolute shrinkage and selection operator (LASSO). Variable selection was performed: (1) by using recursive random forests to rule out a quarter of the least important descriptors at each iteration and (2) by using LASSO modeling with 10-fold inner cross-validation to tune its penalty λ for each data set. Along with regular statistical parameters of model performance, we proposed the highest pairwise correlation rate, average pairwise Pearson's correlation coefficient, and Tanimoto coefficient to evaluate the optimal by RF and LASSO in an extensive way. Results showed that variable selection could allow a tremendous reduction of noisy descriptors (at most 96% with RF method in this study) and apparently enhance model's predictive performance as well. Furthermore, random forests showed property of gathering important predictors without restricting their pairwise correlation, which is contrary to LASSO. The mutual exclusion of highly correlated variables in LASSO modeling tends to skip important variables that are highly related to response endpoints and thus undermine the model's predictive performance. The optimal variables selected by RF share low similarity with those by LASSO (e.g., the Tanimoto coefficients were smaller than 0.20 in seven out of eight data sets). We found that the differences between RF and LASSO predictive performances mainly resulted from the variables selected by different strategies rather than the learning algorithms. Our study showed that the right selection of variables is more important than the learning algorithm for modeling. We hope that a standard procedure could be developed based on these proposed statistical metrics to select the truly important variables for model interpretation, as well as for further use to facilitate drug discovery and environmental toxicity assessment.