Variable selection in linear regression models: Choosing the best subset is not always the best choice

We consider the question of variable selection in linear regressions, in the sense of identifying the correct direct predictors (those variables that have nonzero coefficients given all candidate predictors). Best subset selection (BSS) is often considered the "gold standard," with its use being restricted only by its NP-hard nature. Alternatives such as the least absolute shrinkage and selection operator (Lasso) or the Elastic net (Enet) have become methods of choice in high-dimensional settings. A recent proposal represents BSS as a mixed-integer optimization problem so that large problems have become computationally feasible. We present an extensive neutral comparison assessing the ability to select the correct direct predictors of BSS compared to forward stepwise selection (FSS), Lasso, and Enet. The simulation considers a range of settings that are challenging regarding dimensionality (number of observations and variables), signal-to-noise ratios, and correlations between predictors. As fair measure of performance, we primarily used the best possible F1-score for each method, and results were confirmed by alternative performance measures and practical criteria for choosing the tuning parameters and subset sizes. Surprisingly, it was only in settings where the signal-to-noise ratio was high and the variables were uncorrelated that BSS reliably outperformed the other methods, even in low-dimensional settings. Furthermore, FSS performed almost identically to BSS. Our results shed new light on the usual presumption of BSS being, in principle, the best choice for selecting the correct direct predictors. Especially for correlated variables, alternatives like Enet are faster and appear to perform better in practical settings.

Stability of clinical prediction models developed using statistical or machine learning methods

Clinical prediction models estimate an individual's risk of a particular health outcome. A developed model is a consequence of the development dataset and model-building strategy, including the sample size, number of predictors, and analysis method (e.g., regression or machine learning). We raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model-building steps (those used to develop the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and deriving (i) a prediction instability plot of bootstrap model versus original model predictions; (ii) the mean absolute prediction error (mean absolute difference between individuals' original and bootstrap model predictions), and (iii) calibration, classification, and decision curve instability plots of bootstrap models applied in the original sample. A case study illustrates how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), while informing a model's critical appraisal (risk of bias rating), fairness, and further validation requirements.

Construction and evaluation of hourly average indoor PM2.5 concentration prediction models based on multiple types of places

Background

People usually spend most of their time indoors, so indoor fine particulate matter (PM2.5) concentrations are crucial for refining individual PM2.5 exposure evaluation. The development of indoor PM2.5 concentration prediction models is essential for the health risk assessment of PM2.5 in epidemiological studies involving large populations.

Methods

In this study, based on the monitoring data of multiple types of places, the classical multiple linear regression (MLR) method and random forest regression (RFR) algorithm of machine learning were used to develop hourly average indoor PM2.5 concentration prediction models. Indoor PM2.5 concentration data, which included 11,712 records from five types of places, were obtained by on-site monitoring. Moreover, the potential predictor variable data were derived from outdoor monitoring stations and meteorological databases. A ten-fold cross-validation was conducted to examine the performance of all proposed models.

Results

The final predictor variables incorporated in the MLR model were outdoor PM2.5 concentration, type of place, season, wind direction, surface wind speed, hour, precipitation, air pressure, and relative humidity. The ten-fold cross-validation results indicated that both models constructed had good predictive performance, with the determination coefficients (R2) of RFR and MLR were 72.20 and 60.35%, respectively. Generally, the RFR model had better predictive performance than the MLR model (RFR model developed using the same predictor variables as the MLR model, R2 = 71.86%). In terms of predictors, the importance results of predictor variables for both types of models suggested that outdoor PM2.5 concentration, type of place, season, hour, wind direction, and surface wind speed were the most important predictor variables.

Conclusion

In this research, hourly average indoor PM2.5 concentration prediction models based on multiple types of places were developed for the first time. Both the MLR and RFR models based on easily accessible indicators displayed promising predictive performance, in which the machine learning domain RFR model outperformed the classical MLR model, and this result suggests the potential application of RFR algorithms for indoor air pollutant concentration prediction.

Comparison of variable selection procedures and investigation of the role of shrinkage in linear regression-protocol of a simulation study in low-dimensional data

In low-dimensional data and within the framework of a classical linear regression model, we intend to compare variable selection methods and investigate the role of shrinkage of regression estimates in a simulation study. Our primary aim is to build descriptive models that capture the data structure parsimoniously, while our secondary aim is to derive a prediction model. Simulation studies are an important tool in statistical methodology research if they are well designed, executed, and reported. However, bias in favor of an “own” preferred method is prevalent in most simulation studies in which a new method is proposed and compared with existing methods. To overcome such bias, neutral comparison studies, which disregard the superiority or inferiority of a particular method, have been proposed. In this paper, we designed a simulation study with key principles of neutral comparison studies in mind, though certain unintentional biases cannot be ruled out. To improve the design and reporting of a simulation study, we followed the recently proposed ADEMP structure, which entails defining the aims (A), data-generating mechanisms (D), estimand/target of analysis (E), methods (M), and performance measures (P). To ensure the reproducibility of results, we published the protocol before conducting the study. In addition, we presented earlier versions of the design to several experts whose feedback influenced certain aspects of the design. We will compare popular penalized regression methods (lasso, adaptive lasso, relaxed lasso, and nonnegative garrote) that combine variable selection and shrinkage with classical variable selection methods (best subset selection and backward elimination) with and without post-estimation shrinkage of parameter estimates.

Classification of MR-Detected Additional Lesions in Patients With Breast Cancer Using a Combination of Radiomics Analysis and Machine Learning

Objective

This study was conducted in order to investigate the feasibility of using radiomics analysis (RA) with machine learning algorithms based on breast magnetic resonance (MR) images for discriminating malignant from benign MR-detected additional lesions in patients with primary breast cancer.

Materials and Methods

One hundred seventy-four MR-detected additional lesions (benign, n = 86; malignancy, n = 88) from 158 patients with ipsilateral primary breast cancer from a tertiary medical center were included in this retrospective study. The entire data were randomly split to training (80%) and independent test sets (20%). In addition, 25 patients (benign, n = 21; malignancy, n = 15) from another tertiary medical center were included for the external test. Radiomics features that were extracted from three regions-of-interest (ROIs; intratumor, peritumor, combined) using fat-saturated T1-weighted images obtained by subtracting pre- from postcontrast images (SUB) and T2-weighted image (T2) were utilized to train the support vector machine for the binary classification. A decision tree method was utilized to build a classifier model using clinical imaging interpretation (CII) features assessed by radiologists. Area under the receiver operating characteristic curve (AUROC), accuracy, sensitivity, and specificity were used to compare the diagnostic performance.

Results

The RA models trained using radiomics features from the intratumor-ROI showed comparable performance to the CII model (accuracy, AUROC: 73.3%, 69.6% for the SUB RA model; 70.0%, 75.1% for the T2 RA model; 73.3%, 72.0% for the CII model). The diagnostic performance increased when the radiomics and CII features were combined to build a fusion model. The fusion model that combines the CII features and radiomics features from multiparametric MRI data demonstrated the highest performance with an accuracy of 86.7% and an AUROC of 91.1%. The external test showed a similar pattern where the fusion models demonstrated higher levels of performance compared with the RA- or CII-only models. The accuracy and AUROC of the SUB+T2 RA+CII model in the external test were 80.6% and 91.4%, respectively.

Conclusion

Our study demonstrated the feasibility of using RA with machine learning approach based on multiparametric MRI for quantitatively characterizing MR-detected additional lesions. The fusion model demonstrated an improved diagnostic performance over the models trained with either RA or CII alone.

Self-Reported Obsession Toward COVID-19 Preventive Measures Among Undergraduate Medical Students During the Early Phase of Pandemic in Jordan

Background: Coronavirus disease 2019 (COVID-19) pandemic and its associated precautionary measures have substantial impacts not only on the medical, economic, and social context but also on psychological health. This study aimed to assess the obsession toward COVID-19 preventive measures among undergraduate medical students during the early phase of the pandemic in Jordan.

Methods: Online questionnaires were distributed between March 16, 2020 and March 19, 2020. Socio-demographic characteristics were collected, and self-reported obsession toward COVID-19 preventive measures was assessed using a single question.COVID-19 knowledge, risk perception, and precautionary measures were evaluated using scales. Using the chi-square test, Student t-test, and one-way ANOVA, we assessed the differences in the obsession of students with socio-demographic characteristics and scores of the scales.

Results: A total of 1,404 participants (60% were female participants) completed the survey with a participation rate of 15.6%. Obsession with preventive measures was reported by 6.8%. Obsession was significantly more common among women (9.2%) than men (3.3%) and students who attended COVID-19 lectures (9.5%) than those who did not attend such lectures (5.8%) (p < 0.001 and p = 0.015, respectively). Obsessed participants reported significantly higher levels of COVID-19 knowledge (p = 0.012) and precautionary measures (p < 0.001). COVID-19 risk perception had a mild effect size difference but with no statistical significance (p = 0.075). There were no significant differences in the academic levels of participants (p = 0.791) and universities (p = 0.807) between students who were obsessed and those who were not.

Conclusions: Obsession is one of the significant but unspoken psychological effects of COVID-19 precautionary measures among undergraduate medical students. Medical schools should be equipped with means to handle pandemic psychological effects.

Work with cancer or stop working after diagnosis? Variables affecting the decision

BACKGROUND

Studies about work and cancer predominantly considered the return to work of cancer survivors. However, some studies highlighted that many patients work with cancer even immediately after the diagnosis. Little is known about the frequency, causes, and consequences of this behavior.

OBJECTIVE

This study aimed to estimate how many cancer patients continue working in the month after the diagnosis in an Italian context and to determine which factors affect the decision to stop working in the same period.

METHODS

One hundred seventy-six patients with breast, gastrointestinal, prostate, or female reproductive system cancer completed a survey with demographic, occupational, and psychosocial information. Clinical information was collected from medical records. We measured how many workers continued working in the month after cancer diagnosis without substantial interruptions and selected the best logistic regression model of this behavior's predictors.

RESULTS

Sixty-eight percent of the patients continued working in the month after the diagnosis. Patients were more likely to stop working with a higher level of perceived work-health incompatibility (OR = 2.64; 95%CI: 1.48-4.69), an open-ended contract (OR = 3.20; CI: 1.13-9.09), and a complex treatment (surgery+chemo-/radio-therapy, OR = 4.25; CI: 1.55-11.65) and less likely with breast cancer (OR = 0.20; CI: 0.07-0.56), and more children (OR = 0.59; CI: 0.37-0.96).

CONCLUSIONS

To continue working with cancer is a common practice among the newly diagnosed. The decision to suspend work activity relates to evaluating how much work activities hamper one's health care needs and the practical difficulties expected in handling cancer care and work.

The development and validation of a prognostic model to PREDICT Relapse of depression in adult patients in primary care: protocol for the PREDICTR study

BACKGROUND

Most patients who present with depression are treated in primary care by general practitioners (GPs). Relapse of depression is common (at least 50% of patients treated for depression will relapse after a single episode) and leads to considerable morbidity and decreased quality of life for patients. The majority of patients will relapse within 6 months, and those with a history of relapse are more likely to relapse in the future than those with no such history. GPs see a largely undifferentiated case-mix of patients, and once patients with depression reach remission, there is limited guidance to help GPs stratify patients according to risk of relapse. We aim to develop a prognostic model to predict an individual's risk of relapse within 6-8 months of entering remission. The long-term objective is to inform the clinical management of depression after the acute phase.

METHODS

We will develop a prognostic model using secondary analysis of individual participant data drawn from seven RCTs and one longitudinal cohort study in primary or community care settings. We will use logistic regression to predict the outcome of relapse of depression within 6-8 months. We plan to include the following established relapse predictors in the model: residual depressive symptoms, number of previous depressive episodes, co-morbid anxiety and severity of index episode. We will use a "full model" development approach, including all available predictors. Performance statistics (optimism-adjusted C-statistic, calibration-in-the-large, calibration slope) and calibration plots (with smoothed calibration curves) will be calculated. Generalisability of predictive performance will be assessed through internal-external cross-validation. Clinical utility will be explored through net benefit analysis.

DISCUSSION

We will derive a statistical model to predict relapse of depression in remitted depressed patients in primary care. Assuming the model has sufficient predictive performance, we outline the next steps including independent external validation and further assessment of clinical utility and impact.

STUDY REGISTRATION

ClinicalTrials.gov ID: NCT04666662.

Missing Data in Prediction Research: A Five-Step Approach for Multiple Imputation, Illustrated in the CENTER-TBI Study

In medical research, missing data is common. In acute diseases, such as traumatic brain injury (TBI), even well-conducted prospective studies may suffer from missing data in baseline characteristics and outcomes. Statistical models may simply drop patients with any missing values, potentially leaving a selected subset of the original cohort. Imputation is widely accepted by methodologists as an appropriate way to deal with missing data. We aim to provide practical guidance on handling missing data for prediction modeling. We hereto propose a five-step approach, centered around single and multiple imputation: 1) explore the missing data patterns; 2) choose a method of imputation; 3) perform imputation; 4) assess diagnostics of the imputation; and 5) analyze the imputed data sets. We illustrate these five steps with the estimation and validation of the IMPACT (International Mission on Prognosis and Analysis of Clinical Trials in Traumatic Brain Injury) prognostic model in 1375 patients from the CENTER-TBI database, included in 53 centers across 17 countries, with moderate or severe TBI in the prospective European CENTER-TBI study. Future prediction modeling studies in acute diseases may benefit from following the suggested five steps for optimal statistical analysis and interpretation, after maximal effort has been made to minimize missing data.

Selection of variables for multivariable models: Opportunities and limitations in quantifying model stability by resampling

Statistical models are often fitted to obtain a concise description of the association of an outcome variable with some covariates. Even if background knowledge is available to guide preselection of covariates, stepwise variable selection is commonly applied to remove irrelevant ones. This practice may introduce additional variability and selection is rarely certain. However, these issues are often ignored and model stability is not questioned. Several resampling-based measures were proposed to describe model stability, including variable inclusion frequencies (VIFs), model selection frequencies, relative conditional bias (RCB), and root mean squared difference ratio (RMSDR). The latter two were recently proposed to assess bias and variance inflation induced by variable selection. Here, we study the consistency and accuracy of resampling estimates of these measures and the optimal choice of the resampling technique. In particular, we compare subsampling and bootstrapping for assessing stability of linear, logistic, and Cox models obtained by backward elimination in a simulation study. Moreover, we exemplify the estimation and interpretation of all suggested measures in a study on cardiovascular risk. The VIF and the model selection frequency are only consistently estimated in the subsampling approach. By contrast, the bootstrap is advantageous in terms of bias and precision for estimating the RCB as well as the RMSDR. Though, unbiased estimation of the latter quantity requires independence of covariates, which is rarely encountered in practice. Our study stresses the importance of addressing model stability after variable selection and shows how to cope with it.

Spanish Influenza Score (SIS): Usefulness of machine learning in the development of an early mortality prediction score in severe influenza

OBJECTIVE

To develop a mortality prediction score (Spanish Influenza Score [SIS]) for patients with severe influenza considering only variables at ICU admission, and compare its performance respect of Random Forest (RF).

DESIGN

Sub-analysis from the GETGAG/SEMICYUC database.

SCOPE

Intensive Care Medicine.

PATIENTS

Patients admitted to 184 Spanish ICUs (2009-2018) with influenza infection Intervention: None.

VARIABLES

Demographic data, severity of illness, times from symptoms onset until hospital admission (Gap-H), hospital to ICU (Gap-ICU) or hospital to diagnosis (Gap-Dg), antiviral vaccination, number of quadrants infiltrated, acute renal failure, invasive or noninvasive ventilation, shock and comorbidities. The study variable cut-off points and importance were obtained automatically. Logistic regression analysis with cross-validation was performed to develop the SIS score using the output coefficients. Accuracy and discrimination (AUC-ROC) were applied to evaluate SIS and RF. All analyses were performed using R (CRAN-R Project).

RESULTS

A total of 3959 patients were included. The mean age was 55 years (range 43-67), 60% were men, APACHE II 16 (12-21) and SOFA 5 (4-8), with ICU mortality 21.3%. Mechanical ventilation, shock, APACHE II, SOFA, acute renal failure and Gap-ICU were included in the SIS. The latter was generated according to the ORs obtained by logistic regression, and showed an accuracy of 83% with an AUC-ROC of 82%, similar to RF (AUC-ROC 82%).

CONCLUSIONS

The SIS score is easy to apply and shows adequate capacity to stratify the risk of ICU mortality. However, further studies are needed to validate the tool prospectively.

State of the art in selection of variables and functional forms in multivariable analysis-outstanding issues

BACKGROUND

How to select variables and identify functional forms for continuous variables is a key concern when creating a multivariable model. Ad hoc 'traditional' approaches to variable selection have been in use for at least 50 years. Similarly, methods for determining functional forms for continuous variables were first suggested many years ago. More recently, many alternative approaches to address these two challenges have been proposed, but knowledge of their properties and meaningful comparisons between them are scarce. To define a state of the art and to provide evidence-supported guidance to researchers who have only a basic level of statistical knowledge, many outstanding issues in multivariable modelling remain. Our main aims are to identify and illustrate such gaps in the literature and present them at a moderate technical level to the wide community of practitioners, researchers and students of statistics.

METHODS

We briefly discuss general issues in building descriptive regression models, strategies for variable selection, different ways of choosing functional forms for continuous variables and methods for combining the selection of variables and functions. We discuss two examples, taken from the medical literature, to illustrate problems in the practice of modelling.

RESULTS

Our overview revealed that there is not yet enough evidence on which to base recommendations for the selection of variables and functional forms in multivariable analysis. Such evidence may come from comparisons between alternative methods. In particular, we highlight seven important topics that require further investigation and make suggestions for the direction of further research.

CONCLUSIONS

Selection of variables and of functional forms are important topics in multivariable analysis. To define a state of the art and to provide evidence-supported guidance to researchers who have only a basic level of statistical knowledge, further comparative research is required.

Perceived acceptability of wearable devices for the treatment of mental health problems

OBJECTIVE

This study examined the potential acceptability of wearable devices (e.g., smart headbands, wristbands, and watches) aimed at treating mental health disorders, relative to conventional approaches.

METHODS

A questionnaire assessed perceptions of wearable and nonwearable treatments, along with demographic and psychological information. Respondents (N = 427) were adults from a community sample (M_age  = 44.6, SD_age  = 15.3) which included current (30.2%) and former (53.9%) mental health help-seekers.

RESULTS

Perceived effectiveness of wearables was a strong predictor of interest in using them as adjuncts to talk therapies, or as an alternative to self-help options (e.g., smartphone applications). Devices were more appealing to those with negative evaluations of psychological therapy and less experience in help-seeking.

CONCLUSIONS

Interest in using wearable devices was strong, particularly when devices were seen as effective. Clients with negative attitudes to conventional therapies may be more responsive to using wearable devices as a less directive treatment approach.

Visuo-tactile congruency influences the body schema during full body ownership illusion

•

Visuo-tactile congruency influences the body schema, but not the body image.

•

Movement behavior may provide a window into the unconscious body schema.

•

We present novel methods to distort avatars and Euclidean space in virtual reality.

Previous research showed that full body ownership illusions in virtual reality (VR) can be robustly induced by providing congruent visual stimulation, and that congruent tactile experiences provide a dispensable extension to an already established phenomenon. Here we show that visuo-tactile congruency indeed does not add to already high measures for body ownership on explicit measures, but does modulate movement behavior when walking in the laboratory. Specifically, participants who took ownership over a more corpulent virtual body with intact visuo-tactile congruency increased safety distances towards the laboratory’s walls compared to participants who experienced the same illusion with deteriorated visuo-tactile congruency. This effect is in line with the body schema more readily adapting to a more corpulent body after receiving congruent tactile information. We conclude that the action-oriented, unconscious body schema relies more heavily on tactile information compared to more explicit aspects of body ownership.

Do sensorimotor cortex activity, an individual's capacity for neuroplasticity, and psychological features during an episode of acute low back pain predict outcome at 6 months: a protocol for an Australian, multisite prospective, longitudinal cohort study

INTRODUCTION

Low back pain (LBP) is the leading cause of disability worldwide, with prevalence doubling in the past 14 years. To date, prognostic screening tools display poor discrimination and offer no net benefit of screening over and above a 'treat all' approach. Characteristics of the primary sensory (S1) and motor (M1) cortices may predict the development of chronic LBP, yet the prognostic potential of these variables remains unknown. The Understanding persistent Pain Where it ResiDes (UPWaRD) study aims to determine whether sensorimotor cortex activity, an individual's capacity for plasticity and psychosocial factors in the acute stage of pain, predict LBP outcome at 6 months. This paper describes the methods and analysis plan for the development of the prediction model.

METHODS AND ANALYSIS

The study uses a multicentre prospective longitudinal cohort design with 6-month follow-up. 120 participants, aged 18 years or older, experiencing an acute episode of LBP (less than 6 weeks duration) will be included. Primary outcomes are pain and disability.

ETHICS AND DISSEMINATION

Ethical approval has been obtained from Western Sydney University Human Research Ethics Committee (H10465) and from Neuroscience Research Australia (SSA: 16/002). Dissemination will occur through presentations at national and international conferences and publications in international peer-reviewed journals.

TRIAL REGISTRATION NUMBER

ACTRN12619000002189; Pre-results.

A comparison of model selection methods for prediction in the presence of multiply imputed data

Many approaches for variable selection with multiply imputed data in the development of a prognostic model have been proposed. However, no method prevails as uniformly best. We conducted a simulation study with a binary outcome and a logistic regression model to compare two classes of variable selection methods in the presence of MI data: (I) Model selection on bootstrap data, using backward elimination based on AIC or lasso, and fit the final model based on the most frequently (e.g. ≥50%) selected variables over all MI and bootstrap data sets; (II) Model selection on original MI data, using lasso. The final model is obtained by (i) averaging estimates of variables that were selected in any MI data set or (ii) in 50% of the MI data; (iii) performing lasso on the stacked MI data, and (iv) as in (iii) but using individual weights as determined by the fraction of missingness. In all lasso models, we used both the optimal penalty and the 1‐se rule. We considered recalibrating models to correct for overshrinkage due to the suboptimal penalty by refitting the linear predictor or all individual variables. We applied the methods on a real dataset of 951 adult patients with tuberculous meningitis to predict mortality within nine months. Overall, applying lasso selection with the 1‐se penalty shows the best performance, both in approach I and II. Stacking MI data is an attractive approach because it does not require choosing a selection threshold when combining results from separate MI data sets

Characterization of dried and freeze-dried sea fennel (Crithmum maritimum L.) samples with headspace gas-chromatography/mass spectrometry and evaluation of an electronic nose discrimination potential

Processed samples (air-dried @ 40 and @ 60 °C and freeze-dried) of sea fennel (Crithmum maritimum L.), an autochthonous spice with interesting market potential, were analyzed by headspace gas-chromatography/mass spectrometry and classification capabilities of an electronic nose in discriminating between samples with stepwise forward statistics were evaluated as well. Freeze-drying process was the most preservative in terms of limiting darkening without compromising appearance of the final product, providing weight loss of about 85% and water activity below the limit for mold growth issues. Headspace analysis of samples highlighted the presence of 35 volatiles grouped as terpene hydrocarbons, oxygenated terpenes, sesquiterpen hydrocarbons, phenyl propanoids, not-terpenic aldehydes and not-terpenic ketones. Correlations emerged between selected sensors and some detected volatile organic compounds. Stepwise linear discriminant analysis and simple K-nearest neighbors obtained a 100% overall correct classification rate in cross-validation of the electronic nose in classifying samples, whereas stepwise quadratic discriminant analysis and Naive-Bayes gave 93.3%. The sea fennel could be a new interesting spice to launch in the food market and the electronic nose showed the potential to be used in monitoring the industrial process aimed at extending its shelf-life.

Variable selection - A review and recommendations for the practicing statistician

Statistical models support medical research by facilitating individualized outcome prognostication conditional on independent variables or by estimating effects of risk factors adjusted for covariates. Theory of statistical models is well-established if the set of independent variables to consider is fixed and small. Hence, we can assume that effect estimates are unbiased and the usual methods for confidence interval estimation are valid. In routine work, however, it is not known a priori which covariates should be included in a model, and often we are confronted with the number of candidate variables in the range 10-30. This number is often too large to be considered in a statistical model. We provide an overview of various available variable selection methods that are based on significance or information criteria, penalized likelihood, the change-in-estimate criterion, background knowledge, or combinations thereof. These methods were usually developed in the context of a linear regression model and then transferred to more generalized linear models or models for censored survival data. Variable selection, in particular if used in explanatory modeling where effect estimates are of central interest, can compromise stability of a final model, unbiasedness of regression coefficients, and validity of p-values or confidence intervals. Therefore, we give pragmatic recommendations for the practicing statistician on application of variable selection methods in general (low-dimensional) modeling problems and on performing stability investigations and inference. We also propose some quantities based on resampling the entire variable selection process to be routinely reported by software packages offering automated variable selection algorithms.

Application of Penalized Regression Techniques in Modelling Insulin Sensitivity by Correlated Metabolic Parameters

This paper aims to introduce penalized estimation techniques in clinical investigations of diabetes, as well as to assess their possible advantages and limitations. Data from a previous study was used to carry out the simulations to assess: a) which procedure results in the lowest prediction error of the final model in the setting of a large number of predictor variables with high multicollinearity (of importance if insulin sensitivity should be predicted) and b) which procedure achieves the most accurate estimate of regression coefficients in the setting of fewer predictors with small unidirectional effects and moderate correlation between explanatory variables (of importance if the specific relation between an independent variable and insulin sensitivity should be examined). Moreover a special focus is on the correct direction of estimated parameter effects, a non-negligible source of error and misinterpretation of study results. The simulations were performed for varying sample size to evaluate the performance of LASSO, Ridge as well as different algorithms for Elastic Net. These methods were also compared with automatic variable selection procedures (i.e. optimizing AIC or BIC).We were not able to identify one method achieving superior performance in all situations. However, the improved accuracy of estimated effects underlines the importance of using penalized regression techniques in our example (e.g. if a researcher aims to compare relations of several correlated parameters with insulin sensitivity). However, the decision which procedure should be used depends on the specific context of a study (accuracy versus complexity) and moreover should involve clinical prior knowledge.

Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration

The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) Statement includes a 22-item checklist, which aims to improve the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. This explanation and elaboration document describes the rationale; clarifies the meaning of each item; and discusses why transparent reporting is important, with a view to assessing risk of bias and clinical usefulness of the prediction model. Each checklist item of the TRIPOD Statement is explained in detail and accompanied by published examples of good reporting. The document also provides a valuable reference of issues to consider when designing, conducting, and analyzing prediction model studies. To aid the editorial process and help peer reviewers and, ultimately, readers and systematic reviewers of prediction model studies, it is recommended that authors include a completed checklist in their submission. The TRIPOD checklist can also be downloaded from www.tripod-statement.org.

Validation of prediction models based on lasso regression with multiply imputed data

Background

In prognostic studies, the lasso technique is attractive since it improves the quality of predictions by shrinking regression coefficients, compared to predictions based on a model fitted via unpenalized maximum likelihood. Since some coefficients are set to zero, parsimony is achieved as well. It is unclear whether the performance of a model fitted using the lasso still shows some optimism. Bootstrap methods have been advocated to quantify optimism and generalize model performance to new subjects. It is unclear how resampling should be performed in the presence of multiply imputed data.

Method

The data were based on a cohort of Chronic Obstructive Pulmonary Disease patients. We constructed models to predict Chronic Respiratory Questionnaire dyspnea 6 months ahead. Optimism of the lasso model was investigated by comparing 4 approaches of handling multiply imputed data in the bootstrap procedure, using the study data and simulated data sets. In the first 3 approaches, data sets that had been completed via multiple imputation (MI) were resampled, while the fourth approach resampled the incomplete data set and then performed MI.

Results

The discriminative model performance of the lasso was optimistic. There was suboptimal calibration due to over-shrinkage. The estimate of optimism was sensitive to the choice of handling imputed data in the bootstrap resampling procedure. Resampling the completed data sets underestimates optimism, especially if, within a bootstrap step, selected individuals differ over the imputed data sets. Incorporating the MI procedure in the validation yields estimates of optimism that are closer to the true value, albeit slightly too larger.

Conclusion

Performance of prognostic models constructed using the lasso technique can be optimistic as well. Results of the internal validation are sensitive to how bootstrap resampling is performed.

Electronic supplementary material

The online version of this article (doi:10.1186/1471-2288-14-116) contains supplementary material, which is available to authorized users.