STRengthening analytical thinking for observational studies: the STRATOS initiative

Common Critiques and Recommendations for Studies in Neurology Using Machine Learning Methods

Machine learning (ML) methods are becoming more prevalent in the neurology literature as alternatives to traditional statistical methods to address challenges in the analysis of modern data sets. Despite the increase in the popularity of ML methods in neurology studies, some authors do not fully address all items recommended in reporting guidelines. The authors of this Research Methods article are members of the Neurology® editorial board and have reviewed many studies using ML methods. In their review reports, several critiques often appear, which could be avoided if guidance were available. In this article, we detail common critiques found in ML research studies and make recommendations for how to avoid them. The first critique involves misalignment of the study goals and the analysis conducted. The second critique focuses on ML terminology being appropriately used. Critiques 3-6 are related to the study design: justifying sample sizes and the suitability of the data set for the study goals, describing the ML analysis pipeline sufficiently, quantifying the amount of missing data and providing information about missing data handling, and including uncertainty estimates for key metrics. The seventh critique focuses on fairly describing both strengths and limitations of the ML study, including the analysis methodology and results. We provide examples in neurology for each critique and guidance on how to avoid the critique. Overall, we recommend that authors use ML-specific checklists developed by research consortia for designing and reporting studies using ML. We also recommend that authors involve both a statistician and an ML expert in work that uses ML. Although our list of critiques is not exhaustive, our recommendations should help improve the quality and rigor of ML studies. ML has great potential to revolutionize neurology, but investigators need to conduct and report the results in a way that allows readers to fully evaluate the benefits and limitations of ML approaches.

Limitations of Separating Athletes into High or Low-Risk Groups based on a Cut-Off. A Clinical Commentary

Background

Athlete injury risk assessment and management is an important, yet challenging task for sport and exercise medicine professionals. A common approach to injury risk screening is to stratify athletes into risk groups based on their performance on a test relative to a cut-off threshold. However, one potential reason for ineffective injury prevention efforts is the over-reliance on identifying these 'at-risk' groups using arbitrary cut-offs for these tests and measures. The purpose of this commentary is to discuss the conceptual and technical issues related to the use of a cut-off in both research and clinical practice.

Clinical Question

How can we better assess and interpret clinical tests or measures to enable a more effective injury risk assessment in athletes?

Key Results

Cut-offs typically lack strong biologic plausibility to support them; and are typically derived in a data-driven manner and thus not generalizable to other samples. When a cut-off is used in analyses, information is lost, leading to potentially misleading results and less accurate injury risk prediction. Dichotomizing a continuous variable using a cut-off should be avoided. Using continuous variables on its original scale is advantageous because information is not discarded, outcome prediction accuracy is not lost, and personalized medicine can be facilitated.

Clinical Application

Researchers and clinicians are encouraged to analyze and interpret the results of tests and measures using continuous variables and avoid relying on singular cut-offs to guide decisions. Injury risk can be predicted more accurately when using continuous variables in their natural form. A more accurate risk prediction will facilitate personalized approaches to injury risk mitigation and may lead to a decline in injury rates.

Level of Evidence

5.

Evaluating variable selection methods for multivariable regression models: A simulation study protocol

Researchers often perform data-driven variable selection when modeling the associations between an outcome and multiple independent variables in regression analysis. Variable selection may improve the interpretability, parsimony and/or predictive accuracy of a model. Yet variable selection can also have negative consequences, such as false exclusion of important variables or inclusion of noise variables, biased estimation of regression coefficients, underestimated standard errors and invalid confidence intervals, as well as model instability. While the potential advantages and disadvantages of variable selection have been discussed in the literature for decades, few large-scale simulation studies have neutrally compared data-driven variable selection methods with respect to their consequences for the resulting models. We present the protocol for a simulation study that will evaluate different variable selection methods: forward selection, stepwise forward selection, backward elimination, augmented backward elimination, univariable selection, univariable selection followed by backward elimination, and penalized likelihood approaches (Lasso, relaxed Lasso, adaptive Lasso). These methods will be compared with respect to false inclusion and/or exclusion of variables, consequences on bias and variance of the estimated regression coefficients, the validity of the confidence intervals for the coefficients, the accuracy of the estimated variable importance ranking, and the predictive performance of the selected models. We consider both linear and logistic regression in a low-dimensional setting (20 independent variables with 10 true predictors and 10 noise variables). The simulation will be based on real-world data from the National Health and Nutrition Examination Survey (NHANES). Publishing this study protocol ahead of performing the simulation increases transparency and allows integrating the perspective of other experts into the study design.

Initial data analysis for longitudinal studies to build a solid foundation for reproducible analysis

Initial data analysis (IDA) is the part of the data pipeline that takes place between the end of data retrieval and the beginning of data analysis that addresses the research question. Systematic IDA and clear reporting of the IDA findings is an important step towards reproducible research. A general framework of IDA for observational studies includes data cleaning, data screening, and possible updates of pre-planned statistical analyses. Longitudinal studies, where participants are observed repeatedly over time, pose additional challenges, as they have special features that should be taken into account in the IDA steps before addressing the research question. We propose a systematic approach in longitudinal studies to examine data properties prior to conducting planned statistical analyses. In this paper we focus on the data screening element of IDA, assuming that the research aims are accompanied by an analysis plan, meta-data are well documented, and data cleaning has already been performed. IDA data screening comprises five types of explorations, covering the analysis of participation profiles over time, evaluation of missing data, presentation of univariate and multivariate descriptions, and the depiction of longitudinal aspects. Executing the IDA plan will result in an IDA report to inform data analysts about data properties and possible implications for the analysis plan-another element of the IDA framework. Our framework is illustrated focusing on hand grip strength outcome data from a data collection across several waves in a complex survey. We provide reproducible R code on a public repository, presenting a detailed data screening plan for the investigation of the average rate of age-associated decline of grip strength. With our checklist and reproducible R code we provide data analysts a framework to work with longitudinal data in an informed way, enhancing the reproducibility and validity of their work.

Developing partnerships for academic data science consulting and collaboration units

Data science consulting and collaboration units (DSUs) are core infrastructure for research at universities. Activities span data management, study design, data analysis, data visualization, predictive modelling, preparing reports, manuscript writing and advising on statistical methods and may include an experiential or teaching component. Partnerships are needed for a thriving DSU as an active part of the larger university network. Guidance for identifying, developing and managing successful partnerships for DSUs can be summarized in six rules: (1) align with institutional strategic plans, (2) cultivate partnerships that fit your mission, (3) ensure sustainability and prepare for growth, (4) define clear expectations in a partnership agreement, (5) communicate and (6) expect the unexpected. While these rules are not exhaustive, they are derived from experiences in a diverse set of DSUs, which vary by administrative home, mission, staffing and funding model. As examples in this paper illustrate, these rules can be adapted to different organizational models for DSUs. Clear expectations in partnership agreements are essential for high quality and consistent collaborations and address core activities, duration, staffing, cost and evaluation. A DSU is an organizational asset that should involve thoughtful investment if the institution is to gain real value.

Phases of methodological research in biostatistics-Building the evidence base for new methods

Although new biostatistical methods are published at a very high rate, many of these developments are not trustworthy enough to be adopted by the scientific community. We propose a framework to think about how a piece of methodological work contributes to the evidence base for a method. Similar to the well-known phases of clinical research in drug development, we propose to define four phases of methodological research. These four phases cover (I) proposing a new methodological idea while providing, for example, logical reasoning or proofs, (II) providing empirical evidence, first in a narrow target setting, then (III) in an extended range of settings and for various outcomes, accompanied by appropriate application examples, and (IV) investigations that establish a method as sufficiently well-understood to know when it is preferred over others and when it is not; that is, its pitfalls. We suggest basic definitions of the four phases to provoke thought and discussion rather than devising an unambiguous classification of studies into phases. Too many methodological developments finish before phase III/IV, but we give two examples with references. Our concept rebalances the emphasis to studies in phases III and IV, that is, carefully planned method comparison studies and studies that explore the empirical properties of existing methods in a wider range of problems.

The Bayesian simulation study (BASIS) framework for simulation studies in statistical and methodological research

Statistical simulation studies are becoming increasingly popular to demonstrate the performance or superiority of new computational procedures and algorithms. Despite this status quo, previous surveys of the literature have shown that the reporting of statistical simulation studies often lacks relevant information and structure. The latter applies in particular to Bayesian simulation studies, and in this paper the Bayesian simulation study framework (BASIS) is presented as a step towards improving the situation. The BASIS framework provides a structured skeleton for planning, coding, executing, analyzing, and reporting Bayesian simulation studies in biometrical research and computational statistics. It encompasses various features of previous proposals and recommendations in the methodological literature and aims to promote neutral comparison studies in statistical research. Computational aspects covered in the BASIS include algorithmic choices, Markov-chain-Monte-Carlo convergence diagnostics, sensitivity analyses, and Monte Carlo standard error calculations for Bayesian simulation studies. Although the BASIS framework focuses primarily on methodological research, it also provides useful guidance for researchers who rely on the results of Bayesian simulation studies or analyses, as current state-of-the-art guidelines for Bayesian analyses are incorporated into the BASIS.

Greedy knot selection algorithm for restricted cubic spline regression

Non-linear regression modeling is common in epidemiology for prediction purposes or estimating relationships between predictor and response variables. Restricted cubic spline (RCS) regression is one such method, for example, highly relevant to Cox proportional hazard regression model analysis. RCS regression uses third-order polynomials joined at knot points to model non-linear relationships. The standard approach is to place knots by a regular sequence of quantiles between the outer boundaries. A regression curve can easily be fitted to the sample using a relatively high number of knots. The problem is then overfitting, where a regression model has a good fit to the given sample but does not generalize well to other samples. A low knot count is thus preferred. However, the standard knot selection process can lead to underperformance in the sparser regions of the predictor variable, especially when using a low number of knots. It can also lead to overfitting in the denser regions. We present a simple greedy search algorithm using a backward method for knot selection that shows reduced prediction error and Bayesian information criterion scores compared to the standard knot selection process in simulation experiments. We have implemented the algorithm as part of an open-source R-package, knutar.

Accounting for time dependency in meta-analyses of concordance probability estimates

Recent years have seen the development of many novel scoring tools for disease prognosis and prediction. To become accepted for use in clinical applications, these tools have to be validated on external data. In practice, validation is often hampered by logistical issues, resulting in multiple small-sized validation studies. It is therefore necessary to synthesize the results of these studies using techniques for meta-analysis. Here we consider strategies for meta analyzing the concordance probability for time-to-event data ("C-index"), which has become a popular tool to evaluate the discriminatory power of prediction models with a right-censored outcome. We show that standard meta-analysis of the C-index may lead to biased results, as the magnitude of the concordance probability depends on the length of the time interval used for evaluation (defined e.g., by the follow-up time, which might differ considerably between studies). To address this issue, we propose a set of methods for random-effects meta-regression that incorporate time directly as covariate in the model equation. In addition to analyzing nonlinear time trends via fractional polynomial, spline, and exponential decay models, we provide recommendations on suitable transformations of the C-index before meta-regression. Our results suggest that the C-index is best meta-analyzed using fractional polynomial meta-regression with logit-transformed C-index values. Classical random-effects meta-analysis (not considering time as covariate) is demonstrated to be a suitable alternative when follow-up times are small. Our findings have implications for the reporting of C-index values in future studies, which should include information on the length of the time interval underlying the calculations.

Poor handling of continuous predictors in clinical prediction models using logistic regression: a systematic review

BACKGROUND AND OBJECTIVES

When developing a clinical prediction model, assuming a linear relationship between the continuous predictors and outcome is not recommended. Incorrect specification of the functional form of continuous predictors could reduce predictive accuracy. We examine how continuous predictors are handled in studies developing a clinical prediction model.

METHODS

We searched PubMed for clinical prediction model studies developing a logistic regression model for a binary outcome, published between July 01, 2020, and July 30, 2020.

RESULTS

In total, 118 studies were included in the review (18 studies (15%) assessed the linearity assumption or used methods to handle nonlinearity, and 100 studies (85%) did not). Transformation and splines were commonly used to handle nonlinearity, used in 7 (n = 7/18, 39%) and 6 (n = 6/18, 33%) studies, respectively. Categorization was most often used method to handle continuous predictors (n = 67/118, 56.8%) where most studies used dichotomization (n = 40/67, 60%). Only ten models included nonlinear terms in the final model (n = 10/18, 56%).

CONCLUSION

Though widely recommended not to categorize continuous predictors or assume a linear relationship between outcome and continuous predictors, most studies categorize continuous predictors, few studies assess the linearity assumption, and even fewer use methodology to account for nonlinearity. Methodological guidance is provided to guide researchers on how to handle continuous predictors when developing a clinical prediction model.

Issues in Implementing Regression Calibration Analyses

Regression calibration is a popular approach for correcting biases in estimated regression parameters when exposure variables are measured with error. This approach involves building a calibration equation to estimate the value of the unknown true exposure given the error-prone measurement and other covariates. The estimated, or calibrated, exposure is then substituted for the unknown true exposure in the health outcome regression model. When used properly, regression calibration can greatly reduce the bias induced by exposure measurement error. Here, we first provide an overview of the statistical framework for regression calibration, specifically discussing how a special type of error, called Berkson error, arises in the estimated exposure. We then present practical issues to consider when applying regression calibration, including: 1) how to develop the calibration equation and which covariates to include; 2) valid ways to calculate standard errors of estimated regression coefficients; and 3) problems arising if one of the covariates in the calibration model is a mediator of the relationship between the exposure and outcome. Throughout, we provide illustrative examples using data from the Hispanic Community Health Study/Study of Latinos (United States, 2008-2011) and simulations. We conclude with recommendations for how to perform regression calibration.

Statistical analysis of high-dimensional biomedical data: a gentle introduction to analytical goals, common approaches and challenges

BACKGROUND

In high-dimensional data (HDD) settings, the number of variables associated with each observation is very large. Prominent examples of HDD in biomedical research include omics data with a large number of variables such as many measurements across the genome, proteome, or metabolome, as well as electronic health records data that have large numbers of variables recorded for each patient. The statistical analysis of such data requires knowledge and experience, sometimes of complex methods adapted to the respective research questions.

METHODS

Advances in statistical methodology and machine learning methods offer new opportunities for innovative analyses of HDD, but at the same time require a deeper understanding of some fundamental statistical concepts. Topic group TG9 "High-dimensional data" of the STRATOS (STRengthening Analytical Thinking for Observational Studies) initiative provides guidance for the analysis of observational studies, addressing particular statistical challenges and opportunities for the analysis of studies involving HDD. In this overview, we discuss key aspects of HDD analysis to provide a gentle introduction for non-statisticians and for classically trained statisticians with little experience specific to HDD.

RESULTS

The paper is organized with respect to subtopics that are most relevant for the analysis of HDD, in particular initial data analysis, exploratory data analysis, multiple testing, and prediction. For each subtopic, main analytical goals in HDD settings are outlined. For each of these goals, basic explanations for some commonly used analysis methods are provided. Situations are identified where traditional statistical methods cannot, or should not, be used in the HDD setting, or where adequate analytic tools are still lacking. Many key references are provided.

CONCLUSIONS

This review aims to provide a solid statistical foundation for researchers, including statisticians and non-statisticians, who are new to research with HDD or simply want to better evaluate and understand the results of HDD analyses.

Effects of influential points and sample size on the selection and replicability of multivariable fractional polynomial models

BACKGROUND

The multivariable fractional polynomial (MFP) approach combines variable selection using backward elimination with a function selection procedure (FSP) for fractional polynomial (FP) functions. It is a relatively simple approach which can be easily understood without advanced training in statistical modeling. For continuous variables, a closed test procedure is used to decide between no effect, linear, FP1, or FP2 functions. Influential points (IPs) and small sample sizes can both have a strong impact on a selected function and MFP model.

METHODS

We used simulated data with six continuous and four categorical predictors to illustrate approaches which can help to identify IPs with an influence on function selection and the MFP model. Approaches use leave-one or two-out and two related techniques for a multivariable assessment. In eight subsamples, we also investigated the effects of sample size and model replicability, the latter by using three non-overlapping subsamples with the same sample size. For better illustration, a structured profile was used to provide an overview of all analyses conducted.

RESULTS

The results showed that one or more IPs can drive the functions and models selected. In addition, with a small sample size, MFP was not able to detect some non-linear functions and the selected model differed substantially from the true underlying model. However, when the sample size was relatively large and regression diagnostics were carefully conducted, MFP selected functions or models that were similar to the underlying true model.

CONCLUSIONS

For smaller sample size, IPs and low power are important reasons that the MFP approach may not be able to identify underlying functional relationships for continuous variables and selected models might differ substantially from the true model. However, for larger sample sizes, a carefully conducted MFP analysis is often a suitable way to select a multivariable regression model which includes continuous variables. In such a case, MFP can be the preferred approach to derive a multivariable descriptive model.

Regularization approaches in clinical biostatistics: A review of methods and their applications

A range of regularization approaches have been proposed in the data sciences to overcome overfitting, to exploit sparsity or to improve prediction. Using a broad definition of regularization, namely controlling model complexity by adding information in order to solve ill-posed problems or to prevent overfitting, we review a range of approaches within this framework including penalization, early stopping, ensembling and model averaging. Aspects of their practical implementation are discussed including available R-packages and examples are provided. To assess the extent to which these approaches are used in medicine, we conducted a review of three general medical journals. It revealed that regularization approaches are rarely applied in practical clinical applications, with the exception of random effects models. Hence, we suggest a more frequent use of regularization approaches in medical research. In situations where also other approaches work well, the only downside of the regularization approaches is increased complexity in the conduct of the analyses which can pose challenges in terms of computational resources and expertise on the side of the data analyst. In our view, both can and should be overcome by investments in appropriate computing facilities and educational resources.

Assessing Performance and Clinical Usefulness in Prediction Models With Survival Outcomes: Practical Guidance for Cox Proportional Hazards Models

Risk prediction models need thorough validation to assess their performance. Validation of models for survival outcomes poses challenges due to the censoring of observations and the varying time horizon at which predictions can be made. This article describes measures to evaluate predictions and the potential improvement in decision making from survival models based on Cox proportional hazards regression.

As a motivating case study, the authors consider the prediction of the composite outcome of recurrence or death (the "event") in patients with breast cancer after surgery. They developed a simple Cox regression model with 3 predictors, as in the Nottingham Prognostic Index, in 2982 women (1275 events over 5 years of follow-up) and externally validated this model in 686 women (285 events over 5 years). Improvement in performance was assessed after the addition of progesterone receptor as a prognostic biomarker.

The model predictions can be evaluated across the full range of observed follow-up times or for the event occurring by the end of a fixed time horizon of interest. The authors first discuss recommended statistical measures that evaluate model performance in terms of discrimination, calibration, or overall performance. Further, they evaluate the potential clinical utility of the model to support clinical decision making according to a net benefit measure. They provide SAS and R code to illustrate internal and external validation.

The authors recommend the proposed set of performance measures for transparent reporting of the validity of predictions from survival models.

Centre for Statistical and Methodological Excellence (CESAME): A Consortium Initiative for Improving Methodology in Randomised Clinical Trials

When conducting randomised clinical trials, the choice of methodology and statistical analyses will influence the results. If the planned methodology is not of optimal quality and predefined in detail, there is a risk of biased trial results and interpretation. Even though clinical trial methodology is already at a very high standard, there are many trials that deliver biased results due to the implementation of inadequate methodology, poor data quality and erroneous or biased analyses. To increase the internal and external validity of randomised clinical trial results, several international institutions within clinical intervention research have formed The Centre for Statistical and Methodological Excellence (CESAME). Based on international consensus, the CESAME initiative will develop recommendations for the proper methodological planning, conduct and analysis of clinical intervention research. CESAME aims to increase the validity of randomised clinical trial results which will ultimately benefit patients worldwide across medical specialities. The work of CESAME will be performed within 3 closely interconnected pillars: (1) planning randomised clinical trials; (2) conducting randomised clinical trials; and (3) analysing randomised clinical trials.

Comparison of variable selection procedures and investigation of the role of shrinkage in linear regression-protocol of a simulation study in low-dimensional data

In low-dimensional data and within the framework of a classical linear regression model, we intend to compare variable selection methods and investigate the role of shrinkage of regression estimates in a simulation study. Our primary aim is to build descriptive models that capture the data structure parsimoniously, while our secondary aim is to derive a prediction model. Simulation studies are an important tool in statistical methodology research if they are well designed, executed, and reported. However, bias in favor of an “own” preferred method is prevalent in most simulation studies in which a new method is proposed and compared with existing methods. To overcome such bias, neutral comparison studies, which disregard the superiority or inferiority of a particular method, have been proposed. In this paper, we designed a simulation study with key principles of neutral comparison studies in mind, though certain unintentional biases cannot be ruled out. To improve the design and reporting of a simulation study, we followed the recently proposed ADEMP structure, which entails defining the aims (A), data-generating mechanisms (D), estimand/target of analysis (E), methods (M), and performance measures (P). To ensure the reproducibility of results, we published the protocol before conducting the study. In addition, we presented earlier versions of the design to several experts whose feedback influenced certain aspects of the design. We will compare popular penalized regression methods (lasso, adaptive lasso, relaxed lasso, and nonnegative garrote) that combine variable selection and shrinkage with classical variable selection methods (best subset selection and backward elimination) with and without post-estimation shrinkage of parameter estimates.

Modern Learning from Big Data in Critical Care: Primum Non Nocere

Large and complex data sets are increasingly available for research in critical care. To analyze these data, researchers use techniques commonly referred to as statistical learning or machine learning (ML). The latter is known for large successes in the field of diagnostics, for example, by identification of radiological anomalies. In other research areas, such as clustering and prediction studies, there is more discussion regarding the benefit and efficiency of ML techniques compared with statistical learning. In this viewpoint, we aim to explain commonly used statistical learning and ML techniques and provide guidance for responsible use in the case of clustering and prediction questions in critical care. Clustering studies have been increasingly popular in critical care research, aiming to inform how patients can be characterized, classified, or treated differently. An important challenge for clustering studies is to ensure and assess generalizability. This limits the application of findings in these studies toward individual patients. In the case of predictive questions, there is much discussion as to what algorithm should be used to most accurately predict outcome. Aspects that determine usefulness of ML, compared with statistical techniques, include the volume of the data, the dimensionality of the preferred model, and the extent of missing data. There are areas in which modern ML methods may be preferred. However, efforts should be made to implement statistical frameworks (e.g., for dealing with missing data or measurement error, both omnipresent in clinical data) in ML methods. To conclude, there are important opportunities but also pitfalls to consider when performing clustering or predictive studies with ML techniques. We advocate careful valuation of new data-driven findings. More interaction is needed between the engineer mindset of experts in ML methods, the insight in bias of epidemiologists, and the probabilistic thinking of statisticians to extract as much information and knowledge from data as possible, while avoiding harm.

How to write statistical analysis section in medical research

Reporting of statistical analysis is essential in any clinical and translational research study. However, medical research studies sometimes report statistical analysis that is either inappropriate or insufficient to attest to the accuracy and validity of findings and conclusions. Published works involving inaccurate statistical analyses and insufficient reporting influence the conduct of future scientific studies, including meta-analyses and medical decisions. Although the biostatistical practice has been improved over the years due to the involvement of statistical reviewers and collaborators in research studies, there remain areas of improvement for transparent reporting of the statistical analysis section in a study. Evidence-based biostatistics practice throughout the research is useful for generating reliable data and translating meaningful data to meaningful interpretation and decisions in medical research. Most existing research reporting guidelines do not provide guidance for reporting methods in the statistical analysis section that helps in evaluating the quality of findings and data interpretation. In this report, we highlight the global and critical steps to be reported in the statistical analysis of grants and research articles. We provide clarity and the importance of understanding study objective types, data generation process, effect size use, evidence-based biostatistical methods use, and development of statistical models through several thematic frameworks. We also provide published examples of adherence or non-adherence to methodological standards related to each step in the statistical analysis and their implications. We believe the suggestions provided in this report can have far-reaching implications for education and strengthening the quality of statistical reporting and biostatistical practice in medical research.

Structured reporting to improve transparency of analyses in prognostic marker studies

BACKGROUND

Factors contributing to the lack of understanding of research studies include poor reporting practices, such as selective reporting of statistically significant findings or insufficient methodological details. Systematic reviews have shown that prognostic factor studies continue to be poorly reported, even for important aspects, such as the effective sample size. The REMARK reporting guidelines support researchers in reporting key aspects of tumor marker prognostic studies. The REMARK profile was proposed to augment these guidelines to aid in structured reporting with an emphasis on including all aspects of analyses conducted.

METHODS

A systematic search of prognostic factor studies was conducted, and fifteen studies published in 2015 were selected, three from each of five oncology journals. A paper was eligible for selection if it included survival outcomes and multivariable models were used in the statistical analyses. For each study, we summarized the key information in a REMARK profile consisting of details about the patient population with available variables and follow-up data, and a list of all analyses conducted.

RESULTS

Structured profiles allow an easy assessment if reporting of a study only has weaknesses or if it is poor because many relevant details are missing. Studies had incomplete reporting of exclusion of patients, missing information about the number of events, or lacked details about statistical analyses, e.g., subgroup analyses in small populations without any information about the number of events. Profiles exhibit severe weaknesses in the reporting of more than 50% of the studies. The quality of analyses was not assessed, but some profiles exhibit several deficits at a glance.

CONCLUSIONS

A substantial part of prognostic factor studies is poorly reported and analyzed, with severe consequences for related systematic reviews and meta-analyses. We consider inadequate reporting of single studies as one of the most important reasons that the clinical relevance of most markers is still unclear after years of research and dozens of publications. We conclude that structured reporting is an important step to improve the quality of prognostic marker research and discuss its role in the context of selective reporting, meta-analysis, study registration, predefined statistical analysis plans, and improvement of marker research.

Statistical Analysis in the German National Cohort (NAKO) - Specific Aspects and General Recommendations

The German National Cohort (NAKO) is an ongoing, prospective multicenter cohort study, which started recruitment in 2014 and includes more than 205,000 women and men aged 19-74 years. The study data will be available to the global research community for analyses. Although the ultimate decision about the analytic methods will be made by the respective investigator, in this paper we provide the basis for a harmonized approach to the statistical analyses in the NAKO. We discuss specific aspects of the study (e.g., data collection, weighting to account for the sampling design), but also give general recommendations which may apply to other large cohort studies as well.

Using linear and natural cubic splines, SITAR, and latent trajectory models to characterise nonlinear longitudinal growth trajectories in cohort studies

BACKGROUND

Longitudinal data analysis can improve our understanding of the influences on health trajectories across the life-course. There are a variety of statistical models which can be used, and their fitting and interpretation can be complex, particularly where there is a nonlinear trajectory. Our aim was to provide an accessible guide along with applied examples to using four sophisticated modelling procedures for describing nonlinear growth trajectories.

METHODS

This expository paper provides an illustrative guide to summarising nonlinear growth trajectories for repeatedly measured continuous outcomes using (i) linear spline and (ii) natural cubic spline linear mixed-effects (LME) models, (iii) Super Imposition by Translation and Rotation (SITAR) nonlinear mixed effects models, and (iv) latent trajectory models. The underlying model for each approach, their similarities and differences, and their advantages and disadvantages are described. Their application and correct interpretation of their results is illustrated by analysing repeated bone mass measures to characterise bone growth patterns and their sex differences in three cohort studies from the UK, USA, and Canada comprising 8500 individuals and 37,000 measurements from ages 5-40 years. Recommendations for choosing a modelling approach are provided along with a discussion and signposting on further modelling extensions for analysing trajectory exposures and outcomes, and multiple cohorts.

RESULTS

Linear and natural cubic spline LME models and SITAR provided similar summary of the mean bone growth trajectory and growth velocity, and the sex differences in growth patterns. Growth velocity (in grams/year) peaked during adolescence, and peaked earlier in females than males e.g., mean age at peak bone mineral content accrual from multicohort SITAR models was 12.2 years in females and 13.9 years in males. Latent trajectory models (with trajectory shapes estimated using a natural cubic spline) identified up to four subgroups of individuals with distinct trajectories throughout adolescence.

CONCLUSIONS

LME models with linear and natural cubic splines, SITAR, and latent trajectory models are useful for describing nonlinear growth trajectories, and these methods can be adapted for other complex traits. Choice of method depends on the research aims, complexity of the trajectory, and available data. Scripts and synthetic datasets are provided for readers to replicate trajectory modelling and visualisation using the R statistical computing software.

Review of guidance papers on regression modeling in statistical series of medical journals

Although regression models play a central role in the analysis of medical research projects, there still exist many misconceptions on various aspects of modeling leading to faulty analyses. Indeed, the rapidly developing statistical methodology and its recent advances in regression modeling do not seem to be adequately reflected in many medical publications. This problem of knowledge transfer from statistical research to application was identified by some medical journals, which have published series of statistical tutorials and (shorter) papers mainly addressing medical researchers. The aim of this review was to assess the current level of knowledge with regard to regression modeling contained in such statistical papers. We searched for target series by a request to international statistical experts. We identified 23 series including 57 topic-relevant articles. Within each article, two independent raters analyzed the content by investigating 44 predefined aspects on regression modeling. We assessed to what extent the aspects were explained and if examples, software advices, and recommendations for or against specific methods were given. Most series (21/23) included at least one article on multivariable regression. Logistic regression was the most frequently described regression type (19/23), followed by linear regression (18/23), Cox regression and survival models (12/23) and Poisson regression (3/23). Most general aspects on regression modeling, e.g. model assumptions, reporting and interpretation of regression results, were covered. We did not find many misconceptions or misleading recommendations, but we identified relevant gaps, in particular with respect to addressing nonlinear effects of continuous predictors, model specification and variable selection. Specific recommendations on software were rarely given. Statistical guidance should be developed for nonlinear effects, model specification and variable selection to better support medical researchers who perform or interpret regression analyses.

A systematic review of methodological approaches for evaluating real-world effectiveness of COVID-19 vaccines: Advising resource-constrained settings

Real-world effectiveness studies are important for monitoring performance of COVID-19 vaccination programmes and informing COVID-19 prevention and control policies. We aimed to synthesise methodological approaches used in COVID-19 vaccine effectiveness studies, in order to evaluate which approaches are most appropriate to implement in low- and middle-income countries (LMICs). For this rapid systematic review, we searched PubMed and Scopus for articles published from inception to July 7, 2021, without language restrictions. We included any type of peer-reviewed observational study measuring COVID-19 vaccine effectiveness, for any population. We excluded randomised control trials and modelling studies. All data used in the analysis were extracted from included papers. We used a standardised data extraction form, modified from STrengthening the Reporting of OBservational studies in Epidemiology (STROBE). Study quality was assessed using the REal Life EVidence AssessmeNt Tool (RELEVANT) tool. This study is registered with PROSPERO, CRD42021264658. Our search identified 3,327 studies, of which 42 were eligible for analysis. Most studies (97.5%) were conducted in high-income countries and the majority assessed mRNA vaccines (78% mRNA only, 17% mRNA and viral vector, 2.5% viral vector, 2.5% inactivated vaccine). Thirty-five of the studies (83%) used a cohort study design. Across studies, short follow-up time and limited assessment and mitigation of potential confounders, including previous SARS-CoV-2 infection and healthcare seeking behaviour, were major limitations. This review summarises methodological approaches for evaluating real-world effectiveness of COVID-19 vaccines and highlights the lack of such studies in LMICs, as well as the importance of context-specific vaccine effectiveness data. Further research in LMICs will refine guidance for conducting real-world COVID-19 vaccine effectiveness studies in resource-constrained settings.

Methodological guidelines to estimate population-based health indicators using linked data and/or machine learning techniques

BACKGROUND

The capacity to use data linkage and artificial intelligence to estimate and predict health indicators varies across European countries. However, the estimation of health indicators from linked administrative data is challenging due to several reasons such as variability in data sources and data collection methods resulting in reduced interoperability at various levels and timeliness, availability of a large number of variables, lack of skills and capacity to link and analyze big data. The main objective of this study is to develop the methodological guidelines calculating population-based health indicators to guide European countries using linked data and/or machine learning (ML) techniques with new methods.

METHOD

We have performed the following step-wise approach systematically to develop the methodological guidelines: i. Scientific literature review, ii. Identification of inspiring examples from European countries, and iii. Developing the checklist of guidelines contents.

RESULTS

We have developed the methodological guidelines, which provide a systematic approach for studies using linked data and/or ML-techniques to produce population-based health indicators. These guidelines include a detailed checklist of the following items: rationale and objective of the study (i.e., research question), study design, linked data sources, study population/sample size, study outcomes, data preparation, data analysis (i.e., statistical techniques, sensitivity analysis and potential issues during data analysis) and study limitations.

CONCLUSIONS

This is the first study to develop the methodological guidelines for studies focused on population health using linked data and/or machine learning techniques. These guidelines would support researchers to adopt and develop a systematic approach for high-quality research methods. There is a need for high-quality research methodologies using more linked data and ML-techniques to develop a structured cross-disciplinary approach for improving the population health information and thereby the population health.

Can we use existing guidance to support the development of robust real-world evidence for health technology assessment/payer decision-making?

Advances in the digitization of health systems and expedited regulatory approvals of innovative treatments have led to increased potential for the use of real-world data (RWD) to generate real-world evidence (RWE) to complement evidence from clinical trials. However, health technology assessment (HTA) bodies and payers have concerns about the ability to generate RWE of sufficient quality to be pivotal evidence of relative treatment effectiveness. Consequently, there is a growing need for HTA bodies and payers to develop guidance for the industry and other stakeholders about the use of RWD/RWE to support access, reimbursement, and pricing. We therefore sought to (i) understand barriers to the use of RWD/RWE by HTA bodies and payers; (ii) review potential solutions in the form of published guidance; and (iii) review findings with selected HTA/payer bodies. Four themes considered key to shaping the generation of robust RWE for HTA bodies and payers were identified as: (i) data (availability, governance, and quality); (ii) methodology (design and analytics); (iii) trust (transparency and reproducibility); and (iv) policy and partnerships. A range of guidance documents were found from trusted sources that could address these themes. These were discussed with HTA experts. This commentary summarizes the potential guidance solutions available to help resolve issues faced by HTA decision-makers in the adoption of RWD/RWE. It shows that there is alignment among stakeholders about the areas that need improvement in the development of RWE and that the key priority to move forward is better collaboration to make data usable for multiple purposes.

The importance of the design of observational studies in comparative effectiveness research: Lessons from the GARFIELD-AF and ORBIT-AF registries

Randomized controlled trials (RCTs) are considered the gold standard for estimating the effectiveness of a treatment. However, in many instances they are impractical to conduct because of time limitations, cost restrictions, or ethical reasons. As a consequence, non-randomized observational studies have an important role in comparative effectiveness and safety research since they can address issues that would not be possible using conventional RCT methodology. Observational studies can be strategically designed to reduce the risk of potential sources of bias by emulating the design principles of an equivalent but ideal randomized trial - the target trial - that would answer the research question of interest. In this article, we review some of the necessary components of observational studies required for valid causal inference within the framework of target trial emulation, so as to avoid common methodological pitfalls of study design. We discuss the assumptions of consistency, time-zero specification, exchangeability and positivity. To illustrate these concepts in a context where existing knowledge is well-established through clinical trials, we evaluate and compare the treatment effects of vitamin K antagonists (VKA) against no VKA (No VKA) on the treatment of atrial fibrillation from two real-world observational studies, namely the GARFIELD-AF and ORBIT-AF registries. Results are compared with those of published RCTs.

Flexible extension of the accelerated failure time model to account for nonlinear and time-dependent effects of covariates on the hazard

The accelerated failure time model is an alternative to the Cox proportional hazards model in survival analysis. However, conclusions regarding the associations of prognostic factors with event times are valid only if the underlying modeling assumptions are met. In contrast to several flexible methods for relaxing the proportional hazards and linearity assumptions in the Cox model, formal investigation of the constant-over-time time ratio and linearity assumptions in the accelerated failure time model has been limited. Yet, in practice, prognostic factors may have time-dependent and/or nonlinear effects. Furthermore, parametric accelerated failure time models require correct specification of the baseline hazard function, which is treated as a nuisance parameter in the Cox proportional hazards model, and is rarely known in practice. To address these challenges, we propose a flexible extension of the accelerated failure time model where unpenalized regression B-splines are used to model (i) the baseline hazard function of arbitrary shape, (ii) the time-dependent covariate effects on the hazard, and (iii) nonlinear effects for continuous covariates. Simulations evaluate the accuracy of the time-dependent and/or nonlinear estimates, and of the resulting survival functions, in multivariable settings. The proposed flexible extension of the accelerated failure time model is applied to re-assess the effects of prognostic factors on mortality after septic shock.

A novel tool to assess the quality of RWE to guide the management of retinal disease

Despite the growing importance of real-world evidence (RWE) for guiding clinical decisions in retinal disease, there is currently no widely used guidance available for assessing the quality and relevance of RWE studies in ophthalmology. This paper summarizes the development of a user-friendly tool that facilitates assessment of the quality of available RWE for neovascular age-related macular degeneration (nAMD), diabetic macular oedema (DME) and retinal vein occlusion (RVO). A literature search was conducted to identify tools developed to assess the quality of RWE, in order to identify the most appropriate framework on which to base this tool. The Good Research for Comparative Effectiveness (GRACE) guidelines was chosen for this purpose as it is designed to assess the quality of observational studies and has been extensively validated, including demonstration of strong sensitivity and specificity. The GRACE guidelines were adapted to develop a straightforward tabular tool that allows simple assessment and comparison of the quality of published evidence in retinal disease for researchers and physicians alike, and includes guidance on treatment details, outcome measures, study population, and controlling for bias. The newly developed tool provides a simple method to support assessment of the strength of evidence and certainty of conclusions drawn from RWE in retinal disease, to ensure clinical decision-making is influenced by the highest quality evidence.

Quantifying the influence of bias in reproductive and perinatal epidemiology through simulation

PURPOSE

The application of simulated data in epidemiological studies enables the illustration and quantification of the magnitude of various types of bias commonly found in observational studies. This was a review of the application of simulation methods to the quantification of bias in reproductive and perinatal epidemiology and an assessment of value gained.

METHODS

A search of published studies available in English was conducted in August 2020 using PubMed, Medline, Embase, CINAHL, and Scopus. A gray literature search of Google and Google Scholar, and a hand search using the reference lists of included studies was undertaken.

RESULTS

Thirty-nine papers were included in this study, covering information (n = 14), selection (n = 14), confounding (n = 9), protection (n = 1), and attenuation bias (n = 1). The methods of simulating data and reporting of results varied, with more recent studies including causal diagrams. Few studies included code for replication.

CONCLUSIONS

Although there has been an increasing application of simulation in reproductive and perinatal epidemiology since 2015, overall this remains an underexplored area. Further efforts are required to increase knowledge of how the application of simulation can quantify the influence of bias, including improved design, analysis and reporting. This will improve causal interpretation in reproductive and perinatal studies.

1,3-Butadiene, styrene and selected outcomes among synthetic rubber polymer workers: Updated exposure-response analyses

OBJECTIVE

- To evaluate exposure-response relationships between 1,3-butadiene and styrene and selected diseases among synthetic rubber polymer workers.

METHODS

- 21,087 workers (16,579 men; 4508 women) were followed from 1943 through 2009 to determine mortality outcomes. Cox regression models estimated rate ratios (RRs) and 95% confidence intervals (CIs) by quartile of cumulative exposure to butadiene or styrene and exposure-response trends for cancers of the bladder, lung, kidney, esophagus and pancreas, and for all nonmalignant respiratory disease (NMRD), chronic obstructive pulmonary disease (COPD) and pneumonia.

RESULTS

- Bladder cancer RRs were 2.13 (95% CI = 1.03 to 4.41) and 1.64 (95% CI = 0.76 to 3.54) in the highest quartiles of cumulative exposure to butadiene and styrene, respectively, and exposure-response trends were positive for both monomers (butadiene, trend p = 0.001; styrene, trend p = 0.004). Further analyses indicated that the exposure-response effect of each monomer on bladder cancer was demonstrated clearly only in the subgroup with high cumulative exposure (at or above the median) to the other monomer. Lung cancer was not associated with either monomer among men. Among women, lung cancer RRs were above 1.0 in each quartile of cumulative exposure to each monomer, but exposure-response was not seen for either monomer. Male workers had COPD RRs slightly above 1.0 in each quartile of cumulative exposure to each monomer, but there was no evidence of exposure-response among the exposed. Monomer exposure was not consistently associated with COPD in women or with the other cancer outcomes.

CONCLUSIONS

- This study found a positive exposure-response relationship between monomer exposures and bladder cancer. The independent effects of butadiene and styrene on this cancer could not be delineated. In some analyses, monomer exposure was associated with lung cancer in women and with COPD in men, but inconsistent exposure-response trends and divergent results by sex do not support a causal interpretation of the isolated positive associations.

On the optimistic performance evaluation of newly introduced bioinformatic methods

Most research articles presenting new data analysis methods claim that "the new method performs better than existing methods," but the veracity of such statements is questionable. Our manuscript discusses and illustrates consequences of the optimistic bias occurring during the evaluation of novel data analysis methods, that is, all biases resulting from, for example, selection of datasets or competing methods, better ability to fix bugs in a preferred method, and selective reporting of method variants. We quantitatively investigate this bias using an example from epigenetic analysis: normalization methods for data generated by the Illumina HumanMethylation450K BeadChip microarray.

The multiplicity of analysis strategies jeopardizes replicability: lessons learned across disciplines

For a given research question, there are usually a large variety of possible analysis strategies acceptable according to the scientific standards of the field, and there are concerns that this multiplicity of analysis strategies plays an important role in the non-replicability of research findings. Here, we define a general framework on common sources of uncertainty arising in computational analyses that lead to this multiplicity, and apply this framework within an overview of approaches proposed across disciplines to address the issue. Armed with this framework, and a set of recommendations derived therefrom, researchers will be able to recognize strategies applicable to their field and use them to generate findings more likely to be replicated in future studies, ultimately improving the credibility of the scientific process.

Facilitating harmonized data quality assessments. A data quality framework for observational health research data collections with software implementations in R

Background

No standards exist for the handling and reporting of data quality in health research. This work introduces a data quality framework for observational health research data collections with supporting software implementations to facilitate harmonized data quality assessments.

Methods

Developments were guided by the evaluation of an existing data quality framework and literature reviews. Functions for the computation of data quality indicators were written in R. The concept and implementations are illustrated based on data from the population-based Study of Health in Pomerania (SHIP).

Results

The data quality framework comprises 34 data quality indicators. These target four aspects of data quality: compliance with pre-specified structural and technical requirements (integrity); presence of data values (completeness); inadmissible or uncertain data values and contradictions (consistency); unexpected distributions and associations (accuracy). R functions calculate data quality metrics based on the provided study data and metadata and R Markdown reports are generated. Guidance on the concept and tools is available through a dedicated website.

Conclusions

The presented data quality framework is the first of its kind for observational health research data collections that links a formal concept to implementations in R. The framework and tools facilitate harmonized data quality assessments in pursue of transparent and reproducible research. Application scenarios comprise data quality monitoring while a study is carried out as well as performing an initial data analysis before starting substantive scientific analyses but the developments are also of relevance beyond research.

Doug Altman: Driving critical appraisal and improvements in the quality of methodological and medical research

Doug Altman was a visionary leader and one of the most influential medical statisticians of the last 40 years. Based on a presentation in the "Invited session in memory of Doug Altman" at the 40th Annual Conference of the International Society for Clinical Biostatistics (ISCB) in Leuven, Belgium and our long-standing collaborations with Doug, we discuss his contributions to regression modeling, reporting, prognosis research, as well as some more general issues while acknowledging that we cannot cover the whole spectrum of Doug's considerable methodological output. His statement "To maximize the benefit to society, you need to not just do research but do it well" should be a driver for all researchers. To improve current and future research, we aim to summarize Doug's messages for these three topics.

Analysis of time-to-event for observational studies: Guidance to the use of intensity models

This paper provides guidance for researchers with some mathematical background on the conduct of time-to-event analysis in observational studies based on intensity (hazard) models. Discussions of basic concepts like time axis, event definition and censoring are given. Hazard models are introduced, with special emphasis on the Cox proportional hazards regression model. We provide check lists that may be useful both when fitting the model and assessing its goodness of fit and when interpreting the results. Special attention is paid to how to avoid problems with immortal time bias by introducing time-dependent covariates. We discuss prediction based on hazard models and difficulties when attempting to draw proper causal conclusions from such models. Finally, we present a series of examples where the methods and check lists are exemplified. Computational details and implementation using the freely available R software are documented in Supplementary Material. The paper was prepared as part of the STRATOS initiative.

Formulating causal questions and principled statistical answers

Although review papers on causal inference methods are now available, there is a lack of introductory overviews on what they can render and on the guiding criteria for choosing one particular method. This tutorial gives an overview in situations where an exposure of interest is set at a chosen baseline ("point exposure") and the target outcome arises at a later time point. We first phrase relevant causal questions and make a case for being specific about the possible exposure levels involved and the populations for which the question is relevant. Using the potential outcomes framework, we describe principled definitions of causal effects and of estimation approaches classified according to whether they invoke the no unmeasured confounding assumption (including outcome regression and propensity score-based methods) or an instrumental variable with added assumptions. We mainly focus on continuous outcomes and causal average treatment effects. We discuss interpretation, challenges, and potential pitfalls and illustrate application using a "simulation learner," that mimics the effect of various breastfeeding interventions on a child's later development. This involves a typical simulation component with generated exposure, covariate, and outcome data inspired by a randomized intervention study. The simulation learner further generates various (linked) exposure types with a set of possible values per observation unit, from which observed as well as potential outcome data are generated. It thus provides true values of several causal effects. R code for data generation and analysis is available on www.ofcaus.org, where SAS and Stata code for analysis is also provided.

Systematic review of education and practical guidance on regression modeling for medical researchers who lack a strong statistical background: Study protocol

In the last decades, statistical methodology has developed rapidly, in particular in the field of regression modeling. Multivariable regression models are applied in almost all medical research projects. Therefore, the potential impact of statistical misconceptions within this field can be enormous Indeed, the current theoretical statistical knowledge is not always adequately transferred to the current practice in medical statistics. Some medical journals have identified this problem and published isolated statistical articles and even whole series thereof. In this systematic review, we aim to assess the current level of education on regression modeling that is provided to medical researchers via series of statistical articles published in medical journals. The present manuscript is a protocol for a systematic review that aims to assess which aspects of regression modeling are covered by statistical series published in medical journals that intend to train and guide applied medical researchers with limited statistical knowledge. Statistical paper series cannot easily be summarized and identified by common keywords in an electronic search engine like Scopus. We therefore identified series by a systematic request to statistical experts who are part or related to the STRATOS Initiative (STRengthening Analytical Thinking for Observational Studies). Within each identified article, two raters will independently check the content of the articles with respect to a predefined list of key aspects related to regression modeling. The content analysis of the topic-relevant articles will be performed using a predefined report form to assess the content as objectively as possible. Any disputes will be resolved by a third reviewer. Summary analyses will identify potential methodological gaps and misconceptions that may have an important impact on the quality of analyses in medical research. This review will thus provide a basis for future guidance papers and tutorials in the field of regression modeling which will enable medical researchers 1) to interpret publications in a correct way, 2) to perform basic statistical analyses in a correct way and 3) to identify situations when the help of a statistical expert is required.

Introduction to statistical simulations in health research

In health research, statistical methods are frequently used to address a wide variety of research questions. For almost every analytical challenge, different methods are available. But how do we choose between different methods and how do we judge whether the chosen method is appropriate for our specific study? Like in any science, in statistics, experiments can be run to find out which methods should be used under which circumstances. The main objective of this paper is to demonstrate that simulation studies, that is, experiments investigating synthetic data with known properties, are an invaluable tool for addressing these questions. We aim to provide a first introduction to simulation studies for data analysts or, more generally, for researchers involved at different levels in the analyses of health data, who (1) may rely on simulation studies published in statistical literature to choose their statistical methods and who, thus, need to understand the criteria of assessing the validity and relevance of simulation results and their interpretation; and/or (2) need to understand the basic principles of designing statistical simulations in order to efficiently collaborate with more experienced colleagues or start learning to conduct their own simulations. We illustrate the implementation of a simulation study and the interpretation of its results through a simple example inspired by recent literature, which is completely reproducible using the R-script available from online supplemental file 1.

Research Objectives, Statistical Analyses and Interpretation of Health-Related Quality of Life Data in Glioma Research: A Systematic Review

BACKGROUND

Health-related quality of life (HRQoL) has become an increasingly important patient-reported outcome in glioma studies. Ideally, collected HRQoL data should be exploited to the full, with proper analytical methods. This systematic review aimed to provide an overview on how HRQoL data is currently evaluated in glioma studies, focusing on the research objectives and statistical analyses of HRQoL data.

METHODS

A systematic literature search in the databases PubMed, Embase, Web of Science and Cochrane was conducted up to 5 June 2020. Articles were selected based on predetermined inclusion criteria and information on study design, HRQoL instrument, HRQoL research objective and statistical methods were extracted.

RESULTS

A total of 170 articles describing 154 unique studies were eligible, in which 17 different HRQoL instruments were used. HRQoL was the primary outcome in 62% of the included articles, and 51% investigated ≥1 research question with respect to HRQoL, for which various analytical methods were used. In only 42% of the articles analyzing HRQoL results over time, the minimally clinical important difference was reported and interpreted. Eighty-six percent of articles reported HRQoL results at a group level only, and not at the individual patient level.

CONCLUSION

Currently, the assessment and analysis of HRQoL outcomes in glioma studies is highly variable. Opportunities to maximize information obtained with HRQoL data include appropriate and complementary analyses at both the group and individual level, comprehensive reporting of HRQoL results in separate articles or supplementary material, and adherence to existing guidelines about the assessment, analysis and reporting of patient-reported outcomes.

Evidence-based statistical analysis and methods in biomedical research (SAMBR) checklists according to design features

BACKGROUND

Statistical analysis according to design features and objectives is essential to ensure the validity and reliability of the study findings and conclusions in biomedical research. Heterogeneity in reporting study design elements and conducting statistical analyses is often observed for the same study design and study objective in medical literatures. Sometimes, researchers face a lot of predicaments using appropriate statistical approaches highlighted by methodologists for a specific study design either due to lack of accessibility or understanding of statistical methods or unavailability of checklists related to design and analysis in a concise format. The purpose of this review is to provide the checklist of statistical analysis and methods in biomedical research (SAMBR) to applied researchers.

RECENT FINDINGS

We initially identified the important steps of reporting design features that may influence the choice of statistical analysis in biomedical research and essential steps of data analysis of common studies. We subsequently searched for statistical approaches employed for each study design/study objective available in publications and other resources. Compilation of these steps produced SAMBR guidance document, which includes three parts. Applied researchers can use part (A) and part (B) of SAMBR to describe or evaluate research design features and quality of statistical analysis, respectively, in reviewing studies or designing protocols. Part (C) of SAMBR can be used to perform essential and preferred evidence-based data analysis specific to study design and objective.

CONCLUSIONS

We believe that the statistical methods checklists may improve reporting of research design, standardize methodological practices, and promote consistent application of statistical approaches, thus improving the quality of research studies. The checklists do not enforce the use of suggested statistical methods but rather highlight and encourage to conduct the best statistical practices. There is a need to develop an interactive web-based application of the checklists for users for its wide applications.

STRATOS guidance document on measurement error and misclassification of variables in observational epidemiology: Part 2-More complex methods of adjustment and advanced topics

We continue our review of issues related to measurement error and misclassification in epidemiology. We further describe methods of adjusting for biased estimation caused by measurement error in continuous covariates, covering likelihood methods, Bayesian methods, moment reconstruction, moment-adjusted imputation, and multiple imputation. We then describe which methods can also be used with misclassification of categorical covariates. Methods of adjusting estimation of distributions of continuous variables for measurement error are then reviewed. Illustrative examples are provided throughout these sections. We provide lists of available software for implementing these methods and also provide the code for implementing our examples in the Supporting Information. Next, we present several advanced topics, including data subject to both classical and Berkson error, modeling continuous exposures with measurement error, and categorical exposures with misclassification in the same model, variable selection when some of the variables are measured with error, adjusting analyses or design for error in an outcome variable, and categorizing continuous variables measured with error. Finally, we provide some advice for the often met situations where variables are known to be measured with substantial error, but there is only an external reference standard or partial (or no) information about the type or magnitude of the error.

State of the art in selection of variables and functional forms in multivariable analysis-outstanding issues

BACKGROUND

How to select variables and identify functional forms for continuous variables is a key concern when creating a multivariable model. Ad hoc 'traditional' approaches to variable selection have been in use for at least 50 years. Similarly, methods for determining functional forms for continuous variables were first suggested many years ago. More recently, many alternative approaches to address these two challenges have been proposed, but knowledge of their properties and meaningful comparisons between them are scarce. To define a state of the art and to provide evidence-supported guidance to researchers who have only a basic level of statistical knowledge, many outstanding issues in multivariable modelling remain. Our main aims are to identify and illustrate such gaps in the literature and present them at a moderate technical level to the wide community of practitioners, researchers and students of statistics.

METHODS

We briefly discuss general issues in building descriptive regression models, strategies for variable selection, different ways of choosing functional forms for continuous variables and methods for combining the selection of variables and functions. We discuss two examples, taken from the medical literature, to illustrate problems in the practice of modelling.

RESULTS

Our overview revealed that there is not yet enough evidence on which to base recommendations for the selection of variables and functional forms in multivariable analysis. Such evidence may come from comparisons between alternative methods. In particular, we highlight seven important topics that require further investigation and make suggestions for the direction of further research.

CONCLUSIONS

Selection of variables and of functional forms are important topics in multivariable analysis. To define a state of the art and to provide evidence-supported guidance to researchers who have only a basic level of statistical knowledge, further comparative research is required.

Nonlinear and time-dependent effects of sparsely measured continuous time-varying covariates in time-to-event analysis

Many flexible extensions of the Cox proportional hazards model incorporate time-dependent (TD) and/or nonlinear (NL) effects of time-invariant covariates. In contrast, little attention has been given to the assessment of such effects for continuous time-varying covariates (TVCs). We propose a flexible regression B-spline-based model for TD and NL effects of a TVC. To account for sparse TVC measurements, we added to this model the effect of time elapsed since last observation (TEL), which acts as an effect modifier. TD, NL, and TEL effects are estimated with the iterative alternative conditional estimation algorithm. Furthermore, a simulation extrapolation (SIMEX)-like procedure was adapted to correct the estimated effects for random measurement errors in the observed TVC values. In simulations, TD and NL estimates were unbiased if the TVC was measured with a high frequency. With sparse measurements, the strength of the effects was underestimated but the TEL estimate helped reduce the bias, whereas SIMEX helped further to correct for bias toward the null due to "white noise" measurement errors. We reassessed the effects of systolic blood pressure (SBP) and total cholesterol, measured at two-year intervals, on cardiovascular risks in women participating in the Framingham Heart Study. Accounting for TD effects of SBP, cholesterol and age, the NL effect of cholesterol, and the TEL effect of SBP improved substantially the model's fit to data. Flexible estimates yielded clinically important insights regarding the role of these risk factors. These results illustrate the advantages of flexible modeling of TVC effects.

Joint model robustness compared with the time-varying covariate Cox model to evaluate the association between a longitudinal marker and a time-to-event endpoint

BACKGROUND

The recent progress in medical research generates an increasing interest in the use of longitudinal biomarkers for characterizing the occurrence of an outcome. The present work is motivated by a study, where the objective was to explore the potential of the long pentraxin 3 (PTX3) as a prognostic marker of Acute Graft-versus-Host Disease (GvHD) after haematopoietic stem cell transplantation. Time-varying covariate Cox model was commonly used, despite its limiting assumptions that marker values are constant in time and measured without error. A joint model has been developed as a viable alternative; however, the approach is computationally intensive and requires additional strong assumptions, in which the impacts of their misspecification were not sufficiently studied.

METHODS

We conduct an extensive simulation to clarify relevant assumptions for the understanding of joint models and assessment of its robustness under key model misspecifications. Further, we characterize the extent of bias introduced by the limiting assumptions of the time-varying covariate Cox model and compare its performance with a joint model in various contexts. We then present results of the two approaches to evaluate the potential of PTX3 as a prognostic marker of GvHD after haematopoietic stem cell transplantation.

RESULTS

Overall, we illustrate that a joint model provides an unbiased estimate of the association between a longitudinal marker and the hazard of an event in the presence of measurement error, showing improvement over the time-varying Cox model. However, a joint model is severely biased when the baseline hazard or the shape of the longitudinal trajectories are misspecified. Both the Cox model and the joint model correctly specified indicated PTX3 as a potential prognostic marker of GvHD, with the joint model providing a higher hazard ratio estimate.

CONCLUSIONS

Joint models are beneficial to investigate the capability of the longitudinal marker to characterize time-to-event endpoint. However, the benefits are strictly linked to the correct specification of the longitudinal marker trajectory and the baseline hazard function, indicating a careful consideration of assumptions to avoid biased estimates.

Comparison of model-building strategies for excess hazard regression models in the context of cancer epidemiology

BACKGROUND

Large and complex population-based cancer data are becoming broadly available, thanks to purposeful linkage between cancer registry data and health electronic records. Aiming at understanding the explanatory power of factors on cancer survival, the modelling and selection of variables need to be understood and exploited properly for improving model-based estimates of cancer survival.

METHOD

We assess the performances of well-known model selection strategies developed by Royston and Sauerbrei and Wynant and Abrahamowicz that we adapt to the relative survival data setting and to test for interaction terms.

RESULTS

We apply these to all male patients diagnosed with lung cancer in England in 2012 (N = 15,688), and followed-up until 31/12/2015. We model the effects of age at diagnosis, tumour stage, deprivation, comorbidity and emergency presentation, as well as interactions between age and all of the above. Given the size of the dataset, all model selection strategies favoured virtually the same model, except for a non-linear effect of age at diagnosis selected by the backward-based selection strategies (versus a linear effect selected otherwise).

CONCLUSION

The results from extensive simulations evaluating varying model complexity and sample sizes provide guidelines on a model selection strategy in the context of excess hazard modelling.

A harmonized longitudinal biomarkers and cognition database for assessing the natural history of preclinical Alzheimer's disease from young adulthood and for designing prevention trials

INTRODUCTION

Large longitudinal biomarkers database focusing on middle age is needed for Alzheimer's disease (AD) prevention.

METHODS

Data for cerebrospinal fluid analytes, molecular imaging of cerebral fibrillar β-amyloid with positron emission tomography, magnetic resonance imaging-based brain structures, and clinical/cognitive outcomes were harmonized across eight AD biomarker studies. Statistical power was estimated.

RESULTS

The harmonized database included 7779 participants with clinical/cognitive data: 3542 were 18∼65 years at the baseline, 5865 had longitudinal cognitive data for a median of 4.7 years, 2473 participated in the cerebrospinal fluid studies (906 had longitudinal data), 2496 participated in the magnetic resonance imaging studies (1283 had longitudinal data), and 1498 participated in the positron emission tomography amyloid studies (849 had longitudinal data). The database provides adequate power for detecting early biomarker changes, and demonstrates the feasibility of AD prevention trials on middle-aged individuals.

DISCUSSION

The harmonized database is an optimum resource to design AD prevention trials decades before symptomatic onset.

Evaluating Flexible Modeling of Continuous Covariates in Inverse-Weighted Estimators

Correct specification of the exposure model is essential for unbiased estimation in marginal structural models with inverse-probability-of-treatment weights. However, although flexible modeling is commonplace when estimating effects of continuous covariates in outcome models, its use is less frequent in estimation of inverse probability weights. Using simulations, we assess the accuracy of the treatment effect estimates and covariate balance obtained with different exposure model specifications when the true relationship between a continuous, possibly time-varying covariate Lt and the logit of the probability of exposure is nonlinear. Specifically, we compare 4 approaches to modeling the effect of Lt when estimating inverse probability weights: a linear function, the covariate-balancing propensity score, and 2 easy-to-implement flexible methods that relax the assumption of linearity: cubic regression splines and fractional polynomials. Using data from 2 empirical studies, we compare linear exposure models with flexible exposure models to estimate the effect of sustained virological response to hepatitis C virus treatment on the progression of liver fibrosis. Our simulation results demonstrate that ignoring important nonlinear relationships when fitting the exposure model may provide poorer covariate balance and induce substantial bias in the estimated exposure-outcome associations. Analysts should routinely consider flexible modeling of continuous covariates when estimating inverse-probability-of-treatment weights.