Against the "one method fits all data sets" philosophy for comparison studies in methodological research

Multiple paths to rumination within a network analytical framework

Theories of rumination have proposed different psychological factors to place one at risk for repetitive negative thinking. A comprehensive empirical test that captures the most relevant contributors to rumination is lacking. Building on influential self-regulatory and metacognitive frameworks, we modeled how key constructs in this context relate to ruminative thinking. 498 participants completed online questionnaires including indicators of rumination, metacognition, promotion goal orientation, effortful control, and depression. We estimated regularized partial correlation networks to investigate unique associations between the different constructs and followed these analyses up with directed acyclic graphs to identify potential pathways towards rumination. Results demonstrated that: (1) both self-regulatory and metacognitive factors were directly linked to rumination, amongst these were (2) positive beliefs, negative beliefs about uncontrollability and harm, cognitive self-consciousness, depression, effortful control, perfectionism, and (lack of) cognitive confidence, and (3) we identified multiple directed pathways, suggesting three direct contributors to rumination while controlling for the influence of all other variables: diminished effortful control, positive beliefs, and cognitive self-consciousness. This study is the first to comprehensively assess metacognitive and self-regulatory frameworks of rumination in a data-driven manner. Our findings suggest that there are multiple pathways towards rumination, which should be incorporated in clinical case conceptualization of rumination and related disorders.

Explaining the optimistic performance evaluation of newly proposed methods: A cross-design validation experiment

The constant development of new data analysis methods in many fields of research is accompanied by an increasing awareness that these new methods often perform better in their introductory paper than in subsequent comparison studies conducted by other researchers. We attempt to explain this discrepancy by conducting a systematic experiment that we call "cross-design validation of methods". In the experiment, we select two methods designed for the same data analysis task, reproduce the results shown in each paper, and then reevaluate each method based on the study design (i.e., datasets, competing methods, and evaluation criteria) that was used to show the abilities of the other method. We conduct the experiment for two data analysis tasks, namely cancer subtyping using multiomic data and differential gene expression analysis. Three of the four methods included in the experiment indeed perform worse when they are evaluated on the new study design, which is mainly caused by the different datasets. Apart from illustrating the many degrees of freedom existing in the assessment of a method and their effect on its performance, our experiment suggests that the performance discrepancies between original and subsequent papers may not only be caused by the nonneutrality of the authors proposing the new method but also by differences regarding the level of expertise and field of application. Authors of new methods should thus focus not only on a transparent and extensive evaluation but also on comprehensive method documentation that enables the correct use of their methods in subsequent studies.

On the role of benchmarking data sets and simulations in method comparison studies

Method comparisons are essential to provide recommendations and guidance for applied researchers, who often have to choose from a plethora of available approaches. While many comparisons exist in the literature, these are often not neutral but favor a novel method. Apart from the choice of design and a proper reporting of the findings, there are different approaches concerning the underlying data for such method comparison studies. Most manuscripts on statistical methodology rely on simulation studies and provide a single real-world data set as an example to motivate and illustrate the methodology investigated. In the context of supervised learning, in contrast, methods are often evaluated using so-called benchmarking data sets, that is, real-world data that serve as gold standard in the community. Simulation studies, on the other hand, are much less common in this context. The aim of this paper is to investigate differences and similarities between these approaches, to discuss their advantages and disadvantages, and ultimately to develop new approaches to the evaluation of methods picking the best of both worlds. To this aim, we borrow ideas from different contexts such as mixed methods research and Clinical Scenario Evaluation.

Pitfalls and potentials in simulation studies: Questionable research practices in comparative simulation studies allow for spurious claims of superiority of any method

Comparative simulation studies are workhorse tools for benchmarking statistical methods. As with other empirical studies, the success of simulation studies hinges on the quality of their design, execution, and reporting. If not conducted carefully and transparently, their conclusions may be misleading. In this paper, we discuss various questionable research practices, which may impact the validity of simulation studies, some of which cannot be detected or prevented by the current publication process in statistics journals. To illustrate our point, we invent a novel prediction method with no expected performance gain and benchmark it in a preregistered comparative simulation study. We show how easy it is to make the method appear superior over well-established competitor methods if questionable research practices are employed. Finally, we provide concrete suggestions for researchers, reviewers, and other academic stakeholders for improving the methodological quality of comparative simulation studies, such as preregistering simulation protocols, incentivizing neutral simulation studies, and code and data sharing.