Quantifying the Reliability and Replicability of Psychopathology Network Characteristics

Pairwise Markov random field networks-including Gaussian graphical models (GGMs) and Ising models-have become the "state-of-the-art" method for psychopathology network analyses. Recent research has focused on the reliability and replicability of these networks. In the present study, we compared the existing suite of methods for maximizing and quantifying the stability and consistency of PMRF networks (i.e., lasso regularization, plus the bootnet and NetworkComparisonTest packages in R) with a set of metrics for directly comparing the detailed network characteristics interpreted in the literature (e.g., the presence, absence, sign, and strength of each individual edge). We compared GGMs of depression and anxiety symptoms in two waves of data from an observational study (n = 403) and reanalyzed four posttraumatic stress disorder GGMs from a recent study of network replicability. Taken on face value, the existing suite of methods indicated that overall the network edges were stable, interpretable, and consistent between networks, but the direct metrics of replication indicated that this was not the case (e.g., 39-49% of the edges in each network were unreplicated across the pairwise comparisons). We discuss reasons for these apparently contradictory results (e.g., relying on global summary statistics versus examining the detailed characteristics interpreted in the literature) and conclude that the limited reliability of the detailed characteristics of networks observed here is likely to be common in practice, but overlooked by current methods. Poor replicability underpins our concern surrounding the use of these methods, given that generalizable conclusions are fundamental to the utility of their results.

Evidence that psychopathology symptom networks have limited replicability

Network analysis is quickly gaining popularity in psychopathology research as a method that aims to reveal causal relationships among individual symptoms. To date, 4 main types of psychopathology networks have been proposed: (a) association networks, (b) regularized concentration networks, (c) relative importance networks, and (d) directed acyclic graphs. The authors examined the replicability of these analyses based on symptoms of major depression and generalized anxiety between and within 2 highly similar epidemiological samples (i.e., the National Comorbidity Survey-Replication [n = 9282] and the National Survey of Mental Health and Wellbeing [n = 8841]). Although association networks were stable, the 3 other types of network analysis (i.e., the conditional independence networks) had poor replicability between and within methods and samples. The detailed aspects of the models-such as the estimation of specific edges and the centrality of individual nodes-were particularly unstable. For example, 44% of the symptoms were estimated as the "most influential" on at least 1 centrality index across the 6 conditional independence networks in the full samples, and only 13-21% of the edges were consistently estimated across these networks. One of the likely reasons for the instability of the networks is the predominance of measurement error in the assessment of individual symptoms. The authors discuss the implications of these findings for the growing field of psychopathology network research, and conclude that novel results originating from psychopathology networks should be held to higher standards of evidence before they are ready for dissemination or implementation in the field. (PsycINFO Database Record