Using Observational Data to Calibrate Simulation Models

Identification of optimum scopes of environmental drivers for schistosome-transmitting Oncomelania hupensis using agent-based model in Dongting Lake Region, China

Oncomelania hupensis (O. hupensis), the sole intermediate host of Schistosoma japonicum, greatly influence the prevalence and distribution of schistosomiasis japonica. The distribution area of O. hupensis has remained extensive for numerous years. This study aimed to establish a valid agent-based model of snail density and further explore the environmental conditions suitable for snail breeding. A marshland with O. hupensis was selected as a study site in Dongting Lake Region, and snail surveys were monthly conducted from 2007 to 2016. Combined with the data from historical literature, an agent-based model of snail density was constructed in NetLogo 6.2.0 and validated with the collected survey data. BehaviorSpace was used to identify the optimal ranges of soil temperature, pH, soil water content, and vegetation coverage for snail growth, development and reproduction. An agent-based model of snail density was constructed and showed a strong agreement with the monthly average snail density from the field surveys. As soil temperature increased, the snail density initially rose before declining, reaching its peak at around 21°C. There were similar variation patterns for other environmental factors. The findings from the model suggested that the optimum ranges of soil temperature, pH, soil water content and vegetation coverage were 19°C to 23 °C, 6.4 to 7.6, 42% to 75%, and 70% to 93%, respectively. A valid agent-based model of snail density was constructed, providing more objective information about the optimum ranges of environmental factors for snail growth, development and reproduction.

Projected Life Expectancy for Adolescents With HIV in the US

Importance

Life expectancy is a key measure of overall population health. Life expectancy estimates for youth with HIV in the US are needed in the current HIV care and treatment context to guide health policies and resource allocation.

Objective

To compare life expectancy between 18-year-old youth with perinatally acquired HIV (PHIV), youth with nonperinatally acquired HIV (NPHIV), and youth without HIV.

Design, Setting, and Participants

Using a US-focused adolescent-specific Monte Carlo state-transition HIV model, we simulated individuals from age 18 years until death. We estimated probabilities of HIV treatment and care engagement, HIV progression, clinical events, and mortality from observational cohorts and clinical trials for model input parameters. The simulated individuals were 18-year-old race and ethnicity-matched youth with PHIV, youth with NPHIV, and youth without HIV; 47%, 85%, and 50% were assigned male sex at birth, respectively. Individuals were categorized by US Centers for Disease Control and Prevention-defined HIV acquisition risk: men who have sex with men, people who ever injected drugs, heterosexually active individuals at increased risk for HIV infection, or average risk for HIV infection. Distributions were 3%, 2%, 12%, and 83% for youth with PHIV and youth without HIV, and 80%, 6%, 14%, and 0% for youth with NPHIV, respectively. Among the simulated youth in this analysis, individuals were 61% Black, 24% Hispanic, and 15% White, respectively.

Exposures

HIV status by timing of acquisition.

Main Outcomes

Life expectancy loss for youth with PHIV and youth with NPHIV: difference between mean projected life expectancy under current and ideal HIV care scenarios compared with youth without HIV. Uncertainty intervals reflect varying adolescent HIV-related mortality inputs (95% CIs).

Results

Compared with youth without HIV (life expectancy: male, 76.3 years; female, 81.7 years), male youth with PHIV and youth with NPHIV had projected life expectancy losses of 10.4 years (95% CI, 5.5-18.1) and 15.0 years (95% CI, 9.3-26.8); female youth with PHIV and youth with NPHIV had projected life expectancy losses of 11.8 years (95% CI, 6.4-20.2) and 19.5 years (95% CI, 13.8-31.6), respectively. When receiving ideal HIV care, life expectancy losses were projected to improve for youth with PHIV (male: 0.5 years [95% CI, 0.3-1.8]: female: 0.6 years [95% CI, 0.4-2.1]) but were projected to persist for youth with NPHIV (male: 6.0 years [95% CI, 5.0-9.1]; female: 10.4 years [95% CI, 9.4-13.6]).

Conclusions

This adolescent-focused microsimulation modeling analysis projected that youth with HIV would have shorter life expectancy than youth without HIV. Projected differences were larger for youth with NPHIV compared with youth with PHIV. Differences in mortality by sex at birth, sexual behavior, and injection drug use contributed to lower projected life expectancy among youth with NPHIV. Interventions focused on HIV care and social factors are needed to improve life expectancy for youth with HIV in the US.

Sociodemographic characteristics of women who died by suicide in India from 2014 to 2020: findings from surveillance data

BACKGROUND

Women in India have twice the suicide death rate (SDR) compared with the global average for women. The aim of this study is to present a systematic understanding of sociodemographic risk factors, reasons for suicide deaths, and methods of suicide among women in India at the state level over time.

METHODS

Administrative data on suicide deaths among women by education level, marital status, and occupation, and reason for and method of suicide were extracted from the National Crimes Record Bureau reports for years 2014 to 2020. We extrapolated SDR at the population level for Indian women by education, marital status, and occupation to understand the sociodemography of these suicide deaths for India and its states. We reported the reasons for and methods of suicide deaths among Indian women at the state level over this period.

FINDINGS

SDR was higher among women with education of class 6 or more (10·2; 95% CI 10·1-10·4) than those with no education (3·8; 3·7-3·9) or education until class 5 (5·4; 5·2-5·5) in India in 2020, with similar patterns in most states. SDR declined between 2014 and 2020 for women with education until class 5. Women currently married accounted for 28 085 (63·1%) of 44 498 suicide deaths in India, 8336 (56·2%) of 14 840 in less developed states, and 19 661 (66·9%) of 29 407 in more developed states in 2020. For India, women currently married had a significantly higher SDR (8·1; 8·0-8·2) than those never married in 2014. However, women who never married had a significantly higher SDR (8·4; 8·2-8·5) in 2020 than those who were currently married. Many individual states in 2020 had similar SDR for women who never married and those who are currently married. Housewife as an occupation accounted for 50% or more of suicide deaths from 2014 to 2020 in India and its states. Family problems was the most common reason for suicide from 2014 to 2020, accounting for 16 140 (36·3%) of 44 498 suicide deaths in India, 5268 (35·5%) of 14 840 in less developed states, and 10 803 (36·7%) of 29 407 in more developed states in 2020. Hanging was the leading mean of suicide from 2014 to 2020. Insecticide or poison consumption was the second leading cause of suicide, accounting for 2228 (15·0%) of all 14 840 suicide deaths in less developed states and 5753 (19·6%) of 29 407 in more developed states, with a near 70·0% increase in the use of this method from 2014 to 2020.

INTERPRETATION

The higher SDR among women who have received an education, similar SDR between women currently married and never married, and variations in the reasons for and means of suicide at the state level highlight the need to incorporate sociological insights into how the external social environment can matter for women to better understand the complexity of suicide and determine how to effectively intervene.

FUNDING

Bill & Melinda Gates Foundation.

Causal evidence in health decision making: methodological approaches of causal inference and health decision science

Objectives

Public health decision making is a complex process based on thorough and comprehensive health technology assessments involving the comparison of different strategies, values and tradeoffs under uncertainty. This process must be based on best available evidence and plausible assumptions. Causal inference and health decision science are two methodological approaches providing information to help guide decision making in health care. Both approaches are quantitative methods that use statistical and modeling techniques and simplifying assumptions to mimic the complexity of the real world. We intend to review and lay out both disciplines with their aims, strengths and limitations based on a combination of textbook knowledge and expert experience.

Methods

To help understanding and differentiating the methodological approaches of causal inference and health decision science, we reviewed both methods with the focus on aims, research questions, methods, assumptions, limitations and challenges, and software. For each methodological approach, we established a group of four experts from our own working group to carefully review and summarize each method, followed by structured discussion rounds and written reviews, in which the experts from all disciplines including HTA and medicine were involved. The entire expert group discussed objectives, strengths and limitations of both methodological areas, and potential synergies. Finally, we derived recommendations for further research and provide a brief outlook on future trends.

Results

Causal inference methods aim for drawing causal conclusions from empirical data on the relationship of pre-specified interventions on a specific target outcome and apply a counterfactual framework and statistical techniques to derive causal effects of exposures or interventions from these data. Causal inference is based on a causal diagram, more specifically, a directed acyclic graph (DAG), which encodes the assumptions regarding the causal relations between variables. Depending on the type of confounding and selection bias, traditional statistical methods or more complex g-methods are needed to derive valid causal effects. Besides the correct specification of the DAG and the statistical model, assumptions such as consistency, positivity, and exchangeability must be checked when aiming at causal inference. Health decision science aims for guiding policy decision making regarding health interventions considering and balancing multiple competing objectives of a decision based on data from multiple sources and studies, for example prevalence studies, clinical trials and long-term observational routine effectiveness studies, and studies on preferences and costs. It involves decision analysis, a systematic, explicit and quantitative framework to guide decisions under uncertainty. Decision analyses are based on decision-analytic models to mimic the course of disease as well as aspects and consequences of the intervention in order to quantitatively optimize the decision. Depending on the type of decision problem, decision trees, state-transition models, discrete event simulation models, dynamic transmission models, or other model types are applied. Models must be validated against observed data, and comprehensive sensitivity analyses must be performed to assess uncertainty. Besides the appropriate choice of the model type and the valid specification of the model structure, it must be checked if input parameters of effects can be interpreted as causal parameters in the model. Otherwise results will be biased.

Conclusions

Both causal inference and health decision science aim for providing best causal evidence for informed health decision making. The strengths and limitations of both methods differ and a good understanding of both methods is essential for correct application but also for correct interpretation of findings from the described methods. Importantly, decision-analytic modeling should be combined with causal inference when developing guidance and recommendations regarding decisions on health care interventions.

Rapid-cycle systems modeling to support evidence-informed decision-making during system-wide implementation

Background

To “model and simulate change” is an accepted strategy to support implementation at scale. Much like a power analysis can inform decisions about study design, simulation models offer an analytic strategy to synthesize evidence that informs decisions regarding implementation of evidence-based interventions. However, simulation modeling is under-utilized in implementation science. To realize the potential of simulation modeling as an implementation strategy, additional methods are required to assist stakeholders to use models to examine underlying assumptions, consider alternative strategies, and anticipate downstream consequences of implementation. To this end, we propose Rapid-cycle Systems Modeling (RCSM)—a form of group modeling designed to promote engagement with evidence to support implementation. To demonstrate its utility, we provide an illustrative case study with mid-level administrators developing system-wide interventions that aim to identify and treat trauma among children entering foster care.

Methods

RCSM is an iterative method that includes three steps per cycle: (1) identify and prioritize stakeholder questions, (2) develop or refine a simulation model, and (3) engage in dialogue regarding model relevance, insights, and utility for implementation. For the case study, 31 key informants were engaged in step 1, a prior simulation model was adapted for step 2, and six member-checking group interviews (n = 16) were conducted for step 3.

Results

Step 1 engaged qualitative methods to identify and prioritize stakeholder questions, specifically identifying a set of inter-related decisions to promote implementing trauma-informed screening. In step 2, the research team created a presentation to communicate key findings from the simulation model that addressed decisions about programmatic reach, optimal screening thresholds to balance demand for treatment with supply, capacity to start-up and sustain screening, and availability of downstream capacity to provide treatment for those with indicated need. In step 3, member-checking group interviews with stakeholders documented the relevance of the model results to implementation decisions, insight regarding opportunities to improve system performance, and potential to inform conversations regarding anticipated implications of implementation choices.

Conclusions

By embedding simulation modeling in a process of stakeholder engagement, RCSM offers guidance to realize the potential of modeling not only as an analytic strategy, but also as an implementation strategy.

Emulating Target Trials to Improve Causal Inference From Agent-Based Models

Agent-based models are a key tool for investigating the emergent properties of population health settings, such as infectious disease transmission, where the exposure often violates the key "no interference" assumption of traditional causal inference under the potential outcomes framework. Agent-based models and other simulation-based modeling approaches have generally been viewed as a separate knowledge-generating paradigm from the potential outcomes framework, but this can lead to confusion about how to interpret the results of these models in real-world settings. By explicitly incorporating the target trial framework into the development of an agent-based or other simulation model, we can clarify the causal parameters of interest, as well as make explicit the assumptions required for valid causal effect estimation within or between populations. In this paper, we describe the use of the target trial framework for designing agent-based models when the goal is estimation of causal effects in the presence of interference, or spillover.

Complex systems models for causal inference in social epidemiology

Systems models, which by design aim to capture multi-level complexity, are a natural choice of tool for bridging the divide between social epidemiology and causal inference. In this commentary, we discuss the potential uses of complex systems models for improving our understanding of quantitative causal effects in social epidemiology. To put systems models in context, we will describe how this approach could be used to optimise the distribution of COVID-19 response resources to minimise social inequalities during and after the pandemic.

Calibration of individual-based models to epidemiological data: A systematic review

Individual-based models (IBMs) informing public health policy should be calibrated to data and provide estimates of uncertainty. Two main components of model-calibration methods are the parameter-search strategy and the goodness-of-fit (GOF) measure; many options exist for each of these. This review provides an overview of calibration methods used in IBMs modelling infectious disease spread. We identified articles on PubMed employing simulation-based methods to calibrate IBMs informing public health policy in HIV, tuberculosis, and malaria epidemiology published between 1 January 2013 and 31 December 2018. Articles were included if models stored individual-specific information, and calibration involved comparing model output to population-level targets. We extracted information on parameter-search strategies, GOF measures, and model validation. The PubMed search identified 653 candidate articles, of which 84 met the review criteria. Of the included articles, 40 (48%) combined a quantitative GOF measure with an algorithmic parameter-search strategy–either an optimisation algorithm (14/40) or a sampling algorithm (26/40). These 40 articles varied widely in their choices of parameter-search strategies and GOF measures. For the remaining 44 (52%) articles, the parameter-search strategy could either not be identified (32/44) or was described as an informal, non-reproducible method (12/44). Of these 44 articles, the majority (25/44) were unclear about the GOF measure used; of the rest, only five quantitatively evaluated GOF. Only a minority of the included articles, 14 (17%) provided a rationale for their choice of model-calibration method. Model validation was reported in 31 (37%) articles. Reporting on calibration methods is far from optimal in epidemiological modelling studies of HIV, malaria and TB transmission dynamics. The adoption of better documented, algorithmic calibration methods could improve both reproducibility and the quality of inference in model-based epidemiology. There is a need for research comparing the performance of calibration methods to inform decisions about the parameter-search strategies and GOF measures.

Calibration—that is, “fitting” the model to data—is a crucial part of using mathematical models to better forecast and control the population-level spread of infectious diseases. Evidence that the mathematical model is well-calibrated improves confidence that the model provides a realistic picture of the consequences of health policy decisions. To make informed decisions, Policymakers need information about uncertainty: i.e., what is the range of likely outcomes (rather than just a single prediction). Thus, modellers should also strive to provide accurate measurements of uncertainty, both for their model parameters and for their predictions. This systematic review provides an overview of the methods used to calibrate individual-based models (IBMs) of the spread of HIV, malaria, and tuberculosis. We found that less than half of the reviewed articles used reproducible, non-subjective calibration methods. For the remaining articles, the method could either not be identified or was described as an informal, non-reproducible method. Only one-third of the articles obtained estimates of parameter uncertainty. We conclude that the adoption of better-documented, algorithmic calibration methods could improve both reproducibility and the quality of inference in model-based epidemiology.

Guidance for a causal comparative effectiveness analysis emulating a target trial based on big real world evidence: when to start statin treatment

Aim: The aim of this project is to describe a causal (counterfactual) approach for analyzing when to start statin treatment to prevent cardiovascular disease using real-world evidence. Methods: We use directed acyclic graphs to operationalize and visualize the causal research question considering selection bias, potential time-independent and time-dependent confounding. We provide a study protocol following the 'target trial' approach and describe the data structure needed for the causal assessment. Conclusion: The study protocol can be applied to real-world data, in general. However, the structure and quality of the database play an essential role for the validity of the results, and database-specific potential for bias needs to be explicitly considered.

DAG-informed regression modelling, agent-based modelling and microsimulation modelling: a critical comparison of methods for causal inference

The current paradigm for causal inference in epidemiology relies primarily on the evaluation of counterfactual contrasts via statistical regression models informed by graphical causal models (often in the form of directed acyclic graphs, or DAGs) and their underlying mathematical theory. However, there have been growing calls for supplementary methods, and one such method that has been proposed is agent-based modelling due to its potential for simulating counterfactuals. However, within the epidemiological literature, there currently exists a general lack of clarity regarding what exactly agent-based modelling is (and is not) and, importantly, how it differs from microsimulation modelling-perhaps its closest methodological comparator. We clarify this distinction by briefly reviewing the history of each method, which provides a context for their similarities and differences, and casts light on the types of research questions that they have evolved (and thus are well suited) to answering; we do the same for DAG-informed regression methods. The distinct historical evolutions of DAG-informed regression modelling, microsimulation modelling and agent-based modelling have given rise to distinct features of the methods themselves, and provide a foundation for critical comparison. Not only are the three methods well suited to addressing different types of causal questions, but, in doing so, they place differing levels of emphasis on fixed and random effects, and also tend to operate on different timescales and in different timeframes.