Performance of a mathematical model to forecast lives saved from HIV treatment expansion in resource-limited settings

Development and calibration of a mathematical model of HIV outcomes among Rwandan adults: informing equitable achievement of targets in Rwanda

Background

We developed and calibrated the Central Africa-International epidemiology Databases to Evaluate AIDS (CA-IeDEA) HIV policy model to inform equitable achievement of global goals, overall and across sub-populations, in Rwanda.

Methods

We created a deterministic dynamic model to project adult HIV epidemic and care continuum outcomes, overall and for 25 subpopulations (age group, sex, HIV acquisition risk, urbanicity). Data came from the Rwanda cohort of CA-IeDEA, 2004-2020; Rwanda Demographic and Health Surveys, 2005, 2010, 2015; Rwanda Population-based HIV Impact Assessment, 2019; and the literature and reports. We calibrated the model to 47 targets by selecting the 50 best-fitting parameter sets among 20,000 simulations. Calibration targets reflected epidemic (HIV prevalence, incidence), global goals (percentage on antiretroviral therapy (ART) among diagnosed, percentage virally suppressed among on ART) and other (number on ART, percentage virally suppressed) indicators, overall and by sex. Best-fitting sets minimized the summed absolute value of the percentage deviation (AVPD) between model projections and calibration targets. Good model performance was mean AVPD ≤5% across the 50 best-fitting sets and/or projections within the target confidence intervals; acceptable was mean AVPD >5% and ≤15%.

Results

Across indicators, 1,841 of 2,350 (78.3%) model projections were a good or acceptable fit to calibration targets. For HIV epidemic indicators, 256 of 300 (85.3%) projections were a good fit to targets, with the model performing better for women (83.3% a good fit) than for men (71.7% a good fit). For global goals indicators, 96 of 100 (96.0%) projections were a good fit; model performance was similar for women and men. For other indicators, 653 of 950 (68.7%) projections were a good or acceptable fit. Fit was better for women than for men (percentage virally suppressed only) and when restricting targets for number on ART to 2013 and beyond.

Conclusions

The CA-IeDEA HIV policy model fits historical data and can inform policy solutions for equitably achieving global goals to end the HIV epidemic in Rwanda. High-quality, unbiased population-based data, as well as novel approaches that account for calibration target quality, are critical to ongoing use of mathematical models for programmatic planning.

Implementing parallel spreadsheet models for health policy decisions: The impact of unintentional errors on model projections

Background

Spreadsheet software is increasingly used to implement systems science models informing health policy decisions, both in academia and in practice where technical capacity may be limited. However, spreadsheet models are prone to unintentional errors that may not always be identified using standard error-checking techniques. Our objective was to illustrate, through a methodologic case study analysis, the impact of unintentional errors on model projections by implementing parallel model versions.

Methods

We leveraged a real-world need to revise an existing spreadsheet model designed to inform HIV policy. We developed three parallel versions of a previously validated spreadsheet-based model; versions differed by the spreadsheet cell-referencing approach (named single cells; column/row references; named matrices). For each version, we implemented three model revisions (re-entry into care; guideline-concordant treatment initiation; immediate treatment initiation). After standard error-checking, we identified unintentional errors by comparing model output across the three versions. Concordant model output across all versions was considered error-free. We calculated the impact of unintentional errors as the percentage difference in model projections between model versions with and without unintentional errors, using +/-5% difference to define a material error.

Results

We identified 58 original and 4,331 propagated unintentional errors across all model versions and revisions. Over 40% (24/58) of original unintentional errors occurred in the column/row reference model version; most (23/24) were due to incorrect cell references. Overall, >20% of model spreadsheet cells had material unintentional errors. When examining error impact along the HIV care continuum, the percentage difference between versions with and without unintentional errors ranged from +3% to +16% (named single cells), +26% to +76% (column/row reference), and 0% (named matrices).

Conclusions

Standard error-checking techniques may not identify all errors in spreadsheet-based models. Comparing parallel model versions can aid in identifying unintentional errors and promoting reliable model projections, particularly when resources are limited.