Predictive Analytics Using Machine Learning to Identify ART Clients at Health System Level at Greatest Risk of Treatment Interruption in Mozambique and Nigeria

Rudi Kundini, Pamoja Kundini (RKPK): study protocol for a hybrid type 1 randomized effectiveness-implementation trial using data science and economic incentive strategies to strengthen the continuity of care among people living with HIV in Tanzania

BACKGROUND

Economic incentives can improve clinical outcomes among in-care people living with HIV (PLHIV), but evidence is limited for their effectiveness among out-of-care PLHIV or those at risk of disengagement. We propose a type 1 hybrid effectiveness-implementation study to advance global knowledge about the use of economic incentives to strengthen the continuity of HIV care and accelerate global goals for HIV epidemic control.

METHODS

The Rudi Kundini, Pamoja Kundini study will evaluate two implementation models of an economic incentive strategy for supporting two groups of PLHIV in Tanzania. Phase 1 of the study consists of a two-arm, cluster randomized trial across 32 health facilities to assess the effectiveness of a home visit plus one-time economic incentive on the proportion of out-of-care PLHIV with viral load suppression (< 1000 copies/ml) 6 months after enrollment (n = 640). Phase 2 is an individual 1:1 randomized controlled trial designed to determine the effectiveness of a short-term counseling and economic incentive program offered to in-care PLHIV who are predicted through machine learning to be at risk of disengaging from care on the outcome of viral load suppression at 12 months (n = 692). The program includes up to three incentives conditional upon visit attendance coupled with adapted counselling sessions for this population of PLHIV. Consistent with a hybrid effectiveness-implementation study design, phase 3 is a mixed methods evaluation to explore barriers and facilitators to strategy implementation in phases 1 and 2. Results will be used to guide optimization and scale-up of the incentive strategies, if effective, to the larger population of Tanzanian PLHIV who struggle with continuity of HIV care.

DISCUSSION

Innovative strategies that recognize the dynamic process of lifelong retention in HIV care are urgently needed. Strategies such as conditional economic incentives are a simple and effective method for improving many health outcomes, including those on the HIV continuum. If coupled with other supportive services such as home visits (phase 1) or with tailored counselling (phase 2), economic incentives have the potential to strengthen engagement among the subpopulation of PLHIV who struggle with retention in care and could help to close the gap towards reaching global "95-95-95" goals for ending the AIDS epidemic.

TRIAL REGISTRATION

Phase 1: ClinicalTrials.gov, NCT05248100 , registered 2/21/2022. Phase 2: ClinicalTrials.gov, NCT05373095 , registered 5/13/2022.

Development and validation of a prediction tool to support engagement in HIV care among young people ages 10-24 years in Kenya

INTRODUCTION

Loss to follow-up (LTFU) among adolescents and young adults living with HIV (AYALWH) is a barrier to optimal health and HIV services. We developed and validated a clinical prediction tool to identify AYALWH at risk of LTFU.

METHODS

We used electronic medical records (EMR) of AYALWH ages 10 to 24 in HIV care at 6 facilities in Kenya and surveys from a subset of participants. Early LTFU was defined as >30 days late for a scheduled visit in the last 6 months, which accounts for clients with multi-month refills. We developed a tool combining surveys with EMR ('survey-plus-EMR tool'), and an 'EMR-alone' tool to predict high, medium, and low risk of LTFU. The survey-plus-EMR tool included candidate sociodemographics, partnership status, mental health, peer support, any unmet clinic needs, WHO stage, and time in care variables for tool development, while the EMR-alone included clinical and time in care variables only. Tools were developed in a 50% random sample of the data and internally validated using 10-fold cross-validation of the full sample. Tool performance was evaluated using Hazard Ratios (HR), 95% Confidence Intervals (CI), and area under the curve (AUC) ≥ 0.7 for good performance and ≥0.60 for modest performance.

RESULTS

Data from 865 AYALWH were included in the survey-plus-EMR tool and early LTFU was (19.2%, 166/865). The survey-plus-EMR tool ranged from 0 to 4, including PHQ-9 ≥5, lack of peer support group attendance, and any unmet clinical need. High (3 or 4) and medium (2) prediction scores were associated with greater risk of LTFU (high, 29.0%, HR 2.16, 95%CI: 1.25-3.73; medium, 21.4%, HR 1.52, 95%CI: 0.93-2.49, global p-value = 0.02) in the validation dataset. The 10-fold cross validation AUC was 0.66 (95%CI: 0.63-0.72). Data from 2,696 AYALWH were included in the EMR-alone tool and early LTFU was 28.6% (770/2,696). In the validation dataset, high (score = 2, LTFU = 38.5%, HR 2.40, 95%CI: 1.17-4.96) and medium scores (1, 29.6%, HR 1.65, 95%CI: 1.00-2.72) predicted significantly higher LTFU than low-risk scores (0, 22.0%, global p-value = 0.03). Ten-fold cross-validation AUC was 0.61 (95%CI: 0.59-0.64).

CONCLUSIONS

Clinical prediction of LTFU was modest using the surveys-plus-EMR tool and the EMR-alone tool, suggesting limited use in routine care. However, findings may inform future prediction tools and intervention targets to reduce LTFU among AYALWH.

Predicting Treatment Interruption Among People Living With HIV in Nigeria: Machine Learning Approach

BACKGROUND

Antiretroviral therapy (ART) has transformed HIV from a fatal illness to a chronic disease. Given the high rate of treatment interruptions, HIV programs use a range of approaches to support individuals in adhering to ART and in re-engaging those who interrupt treatment. These interventions can often be time-consuming and costly, and thus providing for all may not be sustainable.

OBJECTIVE

This study aims to describe our experiences developing a machine learning (ML) model to predict interruption in treatment (IIT) at 30 days among people living with HIV newly enrolled on ART in Nigeria and our integration of the model into the routine information system. In addition, we collected health workers' perceptions and use of the model's outputs for case management.

METHODS

Routine program data collected from January 2005 through February 2021 was used to train and test an ML model (boosting tree and Extreme Gradient Boosting) to predict future IIT. Data were randomly sampled using an 80/20 split into training and test data sets, respectively. Model performance was estimated using sensitivity, specificity, and positive and negative predictive values. Variables considered to be highly associated with treatment interruption were preselected by a group of HIV prevention researchers, program experts, and biostatisticians for inclusion in the model. Individuals were defined as having IIT if they were provided a 30-day supply of antiretrovirals but did not return for a refill within 28 days of their scheduled follow-up visit date. Outputs from the ML model were shared weekly with health care workers at selected facilities.

RESULTS

After data cleaning, complete data for 136,747 clients were used for the analysis. The percentage of IIT cases decreased from 58.6% (36,663/61,864) before 2017 to 14.2% (3690/28,046) from October 2019 through February 2021. Overall IIT was higher among clients who were sicker at enrollment. Other factors that were significantly associated with IIT included pregnancy and breastfeeding status and facility characteristics (location, service level, and service type). Several models were initially developed; the selected model had a sensitivity of 81%, specificity of 88%, positive predictive value of 83%, and negative predictive value of 87%, and was successfully integrated into the national electronic medical records database. During field-testing, the majority of users reported that an IIT prediction tool could lead to proactive steps for preventing IIT and improving patient outcomes.

CONCLUSIONS

High-performing ML models to identify patients with HIV at risk of IIT can be developed using routinely collected service delivery data and integrated into routine health management information systems. Machine learning can improve the targeting of interventions through differentiated models of care before patients interrupt treatment, resulting in increased cost-effectiveness and improved patient outcomes.

Artificial Intelligence and Machine Learning Based Prediction of Viral Load and CD4 Status of People Living with HIV (PLWH) on Anti-Retroviral Treatment in Gedeo Zone Public Hospitals

Background

Despite the success made in scaling up HIV treatment activities, there remains a tremendous unmet demand for the monitoring of the disease progression and treatment success, which threatens HIV/AIDS treatment and control. This research presented the assessments of viral load and CD4 classification of adults enrolled in ART care using machine learning algorithms.

Methods

We trained, validated, and tested eight machine learning (ML) classifier algorithms with historical data, including demographics, clinical, and laboratory data. Data were extracted from the ART registry database of Yirgacheffe Primary Hospital and Dilla University Referral Hospital. ML classifiers were trained to predict virological failure (viral load >1000 copies/mL) and poor CD4 (CD4 cell count <200 cells/mL). The model predictive performances were evaluated using accuracy, sensitivity, specificity, precision, f1-score, F-beta scores, and AUC.

Results

The mean age of the sample participants was 41.6 years (SD = 10.9). The experimental results showed that XGB classifier ranked as the best algorithm for viral load prediction in terms of sensitivity (97%), f1-score (96%), AUC (0.99), accuracy (96%), followed by RF. The GB classifier exhibited a better predictive capability in predicting participants with a CD4 cell count <200 cells/mL.

Conclusion

In this study, the XGB and RF models had the highest accuracy and outperformed on various evaluation metrics among the models examined for viral load classification. In the prediction of participants CD4, GB model had the highest accuracy.

Validation and Improvement of a Machine Learning Model to Predict Interruptions in Antiretroviral Treatment in South Africa

INTRODUCTION

Machine learning algorithms are increasingly being used to inform HIV prevention and detection strategies. We validated and extended a previously developed machine learning model for patient retention on antiretroviral therapy in a new geographic catchment area in South Africa.

METHODS

We compared the ability of an adaptive boosting algorithm to predict interruption in treatment (IIT) in 2 South African cohorts from the Free State and Mpumalanga and Gauteng and North West (GA/NW) provinces. We developed a novel set of predictive features for the GA/NW cohort using a categorical boosting model. We evaluated the ability of the model to predict IIT over all visits and across different periods within a patient's treatment trajectory.

RESULTS

When predicting IIT, the GA/NW and Free State and Mpumalanga models demonstrated a sensitivity of 60% and 61%, respectively, able to correctly predict nearly two-thirds of all missed visits with a positive predictive value of 18% and 19%. Using predictive features generated from the GA/NW cohort, the categorical boosting model correctly predicted 22,119 of a total of 35,985 missed next visits, yielding a sensitivity of 62%, specificity of 67%, and positive predictive value of 20%. Model performance was highest when tested on visits within the first 6 months.

CONCLUSIONS

Machine learning algorithms may be useful in informing tools to increase antiretroviral therapy patient retention and efficiency of HIV care interventions. This is particularly relevant in developing countries where health data systems are being strengthened to collect data on a scale that is large enough to apply novel analytical methods.