The effectiveness of pay-for-performance contracts with non-governmental organizations in Afghanistan - results of a controlled interrupted time series analysis

BACKGROUND

In many contexts, including fragile settings like Afghanistan, the coverage of basic health services is low. To address these challenges there has been considerable interest in working with NGOs and examining the effect of financial incentives on service providers. The Government of Afghanistan has used contracting with NGOs for more than 15 years and in 2019 introduced pay-for-performance (P4P) into the contracts. This study examines the impact of P4P on health service delivery in Afghanistan.

METHODS

We conducted an interrupted time series (ITS) analysis with a non-randomized comparison group that employed segmented regression models and used independently verified health management information system (HMIS) data from 2015 to 2021. We compared 31 provinces with P4P contracts to 3 provinces where the Ministry of Public Health (MOPH) continued to deliver services without P4P. We used data from annual health facility surveys to assess the quality of care.

FINDINGS

Independent verification of the HMIS data found that consistency and accuracy was greater than 90% in the contracted provinces. The introduction of P4P increased the 10 P4P-compensated service delivery outcomes by a median of 22.1 percentage points (range 10.2 to 43.8) for the two-arm analysis and 19.9 percentage points (range: - 8.3 to 56.1) for the one-arm analysis. There was a small decrease in quality of care initially, but it was short-lived. We found few other unintended consequences.

INTERPRETATION

P4P contracts with NGOs led to a substantial improvement in service delivery at lower cost despite a very difficult security situation. The promising results from this large-scale experience warrant more extensive application of P4P contracts in other fragile settings or wherever coverage remains low.

Surveillance as a Core Intervention to Strengthen Malaria Control Programs in Moderate to High Transmission Settings

New tools are needed for malaria control, and recent improvements in malaria surveillance have opened the possibility of transforming surveillance into a core intervention. Implementing this strategy can be challenging in moderate to high transmission settings. However, there is a wealth of practical experience among national malaria control programs and partners working to improve and use malaria surveillance data to guide programming. Granular and timely data are critical to understanding geographic heterogeneity, appropriately defining and targeting interventions packages, and enabling timely decision-making at the operational level. Resources to be targeted based on surveillance data include vector control, case management commodities, outbreak responses, quality improvement interventions, and human resources, including community health workers, as they contribute to a more refined granularity of the surveillance system. Effectively transforming malaria surveillance into a core intervention will require strong global and national leadership, empowerment of subnational and local leaders, collaboration among development partners, and global coordination. Ensuring that national health systems include community health work can contribute to a successful transformation. It will require a strong supply chain to ensure that all suspected cases can be diagnosed and data reporting tools including appropriate electronic devices to provide timely data. Regular data quality audits, decentralized implementation, supportive supervision, data-informed decision-making processes, and harnessing technology for data analysis and visualization are needed to improve the capacity for data-driven decision-making at all levels. Finally, resources must be available to respond programmatically to these decisions.

How does the effectiveness of strategies to improve healthcare provider practices in low-income and middle-income countries change after implementation? Secondary analysis of a systematic review

BACKGROUND

A recent systematic review evaluated the effectiveness of strategies to improve healthcare provider (HCP) performance in low-income and middle-income countries. The review identified strategies with varying effects, including in-service training, supervision and group problem-solving. However, whether their effectiveness changed over time remained unclear. In particular, understanding whether effects decay over time is crucial to improve sustainability.

METHODS

We conducted a secondary analysis of data from the aforementioned review to explore associations between time and effectiveness. We calculated effect sizes (defined as percentage-point (%-point) changes) for HCP practice outcomes (eg, percentage of patients correctly treated) at each follow-up time point after the strategy was implemented. We estimated the association between time and effectiveness using random-intercept linear regression models with time-specific effect sizes clustered within studies and adjusted for baseline performance.

RESULTS

The primary analysis included 37 studies, and a sensitivity analysis included 77 additional studies. For training, every additional month of follow-up was associated with a 0.19 %-point decrease in effectiveness (95% CI: -0.36 to -0.03). For training combined with supervision, every additional month was associated with a 0.40 %-point decrease in effectiveness (95% CI: -0.68 to -0.12). Time trend results for supervision were inconclusive. For group problem-solving alone, time was positively associated with effectiveness, with a 0.50 %-point increase in effect per month (95% CI: 0.37 to 0.64). Group problem-solving combined with training was associated with large improvements, and its effect was not associated with time.

CONCLUSIONS

Time trends in the effectiveness of different strategies to improve HCP practices vary among strategies. Programmes relying solely on in-service training might need periodical refresher training or, better still, consider combining training with group problem-solving. Although more high-quality research is needed, these results, which are important for decision-makers as they choose which strategies to use, underscore the utility of studies with multiple post-implementation measurements so sustainability of the impact on HCP practices can be assessed.

The effectiveness of supervision strategies to improve health care provider practices in low- and middle-income countries: secondary analysis of a systematic review

BACKGROUND

Although supervision is a ubiquitous approach to support health programs and improve health care provider (HCP) performance in low- and middle-income countries (LMICs), quantitative evidence of its effects is unclear. The objectives of this study are to describe the effect of supervision strategies on HCP practices in LMICs and to identify attributes associated with greater effectiveness of routine supervision.

METHODS

We performed a secondary analysis of data on HCP practice outcomes (e.g., percentage of patients correctly treated) from a systematic review on improving HCP performance. The review included controlled trials and interrupted time series studies. We described distributions of effect sizes (defined as percentage-point [%-point] changes) for each supervision strategy. To identify attributes associated with supervision effectiveness, we performed random-effects linear regression modeling and examined studies that directly compared different approaches of routine supervision.

RESULTS

We analyzed data from 81 studies from 36 countries. For professional HCPs, such as nurses and physicians, primarily working at health facilities, routine supervision (median improvement when compared to controls: 10.7%-points; IQR: 9.9, 27.9) had similar effects on HCP practices as audit with feedback (median improvement: 10.1%-points; IQR: 6.2, 23.7). Two attributes were associated with greater mean effectiveness of routine supervision (p < 0.10): supervisors received supervision (by 8.8-11.5%-points), and supervisors participated in problem-solving with HCPs (by 14.2-20.8%-points). Training for supervisors and use of a checklist during supervision visits were not associated with effectiveness. The effects of supervision frequency (i.e., number of visits per year) and dose (i.e., the number of supervision visits during a study) were unclear. For lay HCPs, the effect of routine supervision was difficult to characterize because few studies existed, and effectiveness in those studies varied considerably. Evidence quality for all findings was low primarily because many studies had a high risk of bias.

CONCLUSIONS

Although evidence is limited, to promote more effective supervision, our study supports supervising supervisors and having supervisors engage in problem-solving with HCPs. Supervision's integral role in health systems in LMICs justifies a more deliberate research agenda to identify how to deliver supervision to optimize its effect on HCP practices.

Opening the 'black box' of collaborative improvement: a qualitative evaluation of a pilot intervention to improve quality of malaria surveillance data in public health centres in Uganda

BACKGROUND

Demand for high-quality surveillance data for malaria, and other diseases, is greater than ever before. In Uganda, the primary source of malaria surveillance data is the Health Management Information System (HMIS). However, HMIS data may be incomplete, inaccurate or delayed. Collaborative improvement (CI) is a quality improvement intervention developed in high-income countries, which has been advocated for low-resource settings. In Kayunga, Uganda, a pilot study of CI was conducted in five public health centres, documenting a positive effect on the quality of HMIS and malaria surveillance data. A qualitative evaluation was conducted concurrently to investigate the mechanisms of effect and unintended consequences of the intervention, aiming to inform future implementation of CI.

METHODS

The study intervention targeted health workers, including brief in-service training, plus CI with 'plan-do-study-act' (PDSA) cycles emphasizing self-reflection and group action, periodic learning sessions, and coaching from a CI mentor. Health workers collected data on standard HMIS out-patient registers. The qualitative evaluation (July 2015 to September 2016) included ethnographic observations at each health centre (over 12-14 weeks), in-depth interviews with health workers and stakeholders (n = 20), and focus group discussions with health workers (n = 6).

RESULTS

The results suggest that the intervention did facilitate improvement in data quality, but through unexpected mechanisms. The CI intervention was implemented as planned, but the PDSA cycles were driven largely by the CI mentor, not the health workers. In this context, characterized by a rigid hierarchy within the health system of limited culture of self-reflection and inadequate training and supervision, CI became an effective form of high-quality training with frequent supervisory visits. Health workers appeared motivated to improve data collection habits by their loyalty to the CI mentor and the potential for economic benefits, rather than a desire for self-improvement.

CONCLUSIONS

CI is a promising method of quality improvement and could have a positive impact on malaria surveillance data. However, successful scale-up of CI in similar settings may require deployment of highly skilled mentors. Further research, focusing on the effectiveness of 'real world' mentors using robust study designs, will be required to determine whether CI can be translated effectively and sustainably to low-resource settings.

Effectiveness of in-service training plus the collaborative improvement strategy on the quality of routine malaria surveillance data: results of a pilot study in Kayunga District, Uganda

BACKGROUND

Surveillance data are essential for malaria control, but quality is often poor. The aim of the study was to evaluate the effectiveness of the novel combination of training plus an innovative quality improvement method-collaborative improvement (CI)-on the quality of malaria surveillance data in Uganda.

METHODS

The intervention (training plus CI, or TCI), including brief in-service training and CI, was delivered in 5 health facilities (HFs) in Kayunga District from November 2015 to August 2016. HF teams monitored data quality, conducted plan-do-study-act cycles to test changes, attended periodic learning sessions, and received CI coaching. An independent evaluation was conducted to assess data completeness, accuracy, and timeliness. Using an interrupted time series design without a separate control group, data were abstracted from 156,707 outpatient department (OPD) records, laboratory registers, and aggregated monthly reports (MR) for 4 time periods: baseline-12 months, TCI scale-up-5 months; CI implementation-9 months; post-intervention-4 months. Monthly OPD register completeness was measured as the proportion of patient records with a malaria diagnosis with: (1) all data fields completed, and (2) all clinically-relevant fields completed. Accuracy was the relative difference between: (1) number of monthly malaria patients reported in OPD register versus MR, and (2) proportion of positive malaria tests reported in the laboratory register versus MR. Data were analysed with segmented linear regression modelling.

RESULTS

Data completeness increased substantially following TCI. Compared to baseline, all-field completeness increased by 60.1%-points (95% confidence interval [CI]: 46.9-73.2%) at mid-point, and clinically-relevant completeness increased by 61.6%-points (95% CI: 56.6-66.7%). A relative - 57.4%-point (95% confidence interval: - 105.5, - 9.3%) change, indicating an improvement in accuracy of malaria test positivity reporting, but no effect on data accuracy for monthly malaria patients, were observed. Cost per additional malaria patient, for whom complete clinically-relevant data were recorded in the OPD register, was $3.53 (95% confidence interval: $3.03, $4.15).

CONCLUSIONS

TCI improved malaria surveillance completeness considerably, with limited impact on accuracy. Although these results are promising, the intervention's effectiveness should be evaluated in more HFs, with longer follow-up, ideally in a randomized trial, before recommending CI for wide-scale use.

The effectiveness of training strategies to improve healthcare provider practices in low-income and middle-income countries

INTRODUCTION

In low/middle-income countries (LMICs), training is often used to improve healthcare provider (HCP) performance. However, important questions remain about how well training works and the best ways to design training strategies. The objective of this study is to characterise the effectiveness of training strategies to improve HCP practices in LMICs and identify attributes associated with training effectiveness.

METHODS

We performed a secondary analysis of data from a systematic review on improving HCP performance. The review included controlled trials and interrupted time series, and outcomes measuring HCP practices (eg, percentage of patients correctly treated). Distributions of effect sizes (defined as percentage-point (%-point) changes) were described for each training strategy. To identify effective training attributes, we examined studies that directly compared training approaches and performed random-effects linear regression modelling.

RESULTS

We analysed data from 199 studies from 51 countries. For outcomes expressed as percentages, educational outreach visits (median effect size when compared with controls: 9.9 %-points; IQR: 4.3-20.6) tended to be somewhat more effective than in-service training (median: 7.3 %-points; IQR: 3.6-17.4), which seemed more effective than peer-to-peer training (4.0 %-points) and self-study (by 2.0-9.3 %-points). Mean effectiveness was greater (by 6.0-10.4 %-points) for training that incorporated clinical practice and training at HCPs' work site. Attributes with little or no effect were: training with computers, interactive methods or over multiple sessions; training duration; number of educational methods; distance training; trainers with pedagogical training and topic complexity. For lay HCPs, in-service training had no measurable effect. Evidence quality for all findings was low.

CONCLUSIONS

Although additional research is needed, by characterising the effectiveness of training strategies and identifying attributes of effective training, decision-makers in LMICs can improve how these strategies are selected and implemented.

The effectiveness of the quality improvement collaborative strategy in low- and middle-income countries: A systematic review and meta-analysis

BACKGROUND

Quality improvement collaboratives (QICs) have been used to improve health care for decades. Evidence on QIC effectiveness has been reported, but systematic reviews to date have little information from low- and middle-income countries (LMICs).

OBJECTIVE

To assess the effectiveness of QICs in LMICs.

METHODS

We conducted a systematic review following Cochrane methods, the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach for quality of evidence grading, and the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement for reporting. We searched published and unpublished studies between 1969 and March 2019 from LMICs. We included papers that compared usual practice with QICs alone or combined with other interventions. Pairs of reviewers independently selected and assessed the risk of bias and extracted data of included studies. To estimate strategy effectiveness from a single study comparison, we used the median effect size (MES) in the comparison for outcomes in the same outcome group. The primary analysis evaluated each strategy group with a weighted median and interquartile range (IQR) of MES values. In secondary analyses, standard random-effects meta-analysis was used to estimate the weighted mean MES and 95% confidence interval (CI) of the mean MES of each strategy group. This review is registered with PROSPERO (International Prospective Register of Systematic Reviews): CRD42017078108.

RESULTS

Twenty-nine studies were included; most (21/29, 72.4%) were interrupted time series studies. Evidence quality was generally low to very low. Among studies involving health facility-based health care providers (HCPs), for "QIC only", effectiveness varied widely across outcome groups and tended to have little effect for patient health outcomes (median MES less than 2 percentage points for percentage and continuous outcomes). For "QIC plus training", effectiveness might be very high for patient health outcomes (for continuous outcomes, median MES 111.6 percentage points, range: 96.0 to 127.1) and HCP practice outcomes (median MES 52.4 to 63.4 percentage points for continuous and percentage outcomes, respectively). The only study of lay HCPs, which used "QIC plus training", showed no effect on patient care-seeking behaviors (MES -0.9 percentage points), moderate effects on non-care-seeking patient behaviors (MES 18.7 percentage points), and very large effects on HCP practice outcomes (MES 50.4 percentage points).

CONCLUSIONS

The effectiveness of QICs varied considerably in LMICs. QICs combined with other invention components, such as training, tended to be more effective than QICs alone. The low evidence quality and large effect sizes for QIC plus training justify additional high-quality studies assessing this approach in LMICs.

The Effect of Text Message Reminders to Health Workers on Quality of Care for Malaria, Pneumonia, and Diarrhea in Malawi: A Cluster-Randomized, Controlled Trial

The use of mobile technologies in medicine, or mHealth, holds promise to improve health worker (HW) performance, but evidence is mixed. We conducted a cluster-randomized controlled trial to evaluate the effect of text message reminders to HWs in outpatient health facilities (HFs) on quality of care for malaria, pneumonia, and diarrhea in Malawi. After a baseline HF survey (2,360 patients) in January 2015, 105 HFs were randomized to three arms: 1) text messages to HWs on malaria case management; 2) text messages to HWs on malaria, pneumonia, and diarrhea case management (latter two for children < 5 years); and 3) control arm (no messages). Messages were sent beginning April 2015 twice daily for 6 months, followed by an endline HF survey (2,536 patients) in November 2015. An intention-to-treat analysis with difference-in-differences binomial regression modeling was performed. The proportion of patients with uncomplicated malaria managed correctly increased from 42.8% to 59.6% in the control arm, from 43.7% to 55.8% in arm 1 (effect size -4.7%-points, 95% confidence interval (CI): -18.2, 8.9, P = 0.50) and from 30.2% to 50.9% in arm 2 (effect size 3.9%-points, 95% CI: -14.1, 22.0, P = 0.67). Prescription of first-line antibiotics to children < 5 years with clinically defined pneumonia increased in all arms, but decreased in arm 2 (effect size -4.1%-points, 95% CI: -42.0, 33.8, P = 0.83). Prescription of oral rehydration solution to children with diarrhea declined slightly in all arms. We found no significant improvements in malaria, pneumonia, or diarrhea treatment after HW reminders, illustrating the importance of rigorously testing new interventions before adoption.

The effect of mobile phone text message reminders on health workers' adherence to case management guidelines for malaria and other diseases in Malawi: lessons from qualitative data from a cluster-randomized trial

Background

Mobile health (mHealth), which uses technology such as mobile phones to improve patient health and health care delivery, is increasingly being tested as an intervention to promote health worker (HW) performance. This study assessed the effect of short messaging services (SMS) reminders in a study setting. Following a trial of text-message reminders to HWs to improve case management of malaria and other childhood diseases in southern Malawi that showed little effect, qualitative data was collected to explore the reasons why the intervention was ineffective and describe lessons learned.

Methods

Qualitative data collection was undertaken to lend insight into quantitative results from a trial in which 105 health facilities were randomized to three arms: (1) twice-daily text-message reminders to HWs, including clinicians and drug dispensers, on case management of malaria; (2) twice-daily text-message reminders to HWs on case management of malaria, pneumonia and diarrhoea; and, (3) a control arm. In-depth interviews were conducted with 50 HWs in the intervention arms across seven districts. HWs were asked about acceptability and feasibility of the text-messaging intervention and its perceived impact on recommended case management. The interviews were recorded, transcribed and translated into English for a thematic and framework analysis. Nvivo 11 software was used for data management and analysis.

Results

A total of 50 HWs were interviewed at 22 facilities. HWs expressed high acceptance of text-message reminders and appreciated messages as job aids and practical reference material for their day-to-day work. However, HWs said that health systems barriers, including very high outpatient workload, commodity stock-outs, and lack of supportive supervision and financial incentives demotivated them, limited their ability to act on messages and therefore adherence to case management guidelines. Drug dispensers were more likely than clinicians to report usage of text-message reminders. Despite these challenges, nearly all HWs expressed a desire for a longer duration of the SMS intervention.

Conclusions

Text-message reminders to HWs can provide a platform to improve understanding of treatment guidelines and case management decision-making skills, but might not improve actual adherence to guidelines. More interaction, for example through targeted supervision or two-way technology communication, might be an essential intervention component to help address structural barriers and facilitate improved clinical practice.

Effectiveness of strategies to improve health-care provider practices in low-income and middle-income countries: a systematic review

Background

Inadequate health-care provider performance is a major challenge to the delivery of high-quality health care in low-income and middle-income countries (LMICs). The Health Care Provider Performance Review (HCPPR) is a comprehensive systematic review of strategies to improve health-care provider performance in LMICs.

Methods

For this systematic review we searched 52 electronic databases for published studies and 58 document inventories for unpublished studies from the 1960s to 2016. Eligible study designs were controlled trials and interrupted time series. We only included strategy-versus-control group comparisons. We present results of improving health-care provider practice outcomes expressed as percentages (eg, percentage of patients treated correctly) or as continuous measures (eg, number of medicines prescribed per patient). Effect sizes were calculated as absolute percentage-point changes. The summary measure for each comparison was the median effect size (MES) for all primary outcomes. Strategy effectiveness was described with weighted medians of MES. This study is registered with PROSPERO, number CRD42016046154.

Findings

We screened 216 477 citations and selected 670 reports from 337 studies of 118 strategies. Most strategies had multiple intervention components. For professional health-care providers (generally, facility-based health workers), the effects were near zero for only implementing a technology-based strategy (median MES 1·0 percentage points, IQR −2·8 to 9·9) or only providing printed information for health-care providers (1·4 percentage points, −4·8 to 6·2). For percentage outcomes, training or supervision alone typically had moderate effects (10·3–15·9 percentage points), whereas combining training and supervision had somewhat larger effects than use of either strategy alone (18·0–18·8 percentage points). Group problem solving alone showed large improvements in percentage outcomes (28·0–37·5 percentage points), but, when the strategy definition was broadened to include group problem solving alone or other strategy components, moderate effects were more typical (12·1 percentage points). Several multifaceted strategies had large effects, but multifaceted strategies were not always more effective than simpler ones. For lay health-care providers (generally, community health workers), the effect of training alone was small (2·4 percentage points). Strategies with larger effect sizes included community support plus health-care provider training (8·2–125·0 percentage points). Contextual and methodological heterogeneity made comparisons difficult, and most strategies had low quality evidence.

Interpretation

The impact of strategies to improve health-care provider practices varied substantially, although some approaches were more consistently effective than others. The breadth of the HCPPR makes its results valuable to decision makers for informing the selection of strategies to improve health-care provider practices in LMICs. These results also emphasise the need for researchers to use better methods to study the effectiveness of interventions.

Funding

Bill & Melinda Gates Foundation, CDC Foundation.

Practical recommendations for the evaluation of improvement initiatives

A lack of clear guidance for funders, evaluators and improvers on what to include in evaluation proposals can lead to evaluation designs that do not answer the questions stakeholders want to know. These evaluation designs may not match the iterative nature of improvement and may be imposed onto an initiative in a way that is impractical from the perspective of improvers and the communities with whom they work. Consequently, the results of evaluations are often controversial, and attribution remains poorly understood. Improvement initiatives are iterative, adaptive and context-specific. Evaluation approaches and designs must align with these features, specifically in their ability to consider complexity, to evolve as the initiative adapts over time and to understand the interaction with local context. Improvement initiatives often identify broadly defined change concepts and provide tools for care teams to tailor these in more detail to local conditions. Correspondingly, recommendations for evaluation are best provided as broad guidance, to be tailored to the specifics of the initiative. In this paper, we provide practical guidance and recommendations that funders and evaluators can use when developing an evaluation plan for improvement initiatives that seeks to: identify the questions stakeholders want to address; develop the initial program theory of the initiative; identify high-priority areas to measure progress over time; describe the context the initiative will be applied within; and identify experimental or observational designs that will address attribution.

Infection prevention and control training and capacity building during the Ebola epidemic in Guinea

Background

During the 2014–2016 Ebola epidemic in West Africa, a key epidemiological feature was disease transmission within healthcare facilities, indicating a need for infection prevention and control (IPC) training and support.

Methods

IPC training was provided to frontline healthcare workers (HCW) in healthcare facilities that were not Ebola treatment units, as well as to IPC trainers and IPC supervisors placed in healthcare facilities. Trainings included both didactic and hands-on components, and were assessed using pre-tests, post-tests and practical evaluations. We calculated median percent increase in knowledge.

Results

From October–December 2014, 20 IPC courses trained 1,625 Guineans: 1,521 HCW, 55 IPC trainers, and 49 IPC supervisors. Median test scores increased 40% (interquartile range [IQR]: 19–86%) among HCW, 15% (IQR: 8–33%) among IPC trainers, and 21% (IQR: 15–30%) among IPC supervisors (all P<0.0001) to post-test scores of 83%, 93%, and 93%, respectively.

Conclusions

IPC training resulted in clear improvements in knowledge and was feasible in a public health emergency setting. This method of IPC training addressed a high demand among HCW. Valuable lessons were learned to facilitate expansion of IPC training to other prefectures; this model may be considered when responding to other large outbreaks.

Quality of Case Management for Pneumonia and Diarrhea Among Children Seen at Health Facilities in Southern Malawi

Pneumonia and diarrhea are leading causes of child deaths in Malawi. Guidelines to manage childhood illnesses in resource-poor settings exist, but studies have reported low health-care worker (HCW) adherence to guidelines. We conducted a health facility survey from January to March 2015 to assess HCW management of pneumonia and diarrhea in children < 5 years of age in southern Malawi, and to determine factors associated with case management quality. Descriptive statistics and multivariable logistic regression models examined patient, HCW, and health facility factors associated with recommended pneumonia and diarrhea management, using Malawi's national guidelines as the gold standard. Of 694 surveyed children 2-59 months of age at 95 health facilities, 132 (19.0%) met survey criteria for pneumonia; HCWs gave recommended antibiotic treatment to 90 (68.2%). Of 723 children < 5 years of age, 222 (30.7%) had uncomplicated diarrhea; HCWs provided recommended treatment to 94 (42.3%). In multivariable analyses, caregivers' spontaneous report of children's symptoms was associated with recommended treatment of both pneumonia (odds ratio [OR]: 2.8, 95% confidence interval [CI]: 1.2-6.8, P = 0.023) and diarrhea (OR: 24.2, 95% CI: 6.0-97.0, P < 0001). Malaria diagnosis was negatively associated with recommended treatment (OR for pneumonia: 0.5, 95% CI: 0.2-1.0, P = 0.046; OR for diarrhea: 0.3, 95% CI: 0.1-0.6, P = 0.003). To improve quality of care, children should be assessed systematically, even when malaria is suspected. Renewed efforts to invigorate such a systematic approach, including HCW training, regular follow-up supervision, and monitoring HCW performance, are needed in Malawi.

Health worker adherence to malaria treatment guidelines at outpatient health facilities in southern Malawi following implementation of universal access to diagnostic testing

BACKGROUND

Appropriate diagnosis and treatment are essential for reducing malaria mortality. A cross-sectional outpatient health facility (HF) survey was conducted in southern Malawi from January to March 2015 to determine appropriate malaria testing and treatment practices four years after implementation of a policy requiring diagnostic confirmation before treatment.

METHODS

Enrolled patients were interviewed, examined and had their health booklet reviewed. Health workers (HWs) were asked about training, supervision and access to the 2013 national malaria treatment guidelines. HFs were assessed for malaria diagnostic and treatment capacity. Weighted descriptive analyses and logistic regression of patient, HW and HF characteristics related to testing and treatment were performed.

RESULTS

An evaluation of 105 HFs, and interviews of 150 HWs and 2342 patients was completed. Of 1427 suspect uncomplicated malaria patients seen at HFs with testing available, 1072 (75.7%) were tested, and 547 (53.2%) tested positive. Testing was more likely if patients spontaneously reported fever (odds ratio (OR) 2.6; 95% confidence interval (CI) 1.7-4.0), headache (OR 1.5; 95% CI 1.1-2.1) or vomiting (OR 2.0; 95% CI 1.0-4.0) to HWs and less likely if they reported skin problems (OR 0.4; 95% CI 0.2-0.6). Altogether, 511 (92.7%) confirmed cases and 98 (60.3%) of 178 presumed uncomplicated malaria patients (at HFs without testing) were appropriately treated, while 500 (96.6%) of 525 patients with negative tests did not receive anti-malarials. Only eight (5.7%) suspect severe malaria patients received appropriate pre-referral treatment. Appropriate treatment was more likely for presumed uncomplicated malaria patients (at HFs without testing) with elevated temperature (OR 1.5/1 °C increase; 95% CI 1.1-1.9), who reported fever to HWs (OR 5.7; 95% CI 1.9-17.6), were seen by HWs with additional supervision visits in the previous 6 months (OR 1.2/additional visit; 95% CI 1.0-1.4), or were seen by older HWs (OR 1.1/year of age; 95% CI 1.0-1.1).

CONCLUSIONS

Correct testing and treatment practices were reasonably good for uncomplicated malaria when testing was available. Pre-referral treatment for suspect severe malaria was unacceptably rare. Encouraging HWs to elicit and appropriately respond to patient symptoms may improve practices.

Predictors of health worker performance after Integrated Management of Childhood Illness training in Benin: a cohort study

Background

Correct treatment of potentially life-threatening illnesses (PLTIs) in children under 5 years, such as malaria, pneumonia, and diarrhea, can substantially reduce mortality. The Integrated Management of Childhood Illness (IMCI) strategy has been shown to improve treatment of child illnesses, but multiple studies have shown that gaps in health worker performance remain after training. To better understand factors related to health worker performance, we analyzed 9,330 patient consultations in Benin from 2001–2002, after training one of the first cohorts of 32 health workers in IMCI.

Methods

With data abstracted from patient registers specially designed for IMCI-trained health workers, we examined associations between health facility-, health worker-, and patient-level factors and 10 case-management outcomes for PLTIs.

Results

Altogether, 63.6 % of children received treatment for all their PLTIs in accordance with IMCI guidelines, and 77.8 % received life-saving treatment (i.e., clinically effective treatment, even if not exactly in accordance with IMCI guidelines). Performance of individual health workers varied greatly, from 15–88 % of patients treated correctly, on average. Multivariate regression analyses identified several factors that might have influenced case-management quality, many outside a manager’s direct control. Younger health workers significantly outperformed older ones, and infants received better care than older children. Children with danger signs, those with more complex illnesses, and those with anemia received worse care. Health worker supervision was associated with improved performance for some outcomes.

Conclusions

A variety of factors, some outside the direct control of program managers, can influence health worker practices. An understanding of these influences can help inform the development of strategies to improve performance.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-015-0910-4) contains supplementary material, which is available to authorized users.

The use of continuous surveys to generate and continuously report high quality timely maternal and newborn health data at the district level in Tanzania and Uganda

BACKGROUND

The lack of high quality timely data for evidence-informed decision making at the district level presents a challenge to improving maternal and newborn survival in low income settings. To address this problem, the EQUIP project (Expanded Quality Management using Information Power) implemented a continuous household and health facility survey for continuous feedback of data in two districts each in Tanzania and Uganda as part of a quality improvement innovation for mothers and newborns.

METHODS

Within EQUIP, continuous survey data were used for quality improvement (intervention districts) and for effect evaluation (intervention and comparison districts). Over 30 months of intervention (November 2011 to April 2014), EQUIP conducted continuous cross-sectional household and health facility surveys using 24 independent probability samples of household clusters to represent each district each month, and repeat censuses of all government health facilities. Using repeat samples in this way allowed data to be aggregated at six four-monthly intervals to track progress over time for evaluation, and for continuous feedback to quality improvement teams in intervention districts.In both countries, one continuous survey team of eight people was employed to complete approximately 7,200 household and 200 facility interviews in year one. Data were collected using personal digital assistants. After every four months, routine tabulations of indicators were produced and synthesized to report cards for use by the quality improvement teams.

RESULTS

The first 12 months were implemented as planned. Completion of household interviews was 96% in Tanzania and 91% in Uganda. Indicators across the continuum of care were tabulated every four months, results discussed by quality improvement teams, and report cards generated to support their work.

CONCLUSIONS

The EQUIP continuous surveys were feasible to implement as a method to continuously generate and report on demand and supply side indicators for maternal and newborn health; they have potential to be expanded to include other health topics. Documenting the design and implementation of a continuous data collection and feedback mechanism for prospective description, quality improvement, and evaluation in a low-income setting potentially represents a new paradigm that places equal weight on data systems for course correction, as well as evaluation.

An evaluation of methods for assessing the quality of case management for inpatients with malaria in Benin

To improve healthcare quality for hospitalized patients with malaria in Benin, a feasible and valid evaluation method is needed. Because observation of inpatients is challenging, chart abstraction is an attractive option. However, the quality of inpatient charts is unknown. We employed three methods in five hospitals to assess 11 signs of malaria and severe disease: 1) chart abstraction (probability sample of inpatients), 2) chart abstraction compared to interviews of inpatients and health workers (HWs), and 3) abstraction from charts of recently discharged inpatients compared to interviews with HWs. Method 1 showed that of 473 malaria signs (from 43 charts), 178 (38%, 95% confidence interval 24-51%) were documented. Method 2 showed that 96% (45 of 47) of documented signs were valid. Method 3 suggests that 65% (36 of 55) of non-documented signs were assessed (but not documented) by HWs. Chart abstraction was feasible and documented data were valid, but results should be interpreted cautiously in consideration of low levels of documentation.

The effect of performance indicator category on estimates of intervention effectiveness

BACKGROUND

A challenge for systematic reviews on improving health worker performance is that included studies often use different performance indicators, and the validity of comparing interventions with different indicators is unclear. One potential solution is to adjust comparisons by indicator category, with categories based on steps of the case-management process that can be easily recognized (assessment of symptoms, treatment etc.) and that might require different levels of effort to bring about improvements. However, this approach would only be useful if intervention effect sizes varied by indicator category. To explore this approach, studies were analyzed that evaluated the Integrated Management of Childhood Illness (IMCI) strategy.

METHODS

Performance indicators were grouped into four categories: patient assessment, diagnosis, treatment and counseling. An effect size of IMCI was calculated for each indicator. Linear regression modeling was used to test for differences among the mean effect sizes of the indicator categories.

RESULTS

Six studies were included, with data from 3136 ill child consultations. Mean effect sizes for 63 assessment indicators, 12 diagnosis indicators, 31 treatment indicators and 34 counseling indicators were 50.9 percentage-points (%-points), 44.7, 36.5 and 46.6%-points, respectively. After adjusting for baseline indicator value, compared with the assessment mean effect size, the diagnosis mean was 7.3%-points lower (P = 0.23), the treatment mean was 15.2%-points lower (P = 0.0004) and the counseling mean was 12.9%-points lower (P = 0.0027).

CONCLUSION

Adjusting the results of systematic reviews for indicator category and baseline indicator value might be useful for improving the validity of intervention comparisons.

Trends in health worker performance after implementing the Integrated Management of Childhood Illness strategy in Benin

OBJECTIVE

Training health workers to use Integrated Management of Childhood Illness (IMCI) guidelines can improve care for ill children in outpatient settings in developing countries. However, even after IMCI training, important performance gaps exist. One potential reason is that the effect of training can rapidly wane. Our aim was to determine if the performance of IMCI-trained health workers deteriorated over 3 years.

METHODS

We studied two departments in Benin. First, we performed a record review of 32 IMCI-trained health workers during the first year of IMCI implementation (2001-2002). Second, we analysed data from cross-sectional health facility surveys from 2001 to 2004 that represented the entire study area. Primary outcomes were the proportion of children under 5 years old with potentially life-threatening illnesses who received either recommended or adequate treatment, and among all children, an index of overall guideline adherence. Secondary outcomes reflected the treatment of individual diseases. Outcomes were calculated monthly, and time trends were evaluated with regression modelling.

RESULTS

The record review included 9393 consultations, and the surveys included 411 consultations performed by 105 health workers. For both data sources, performance trends were essentially flat for nearly all outcomes. Absolute levels of performance revealed substantial performance gaps.

CONCLUSIONS

We found no evidence that performance declined over 3 years after IMCI training. However, important performance gaps found immediately after IMCI training persisted and should be addressed.

Evaluating health worker performance in Benin using the simulated client method with real children

BACKGROUND

The simulated client (SC) method for evaluating health worker performance utilizes surveyors who pose as patients to make surreptitious observations during consultations. Compared to conspicuous observation (CO) by surveyors, which is commonly done in developing countries, SC data better reflect usual health worker practices. This information is important because CO can cause performance to be better than usual. Despite this advantage of SCs, the method's full potential has not been realized for evaluating performance for pediatric illnesses because real children have not been utilized as SCs. Previous SC studies used scenarios of ill children that were not actually brought to health workers. During a trial that evaluated a quality improvement intervention in Benin (the Integrated Management of Childhood Illness [IMCI] strategy), we conducted an SC survey with adult caretakers as surveyors and real children to evaluate the feasibility of this approach and used the results to assess the validity of CO.

METHODS

We conducted an SC survey and a CO survey (one right after the other) of health workers in the same 55 health facilities. A detailed description of the SC survey process was produced. Results of the two surveys were compared for 27 performance indicators using logistic regression modeling.

RESULTS

SC and CO surveyors observed 54 and 185 consultations, respectively. No serious problems occurred during the SC survey. Performance levels measured by CO were moderately higher than those measured by SCs (median CO - SC difference = 16.4 percentage-points). Survey differences were sometimes much greater for IMCI-trained health workers (median difference = 29.7 percentage-points) than for workers without IMCI training (median difference = 3.1 percentage-points).

CONCLUSION

SC surveys can be done safely with real children if appropriate precautions are taken. CO can introduce moderately large positive biases, and these biases might be greater for health workers exposed to quality improvement interventions.

TRIAL NUMBER

http://clinicaltrials.gov Identifier NCT00510679.

The effect of mobile phone text-message reminders on Kenyan health workers' adherence to malaria treatment guidelines: a cluster randomised trial

BACKGROUND

Health workers' malaria case-management practices often differ from national guidelines. We assessed whether text-message reminders sent to health workers' mobile phones could improve and maintain their adherence to treatment guidelines for outpatient paediatric malaria in Kenya.

METHODS

From March 6, 2009, to May 31, 2010, we did a cluster-randomised controlled trial at 107 rural health facilities in 11 districts in coastal and western Kenya. With a computer-generated sequence, health facilities were randomly allocated to either the intervention group, in which all health workers received text messages on their personal mobile phones on malaria case-management for 6 months, or the control group, in which health workers did not receive any text messages. Health workers were not masked to the intervention, although patients were unaware of whether they were in an intervention or control facility. The primary outcome was correct management with artemether-lumefantrine, defined as a dichotomous composite indicator of treatment, dispensing, and counselling tasks concordant with Kenyan national guidelines. The primary analysis was by intention to treat. The trial is registered with Current Controlled Trials, ISRCTN72328636.

FINDINGS

119 health workers received the intervention. Case-management practices were assessed for 2269 children who needed treatment (1157 in the intervention group and 1112 in the control group). Intention-to-treat analysis showed that correct artemether-lumefantrine management improved by 23·7 percentage-points (95% CI 7·6-40·0; p=0·004) immediately after intervention and by 24·5 percentage-points (8·1-41·0; p=0·003) 6 months later.

INTERPRETATION

In resource-limited settings, malaria control programmes should consider use of text messaging to improve health workers' case-management practices.

FUNDING

The Wellcome Trust.

Impact of a malaria-control project in Benin that included the integrated management of childhood illness strategy

OBJECTIVES

To estimate the impact of the Integrated Management of Childhood Illness (IMCI) strategy on early-childhood mortality, we evaluated a malaria-control project in Benin that implemented IMCI and promoted insecticide-treated nets (ITNs).

METHODS

We conducted a before-and-after intervention study that included a nonrandomized comparison group. We used the preceding birth technique to measure early-childhood mortality (risk of dying before age 30 months), and we used health facility surveys and household surveys to measure process indicators.

RESULTS

Most process indicators improved in the area covered by the intervention. Notably, because ITNs were also promoted in the comparison area children's ITN use increased by about 20 percentage points in both areas. Regarding early-childhood mortality, the trend from baseline (1999-2001) to follow-up (2002-2004) for the intervention area (13.0% decrease; P < .001) was 14.1% (P < .001) lower than was the trend for the comparison area (1.3% increase; P = .46).

CONCLUSIONS

Mortality decreased in the intervention area after IMCI and ITN promotion. ITN use increased similarly in both study areas, so the mortality impact of ITNs in the 2 areas might have canceled each other out. Thus, the mortality reduction could have been primarily attributable to IMCI's effect on health care quality and care-seeking.

Does shortening the training on Integrated Management of Childhood Illness guidelines reduce its effectiveness? A systematic review

OBJECTIVE

Implementation of the Integrated Management of Childhood Illness (IMCI) strategy with an 11-day training course for health workers improves care for ill children in outpatient settings in developing countries. The 11-day course duration is recommended by the World Health Organization, which developed IMCI. Our aim was to determine if shortening the training (to reduce cost) reduces its effectiveness.

METHODS

We conducted a systematic review to compare IMCI's effectiveness with standard training (duration ≥ 11 days) versus shortened training (5-10 days). Studies were identified from a search of MEDLINE, two existing systematic reviews, and by contacting investigators. We included published or unpublished studies that evaluated IMCI's effectiveness in developing countries and reported quantitative measures of health worker practices related to managing ill children under 5 years old in public or private health facilities. Summary measures were the median of effect sizes for all outcomes from a given study, and the percentage of patients needing oral antimicrobials or rehydration who were treated according to IMCI guidelines.

FINDINGS

Twenty-nine studies were included. Direct comparisons from three studies showed little difference between standard and shortened training. Indirect comparisons from 26 studies revealed that effect sizes for standard training versus no IMCI were greater than shortened training versus no IMCI. Across all comparisons, differences ranged from -3 to +23 percentage-points, and our best estimate was a 2 to 16 percentage-point advantage for standard training. No result was statistically significant. After IMCI training (of any duration), 34% of ill children needing oral antimicrobials or rehydration were not receiving these treatments according to IMCI guidelines.

CONCLUSIONS

Based on limited evidence, standard IMCI training seemed more effective than shortened training, although the difference might be small. As sizable performance gaps often existed after IMCI training, countries should consider implementing other interventions to support health workers after training, regardless of training duration.

Quality of malaria case management at outpatient health facilities in Angola

Background

Angola's malaria case-management policy recommends treatment with artemether-lumefantrine (AL). In 2006, AL implementation began in Huambo Province, which involved training health workers (HWs), supervision, delivering AL to health facilities, and improving malaria testing with microscopy and rapid diagnostic tests (RDTs). Implementation was complicated by a policy that was sometimes ambiguous.

Methods

Fourteen months after implementation began, a cross-sectional survey was conducted in 33 outpatient facilities in Huambo Province to assess their readiness to manage malaria and the quality of malaria case-management for patients of all ages. Consultations were observed, patients were interviewed and re-examined, and HWs were interviewed.

Results

Ninety-three HWs and 177 consultations were evaluated, although many sampled consultations were missed. All facilities had AL in-stock and at least one HW trained to use AL and RDTs. However, anti-malarial stock-outs in the previous three months were common, clinical supervision was infrequent, and HWs had important knowledge gaps. Except for fever history, clinical assessments were often incomplete. Although testing was recommended for all patients with suspected malaria, only 30.7% of such patients were tested. Correct testing was significantly associated with caseloads < 25 patients/day (odds ratio: 18.4; p < 0.0001) and elevated patient temperature (odds ratio: 2.5 per 1°C increase; p = 0.007). Testing was more common among AL-trained HWs, but the association was borderline significant (p = 0.072). When the malaria test was negative, HWs often diagnosed patients with malaria (57.8%) and prescribed anti-malarials (60.0%). Sixty-six percent of malaria-related diagnoses were correct, 20.1% were minor errors, and 13.9% were major (potentially life-threatening) errors. Only 49.0% of malaria treatments were correct, 5.4% were minor errors, and 45.6% were major errors. HWs almost always dosed AL correctly and gave accurate dosing instructions to patients; however, other aspects of counseling needed improvement.

Conclusion

By late-2007, substantial progress had been made to implement the malaria case-management policy in a setting with weak infrastructure. However, policy ambiguities, under-use of malaria testing, and distrust of negative test results led to many incorrect malaria diagnoses and treatments. In 2009, Angola published a policy that clarified many issues. As problems identified in this survey are not unique to Angola, better strategies for improving HW performance are urgently needed.

The rise and fall of supervision in a project designed to strengthen supervision of Integrated Management of Childhood Illness in Benin

OBJECTIVE

In developing countries, supervision is a widely recognized strategy for improving health worker performance; and anecdotally, maintaining regular, high-quality supervision is difficult. However, remarkably little research has explored in depth why supervision is so challenging.

METHODS

In the context of a trial to improve health worker adherence to Integrated Management of Childhood Illness (IMCI) guidelines and strengthen supervision in southeastern Benin, we used record reviews, focus group discussions, key informant interviews, and cross-sectional surveys to examine the supervision process.

FINDINGS

Initially, little IMCI supervision occurred. The frequency increased substantially after implementing a series of workshops, but then deteriorated. Quantitative and qualitative data revealed obstacles to supervision at multiple levels of the health system. Based on supervisors' opinions, the main problems were: poor coordination; inadequate management skills and ineffective management teams; a lack of motivation; problems related to decentralization; health workers sometimes resisting IMCI implementation; and less priority given to IMCI supervision because of incentives for non-supervision activities, a lack of leadership, and an expectation of integrated supervision. To this list, based on our observations, we add: the increasing supervision workload, time required for non-supervision activities, project interventions not always being implemented as planned, and the loss of particularly effective supervisors. In terms of correctly completing steps of the supervision process, the quality of supervision was generally good.

CONCLUSIONS

Managers should monitor supervision, understand the evolving influences on supervision, and use their resources and authority to both promote supervision and remove impediments to supervision. Support from leaders can be crucial, thus donors and politicians should help make supervision a true priority. As with front-line clinicians, supervisors are health workers who need support. We emphasize the importance of research to identify effective and affordable strategies for improving supervision frequency and quality. (ClinicalTrials.gov number NCT00510679).

Caution is required when using health facility-based data to evaluate the health impact of malaria control efforts in Africa

The global health community is interested in the health impact of the billions of dollars invested to fight malaria in Africa. A recent publication used trends in malaria cases and deaths based on health facility records to evaluate the impact of malaria control efforts in Rwanda and Ethiopia. Although the authors demonstrate the use of facility-based data to estimate the impact of malaria control efforts, they also illustrate several pitfalls of such analyses that should be avoided, minimized, or actively acknowledged. A critique of this analysis is presented because many country programmes and donors are interested in evaluating programmatic impact with facility-based data. Key concerns related to: 1) clarifying the objective of the analysis; 2) data validity; 3) data representativeness; 4) the exploration of trends in factors that could influence malaria rates and thus confound the relationship between intervention scale-up and the observed changes in malaria outcomes; 5) the analytic approaches, including small numbers of patient outcomes, selective reporting of results, and choice of statistical and modeling methods; and 6) internal inconsistency on the strength and interpretation of the data. In conclusion, evaluations of malaria burden reduction using facility-based data could be very helpful, but those data should be collected, analysed, and interpreted with care, transparency, and a full recognition of their limitations.

Improving pneumonia case-management in Benin: a randomized trial of a multi-faceted intervention to support health worker adherence to Integrated Management of Childhood Illness guidelines

BACKGROUND

Pneumonia is a leading cause of death among children under five years of age. The Integrated Management of Childhood Illness strategy can improve the quality of care for pneumonia and other common illnesses in developing countries, but adherence to these guidelines could be improved. We evaluated an intervention in Benin to support health worker adherence to the guidelines after training, focusing on pneumonia case management.

METHODS

We conducted a randomized trial. After a health facility survey in 1999 to assess health care quality before Integrated Management of Childhood Illness training, health workers received training plus either study supports (job aids, non-financial incentives and supervision of workers and supervisors) or "usual" supports. Follow-up surveys were conducted in 2001, 2002 and 2004. Outcomes were indicators of health care quality for Integrated Management-defined pneumonia. Further analyses included a graphical pathway analysis and multivariable logistic regression modelling to identify factors influencing case-management quality.

RESULTS

We observed 301 consultations of children with non-severe pneumonia that were performed by 128 health workers in 88 public and private health facilities. Although outcomes improved in both intervention and control groups, we found no statistically significant difference between groups. However, training proceeded slowly, and low-quality care from untrained health workers diluted intervention effects. Per-protocol analyses suggested that health workers with training plus study supports performed better than those with training plus usual supports (20.4 and 19.2 percentage-point improvements for recommended treatment [p=0.08] and "recommended or adequate" treatment [p=0.01], respectively). Both groups tended to perform better than untrained health workers. Analyses of treatment errors revealed that incomplete assessment and difficulties processing clinical findings led to missed pneumonia diagnoses, and missed diagnoses led to inadequate treatment. Increased supervision frequency was associated with better care (odds ratio for recommended treatment=2.1 [95% confidence interval: 1.13.9] per additional supervisory visit).

CONCLUSION

Integrated Management of Childhood Illness training was useful, but insufficient, to achieve high-quality pneumonia case management. Our study supports led to additional improvements, although large gaps in performance still remained. A simple graphical pathway analysis can identify specific, common errors that health workers make in the case-management process; this information could be used to target quality improvement activities, such as supervision (ClinicalTrials.gov number NCT00510679).

Effect of malaria rapid diagnostic tests on the management of uncomplicated malaria with artemether-lumefantrine in Kenya: a cluster randomized trial

Shortly after Kenya introduced artemether-lumefantrine (AL) for first-line treatment of uncomplicated malaria, we conducted a pre-post cluster randomized controlled trial to assess the effect of providing malaria rapid diagnostic tests (RDTs) on recommended treatment (patients with malaria prescribed AL) and overtreatment (patients without malaria prescribed AL) in outpatients >/= 5 years old. Sixty health facilities were randomized to receive either RDTs plus training, guidelines, and supervision (TGS) or TGS alone. Of 1,540 patients included in the analysis, 7% had uncomplicated malaria. The provision of RDTs coupled with TGS emphasizing AL use only after laboratory confirmation of malaria reduced recommended treatment by 63%-points (P = 0.04), because diagnostic test use did not change (-2%-points), but health workers significantly reduced presumptive treatment with AL for patients with a clinical diagnosis of malaria who did not undergo testing (-36%-points; P = 0.03). Health workers generally adhered to RDT results when prescribing AL: 88% of RDT-positive and 9% of RDT-negative patients were treated with AL, respectively. Overtreatment was low in both arms and was not significantly reduced by the provision of RDTs (-12%-points, P = 0.30). RDTs could potentially improve malaria case management, but we urgently need to develop more effective strategies for implementing guidelines before large scale implementation.

Potential of integrated continuous surveys and quality management to support monitoring, evaluation, and the scale-up of health interventions in developing countries

Well-funded initiatives are challenging developing countries to increase health intervention coverage and show impact. Despite substantial resources, however, major obstacles include weak health systems, a lack of reasonably accurate monitoring data, and inadequate use of data for managing programs. This report discusses how integrated continuous surveys and quality management (I-Q), which are well-recognized approaches in wealthy countries, could support intervention scale-up, monitoring and evaluation, quality control for commodities, capacity building, and implementation research in low-resource settings. Integrated continuous surveys are similar to existing national cross-sectional surveys of households and health facilities, except data are collected over several years by permanent teams, and most results are reported monthly at the national, province, and district levels. Quality management involves conceptualizing work as processes, involving all workers in quality improvement, monitoring quality, and teams that improve quality with "plan-do-study-act" cycles. Implementing and evaluating I-Q in a low-income country would provide critical information on the value of this approach.

A multifaceted intervention to improve health worker adherence to integrated management of childhood illness guidelines in Benin

OBJECTIVES

We evaluated an intervention to support health workers after training in Integrated Management of Childhood Illness (IMCI), a strategy that can improve outcomes for children in developing countries by encouraging workers' use of evidence-based guidelines for managing the leading causes of child mortality.

METHODS

We conducted a randomized trial in Benin. We administered a survey in 1999 to assess health care quality before IMCI training. Health workers then received training plus either study supports (job aids, nonfinancial incentives, and supervision of workers and supervisors) or usual supports. Follow-up surveys conducted in 2001 to 2004 assessed recommended treatment, recommended or adequate treatment, and an index of overall guideline adherence.

RESULTS

We analyzed 1244 consultations. Performance improved in both intervention and control groups, with no significant differences between groups. However, training proceeded slowly, and low-quality care from health workers without IMCI training diluted intervention effects. Per-protocol analyses revealed that workers with IMCI training plus study supports provided better care than did those with training plus usual supports (27.3 percentage-point difference for recommended treatment; P < .05), and both groups outperformed untrained workers.

CONCLUSIONS

IMCI training was useful but insufficient. Relatively inexpensive supports can lead to additional improvements.

Assessing the validity of health facility-based data on insecticide-treated bednet possession and use: comparison of data collected via health facility and household surveys--Lindi region and Rufiji district, Tanzania, 2005

Objective To assess the validity of health facility (HF)-based data on bednet and insecticide-treated bednet possession and use by children <5 years old. Methods We compared estimates based on data collected via HF surveys of under-5s attending well-child visits (e.g. immunizations) and sick-child visits vs. representative household surveys (a 'gold standard' method for measuring insecticide-treated net coverage). In Lindi region, Tanzania, we collected contemporaneous data on 637 under-5s via a HF survey (444 well-child visits and 193 sick-child visits), and on 305 households with an under-5 (including 354 children) via a household survey. In Rufiji district, Tanzania, we collected contemporaneous data on 1433 under-5s via a HF survey (911 well-child visits and 522 sick-child visits), and on 328 households with an under-5 (including 455 children) via a household survey. Results Possession of bednets by households with an under-5 was similar using HF data and household data in both Lindi region and Rufiji district. However, reported use of bednets was significantly higher in HF data than household data in both Lindi and Rufiji, as was reported use of insecticide-treated bednets. HF-based data accurately estimated community-level bednet possession in households with an under-5, but overestimated community-level bednet use by 9-35% and insecticide-treated bednet use by 15-21%. Conclusions Information bias rather than selection bias appears to be a key cause for the overestimation of bednet and insecticide-treated bednet use (e.g. social desirability bias: caretakers of under-5s attending health facilities might be more likely to report using bednets and insecticide-treated bednets). Additional studies of the validity, cost and utility of HF-based data to monitor insecticide-treated bednet use are needed before recommending this monitoring strategy for widespread use. Overestimating insecticide-treated bednet use could lead to inappropriate public health actions and missed opportunities for achieving local and global public health goals.

Viewpoint: evaluating the impact of malaria control efforts on mortality in sub-Saharan Africa

OBJECTIVE

To describe an approach for evaluating the impact of malaria control efforts on malaria-associated mortality in sub-Saharan Africa, where disease-specific mortality trends usually cannot be measured directly and most malaria deaths occur among young children.

METHODS

Methods for evaluating changes in malaria-associated mortality are examined; advantages and disadvantages are presented.

RESULTS

All methods require a plausibility argument-i.e., an assumption that mortality reductions can be attributed to programmatic efforts if improvements are found in steps of the causal pathway between intervention scale-up and mortality trends. As different methods provide complementary information, they can be used together. We recommend following trends in the coverage of malaria control interventions, other factors influencing childhood mortality, malaria-associated morbidity (especially anaemia), and all-cause childhood mortality. This approach reflects decreases in malaria's direct and indirect mortality burden and can be examined in nearly all countries. Adding other information can strengthen the plausibility argument: trends in indicators of malaria transmission, information from demographic surveillance systems and sentinel sites where malaria diagnostics are systematically used, and verbal autopsies linked to representative household surveys. Health facility data on malaria deaths have well-recognized limitations; however, in specific circumstances, they could produce reliable trends. Model-based predictions can help describe changes in malaria-specific burden and assist with program management and advocacy.

CONCLUSIONS

Despite challenges, efforts to reduce malaria-associated mortality in Africa can be evaluated with trends in malaria intervention coverage and all-cause childhood mortality. Where there are resources and interest, complementary data on malaria morbidity and malaria-specific mortality could be added.

Predictions of the impact of malaria control efforts on all-cause child mortality in sub-Saharan Africa

All-cause childhood mortality (ACCM) has been recommended as an indicator for evaluating the impact of malaria control efforts in parts of Africa where the malaria burden is high and vital registration systems are weak. As ACCM is not malaria-specific, we explored the relationship between ACCM and malaria mortality. First, we show that if malaria mortality is reduced by 50% in populations exposed to high-intensity malaria transmission, ACCM is predicted to decrease by approximately 17% (plausible values range from 12% to 25%). Second, if malaria interventions are scaled-up from very low (2%) to reasonably high coverage levels (70%), epidemiologic models predict that ACCM would decrease by approximately 17% or 18% (precision estimate: 15-19%). These results suggest that if malaria interventions are scaled-up to reach or exceed 70% coverage, it is plausible that a goal of reducing malaria mortality by 50% could be achieved. It is also likely that a pair of "typical"-size mortality surveys could detect this ACCM change as being statistically significant. Although existing models have important limitations, they could be improved by incorporating empirical results during scale-up of multiple interventions and by adding precision estimates and sensitivity analyses.

Achieving child survival goals: potential contribution of community health workers

There is renewed interest in the potential contribution of community health workers to child survival. Community health workers can undertake various tasks, including case management of childhood illnesses (eg, pneumonia, malaria, and neonatal sepsis) and delivery of preventive interventions such as immunisation, promotion of healthy behaviour, and mobilisation of communities. Several trials show substantial reductions in child mortality, particularly through case management of ill children by these types of community interventions. However, community health workers are not a panacea for weak health systems and will need focussed tasks, adequate remuneration, training, supervision, and the active involvement of the communities in which they work. The introduction of large-scale programmes for community health workers requires evaluation to document the impact on child survival and cost effectiveness and to elucidate factors associated with success and sustainability.

Care takers' recall of Integrated Management of Childhood Illness counselling messages in Benin

A key goal of the Integrated Management of Childhood Illness (IMCI) strategy is to improve the management of childhood illness at health facilities. IMCIguidelines contain many counselling messages, and as it is not known how well caretakers recall these messages, we studied caretakers' recall of IMCI messages when given under ideal conditions. At a clinic in Benin, a study clinician performed counselling and confirmed caretakers'comprehension of all messages. Caretakers were randomly assigned to be interviewed either immediately after the consultation or a day later. Recall was assessed with general and focused open-ended questions. Recall was assessed for 55 caretakers, 29.1% of whom were literate. Caretakers received 3-75 messages (mean = 38.7). The mean percentage of messages recalled was 89.7% immediately after the consultation and 81.9% one day later. These results support IMCI's recommendation that health workers should verify caretakers' comprehension by asking caretakers to repeat counselling messages during consultations.

Distribution of free untreated bednets bundled with insecticide via an integrated child health campaign in Lindi Region, Tanzania: lessons for future campaigns

Use of insecticide-treated bednets (ITNs) to prevent malaria remains low, and effective distribution strategies are needed. An integrated child health campaign with free distribution of 162,254 untreated bednets bundled with insecticide, measles vaccination, vitamin A, and mebendazole for children < 5 years old ("under-5s") was conducted in Lindi Region, Tanzania. We conducted a representative household survey 3 months after the campaign. Altogether, 574 households with 354 under-5s were visited. In households with an under-5, possession of bednets and ITNs increased from 60.9% to 90.7% (P < 0.001) and from 16.5% to 37.3% (P < 0.001), respectively. Increases occurred in all wealth quintiles and equity improved. Reported bednet and ITN use the previous night among under-5s was 46.3% and 21.5%, respectively. Integrated campaigns rapidly and equitably increase bednet possession and use meriting continued large-scale implementation. However, our study found that bednets were rarely treated; thus, future campaigns should provide factory-treated long-lasting ITNs. Low ITN use underscores the need for further efforts to increase use after campaigns.

Quality of treatment for febrile illness among children at outpatient facilities in sub-Saharan Africa

For the prompt and effective management of malaria cases (a key strategy for reducing the enormous burden of the disease), healthworkers must prescribe antimalarial drugs according to evidence-based guidelines. In sub-Saharan Africa, the guidelines for use in outpatient settings generally recommend that febrile illness in children should be suspected to be malaria and be treated with an antimalarial drug. The quality of treatment offered to febrile children at outpatient facilities in this region has now been investigated in a literature review. The results of five methodologically comparable studies were also used to explore the determinants of malaria-treatment practices. The quality of treatment prescribed to febrile children was found to have been generally sub-optimal, with low levels of adherence to national guidelines, the frequent selection of non-recommended antimalarials, and the use of incorrect dosages. Several factors might be to responsible for these shortcomings. Although interventions such as the Integrated Management of Childhood Illness (IMCI) strategy can lead to improvements, a better understanding of the practices of the healthworkers responsible for treating febrile children will be needed before treatment is made much better. The failure to provide treatment of good quality will become an increasingly important problem as antimalarial policies involving drugs with more complex dosing regimens, such as artemisinin-based combination therapies (ACT), are implemented. If the malaria burden in Africa is to be greatly reduced, the deployment of ACT must be accompanied by interventions to ensure the correct treatment of children at the point of care. Some interventions, such as IMCI, can improve the treatment of not only malaria but also other potentially life-threatening illnesses.

Predictors of treatment error for children with uncomplicated malaria seen as outpatients in Blantyre district, Malawi

BACKGROUND

Past studies have shown that health workers in developing countries often do not follow clinical guidelines, though few studies have explored with appropriate methods why errors occur. To develop interventions that improve health worker performance, factors affecting treatment practices must be better understood.

METHODS

We analysed data from a health facility survey in Blantyre District, Malawi, in which health workers were observed treating ill children, and then children were independently re-examined by 'gold-standard' study clinicians. The analysis was limited to children with uncomplicated malaria (defined according to Malawi's guidelines as fever or anaemia without signs of severe illness), and a treatment error was defined as failure to treat with an effective antimalarial.

RESULTS

Twenty-eight health workers and 349 ill-child consultations were evaluated; 247 (70.8%) children were treated with an effective antimalarial, and 102 (29.2%) were subject to treatment error. Logistic regression analysis revealed that in-service malaria training was not associated with treatment quality (univariate odds ratio (OR) = 1.16, 95% confidence interval (CI): 0.46-2.93); whereas acute respiratory infections training was associated with making an error (adjusted OR (aOR) = 2.42, 95% CI: 1.23-4.76). High fever and chief complaint of fever were associated with fewer errors (aOR = 0.25, 95% CI: 0.10-0.60 and aOR = 0.25, 95% CI: 0.13-0.48, respectively). Errors were more likely to occur in consultations starting before 1 p.m. (aOR = 1.88, 95% CI: 1.07-3.31). Supervision was not associated with better treatment quality.

CONCLUSIONS

These results suggest that the disease-specific training and supervision, performed before the survey, did not lead to long-term improvements in health care quality. Furthermore, case management training for one specific disease may have worsened quality of care for another disease. These results support integration of guidelines for multiple conditions. Interventions should be evaluated for unintended negative effects on overall quality of care.

The burden of malaria mortality among African children in the year 2000

BACKGROUND

Although malaria is a leading cause of child deaths, few well-documented estimates of its direct and indirect burden exist. Our objective was to estimate the number of deaths directly attributable to malaria among children <5 years old in sub-Saharan Africa for the year 2000.

METHODS

We divided the population into six sub-populations and, using results of studies identified in a literature review, estimated a malaria mortality rate for each sub-population. Malaria deaths were estimated by multiplying each sub-population by its corresponding rate. Sensitivity analyses were performed to assess the impact of varying key assumptions.

RESULTS

The literature review identified 31 studies from 14 countries in middle Africa and 17 studies and reports from four countries in southern Africa. In 2000, we estimated that approximately 100 million children lived in areas where malaria transmission occurs and that 803 620 (precision estimate: 705 821-901 418) children died from the direct effects of malaria. For all of sub-Saharan Africa, including populations not exposed to malaria, malaria accounted for 18.0% (precision estimate: 15.8-20.2%) of child deaths. These estimates were sensitive to extreme assumptions about the causes of deaths with no known cause.

CONCLUSIONS

These estimates, based on the best available data and methods, clearly demonstrate malaria's enormous mortality burden. We emphasize that these estimates are an approximation with many limitations and that the estimates do not account for malaria's large indirect burden. We describe information needs that, if filled, might improve the validity of future estimates.

Analysis of deaths with an unknown cause in epidemiologic analyses of mortality burden

OBJECTIVE

To describe options for analysing deaths with an unknown cause, which often occur in community-based studies that are used to estimate disease-specific mortality burden and trends in low-income countries.

METHODS

Mathematical formulae were derived that accommodate deaths with an unknown cause for the disease-specific mortality rate, proportion of deaths attributable to the disease and all-cause mortality rate. Seven specific options are presented, including example calculations from a study of childhood malaria mortality in The Gambia. An algorithm is proposed to help make decisions on analysing deaths with an unknown cause.

RESULTS

In the Gambian study, 25.2% of deaths had an unknown cause. Three options would result in 23.6% (minimum), 48.8% (maximum) and 28.7% (probably the best estimate) of deaths attributed to malaria. The best analysis option depends on the disease of interest: diseases for which the diagnostic method has high sensitivity and specificity (e.g., measles, neonatal tetanus) are best analysed assuming that deaths with an unknown cause never have this cause, while diseases for which specificity and/or sensitivity is low (e.g., malaria) are likely to account for some proportion of deaths with an unknown cause.

CONCLUSIONS

The most important aspects of analysing deaths with unknown cause are choosing appropriate assumptions, describing them explicitly and performing a sensitivity analysis. Studies of causes of death should report several key pieces of information on deaths with unknown cause to aid interpretation.

Effect of the Integrated Management of Childhood Illness strategy on health care quality in Morocco

OBJECTIVE

To evaluate an intervention to promote health workers' use of the World Health Organization's Integrated Management of Childhood Illness clinical guidelines and to identify other factors influencing quality of care received by Moroccan children.

SETTING

Public outpatient health facilities.

DESIGN

Cross-sectional survey of consultations with sick children under 5 years old at facilities in two intervention and two comparison provinces in April 2000 (6-12 months after intervention). Consultations were observed, children's caretakers and health workers were interviewed, and children were re-examined by a 'gold standard' study clinician.

STUDY PARTICIPANTS

Probability sample of 467 consultations (97.9% participation) performed by 101 health workers in 62 facilities.

INTERVENTION

Health workers received in-service training with job aids and a follow-up visit with feedback 4-6 weeks after training.

MAIN OUTCOME MEASURES

Index of overall guideline adherence (mean percentage of recommended tasks that were done per child) and the percentage of children requiring antibiotics correctly prescribed antibiotics.

RESULTS

Quality of care was better in intervention provinces, according to the adherence index (79.7 versus 19.5%, P < 0.0001), correct prescription of antibiotics (60.8 versus 31.3%, P = 0.0013), and other indicators. Multivariate modeling revealed a variety of factors significantly associated with quality, including health worker attributes (pre-service training, residence in government-subsidized housing, sex, and opinions) and child/consultation attributes (child's age and temperature, number of chief complaints, and caretaker type).

CONCLUSIONS

Exposure to the intervention was strongly associated with adherence to the guidelines and correct prescribing of antibiotics 6-12 months after exposure. Many other factors may influence health worker performance.

Treatment of paediatric malaria during a period of drug transition to artemether-lumefantrine in Zambia: cross sectional study

OBJECTIVE

To evaluate treatment practices for uncomplicated malaria after the policy change from chloroquine to sulfadoxine-pyrimethamine and to artemether-lumefantrine in Zambia.

DESIGN

Cross sectional survey.

SETTING

Outpatient departments of all government and mission facilities in four districts in Zambia.

PARTICIPANTS

944 children with uncomplicated malaria seen by 103 health workers at 94 health facilities.

MAIN OUTCOME MEASURES

Antimalarial prescriptions in accordance with national guidelines and influence of factors on health workers' decision to prescribe artemether-lumefantrine.

RESULTS

Artemether-lumefantrine, sulfadoxine-pyrimethamine, and chloroquine were available, respectively, at 48 (51%), 94 (100%), and 71 (76%) of the 94 facilities. Of 944 children with uncomplicated malaria, only one child (0.1%) received chloroquine. Among children weighing less than 10 kg, sulfadoxine-pyrimethamine was commonly prescribed in accordance with guidelines (439/550, 79.8%). Among the children weighing 10 kg or more, sulfadoxine-pyrimethamine was commonly prescribed (266/394, 68%), whereas recommended artemether-lumefantrine was prescribed for only 42/394 (11%) children. Among children weighing 10 kg or more seen at facilities where artemether-lumefantrine was available, the same pattern was observed: artemether-lumefantrine was prescribed for only 42/192 (22%) children and sulfadoxine-pyrimethamine remained the drug of choice (103/192, 54%). Programmatic activities such as in-service training and provision of job aids did not seem to influence the prescribing of artemether with lumefantrine.

CONCLUSION

Although the use of chloroquine for uncomplicated malaria was successfully discontinued in Zambia, the change of drug policy towards artemether-lumefantrine does not necessarily translate into adequate use of this drug at the point of care.