Measuring coverage in MNCH: a prospective validation study in Pakistan and Bangladesh on measuring correct treatment of childhood pneumonia

mHealth Series: mHealth project in Zhao County, rural China - Description of objectives, field site and methods

Background

We set up a collaboration between researchers in China and the UK that aimed to explore the use of mHealth in China. This is the first paper in a series of papers on a large mHealth project part of this collaboration. This paper included the aims and objectives of the mHealth project, our field site, and the detailed methods of two studies.

Field site

The field site for this mHealth project was Zhao County, which lies 280 km south of Beijing in Hebei Province, China.

Methods

We described the methodology of two studies: (i) a mixed methods study exploring factors influencing sample size calculations for mHealth–based health surveys and (ii) a cross–over study determining validity of an mHealth text messaging data collection tool. The first study used mixed methods, both quantitative and qualitative, including: (i) two surveys with caregivers of young children, (ii) interviews with caregivers, village doctors and participants of the cross–over study, and (iii) researchers’ views. We combined data from caregivers, village doctors and researchers to provide an in–depth understanding of factors influencing sample size calculations for mHealth–based health surveys. The second study, a cross–over study, used a randomised cross–over study design to compare the traditional face–to–face survey method to the new text messaging survey method. We assessed data equivalence (intrarater agreement), the amount of information in responses, reasons for giving different responses, the response rate, characteristics of non–responders, and the error rate.

Conclusions

This paper described the objectives, field site and methods of a large mHealth project part of a collaboration between researchers in China and the UK. The mixed methods study evaluating factors that influence sample size calculations could help future studies with estimating reliable sample sizes. The cross–over study comparing face–to–face and text message survey data collection could help future studies with developing their mHealth tools.

Epidemiology and etiology of childhood pneumonia in 2010: estimates of incidence, severe morbidity, mortality, underlying risk factors and causative pathogens for 192 countries

Background

The recent series of reviews conducted within the Global Action Plan for Pneumonia and Diarrhoea (GAPPD) addressed epidemiology of the two deadly diseases at the global and regional level; it also estimated the effectiveness of interventions, barriers to achieving high coverage and the main implications for health policy. The aim of this paper is to provide the estimates of childhood pneumonia at the country level. This should allow national policy–makers and stakeholders to implement proposed policies in the World Health Organization (WHO) and UNICEF member countries.

Methods

We conducted a series of systematic reviews to update previous estimates of the global, regional and national burden of childhood pneumonia incidence, severe morbidity, mortality, risk factors and specific contributions of the most common pathogens: Streptococcus pneumoniae (SP), Haemophilus influenzae type B (Hib), respiratory syncytial virus (RSV) and influenza virus (flu). We distributed the global and regional–level estimates of the number of cases, severe cases and deaths from childhood pneumonia in 2010–2011 by specific countries using an epidemiological model. The model was based on the prevalence of the five main risk factors for childhood pneumonia within countries (malnutrition, low birth weight, non–exclusive breastfeeding in the first four months, solid fuel use and crowding) and risk effect sizes estimated using meta–analysis.

Findings

The incidence of community–acquired childhood pneumonia in low– and middle–income countries (LMIC) in the year 2010, using World Health Organization's definition, was about 0.22 (interquartile range (IQR) 0.11–0.51) episodes per child–year (e/cy), with 11.5% (IQR 8.0–33.0%) of cases progressing to severe episodes. This is a reduction of nearly 25% over the past decade, which is consistent with observed reductions in the prevalence of risk factors for pneumonia throughout LMIC. At the level of pneumonia incidence, RSV is the most common pathogen, present in about 29% of all episodes, followed by influenza (17%). The contribution of different pathogens varies by pneumonia severity strata, with viral etiologies becoming relatively less important and most deaths in 2010 caused by the main bacterial agents – SP (33%) and Hib (16%), accounting for vaccine use against these two pathogens.

Conclusions

In comparison to 2000, the primary epidemiological evidence contributing to the models of childhood pneumonia burden has improved only slightly; all estimates have wide uncertainty bounds. Still, there is evidence of a decreasing trend for all measures of the burden over the period 2000–2010. The estimates of pneumonia incidence, severe morbidity, mortality and etiology, although each derived from different and independent data, are internally consistent – lending credibility to the new set of estimates. Pneumonia continues to be the leading cause of both morbidity and mortality for young children beyond the neonatal period and requires ongoing strategies and progress to reduce the burden further.

Measuring coverage in MNCH: a validation study linking population survey derived coverage to maternal, newborn, and child health care records in rural China

Background

Accurate data on coverage of key maternal, newborn, and child health (MNCH) interventions are crucial for monitoring progress toward the Millennium Development Goals 4 and 5. Coverage estimates are primarily obtained from routine population surveys through self-reporting, the validity of which is not well understood. We aimed to examine the validity of the coverage of selected MNCH interventions in Gongcheng County, China.

Method and Findings

We conducted a validation study by comparing women’s self-reported coverage of MNCH interventions relating to antenatal and postnatal care, mode of delivery, and child vaccinations in a community survey with their paper- and electronic-based health care records, treating the health care records as the reference standard. Of 936 women recruited, 914 (97.6%) completed the survey. Results show that self-reported coverage of these interventions had moderate to high sensitivity (0.57 [95% confidence interval (CI): 0.50–0.63] to 0.99 [95% CI: 0.98–1.00]) and low to high specificity (0 to 0.83 [95% CI: 0.80–0.86]). Despite varying overall validity, with the area under the receiver operating characteristic curve (AUC) ranging between 0.49 [95% CI: 0.39–0.57] and 0.90 [95% CI: 0.88–0.92], bias in the coverage estimates at the population level was small to moderate, with the test to actual positive (TAP) ratio ranging between 0.8 and 1.5 for 24 of the 28 indicators examined. Our ability to accurately estimate validity was affected by several caveats associated with the reference standard. Caution should be exercised when generalizing the results to other settings.

Conclusions

The overall validity of self-reported coverage was moderate across selected MNCH indicators. However, at the population level, self-reported coverage appears to have small to moderate degree of bias. Accuracy of the coverage was particularly high for indicators with high recorded coverage or low recorded coverage but high specificity. The study provides insights into the accuracy of self-reports based on a population survey in low- and middle-income countries. Similar studies applying an improved reference standard are warranted in the future.

Measuring coverage in MNCH: challenges and opportunities in the selection of coverage indicators for global monitoring

Global monitoring of intervention coverage is a cornerstone of international efforts to improve reproductive, maternal, newborn, and child health. In this review, we examine the process and implications of selecting a core set of coverage indicators for global monitoring, using as examples the processes used by the Countdown to 2015 for Maternal, Newborn and Child Survival and the Commission on Accountability for Women's and Children's Health. We describe how the generation of data for global monitoring involves five iterative steps: development of standard indicator definitions and measurement approaches to ensure comparability across countries; collection of high-quality data at the country level; compilation of country data at the global level; organization of global databases; and rounds of data quality checking. Regular and rigorous technical review processes that involve high-level decision makers and experts familiar with indicator measurement are needed to maximize uptake and to ensure that indicators used for global monitoring are selected on the basis of available evidence of intervention effectiveness, feasibility of measurement, and data availability as well as programmatic relevance. Experience from recent initiatives illustrates the challenges of striking this balance as well as strategies for reducing the tensions inherent in the indicator selection process. We conclude that more attention and continued investment need to be directed to global monitoring, to support both the process of global database development and the selection of sets of coverage indicators to promote accountability. The stakes are high, because these indicators can drive policy and program development at the country and global level, and ultimately impact the health of women and children and the communities where they live.

Measuring coverage in MNCH: new findings, new strategies, and recommendations for action

Considerable progress has been made in reducing maternal, newborn, and child mortality worldwide, but many more deaths could be prevented if effective interventions were available to all who could benefit from them. Timely, high-quality measurements of intervention coverage--the proportion of a population in need of a health intervention that actually receives it--are essential to support sound decisions about progress and investments in women's and children's health. The PLOS Medicine "Measuring Coverage in MNCH" Collection of research studies and reviews presents systematic assessments of the validity of health intervention coverage measurement based on household surveys, the primary method for estimating population-level intervention coverage in low- and middle-income countries. In this overview of the Collection, we discuss how and why some of the indicators now being used to track intervention coverage may not provide fully reliable coverage measurements, and how a better understanding of the systematic and random error inherent in these coverage indicators can help in their interpretation and use. We draw together strategies proposed across the Collection for improving coverage measurement, and recommend continued support for high-quality household surveys at national and sub-national levels, supplemented by surveys with lighter tools that can be implemented every 1-2 years and by complementary health-facility-based assessments of service quality. Finally, we stress the importance of learning more about coverage measurement to strengthen the foundation for assessing and improving the progress of maternal, newborn, and child health programs.

Measuring coverage in MNCH: total survey error and the interpretation of intervention coverage estimates from household surveys

Nationally representative household surveys are increasingly relied upon to measure maternal, newborn, and child health (MNCH) intervention coverage at the population level in low- and middle-income countries. Surveys are the best tool we have for this purpose and are central to national and global decision making. However, all survey point estimates have a certain level of error (total survey error) comprising sampling and non-sampling error, both of which must be considered when interpreting survey results for decision making. In this review, we discuss the importance of considering these errors when interpreting MNCH intervention coverage estimates derived from household surveys, using relevant examples from national surveys to provide context. Sampling error is usually thought of as the precision of a point estimate and is represented by 95% confidence intervals, which are measurable. Confidence intervals can inform judgments about whether estimated parameters are likely to be different from the real value of a parameter. We recommend, therefore, that confidence intervals for key coverage indicators should always be provided in survey reports. By contrast, the direction and magnitude of non-sampling error is almost always unmeasurable, and therefore unknown. Information error and bias are the most common sources of non-sampling error in household survey estimates and we recommend that they should always be carefully considered when interpreting MNCH intervention coverage based on survey data. Overall, we recommend that future research on measuring MNCH intervention coverage should focus on refining and improving survey-based coverage estimates to develop a better understanding of how results should be interpreted and used.

Measuring coverage in MNCH: accuracy of measuring diagnosis and treatment of childhood malaria from household surveys in Zambia

BACKGROUND

To assess progress in the scale-up of rapid diagnostic tests and artemisinin-based combination therapies (ACTs) across Africa, malaria control programs have increasingly relied on standardized national household surveys to determine the proportion of children with a fever in the past 2 wk who received an effective antimalarial within 1-2 d of the onset of fever. Here, the validity of caregiver recall for measuring the primary coverage indicators for malaria diagnosis and treatment of children <5 y old is assessed.

METHODS AND FINDINGS

A cross-sectional study was conducted in five public clinics in Kaoma District, Western Provence, Zambia, to estimate the sensitivity, specificity, and accuracy of caregivers' recall of malaria testing, diagnosis, and treatment, compared to a gold standard of direct observation at the health clinics. Compared to the gold standard of clinic observation, for recall for children with fever in the past 2 wk, the sensitivity for recalling that a finger/heel stick was done was 61.9%, with a specificity of 90.0%. The sensitivity and specificity of caregivers' recalling a positive malaria test result were 62.4% and 90.7%, respectively. The sensitivity and specificity of recalling that the child was given a malaria diagnosis, irrespective of whether a laboratory test was actually done, were 76.8% and 75.9%, respectively. The sensitivity and specificity for recalling that an ACT was given were 81.0% and 91.5%, respectively.

CONCLUSIONS

Based on these findings, results from household surveys should continue to be used for ascertaining the coverage of children with a fever in the past 2 wk that received an ACT. However, as recall of a malaria diagnosis remains suboptimal, its use in defining malaria treatment coverage is not recommended.

Measuring coverage in MNCH: tracking progress in health for women and children using DHS and MICS household surveys

Household surveys are the primary data source of coverage indicators for children and women for most developing countries. Most of this information is generated by two global household survey programmes-the USAID-supported Demographic and Health Surveys (DHS) and the UNICEF-supported Multiple Indicator Cluster Surveys (MICS). In this review, we provide an overview of these two programmes, which cover a wide range of child and maternal health topics and provide estimates of many Millennium Development Goal indicators, as well as estimates of the indicators for the Countdown to 2015 initiative and the Commission on Information and Accountability for Women's and Children's Health. MICS and DHS collaborate closely and work through interagency processes to ensure that survey tools are harmonized and comparable as far as possible, but we highlight differences between DHS and MICS in the population covered and the reference periods used to measure coverage. These differences need to be considered when comparing estimates of reproductive, maternal, newborn, and child health indicators across countries and over time and we discuss the implications of these differences for coverage measurement. Finally, we discuss the need for survey planners and consumers of survey results to understand the strengths, limitations, and constraints of coverage measurements generated through household surveys, and address some technical issues surrounding sampling and quality control. We conclude that, although much effort has been made to improve coverage measurement in household surveys, continuing efforts are needed, including further research to improve and refine survey methods and analytical techniques.

Measuring coverage in MNCH: challenges in monitoring the proportion of young children with pneumonia who receive antibiotic treatment

Pneumonia remains a major cause of child death globally, and improving antibiotic treatment rates is a key control strategy. Progress in improving the global coverage of antibiotic treatment is monitored through large household surveys such as the Demographic and Health Surveys (DHS) and the Multiple Indicator Cluster Surveys (MICS), which estimate antibiotic treatment rates of pneumonia based on two-week recall of pneumonia by caregivers. However, these survey tools identify children with reported symptoms of pneumonia, and because the prevalence of pneumonia over a two-week period in community settings is low, the majority of these children do not have true pneumonia and so do not provide an accurate denominator of pneumonia cases for monitoring antibiotic treatment rates. In this review, we show that the performance of survey tools could be improved by increasing the survey recall period or by improving either overall discriminative power or specificity. However, even at a test specificity of 95% (and a test sensitivity of 80%), the proportion of children with reported symptoms of pneumonia who truly have pneumonia is only 22% (the positive predictive value of the survey tool). Thus, although DHS and MICS survey data on rates of care seeking for children with reported symptoms of pneumonia and other childhood illnesses remain valid and important, DHS and MICS data are not able to give valid estimates of antibiotic treatment rates in children with pneumonia.