Measuring coverage in MNCH: total survey error and the interpretation of intervention coverage estimates from household surveys

Recruitment, data collection, participation rate, and representativeness of the international cross-sectional PRICOV-19 study across 38 countries

BACKGROUND

Recruitment for surveys has been a great challenge, especially in general practice.

METHODS

Here, we reported recruitment strategies, data collection, participation rates (PR) and representativeness of the PRICOV-19 study, an international comparative, cross-sectional, online survey among general practices (GP practices) in 37 European countries and Israel.

RESULTS

Nine (24%) countries reported a published invitation; 19 (50%) had direct contact with all GPs/GP practices; 19 (50%) contacted a sample of GPs /GP practices; and 7 (18%) used another invitation strategy. The median participation rate was 22% (IQR = 10%, 28%). Multiple invitation strategies (P-value 0.93) and multiple strategies to increase PR (P-value 0.64) were not correlated with the PR. GP practices in (semi-) rural areas, GP practices serving more than 10,000 patients, and group practices were over-represented (P-value < 0.001). There was no significant correlation between the PR and strength of the primary care (PC) system [Spearman's r 0.13, 95% CI (-0.24, 0.46); P-value 0.49]; the COVID-19 morbidity [Spearman's r 0.19, 95% CI (-0.14, 0.49); P-value 0.24], or COVID-19 mortality [Spearman's r 0.19, 95% CI (-0.02, 0.58); P-value 0.06] during the three months before country-specific study commencement.

CONCLUSION

Our main contribution here was to describe the survey recruitment and representativeness of PRICOV-19, an important and novel study.

Antenatal care coverage and early childhood mortality in Zimbabwe: new interpretations from nationally representative household surveys

Zimbabwe has implemented universal antenatal care (ANC) policies since 1980 that have significantly contributed to improvements in ANC access and early childhood mortality rates. However, Demographic and Health Surveys (DHS) and Multiple Indicator Cluster Surveys (MICS), two of Zimbabwe's main sources of health data and evidence, often provide seemingly different estimates of ANC coverage and under-five mortality rates. This creates confusion that can result in disparate policies and practices, with potential negative impacts on mother and child health in Zimbabwe. We conducted a comparability analysis of multiple DHS and MICS datasets to enhance the understanding of point estimates, temporal changes, rural-urban differences and reliability of estimates of ANC coverage and neonatal, infant and under-five mortality rates (NMR, IMR and U5MR, separately) from 2009 to 2019 in Zimbabwe. Our two samples z-tests revealed that both DHS and MICS indicated significant increases in ANC coverage and declines in IMR and U5MR but only from 2009 to 2015. NMR neither increased nor declined from 2009 to 2019. Rural-urban differences were significant for ANC coverage (2009-15 only) but not for NMR, IMR and U5MR. We found that there is a need for more precise DHS and MICS estimates of urban ANC coverage and all estimates of NMR, IMR and U5MR, and that shorter recall periods provide more reliable estimates of ANC coverage in Zimbabwe. Our findings represent new interpretations and clearer insights into progress and gaps around ANC coverage and under-five mortality rates that can inform the development, implementation, monitoring and evaluation of policy and practice responses and further research in Zimbabwe.

Are Clinical Prediction Rules Used in Spinal Cord Injury Care? A Survey of Practice

Background

Accurate outcome prediction is desirable post spinal cord injury (SCI), reducing uncertainty for patients and supporting personalized treatments. Numerous attempts have been made to create clinical prediction rules that identify patients who are likely to recover function. It is unknown to what extent these rules are routinely used in clinical practice.

Objectives

To better understand knowledge of, and attitudes toward, clinical prediction rules amongst SCI clinicians in the United Kingdom.

Methods

An online survey was distributed via mailing lists of clinical special interest groups and relevant National Health Service Trusts. Respondents answered questions about their knowledge of existing clinical prediction rules and their general attitudes to using them. They also provided information about their level of experience with SCI patients.

Results

One hundred SCI clinicians completed the survey. The majority (71%) were unaware of clinical prediction rules for SCI; only 8% reported using them in clinical practice. Less experienced clinicians were less likely to be aware. Lack of familiarity with prediction rules was reported as being a barrier to their use. The importance of clinical expertise when making prognostic decisions was emphasized. All respondents reported interest in using clinical prediction rules in the future.

Conclusion

The results show widespread lack of awareness of clinical prediction rules amongst SCI clinicians in the United Kingdom. However, clinicians were positive about the potential for clinical prediction rules to support decision-making. More focus should be directed toward refining current rules and improving dissemination within the SCI community.

Can We Use Routine Data for Strategic Decision Making? A Time Trend Comparison Between Survey and Routine Data in Mali

Routine data, which is available more regularly than the "gold standard" survey data, can be used to inform programmatic decisions in Mali at the national level. However, caution must be used if using data at a subnational level.

Background:

Countries with scarce resources need timely and high-quality data on coverage of health interventions to make strategic decisions about where to allocate investments in health. Household survey data are generally regarded as “gold standard,” high-quality data. This study assessed the comparability of intervention coverage time trends from routine and survey data at national and subnational levels in Mali.

Methods:

We compared 3 coverage indicators: contraceptive prevalence rate, institutional delivery, and 3 doses of diphtheria, pertussis, and tetanus (DPT3) vaccine, using 3 Mali Demographic and Health Surveys (DHS 2001, 2006, and 2012–2013) and routine health system data covering 2001–2012. For routine data, we used local health information system (HIS) annual reports and an HIS database. To compare time trends between the data sources, we calculated the percentage point change and 95% confidence interval from 2001–2006 and 2006–2012. We then computed the absolute and relative differences between the 2 data sources for each indicator over time at national and regional levels and assessed their level of significance.

Results:

The direction and magnitude of the time trends of contraceptive prevalence rate, institutional delivery, and DPT3 vaccine from 2001 to 2012 were similar at the national level between data sources. At the regional level, there were significant differences in the magnitude and direction of time trends for institutional delivery and the DPT3 vaccine; contraceptive prevalence trends were more consistent. Routine data tended to overestimate DPT3 coverage, and underestimate institutional delivery and contraceptive prevalence relative to survey data.

Conclusion:

Routine data in Mali—particularly at the national level—appear to be appropriate for use to inform program planning and prioritization, but routine time trends should be interpreted with caution at the subnational level. For program evaluations, routine data may not be appropriate to draw accurate inferences about program impact.

Estimation of Ethiopia's immunization coverage - 20 years of discrepancies

Background

Coverage with the third dose of diphtheria-pertussis-tetanus-containing vaccine (DPT3) is a widely used measure of the performance of routine immunization systems. Since 2015, data reported by Ethiopia’s health facilities have suggested DPT3 coverage to be greater than 95%. Yet, Demographic and Health Surveys in 2016 and 2019 found DPT3 coverage to be 53 and 61% respectively for years during this period. This case study reviews the last 20 years of administrative (based on facility data), survey and United Nations (UN) estimates of Ethiopia’s nationwide immunization coverage to document long-standing discrepancies in these statistics.

Methods

Published estimates were compiled of Ethiopia’s nationwide DPT3 coverage from 1999 to 2018. These estimates come from the Joint Reporting Form submitted annually to WHO and UNICEF, a series of 8 population-based surveys and the annual reports of the WHO/UNICEF Estimates of National Immunization Coverage (WUENIC). Possible reasons for variation in survey findings were explored through secondary analysis of data from the 2012 immunization coverage survey. In addition, selected health officials involved with management of the immunization program were interviewed to obtain their perspectives on the reliability of various methods for estimation of immunization coverage.

Findings

Comparison of Ethiopia’s estimates for the same year from different sources shows major and persistent discrepancies between administrative, survey and WUENIC estimates. Moreover, the estimates from each of these sources have repeatedly shown erratic year-to-year fluctuations. Those who were interviewed expressed scepticism of Demographic and Health Survey (DHS) statistics. Officials of the national immunization programme have repeatedly shown a tendency to overlook all survey statistics when reporting on programme performance.

Conclusions

The present case study raises important questions, not only about the estimation methods of national and UN agencies, but about the reliability and comparability of widely trusted coverage surveys. Ethiopia provides an important example of a country where no data source provides a truly robust “gold standard” for estimation of immunization coverage. It is essential to identify and address the reasons for these discrepancies and arrive at a consensus on how to improve the reliability and acceptability of each data source and how best to “triangulate” between them.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-06568-0.

Pairs of independent nationally representative vaccination coverage surveys conducted within one year of each other: A global overview covering 2000-2019

Background

Population-based surveys play an important role in measuring vaccination coverage. Surveys measuring vaccination coverage may be commissioned by the Expanded Programme on Immunization (EPI surveys) or part of multi-domain non-EPI surveys such as Demographic and Health Surveys (DHS) or Multiple Indicator Cluster Surveys (MICS). Surveys conducted too close in time to each other may not only be an inefficient use of resources but may also create problems for programme staff when results suggest inconsistent patterns of programme performance for similar time periods.

Objective

To summarize the occurrence of vaccination coverage surveys conducted close in time during 2000–2019 and compare results of EPI and non-EPI coverage surveys when the surveys were conducted within one year of each other.

Methods

Using a database of published national-level vaccination coverage survey results compiled by the World Health Organization (WHO) and the United Nations Children’s Fund (UNICEF), the authors abstracted information on survey field work dates, sample size, percentage of children with documented history of vaccination and the percent coverage, as well as published uncertainty intervals from DHS and MICS, for the first and third doses of diphtheria-tetanus toxoid-pertussis containing vaccine (DTP1, DTP3) and first dose of measles containing vaccine (MCV1). Survey results of EPI and non-EPI surveys were compared.

Results

The authors identified 646 surveys with final reports and estimates of national-level vaccination coverage for DTP1, DTP3, or MCV1 from a total of 687 surveys with data collection start date from 2000 to 2019. Of the 140 countries with at least one vaccination coverage survey, a median of four surveys was observed. Most countries were Gavi-eligible and located in the WHO Africa Region. Sixty-six survey dyads were identified where an EPI survey occurred within one year of a non-EPI survey. For the 66 dyads, in 49 of 59 with information available, EPI surveys reported higher proportion of documented evidence of vaccination and EPI survey results tended to suggest higher levels of vaccination coverage compared to the non-EPI surveys; quite often, differences were substantial. Surveys that found higher proportions of children with documented vaccination evidence tended to also find higher proportions of children who had been vaccinated.

Summary

Opportunities exist to improve overall planning of vaccination coverage measurement in population-based household surveys so that both EPI and non-EPI surveys are more comparable and survey coverage estimates are more appropriately spaced in time. When surveys occur too close in time, careful attention is warranted to ensure comparability and assess sources of documented evidence of vaccination and related coverage differences.

Performance monitoring and accountability: The Agile Project's protocol, record and experience

The Performance Monitoring and Accountability 2020 (PMA2020) project implemented a multi-country sub-project called PMA Agile, a system of continuous data collection for a probability sample of urban public and private health facilities and their clients that began November 2017 and concluded December 2019.  The objective was to monitor the supply, quality and consumption of family planning services.  In total, across 14 urban settings, nearly 2300 health facilities were surveyed three to six times in two years and a total sample of 48,610 female and male clients of childbearing age were interviewed in Burkina Faso, Democratic Republic of Congo, India, Kenya, Niger and Nigeria.  Consenting female clients with access to a cellphone were re-interviewed by telephone after four months; two rounds of the client exit, and follow-up interviews were conducted in nearly all settings.  This paper reports on the PMA Agile data system protocols, coverage and early experiences.  An online dashboard is publicly accessible, analyses of measured trends are underway, and the data are publicly available.

Performance monitoring and accountability: The Agile Project’s protocol, record and experience

The Performance Monitoring and Accountability 2020 (PMA2020) project implemented a multi-country sub-project called PMA Agile, a system of continuous data collection for a probability sample of urban public and private health facilities and their clients that began November 2017 and concluded December 2019.  The objective was to monitor the supply, quality and consumption of family planning services.  In total, across 14 urban settings, nearly 2300 health facilities were surveyed three to six times in two years and a total sample of 48,610 female and male clients of childbearing age were interviewed in Burkina Faso, Democratic Republic of Congo, India, Kenya, Niger and Nigeria.  Consenting female clients with access to a cellphone were re-interviewed by telephone after four months; two rounds of the client exit, and follow-up interviews were conducted in nearly all settings.  This paper reports on the PMA Agile data system protocols, coverage and early experiences.  An online dashboard is publicly accessible, analyses of measured trends are underway, and the data are publicly available.

How well do mothers recall their own and their infants' perinatal events? A two-district study using cross-sectional stratified random sampling in Bihar, India

OBJECTIVE

Global monitoring of maternal, newborn and child health (MNCH) programmes use self-reported data subject to recall error which may lead to incorrect decisions for improving health services and wasted resources. To minimise this risk, samples of mothers of infants aged 0-2 and 3-5 months are sometimes used. We test whether a single sample of mothers of infants aged 0-5 months provides the same information.

DESIGN

An annual MNCH household survey in two districts of Bihar, India (n=6 million).

PARTICIPANTS

Independent samples (n=475 each) of mothers of infants aged 0-5, 0-2 and 3-5 months.

OUTCOME MEASURES

Main analyses compare responses from the samples of infants aged 0-5 and 0-2 months with Mantel-Haenszel-Cochran statistics using 51 indicators in two districts.

RESULTS

No measurable differences are detected in 79.4% (81/102) comparisons; 20.6% (21/102) display differences for the main comparison. Subanalyses produce similar results. A difference detected for exclusive breast feeding is due to premature complementary feeding by older infants. Measurable differences are detected in 33% (8/24) of the indicators on Front Line Worker (FLW) support, 26.9% (7/26) of indicators of birth preparedness and place of birth and attendant, and 9.5% (4/42) of the indicators on neonatal and antenatal care.

CONCLUSIONS

Differences in FLW visits and compliance with their advice may be due to seasonal effects: mothers of older infants aged 3-5 months were pregnant during the dry season; mothers of infants aged 0-2 months were pregnant during the monsoons, making transportation difficult. Useful coverage estimates can be obtained by sampling mothers with infants aged 0-5 months as with two samples suggesting that mothers of young infants recall their own perinatal events and those of their children. For some indicators (eg, exclusive breast feeding), it may be necessary to adjust targets. Excessive stratification wastes resources, does not improve the quality of information and increases the burden placed on data collectors and communities which can increase non-sampling error.

Estimating access to drinking water and sanitation: The need to account for uncertainty in trend analysis

Nationally representative household surveys are the main source of data for tracking drinking water, sanitation and hygiene (WASH) coverage. However, all survey point estimates have a certain degree of error that must be considered when interpreting survey results for policy and decision making. In this article, we develop an approach to characterize and quantify uncertainty around WASH estimates. We apply it to four countries - Bolivia, Gambia, Morocco and India - representing different regions, number of data points available and types of trajectories, in order to illustrate the importance of communicating uncertainty for temporal estimates, as well as taking into account both the compositional nature and non-linearity of JMP data. The approach is found to be versatile and particularly useful in the WASH sector, where the dissemination and analysis of standard errors lag behind. While it only considers the uncertainty arising from sampling, the proposed approach can help improve the interpretation of WASH data when evaluating trends in coverage and informing decision making.

Coverage and timeliness of vaccination and the validity of routine estimates: Insights from a vaccine registry in Kenya

BACKGROUND

The benefits of childhood vaccines are critically dependent on vaccination coverage. We used a vaccine registry (as gold standard) in Kenya to quantify errors in routine coverage methods (surveys and administrative reports), to estimate the magnitude of survivor bias, contrast coverage with timeliness and use both measures to estimate population immunity.

METHODS

Vaccination records of children in the Kilifi Health and Demographic Surveillance System (KHDSS), Kenya were combined with births, deaths, migration and residence data from 2010 to 17. Using inverse survival curves, we estimated up-to-date and age-appropriate vaccination coverage, calculated mean vaccination coverage in infancy as the area under the inverse survival curves, and estimated the proportion of fully immunised children (FIC). Results were compared with published coverage estimates. Risk factors for vaccination were assessed using Cox regression models.

RESULTS

We analysed data for 49,090 infants and 48,025 children aged 12-23 months in 6 birth cohorts and 6 cross-sectional surveys respectively, and found 2nd year of life surveys overestimated coverage by 2% compared to birth cohorts. Compared to mean coverage in infants, static coverage at 12 months was exaggerated by 7-8% for third doses of oral polio, pentavalent (Penta3) and pneumococcal conjugate vaccines, and by 24% for the measles vaccine. Surveys and administrative coverage also underestimated the proportion of the fully immunised child by 10-14%. For BCG, Penta3 and measles, timeliness was 23-44% higher in children born in a health facility but 20-37% lower in those who first attended during vaccine stock outs.

CONCLUSIONS

Standard coverage surveys in 12-23 month old children overestimate protection by ignoring timeliness, and survivor and recall biases. Where delayed vaccination is common, up-to-date coverage will give biased estimates of population immunity. Surveys and administrative methods also underestimate FIC prevalence. Better measurement of coverage and more sophisticated analyses are required to control vaccine preventable diseases.

Validity of parental recalls to estimate vaccination coverage: evidence from Tanzania

Background

The estimates of vaccination coverage are measured from administrative data and from population based survey. While both card-based and recall data are collected through population survey, and the recall is when the card is missing, the preferred estimates remain of the card-based due to limited validity of parental recalls. As there is a concern of missing cards in poor settings, the evidence on validity of parental recalls is limited and varied across vaccine types, and therefore timely and needed. We validated the recalls against card-based data based on population survey in Tanzania.

Methods

We used a cross-sectional survey of about 3000 households with women who delivered in the last 12 months prior to the interview in 2012 from three regions in Tanzania. Data on the vaccination status on four vaccine types were collected using two data sources, card and recall-based. We compared the level of agreement and identified the recall bias between the two data sources. We further computed the sensitivity and specificity of parental recalls, and used a multivariate logit model to identify the determinants of parental recall bias.

Results

Most parents (85.4%) were able to present the vaccination cards during the survey, and these were used for analysis. Although the coverage levels were generally similar across data sources, the recall-based data slightly overestimated the coverage estimates. The level of agreement between the two data sources was high above 94%, with minimal recall bias of less than 6%. The recall bias due to over-reporting were slightly higher than that due to under-reporting. The sensitivity of parental recalls was generally high for all vaccine types, while the specificity was generally low across vaccine types except for measles. The minimal recall bias for DPT and measles were associated with the mother’s age, education level, health insurance status, region location and child age.

Conclusion

Parental recalls when compared to card-based data are hugely accurate with minimal recall bias in Tanzania. Our findings support the use of parental recall collected through surveys to identify the child vaccination status in the absence of vaccination cards. The use of recall data alongside card-based estimates also ensures more representative coverage estimates.

Improving coverage measurement for reproductive, maternal, neonatal and child health: gaps and opportunities

Background

Regular monitoring of coverage for reproductive, maternal, neonatal, and child health (RMNCH) is central to assessing progress toward health goals. The objectives of this review were to describe the current state of coverage measurement for RMNCH, assess the extent to which current approaches to coverage measurement cover the spectrum of RMNCH interventions, and prioritize interventions for a novel approach to coverage measurement linking household surveys with provider assessments.

Methods

We included 58 interventions along the RMNCH continuum of care for which there is evidence of effectiveness against cause–specific mortality and stillbirth. We reviewed household surveys and provider assessments used in low– and middle–income countries (LMICs) to determine whether these tools generate measures of intervention coverage, readiness, or quality. For facility–based interventions, we assessed the feasibility of linking provider assessments to household surveys to provide estimates of intervention coverage.

Results

Fewer than half (24 of 58) of included RMNCH interventions are measured in standard household surveys. The periconceptional, antenatal, and intrapartum periods were poorly represented. All but one of the interventions not measured in household surveys are facility–based, and 13 of these would be highly feasible to measure by linking provider assessments to household surveys.

Conclusions

We found important gaps in coverage measurement for proven RMNCH interventions, particularly around the time of birth. Based on our findings, we propose three sets of actions to improve coverage measurement for RMNCH, focused on validation of coverage measures and development of new measurement approaches feasible for use at scale in LMICs.

The Effects of an Integrated Community Case Management Strategy on the Appropriate Treatment of Children and Child Mortality in Kono District, Sierra Leone: A Program Evaluation

Integrated community case management (iCCM) aims to reduce child mortality in areas with poor access to health care. iCCM was implemented in 2009 in Kono district, Sierra Leone, a postconflict area with high under-five mortality rates (U5MRs). We evaluated iCCM's impact and effects on child health using cluster surveys in 2010 (midterm) and 2013 (endline) to compare indicators on child mortality, coverage of appropriate treatment, timely access to care, quality of care, and recognition of community health workers (CHWs). The sample size was powered to detect a 28% decline in U5MR. Clusters were selected proportional to population size. All households were sampled to measure mortality and systematic random sampling was used to measure coverage in a subset of households. We used program data to evaluate utilization and access; 5,257 (2010) and 3,649 (2013) households were surveyed. U5MR did not change significantly (4.54 [95% confidence interval [CI]: 3.47-5.60] to 3.95 [95% CI: 3.06-4.83] deaths per 1,000 per month (P = 0.4)) though a relative change smaller than 28% could not be detected. CHWs were the first source of care for 52% (2010) and 50.9% (2013) of children. Coverage of appropriate treatment of fever by CHWs or peripheral health units increased from 45.5% [95% CI: 39.2-52.0] to 58.2% [95% CI: 50.5-65.5] (P = 0.01); changes for diarrhea and pneumonia were not significant. The continued reliance on the CHW as the first source of care and improved coverage for the appropriate treatment of fever support iCCM's role in Kono district.

Survey on Mycoplasma hyopneumoniae gilt acclimation practices in Europe

Gilts are considered to play a key role in Mycoplasma hyopneumoniae (M. hyopneumoniae) transmission and control. An effective gilt acclimation program should ideally reduce M. hyopneumoniae shedding at first farrowing, decreasing pre-weaning colonization prevalence and potential respiratory problems in fatteners. However, information on gilt acclimation practices is scarce in Europe. The aim of this study was to identify current acclimation strategies for M. hyopneumoniae in Europe using a questionnaire designed to assess 15 questions focused on gilt replacement status, acclimation strategies and methods used to ascertain its effect. A total of 321 questionnaires (representing 321 farms) were voluntarily completed by 108 veterinarians (from 18 European countries). From these farms, 280 out of 321 (87.2%) were aware of the health status of gilts on arrival. From these 280 farms, 161 (57.5%) introduced M. hyopneumoniae positive replacements. In addition, 249 out of 321 (77.6%) farms applied an acclimation process using different strategies, being M. hyopneumoniae vaccination (145 out of 249, 58.2%) and the combination of vaccine and exposure to sows selected for slaughter (53 out of 249, 21.3%) the most commonly used. Notwithstanding, only 53 out of 224 (23.6%) farms, knowing the M. hyopneumoniae initial status and performing acclimation strategies against it, verified the effect of the acclimation by ELISA (22 out of 53, 41.5%), PCR (4 out of 53, 7.5%) or both (27 out of 53, 50.9%). This study showed that three fourths of the farms represented in this European survey have M. hyopneumoniae acclimation strategies for gilts, and one fifth of them verify to some extent the effect of the process. Taking into account that the assessment of acclimation efficacy could help in optimizing replacement gilt introduction into the breeding herd, it seems these practices for M. hyopneumoniae are still poorly developed in Europe.

Comparison of inflation of third dose diphtheria tetanus pertussis (DTP3) administrative coverage to other vaccine antigens

Third dose diphtheria tetanus pertussis (DTP3) administrative coverage is a commonly used indicator for immunization program performance, although studies have demonstrated data quality issues with administrative DTP3 coverage. It is possible that administrative coverage for DTP3 may be inflated more than for other antigens. To examine this, theory, we compiled immunization coverage estimates from recent country surveys (n=71) and paired these with corresponding administrative coverage estimates, by country and cohort year, for DTP3 and 4 other antigens. Median administrative coverage was higher than survey estimates of coverage for all antigens (median differences from 26 to 30%), however this difference was similar for DTP3 as for all other antigens. These findings were consistent when countries were stratified by income level and eligibility for Gavi funding. Our findings demonstrate that while country administrative coverage estimates tend to be higher than survey estimates, DTP3 administrative coverage is not inflated more than other antigens.

Does adjusting for recall in trend analysis affect coverage estimates for maternal and child health indicators? An analysis of DHS and MICS survey data

Background

The Demographic and Health Surveys (DHS) and Multiple Indicator Cluster Surveys (MICS) are the major data sources in low- and middle-income countries (LMICs) for evaluating health service coverage. For certain maternal and child health (MCH) indicators, the two surveys use different recall periods: 5 years for DHS and 2 years for MICS.

Objective

We explored whether the different recall periods for DHS and MICS affect coverage trend analyses as well as missing data and coverage estimates.

Designs

We estimated coverage, using proportions with 95% confidence intervals, for four MCH indicators: intermittent preventive treatment of malaria in pregnancy, tetanus vaccination, early breastfeeding and postnatal care. Trends in coverage were compared using data from 1) standard 5-year DHS and 2-year MICS recall periods (unmatched) and 2) DHS restricted to 2-year recall to match the MICS 2-year recall periods (matched). Linear regression was used to explore the relationship between length of recall, missing data and coverage estimates.

Results

Differences in coverage trends were observed between matched and unmatched data in 7 of 18 (39%) comparisons performed. The differences were in the direction of the trend over time, the slope of the coverage change or the significance levels. Consistent trends were seen in 11 of the 18 (61%) comparisons. Proportion of missing data was inversely associated with coverage estimates in both short (2 years) and longer (5 years) recall of the DHS (r=−0.3, p=0.02 and r=−0.4, p=0.004, respectively). The amount of missing information was increased for longer recall compared with shorter recall for all indicators (significant odds ratios ranging between 1.44 and 7.43).

Conclusions

In a context where most LMICs are dependent on population-based household surveys to derive coverage estimates, users of these types of data need to ensure that variability in recall periods and the proportion of missing data across data sources are appropriately accounted for when trend analyses are conducted.

Monitoring vaccination coverage: Defining the role of surveys

•

High quality community-based vaccination coverage surveys are resource-intensive.

•

Other monitoring methods provide useful data for programme managers.

•

Health facility-based assessments evaluate multiple aspects of service provision.

•

Purposeful community samples give local health workers programmatic insights.

•

To be useful, monitoring should lead to action to improve performance.

Vaccination coverage is a widely used indicator of programme performance, measured by registries, routine administrative reports or household surveys. Because the population denominator and the reported number of vaccinations used in administrative estimates are often inaccurate, survey data are often considered to be more reliable. Many countries obtain survey data on vaccination coverage every 3–5 years from large-scale multi-purpose survey programs. Additional surveys may be needed to evaluate coverage in Supplemental Immunization Activities such as measles or polio campaigns, or after major changes have occurred in the vaccination programme or its context.

When a coverage survey is undertaken, rigorous statistical principles and field protocols should be followed to avoid selection bias and information bias. This requires substantial time, expertise and resources hence the role of vaccination coverage surveys in programme monitoring needs to be carefully defined. At times, programmatic monitoring may be more appropriate and provides data to guide program improvement. Practical field methods such as health facility-based assessments can evaluate multiple aspects of service provision, costs, coverage (among clinic attendees) and data quality. Similarly, purposeful sampling or censuses of specific populations can help local health workers evaluate their own performance and understand community attitudes, without trying to claim that the results are representative of the entire population. Administrative reports enable programme managers to do real-time monitoring, investigate potential problems and take timely remedial action, thus improvement of administrative estimates is of high priority. Most importantly, investment in collecting data needs to be complemented by investment in acting on results to improve performance.

Improved measurement for mothers, newborns and children in the era of the Sustainable Development Goals

BACKGROUND

An urgent priority in maternal, newborn and child health is to accelerate the scale-up of cost-effective essential interventions, especially during labor, the immediate postnatal period and for the treatment of serious infectious diseases and acute malnutrition.  Tracking intervention coverage is a key activity to support scale-up and in this paper we examine priorities in coverage measurement, distinguishing between essential interventions that can be measured now and those that require methodological development.

METHODS

We conceptualized a typology of indicators related to intervention coverage that distinguishes access to care from receipt of an intervention by the population in need.  We then built on documented evidence on coverage measurement to determine the status of indicators for essential interventions and to identify areas for development.

RESULTS

Contact indicators from pregnancy to childhood were identified as current indicators for immediate use, but indicators reflecting the quality of care provided during these contacts need development. At each contact point, some essential interventions can be measured now, but the need for development of indicators predominates around interventions at the time of birth and interventions to treat infections. Addressing this need requires improvements in routine facility based data capture, methods for linking provider and community-based data, and improved guidance for effective coverage measurement that reflects the provision of high-quality care.

CONCLUSION

Coverage indicators for some essential interventions can be measured accurately through household surveys and be used to track progress in maternal, newborn and child health.  Other essential interventions currently rely on contact indicators as proxies for coverage but urgent attention is needed to identify new measurement approaches that directly and reliably measure their effective coverage.

Birth registration and child undernutrition in sub-Saharan Africa

Objective

In many countries of the world millions of people are not registered at birth. However, in order to assess children’s nutritional status it is necessary to have an exact knowledge of their age. In the present paper we discuss the effects of insufficient or imprecise age data on estimates of undernutrition prevalence.

Design

Birth registration rates and levels of stunting, underweight and wasting were retrieved from Multiple Indicator Cluster Surveys and Demographic and Health Surveys of thirty-seven sub-Saharan African countries, considering the subdivision in wealth quintiles. The composition of the cross-sectional sample used for nutritional evaluation was analysed using a permutation test. Logistic regression was applied to analyse the relationship between birth registration and undernutrition. The 95 % probability intervals and Student’s t test were used to evaluate the effect of age bias and error.

Results

Heterogeneous sampling designs were detected among countries, with different percentages of children selected for anthropometry. Further, registered children were slightly more represented within samples used for nutritional analysis than in the total sample. A negative relationship between birth registration and undernutrition was recognized, with registered children showing a better nutritional status than unregistered ones, even within each wealth quintile. The over- or underestimation of undernutrition in the case of systematic over- or underestimation of age, respectively, the latter being more probable, was quantified up to 28 %. Age imprecision was shown to slightly overestimate undernutrition.

Conclusions

Selection bias towards registered children and underestimation of children’s age can lead to an underestimation of the prevalence of undernutrition.

What can we learn about postnatal care in Ghana if we ask the right questions? A qualitative study

Background

There are increasing efforts to monitor progress in maternal and neonatal care, with household surveys the main mode of data collection. Postnatal care (PNC) is considered a priority indicator yet few countries report on it, and the need to improve the construct validity associated with PNC questions is recognized.

Objectives

To determine women's knowledge of what happens to the baby after delivery, women's comprehension of terms and question phrasing related to PNC, and issues with recall periods.

Design

Forty qualitative interviews and four focus group discussions were conducted with mothers, and 10 interviews with health workers in rural Ghana. Data were collected on knowledge and recall of postnatal health checks and language used to describe these health checks.

Results

Mothers required specific probing using appropriate language to report postnatal checks. They only had adequate knowledge of postnatal checks, which were easily observed or required asking them a question. Respondents reported that health workers rarely communicated with mothers about what they were doing, and most women did not know the purpose of the equipment used during health checks, such as why a thermometer was being used. Knowledge of neonatal checks in the first hours after a facility delivery was low if the mother and child were separated, or if the mother was tired or weak. Many women reported that they could remember events clearly, but long recall periods affected reporting for some, especially those who had multiple checks or for those with no problems.

Conclusions

Direct questions about PNC or health checks are likely to underestimate coverage. Validity of inferences can be enhanced by using appropriate verbal probes during surveys on commonly performed checks that are clear and visible to the mother.

A mediation approach to understanding socio-economic inequalities in maternal health-seeking behaviours in Egypt

BACKGROUND

The levels and origins of socio-economic inequalities in health-seeking behaviours in Egypt are poorly understood. This paper assesses the levels of health-seeking behaviours related to maternal care (antenatal care [ANC] and facility delivery) and their accumulation during pregnancy and childbirth. Secondly, it explores the mechanisms underlying the association between socio-economic position (SEP) and maternal health-seeking behaviours. Thirdly, it examines the effectiveness of targeting of free public ANC and delivery care.

METHODS

Data from the 2008 Demographic and Health Survey were used to capture two latent constructs of SEP: individual socio-cultural capital and household-level economic capital. These variables were entered into an adjusted mediation model, predicting twelve dimensions of maternal health-seeking; including any ANC, private ANC, first ANC visit in first trimester, regular ANC (four or more visits during pregnancy), facility delivery, and private delivery. ANC and delivery care costs were examined separately by provider type (public or private).

RESULTS

While 74.2% of women with a birth in the 5-year recall period obtained any ANC and 72.4% delivered in a facility, only 48.8% obtained the complete maternal care package (timely and regular facility-based ANC as well as facility delivery) for their most recent live birth. Both socio-cultural capital and economic capital were independently positively associated with receiving any ANC and delivering in a facility. The strongest direct effect of socio-cultural capital was seen in models predicting private provider use of both ANC and delivery. Despite substantial proportions of women using public providers reporting receipt of free care (ANC: 38%, delivery: 24%), this free-of-charge public care was not effectively targeted to women with lowest economic resources.

CONCLUSIONS

Socio-cultural capital is the primary mechanism leading to inequalities in maternal health-seeking in Egypt. Future studies should therefore examine the objective and perceived quality of care from different types of providers. Improvements in the targeting of free public care could help reduce the existing SEP-based inequalities in maternal care coverage in the short term.

Limitations of using administratively reported immunization data for monitoring routine immunization system performance in Nigeria

BACKGROUND

Efforts are underway to strengthen Nigeria's routine immunization system, yet measuring impact poses a challenge. We document limitations in using administrative data from 12 states in Nigeria and explore alternative approaches.

METHODS

We compared state-reported coverage with the third dose of diphtheria-tetanus-pertussis vaccine (DTP3) to district-reported coverage and data from coverage surveys conducted during 2006-2013. We used district-reported data during 2010-2013 to calculate the annual change in immunization coverage, the percentage of the target population that was unimmunized, and the number of vaccine doses administered. Data quality indicators were also assessed.

RESULTS

State-reported DTP3 coverage was 66%-102% in 2010, 49%-98% in 2011, 38%-84% in 2012, and 75%-123% in 2013 and was a median 46%-114% greater than survey coverage during 2006-2013. The mean local government area (LGA)-reported coverage varied substantially (standard deviation range, 10%-33% across years). For 2010-2013, the mean annual percentage change in LGA-reported DTP3 coverage was -15% from 2010 to 2011, -9% from 2011 to 2012, and 74% from 2012 to 2013; the mean annual percentage change in the percentage of the target population unimmunized was -62%, 426%, and -62%, respectively; and the mean annual percentage change in the number of doses administered was -13%, -7%, and 90%, respectively. Annually, a mean 14% of LGAs reported DTP3 coverage of >100%.

DISCUSSION

Assessing immunization system performance by using administrative data has notable limitations. In addition to long-term improvements in administrative data management, alternatives for measuring routine immunization performance should be considered.

Issues and considerations in the use of serologic biomarkers for classifying vaccination history in household surveys

Accurate estimates of vaccination coverage are crucial for assessing routine immunization program performance. Community based household surveys are frequently used to assess coverage within a country. In household surveys to assess routine immunization coverage, a child's vaccination history is classified on the basis of observation of the immunization card, parental recall of receipt of vaccination, or both; each of these methods has been shown to commonly be inaccurate. The use of serologic data as a biomarker of vaccination history is a potential additional approach to improve accuracy in classifying vaccination history. However, potential challenges, including the accuracy of serologic methods in classifying vaccination history, varying vaccine types and dosing schedules, and logistical and financial implications must be considered. We provide historic and scientific context for the potential use of serologic data to assess vaccination history and discuss in detail key areas of importance for consideration in the context of using serologic data for classifying vaccination history in household surveys. Further studies are needed to directly evaluate the performance of serologic data compared with use of immunization cards or parental recall for classification of vaccination history in household surveys, as well assess the impact of age at the time of sample collection on serologic titers, the predictive value of serology to identify a fully vaccinated child for multi-dose vaccines, and the cost impact and logistical issues on outcomes associated with different types of biological samples for serologic testing.

Defining the malaria burden in Nchelenge District, northern Zambia using the World Health Organization malaria indicators survey

BACKGROUND

Malaria is considered as one of the major public health problems and among the diseases of poverty. In areas of stable and relatively high transmission, pregnant women and their newborn babies are among the higher risk groups. A multicentre trial on the safety and efficacy of several formulations of artemisinin-based combination therapy (ACT) during pregnancy is currently on-going in four African countries, including Zambia, whose study site is in Nchelenge district. As the study outcomes may be influenced by the local malaria endemicity, this needs to be characterized. A cross-sectional survey to determine the prevalence and intensity of infection among <10 years old was carried out in March-April 2012 in Nchelenge district.

METHODS

The sampling unit was the household where all children < 10 years of age were included in the survey using simple random household selection on a GPS coded list. A blood sample for determining haemoglobin concentration and identifying malaria infection was collected from each recruited child.

RESULTS

Six hundred thirty households were selected and 782 children tested for malaria and anaemia. Prevalence of malaria infection was 30.2% (236/782), the large majority (97.9%, 231/236) being Plasmodium falciparum and the remaining ones (2.1%, 5/236) Plasmodium malariae. Anaemia, defined as haemoglobin concentration <11 g/dl, was detected in 51.2% (398/782) children.

CONCLUSION

In Zambia, despite the reported decline in malaria burden, pockets of high malaria endemicity, such as Nchelenge district, still remain. This is a border area and significant progress can be achieved only by concerted efforts aimed at increasing coverage of current control interventions across the border.

Measuring coverage in MNCH: evaluation of community-based treatment of childhood illnesses through household surveys

Community case management (CCM) is a strategy for training and supporting workers at the community level to provide treatment for the three major childhood diseases--diarrhea, fever (indicative of malaria), and pneumonia--as a complement to facility-based care. Many low- and middle-income countries are now implementing CCM and need to evaluate whether adoption of the strategy is associated with increases in treatment coverage. In this review, we assess the extent to which large-scale, national household surveys can serve as sources of baseline data for evaluating trends in community-based treatment coverage for childhood illnesses. Our examination of the questionnaires used in Demographic and Health Surveys (DHS) and Multiple Indicator Cluster Surveys (MICS) conducted between 2005 and 2010 in five sub-Saharan African countries shows that questions on care seeking that included a locally adapted option for a community-based provider were present in all the DHS surveys and in some MICS surveys. Most of the surveys also assessed whether appropriate treatments were available, but only one survey collected information on the place of treatment for all three illnesses. This absence of baseline data on treatment source in household surveys will limit efforts to evaluate the effects of the introduction of CCM strategies in the study countries. We recommend alternative analysis plans for assessing CCM programs using household survey data that depend on baseline data availability and on the timing of CCM policy implementation.

Measuring coverage in MNCH: new findings, new strategies, and recommendations for action

Considerable progress has been made in reducing maternal, newborn, and child mortality worldwide, but many more deaths could be prevented if effective interventions were available to all who could benefit from them. Timely, high-quality measurements of intervention coverage--the proportion of a population in need of a health intervention that actually receives it--are essential to support sound decisions about progress and investments in women's and children's health. The PLOS Medicine "Measuring Coverage in MNCH" Collection of research studies and reviews presents systematic assessments of the validity of health intervention coverage measurement based on household surveys, the primary method for estimating population-level intervention coverage in low- and middle-income countries. In this overview of the Collection, we discuss how and why some of the indicators now being used to track intervention coverage may not provide fully reliable coverage measurements, and how a better understanding of the systematic and random error inherent in these coverage indicators can help in their interpretation and use. We draw together strategies proposed across the Collection for improving coverage measurement, and recommend continued support for high-quality household surveys at national and sub-national levels, supplemented by surveys with lighter tools that can be implemented every 1-2 years and by complementary health-facility-based assessments of service quality. Finally, we stress the importance of learning more about coverage measurement to strengthen the foundation for assessing and improving the progress of maternal, newborn, and child health programs.

Measuring coverage in MNCH: tracking progress in health for women and children using DHS and MICS household surveys

Household surveys are the primary data source of coverage indicators for children and women for most developing countries. Most of this information is generated by two global household survey programmes-the USAID-supported Demographic and Health Surveys (DHS) and the UNICEF-supported Multiple Indicator Cluster Surveys (MICS). In this review, we provide an overview of these two programmes, which cover a wide range of child and maternal health topics and provide estimates of many Millennium Development Goal indicators, as well as estimates of the indicators for the Countdown to 2015 initiative and the Commission on Information and Accountability for Women's and Children's Health. MICS and DHS collaborate closely and work through interagency processes to ensure that survey tools are harmonized and comparable as far as possible, but we highlight differences between DHS and MICS in the population covered and the reference periods used to measure coverage. These differences need to be considered when comparing estimates of reproductive, maternal, newborn, and child health indicators across countries and over time and we discuss the implications of these differences for coverage measurement. Finally, we discuss the need for survey planners and consumers of survey results to understand the strengths, limitations, and constraints of coverage measurements generated through household surveys, and address some technical issues surrounding sampling and quality control. We conclude that, although much effort has been made to improve coverage measurement in household surveys, continuing efforts are needed, including further research to improve and refine survey methods and analytical techniques.

Measuring coverage in MNCH: design, implementation, and interpretation challenges associated with tracking vaccination coverage using household surveys

Vaccination coverage is an important public health indicator that is measured using administrative reports and/or surveys. The measurement of vaccination coverage in low- and middle-income countries using surveys is susceptible to numerous challenges. These challenges include selection bias and information bias, which cannot be solved by increasing the sample size, and the precision of the coverage estimate, which is determined by the survey sample size and sampling method. Selection bias can result from an inaccurate sampling frame or inappropriate field procedures and, since populations likely to be missed in a vaccination coverage survey are also likely to be missed by vaccination teams, most often inflates coverage estimates. Importantly, the large multi-purpose household surveys that are often used to measure vaccination coverage have invested substantial effort to reduce selection bias. Information bias occurs when a child's vaccination status is misclassified due to mistakes on his or her vaccination record, in data transcription, in the way survey questions are presented, or in the guardian's recall of vaccination for children without a written record. There has been substantial reliance on the guardian's recall in recent surveys, and, worryingly, information bias may become more likely in the future as immunization schedules become more complex and variable. Finally, some surveys assess immunity directly using serological assays. Sero-surveys are important for assessing public health risk, but currently are unable to validate coverage estimates directly. To improve vaccination coverage estimates based on surveys, we recommend that recording tools and practices should be improved and that surveys should incorporate best practices for design, implementation, and analysis.