Validation studies for population-based intervention coverage indicators: design, analysis, and interpretation

Assessing the validity of maternal report on breastfeeding counselling in Kosovo's primary health facilities

BACKGROUND

Improving the quality of breastfeeding counselling delivered by primary care providers can improve breastfeeding outcomes and ultimately reduce mortality and morbidity of children and mothers. Accurate data on coverage and quality of primary care breastfeeding counselling is essential for monitoring progress; however, global and national indicators are limited. To help address this gap, this study validated indicators of receipt and quality of breastfeeding counselling during routine consultations for infant care at seven primary health facilities across Kosovo.

METHODS

Mothers' reports of breastfeeding counselling received during routine consultations for their infants (0-12 months of age) were collected by exit interview in 2019 and 2021 (n = 609). Responses were compared against direct observation of their consultation using a structured checklist (reference standard) by a trained third-party observer at the primary care facility. We assessed 13 indicators; ten were related to the receipt and content of breastfeeding counselling, and three were specific to the provider's interpersonal skills. We calculated sensitivity, specificity, and area under the receiver operating curve (AUC) to determine individual-level reporting accuracy.

RESULTS

Ten indicators had an agreement rate above 70% and seven indicators had high overall individual-level validity (AUC ≥ 0.7). High prevalence indicators recorded high sensitivity and low specificity, and the inverse for low prevalence indicators. More subjective indicators were less reliable, e.g., mothers over-reported the prevalence of all three indicators related to providers' interpersonal skills.

CONCLUSIONS

This study offers evidence on breastfeeding counselling quality by validating maternal reports of whether a provider discussed breastfeeding, the clinical content of that counselling, and how it was delivered. It is also situated in a primary care setting within a fragile state of which there is limited evidence. We observed that mothers reported accurately when asked directly to recall breastfeeding counselling services received. However, there is a need to further validate subjective questions about interpersonal skills and other measures for the 'experience of care' quality dimension.

Validation of the Brazilian Version of the Modified Scale for Delineating Advanced Practice Nursing Roles

OBJECTIVES

to validate the Brazilian version of the Modified Scale for Delineating Advanced Practice Nursing Roles.

METHODS

this was a methodological study for the clinical validation of an instrument, conducted with 207 nurses working in primary care. Exploratory and confirmatory factor analysis, Cronbach's alpha test, and z-test for proportion comparison were used.

RESULTS

the internal reliability of the scale was 0.944, with alpha greater than 0.80 in most domains, except for Education (0.786). In the exploratory factor analysis, considering the criterion of eigenvalue greater than one, eight factors were identified, explaining 79.38% of the variance. In the comparison of proportions, the adequate responses (≥ 2) in the domain of Comprehensive Direct Care, in both analyzed groups, were statistically equal. This domain had the highest score of adequate responses, followed by Education and Systems Support. Insufficient scoring was observed in the domains of Publication and Professional Leadership.

CONCLUSIONS

the instrument demonstrated stability and reliability to be used in the evaluation of advanced nursing practice.

Validation of maternal recall of number of antenatal care visits attended in rural Southern Nepal: a longitudinal cohort study

OBJECTIVES

This study aimed to examine the validity of maternal recall of total number of antenatal care (ANC) visits during pregnancy and factors associated with the accuracy of maternal recall.

DESIGN

This was a longitudinal cohort study conducted from December 2018 through November 2020.

SETTING

Five government health posts in the Sarlahi district of Southern Nepal.

PARTICIPANTS

402 pregnant women between ages 15 and 49 who presented for their first ANC visit at the study health posts.

MAIN OUTCOMES

The observed number of ANC visits (gold standard) and the reported number of ANC visits at the postpartum interview (maternal recall).

RESULTS

On average, women in the study who had a live birth attended 4.7 ANC visits. About 65% of them attended four or more ANC visits during pregnancy as recommended by the Nepal government, and 38.3% of maternal report matched the categorical ANC visits as observed by the gold standard. The individual validity was poor to moderate, with the highest area under the receiver operating characteristic curve (AUC) being 0.69 (95% CI: 0.65 to 0.74) in the 1-3 visits group. Population-level bias (as distinct from individual-level bias) was observed in the 1-3 visits and 4 visits groups, where 1-3 visits were under-reported (inflation factor (IF): 0.69) and 4 ANC visits were highly over-reported (IF: 2.12). The binary indicator ANC4+ (1-3 visits vs 4+ visits) showed better population-level validity (AUC: 0.69; IF: 1.17) compared with the categorical indicators (1-3 visits, 4 visits, 5-6 visits and more than 6 visits). Report accuracy was not associated with maternal characteristics but was related to ANC frequency. Women who attended more ANC visits were less likely to correctly report their total number of visits.

CONCLUSION

Maternal report of number of ANC visits during pregnancy may not be a valid indicator for measuring ANC coverage. Improvements are needed to measure the frequency of ANC visits.

Can women's reports in client exit interviews be used to measure and track progress of antenatal care services quality? Evidence from a facility assessment census in Malawi

INTRODUCTION

Unlike household surveys, client exit interviews are conducted immediately after a consultation and therefore provides an opportunity to capture routine performance and level of service quality. This study examines the validity and reliability of women's reports on selected ANC interventions in exit interviews conducted in Malawi.

METHODS

Using data from the 2013-2014 Malawi service provision facility census, we compared women's reports in exit interviews regarding the contents of ANC received with reports obtained through direct observation by a trained healthcare professional. The validity of six indicators was tested using two measures: the area under the receiver operating characteristic curve (AUC), and the inflation factor (IF). Reliability of women's reports was measured using the Kappa coefficient (κ) and the prevalence-adjusted bias-adjusted kappa (PABAK). Finally, we examined whether reporting reliability varied significantly by individual and facility characteristics.

RESULTS

Of the six indicators, two concrete and observable measures had high reporting accuracy and met the validity criteria for both AUC ≥ 0.7 and 0.75>IF>1.25, namely whether the provider prescribed or gave malaria prophylaxis (AUC: 0.84, 95% CI: 0.83-0.86; IF: 0.96) or iron/folic tablets (AUC: (0.84 95% CI: 0.81-0.87; IF:1.00). Whereas four measures related to counselling had lower reporting accuracy: whether the provider offered counselling about nutrition in pregnancy (AUC: 0.69, 95%CI: 0.67-0.71; IF = 1.26), delivery preparation (AUC: 0.62, 95% CI: 0.60-065; IF = 0.99), pregnancy related complications (AUC: 0.59, 95%CI: 0.56-0.61; IF = 1.11), and iron/folic acid side effects (AUC:0.58, 95% CI: 0.55-0.60; IF = 1.42). Similarly, the observable measures had high reliability with both κ and PABAK values in the ranges of ≥ 0.61 and ≥ 0.80. Respondent's age, primiparous status, number of antenatal visits, and the type of health provider increased the likelihood of reporting reliability.

CONCLUSION

In order to enhance the measurement of quality of ANC services, our study emphasizes the importance of carefully considering the type of information women are asked to recall and the timing of the interviews. While household survey programmes such as the demographic health survey and multiple indicator cluster survey are commonly used as data sources for measuring intervention coverage and quality, policy makers should complement such data with more reliable sources like routine data from health information systems.

How does social desirability bias influence survey-based estimates of the use of antenatal care in rural Nepal? A validation study

OBJECTIVES

Social desirability bias is often speculated to influence survey responses but seldom studied in healthcare. The objective was to explore whether social desirability scores (SDS) or the presence of interview observers is associated with inaccurate recall and overestimation of antenatal care (ANC) services.

DESIGN

Longitudinal validation study comparing recalled receipt of ANC services and nutrition components of ANC against direct observations of care. An adapted short form Marlowe-Crowne questionnaire was used to generate an SDS, and the presence of interview observers was treated as a separate exposure. We assessed accuracy and overestimation of recalled receipt of ANC services against observed receipt using log-binomial regression, adjusting for age, education, first-pregnancy and socioeconomic status.

SETTING

Rural Southern Nepal with recruitment from five government health posts.

PARTICIPANTS

401 pregnant women.

RESULTS

Social desirability scores did not significantly predict accuracy or overestimation of most types of ANC care except counselling on nausea. Higher SDS was associated with more accurate recall (adjusted RR, aRR 1.08 (95% CI 1.03, 1.12)) and less overestimation (aRR 0.85 (0.80, 0.91)). The presence of mothers-in-law or husbands during interviews was associated with greater overestimation of the number of ANC visits received by more than three visits (aRR 2.07 (1.11, 3.84)) and (aRR 4.19 (2.17, 8.10)), respectively. Those interviewed with friends present tended to overestimate the receipt of counselling on nausea, avoiding alcohol and not smoking.

CONCLUSION

The presence of observers can lead to overestimation of the receipt of ANC care and support the conduct of interviews in private settings despite challenges of doing so in village contexts. Findings that the SDS did not predict the accuracy of most types of ANC care might reflect a reality that such questions may not be sensitive from a social-norms perspective. Additional local adaptation of SDS is recommended.

Comparing observed occurrence of mistreatment during childbirth with women's self-report: a validation study in Ghana, Guinea and Nigeria

BACKGROUND

There has been substantial progress in developing approaches to measure mistreatment of women during childbirth. However, less is known about the differences in measurement approaches. In this study, we compare measures of mistreatment obtained from the same women using labour observations and community-based surveys in Ghana, Guinea and Nigeria.

METHODS

Experiences of mistreatment during childbirth are person-centred quality measures. As such, we assessed individual-level and population-level accuracy of labour observation relative to women's self-report for different types of mistreatment. We calculated sensitivity, specificity, percent agreement and population-level inflation factor (IF), assessing prevalence of mistreatment in labour observation divided by 'true' prevalence in women's self-report. We report the IF degree of bias as: low (0.75

RESULTS

1536 women across Ghana (n=779), Guinea (n=425) and Nigeria (n=332) were included. Most mistreatment items demonstrated better specificity than sensitivity: observation of any physical abuse (44% sensitive, 89% specific), any verbal abuse (61% sensitive, 73% specific) and presence of a labour companion (19% sensitive, 93% specific). Items for stigma (IF 0.16), pain relief requested (IF 0.38), companion present (IF 0.32) and lack of easy access to fluids (IF 0.46) showed high risk of bias, meaning labour observations would substantially underestimate true prevalence. Other items showed low or moderate bias.

CONCLUSION

Using self-report as the reference standard, labour observations demonstrated moderate-to-high specificity (accurately identifying lack of mistreatment) but low-to-moderate sensitivity (accurately identifying presence of mistreatment) among women. For overall prevalence, either women's self-report or observations can be used with low-moderate bias for most mistreatment items. However, given the dynamicity, complexity, and limitations in 'objectivity', some experiences of mistreatment (stigma, pain relief, labour companionship, easy access to fluids) require measurement via women's self-report. More work is needed to understand how subjectivity influences how well a measure represents individual's experiences.

Collapse

Key Words

• cross-sectional survey
• maternal health
• public health

Collapse

MESH Headings

• Female
• Humans
• Pregnancy
• Self Report
• Ghana
• Guinea/epidemiology
• Nigeria
• Parturition
• Pain

Collapse

Grants

• 001 World Health Organization
• UNDP/UNFPA/UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training
• United States Agency for International Development (USAID)

Collapse

Affiliation(s)

Hedieh Mehrtash Department of Sexual and Reproductive Health and Research, WHO, Geneva, Switzerland Department of Global Health, University of Washington School of Public Health, Seattle, Washington, USA
Meghan A Bohren Gender and Women's Health Unit, Nossal Institute for Global Health, University of Melbourne, Melbourne, Victoria, Australia
Kwame Adu-Bonsaffoh Department of Obstetrics and Gynecology, University of Ghana Medical School, Accra, Ghana
Theresa Azonima Irinyenikan Department of Obstetrics and Gynaecology, University of Medical Sciences Teaching Hospital, Akure, Nigeria
Blair O Berger Department of Population, Family and Reproductive Health, Johns Hopkins University Bloomberg School of Public Health, Baltimore, Maryland, USA
Ernest Maya Department of Population Health, School of Public Health, University of Ghana, Accra, Ghana
Mamadou Dioulde Balde Cellulle de Recherche en Sante de la Reproduction en Guinee (CERREGUI), Conakry, Guinea
Thae Maung Maung Department of Medical Research, Ministry of Health and Sports, Yangon, Myanmar
Adeniyi Kolade Aderoba National Perinatal Epidemiology Unit, Nuffield Department of Population Health, University of Oxford, Oxford, UK Department of Obstetrics and Gynaecology, Mother and Child Hospital Akure, Akure, Nigeria
Özge Tuncalp Department of Sexual and Reproductive Health and Research, WHO, Geneva, Switzerland
Hannah H Leslie Division of Prevention Science, University of California San Francisco, San Francisco, California, USA

Collapse

Exploring the accuracy of self-reported maternal and newborn care in select studies from low and middle-income country settings: do respondent and facility characteristics affect measurement?

BACKGROUND

Accurate data on the receipt of essential maternal and newborn health interventions is necessary to interpret and address gaps in effective coverage. Validation results of commonly used content and quality of care indicators routinely implemented in international survey programs vary across settings. We assessed how respondent and facility characteristics influenced the accuracy of women's recall of interventions received in the antenatal and postnatal periods.

METHODS

We synthesized reporting accuracy using data from a known sample of validation studies conducted in Sub-Saharan Africa and Southeast Asia, which assessed the validity of women's self-report of received antenatal care (ANC) (N = 3 studies, 3,169 participants) and postnatal care (PNC) (N = 5 studies, 2,462 participants) compared to direct observation. For each study, indicator sensitivity and specificity are presented with 95% confidence intervals. Univariate fixed effects and bivariate random effects models were used to examine whether respondent characteristics (e.g., age group, parity, education level), facility quality, or intervention coverage level influenced the accuracy of women's recall of whether interventions were received.

RESULTS

Intervention coverage was associated with reporting accuracy across studies for the majority (9 of 12) of PNC indicators. Increasing intervention coverage was associated with poorer specificity for 8 indicators and improved sensitivity for 6 indicators. Reporting accuracy for ANC or PNC indicators did not consistently differ by any other respondent or facility characteristic.

CONCLUSIONS

High intervention coverage may contribute to higher false positive reporting (poorer specificity) among women who receive facility-based maternal and newborn care while low intervention coverage may contribute to false negative reporting (lower sensitivity). While replication in other country and facility settings is warranted, results suggest that monitoring efforts should consider the context of care when interpreting national estimates of intervention coverage.

Moderate Accuracy of Survey Responses about Infant and Young Child Feeding Counseling Reported by Mothers with Children Less than 1 Year of Age in India

BACKGROUND

Counseling on infant and young child feeding (IYCF) to support optimal breastfeeding and complementary feeding practices is an essential intervention, and accurate coverage data is needed to identify gaps and monitor progress. However, coverage information captured during household surveys has not yet been validated.

OBJECTIVES

We examined the validity of maternal reports of IYCF counseling received during community-based contacts and factors associated with reporting accuracy.

METHODS

Direct observations of home visits conducted by community workers in 40 villages in Bihar, India served as the "gold standard" to maternal reports of IYCF counseling received during 2-wk follow-up surveys (n = 444 mothers with children less than 1 y of age, interviews matched to direct observations). Individual-level validity was assessed by calculating sensitivity, specificity, and AUC. Population-level bias was measured using the inflation factor (IF). Multivariable regression models were used to examine factors associated with response accuracy.

RESULTS

Prevalence of IYCF counseling during home visits was very high (90.1%). Maternal report of any IYCF counseling received in the past 2 wk was moderate (AUC: 0.60; 95% CI: 0.52, 0.67), and population bias was low (IF = 0.90). However, the recall of specific counseling messages varied. Maternal report of any breastfeeding, exclusive breastfeeding, and dietary diversity messages had moderate validity (AUC > 0.60), but other child feeding messages had low individual validity. Child age, maternal age, maternal education, mental stress, and social desirability were associated with reporting accuracy of multiple indicators.

CONCLUSIONS

Validity of IYCF counseling coverage was moderate for several key indicators. IYCF counseling is an information-based intervention that may be received from various sources, and it may be challenging to achieve higher reporting accuracy over a longer recall period. We consider the modest validity results as positive and suggest that these coverage indicators may be useful for measuring coverage and tracking progress over time.

Measures to assess quality of postnatal care: A scoping review

High quality postnatal care is key for the health and wellbeing of women after childbirth and their newborns. In 2022, the World Health Organization (WHO) published global recommendations on maternal and newborn care for a positive postnatal care experience in a new WHO PNC guideline. Evidence regarding appropriate measures to monitor implementation of postnatal care (PNC) according to the WHO PNC guideline is lacking. This scoping review aims to document the measures used to assess the quality of postnatal care and their validity. The review was conducted according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR). Five electronic bibliographic databases were searched together with a grey literature search. Two reviewers independently screened and appraised identified articles. All data on PNC measures were extracted and mapped to the 2022 WHO PNC recommendations according to three categories: i) maternal care, ii) newborn care, iii) health system and health promotion interventions. We identified 62 studies providing measures aligning with the WHO PNC recommendations. For most PNC recommendations there were measures available and the highest number of recommendations were found for breastfeeding and the assessment of the newborn. No measures were found for recommendations related to sedentary behavior, criteria to be assessed before discharge, retention of staff in rural areas and use of digital communication. Measure validity assessment was described in 24 studies (39%), but methods were not standardized. Our review highlights a gap in existing PNC measures for several recommendations in the WHO PNC guideline. Assessment of the validity of PNC measures was limited. Consensus on how the quality of PNC should be measured is needed, involving a selection of priority measures and the development of new measures as appropriate.

How valid are women's reports of the antenatal health services they receive from Community Health Workers in Gombe State north-eastern Nigeria?

BACKGROUND

Community health workers (CHWs) in low- and middle-income countries are key to increasing coverage of maternal and newborn interventions through home visits to counsel families about healthy behaviours. Household surveys enable tracking the progress of CHW programmes but recent evidence questions the accuracy of maternal reports. We measured the validity of women's responses about the content of care they received during CHW home visits and examined whether the accuracy of women's responses was affected by CHW counselling skills.

METHODS

We conducted a criterion validity study in 2019, in Gombe State-Nigeria, and collected data from 362 pregnant women. During accompanied CHW home visits the content of CHW care and the presence or absence of 18 positive counselling skills were observed and documented by a researcher. In a follow-up interview three months later, the same women were asked about the care received during the CHW home visit. Women's reports were compared with observation data and the sensitivity, specificity, and area under receiver curve (AUC) calculated. We performed a covariate validity analysis that adjusted for a counselling skill score to assess the variation in accuracy of women's reports with CHW counselling skills.

RESULTS

Ten indicators were included in the validity analysis. Women consistently overestimated the content of care CHWs provided and no indicator met the condition for individual-level accuracy set at AUC ≥ 0.6. The CHW counselling skill score ranged from 9-18 points from a possible 18, with a mean of 14.3; checking on client history or concerns were the most frequently missed item. There was evidence that unmarried women and the relatively most poor women received less skilled counselling than other women (mean counselling scores of 13.2 and 13.7 respectively). There was no consistent evidence of an association between higher counselling skill scores and better accuracy of women's reports.

CONCLUSIONS

The validity of women's responses about CHW care content was poor and consistently overestimated coverage. We discuss several challenges in applying criterion validity study methods to examine measures of community-based care and make only cautious interpretation of the findings that may be relevant to other researchers interested in developing similar studies.

Measurement of respectful maternity care in exit interviews following facility childbirth: a criterion validity assessment in Nigeria

Background

Valid methods for assessing women’s experiences of maternity care are essential to the global efforts toward providing positive childbirth environments for all women in all health facilities.

Methods

This criterion validity study used observation of childbirth as the reference standard compared to exit interviews with women upon discharge with a live baby, usually within 24 hours of childbirth. We investigated eight positive and sixteen negative maternity care experience indicators. Data were collected from ten primary healthcare facilities in Gombe State, northeast Nigeria, in August 2018 and August 2019. Data analysis involved tabulation of demographic characteristics of women and childbirth context, computation of individual level validity metrics including the area under the receiver operating curve (AUC) and estimating population level validity using the inflation factor (IF).

Results

A total of 724 women were observed and interviewed at the time of discharge (exit) following facility childbirth, 15% of whom were adolescents, 99% were married, 43% had no formal education, and a skilled birth attendant had attended only 12%. The prevalence of positive maternity care experience indicators ranged between 25% and 96% in childbirth observations. For these positive indicators, the agreement between childbirth observations and exit interviews ranged from 55% to 97%. Six of the eight positive maternity care experience indicators had high overall validity, meaning AUC≥0.70 and 0.75<IF<1.25, with high sensitivity (89% to 99%) and moderate to high specificity (44% to 84%). The prevalence of the 16 negative maternity care experience indicators ranged between 0.1% and 18% in childbirth observations. For these negative indicators, agreement ranged from 87% to 99%. Just six of the 16 negative maternity care experience indicators met the criteria for validation analysis; and these showed low to moderate sensitivity (32% to 74%), high specificity (97% to 100%) and moderate (0.60<AUC<0.70) to high (AUC≥0.70) individual-level validity.

Conclusions

In this high mortality setting with relatively low coverage of skilled attendance at birth, exit interviews with women following facility-based childbirth care provided responses consistent with the observation of childbirth for eight positive and six negative maternity care experience indicators.

Cognitive testing of questions about antenatal care and nutrition interventions in southern Nepal

Large scale surveys such as the Demographic and Health Surveys (DHS) are used to measure the coverage and quality of antenatal care (ANC)-related services. Studies have increasingly validated questions from these surveys, though few have explored respondent comprehension or associated thought processes. This study aimed to use cognitive testing and validation approaches to understand how survey respondents understand questions related to ANC-related nutrition services. The study was nested within a larger validation study in southern Nepal. Pregnant women's receipt of ANC related services was directly observed at five health posts followed by a recall interview at 6 months postpartum. A week later, a survey module was re-administered to 30 women containing 15 questions about receipt of ANC care and specifically nutrition-related services. Detailed probing was used to identify cognitive challenges related to comprehension, retrieval, judgement, and response. Respondents accurately recalled the four specific ANC visits recommended by the government of Nepal but those with more visits struggled to estimate the total number of ANC visits they had made. A number of terms including "antenatal care, "nutrition" and "breastfeeding" were challenging for many respondents to understand. Visits to private providers including for ultrasounds were inconsistently included in ANC visit counts suggesting that question wording could better specify the type of care. Many respondents over-estimated the number of iron folic acid (IFA) supplements taken during pregnancy, and recall was challenging. Calculations were based on estimating the number of months between first ANC visit to delivery, and only sometimes factored in missed tablets. Opportunities exist to improve questions to facilitate better comprehension by respondents through a combination of using local terms and explanations, reordering some questions, and adapting questions to better match respondents' approaches to estimating numeric responses.

The RADAR coverage tool: developing a toolkit for rigorous household surveys for reproductive, maternal, newborn, and child health & nutrition indicators

Population-based intervention coverage data are used to inform the design of projects, programs, and policies and to evaluate their impact. In low- and middle-income countries (LMICs), household surveys are the primary source of coverage data. Many coverage surveys are implemented by organizations with limited experience or resources in population-based data collection. We developed a streamlined survey and set of supporting materials to facilitate rigorous survey design and implementation. The RADAR coverage survey tool aimed to 1) rigorously measure priority reproductive, maternal, newborn, child health & nutrition coverage indicators, and allow for equity and gender analyses; 2) use standard, valid questions, to the extent possible; 3) be as light as possible; 4) be flexible to address users’ needs; and 5) be compatible with the Lives Saved Tool for analysis of program impact. Early interactions with stakeholders also highlighted survey planning, implementation, and analysis as challenging areas. We therefore developed a suite of resources to support implementers in these areas. The toolkit was piloted by implementers in Tanzania and in Burkina Faso. Although the toolkit was successfully implemented in these settings and facilitated survey planning and implementation, we found that implementers must still have access to sufficient resources, time, and technical expertise in order to use the tool appropriately. This potentially limits the use of the tool to situations where high-quality surveys or evaluations have been prioritized and adequately resourced.

Strengthening effectiveness evaluations to improve programs for women, children and adolescents

A full understanding of the pathways from efficacious interventions to population impact requires rigorous effectiveness evaluations conducted under realistic scale-up conditions at country level. In this paper, we introduce a deductive framework that underpins effectiveness evaluations. This framework forms the theoretical and conceptual basis for the 'Real Accountability: Data Analysis for Results' (RADAR) project, intended to address gaps in guidance and tools for the evaluation of projects being implemented at scale to reduce mortality among women and children. These gaps include needs for a framework to guide decisions about evaluations and practical measurement tools, as well as increased capacity in evaluation practice among donors and program planners at global, national and project levels. RADAR aimed to improve the evidence base for program and policy decisions in reproductive, maternal, newborn and child health and nutrition (RMNCH&N). We focus on five linked methodological steps - presented as core evaluation questions - for designing and implementing effectiveness evaluation of large-scale programs that support both the needs of program managers to improve their programs and the needs of donors to meet their accountability responsibilities. RADAR has operationalized each step with a tool to facilitate its application. We also describe cross-cutting methodological issues and broader contextual factors that affect the planning and implementation of such evaluations. We conclude with proposals for how the global RMNCH&N community can support rigorous program evaluations and make better use of the resulting evidence.

Validation of maternal report of nutrition-related interventions and counselling during antenatal care in southern Nepal

The delivery of nutrition-related interventions and counselling during antenatal care is critical for a healthy pregnancy for both mother and child. However, the accuracy of maternal reports of many of these services during household surveys has not yet been examined. Our objectives were to assess the validity of the maternal reports of 10 antenatal nutrition interventions, including counselling, and examine associates between maternal characteristics and accuracy. Maternal report of services received collected during a post-partum survey was compared to the gold standard, the direct observation of all women's antenatal care visits. Individual-level validity was assessed by calculating indicator sensitivity, specificity and area under the operating curve (AUC). The inflation factor (IF) measured population-level bias. For five indicators, the high true coverage limited our ability to assess the validity of the maternal reports. There were no indicators that had both high individual-level validity (AUC > 0.70) and low population bias (0.75 < IF < 1.25). Indicators with greater true coverage estimates had higher sensitivity and lower specificity estimates compared to those indicators with lower true coverage. There were no maternal characteristics associated with the accuracy of the report. Maternal report of antenatal nutrition-related interventions and counselling during household surveys was found to have variable validity across indicators. Additional research in settings with varying coverage levels should be considered to best inform antenatal care coverage measurement in household surveys.

To call or not to call: exploring the validity of telephone interviews to derive maternal self-reports of experiences with facility childbirth care in northern Nigeria

Background

To institutionalise respectful maternity care, frequent data on the experience of childbirth care is needed by health facility staff and managers. Telephone interviews have been proposed as a low-cost alternative to derive timely and actionable maternal self-reports of experience of care. However, evidence on the validity of telephone interviews for this purpose is limited.

Methods

Eight indicators of positive maternity care experience and 18 indicators of negative maternity care experience were investigated. We compared the responses from exit interviews with women about their childbirth care experience (reference standard) to follow-up telephone interviews with the same women 14 months after childbirth. We calculated individual-level validity metrics including, agreement, sensitivity, specificity, area under the receiver operating characteristic curve (AUC). We compared the characteristics of women included in the telephone follow-up interviews to those from the exit interviews.

Results

Demographic characteristics were similar between the original exit interview group (n=388) and those subsequently reached for telephone interview (n=294). Seven of the eight positive maternity care experience indicators had reported prevalence higher than 50% at both exit and telephone interviews. For these indicators, agreement between the exit and the telephone interviews ranged between 50% and 92%; seven positive indicators met the criteria for validation analysis, but all had an AUC below 0.6. Reported prevalence for 15 of the 18 negative maternity care experience indicators was lower than 5% at exit and telephone interviews. For these 15 indicators, agreement between exit and telephone interview was high at over 80%. Just three negative indicators met the criteria for validation analysis, and all had an AUC below 0.6.

Conclusions

The telephone interviews conducted 14 months after childbirth did not yield results that were consistent with exit interviews conducted at the time of facility discharge. Women’s reports of experience of childbirth care may be influenced by the location of reporting or changes in the recall of experiences of care over time.

Validation of Maternal Report of Receipt of Iron-Folic Acid Supplementation during Antenatal Care in Rural Southern Nepal

BACKGROUND

Coverage of iron-folic acid (IFA) supplementation is a key indicator for tracking programmatic progress within and across countries. However, the validity of maternal report of this information during household surveys has yet to be determined.

OBJECTIVES

This study aimed to examine the validity of maternal recall of receipt of IFA supplementation during antenatal care (ANC) and factors associated with accuracy of maternal recall.

METHODS

A longitudinal cohort design was employed. The direct observation of the IFA received during all ANC visits at the 5 study health posts served as the "gold standard" to the maternal report of IFA received during the postpartum interview. Individual-level validity was assessed by calculating indicator sensitivity, specificity, and AUC. The inflation factor (IF) measured population-level bias. A multivariable log-binomial model was used to assess factors associated with accurate recall.

RESULTS

The majority (95.8%) of women were observed receiving IFA during pregnancy. Women overreported the number of IFA tablets received compared with what was observed during ANC visits (mean difference: 45 tablets). Maternal report of any IFA receipt was moderate (AUC = 0.60; 95% CI: 0.50, 0.71), and population bias was low (IF = 1.01). However, the individual-level validity was poor across the 7 IFA tablet count categories; the AUC for categories ranged from misleading to moderate. Driven by the trend of maternal overreport, the IF indicated that maternal report drastically underestimated the coverage of lower tablet categories and overestimated the coverage of higher tablet counts. Accuracy of maternal report was not associated with months since last ANC observation nor any maternal characteristics.

CONCLUSIONS

Maternal report of the amount of IFA supplementation received during pregnancy produced extremely biased population coverage and performed poorly to moderately for individual-level validity. It is imperative to improve this indicator because it is used in global frameworks and national program planning.

Comparability of family planning quality of care measurement tools in low-and-middle income country settings: a systematic review

BACKGROUND

In low-and-middle income countries (LMICs), accurate measures of the elements of quality care provided by a health worker through family planning services (also known as process quality) are required to ensure family's contraceptives needs are being met. There are many tools used to assess family planning process quality of care (QoC) but no one standardized method. Those measuring QoC in LMICs should select an appropriate tool based the program context and financial/logistical parameters, but they require data on how well each tool measures routine clinical care. We aim to synthesize the literature on validity/comparability of family planning process QoC measurement tools through a quantitative systematic review with no meta-analysis.

METHODS

We searched six literature databases for studies that compared quality measurements from different tools using quantitative statistics such as sensitivity/specificity, kappa statistic or absolute difference. We extracted the comparative measure along with other relevant study information, organized by quality indicator domain (e.g. counseling and privacy), and then classified the measure by low, medium, and high agreement.

RESULTS

We screened 8172 articles and identified eight for analysis. Studies comparing quality measurements from simulated clients, direct observation, client exit interview, provider knowledge quizzes, and medical record review were included. These eight studies were heterogenous in their methods and the measurements compared. There was insufficient data to estimate overall summary measures of validity for the tools. Client exit interviews compared to direct observation or simulated client protocols had the most data and they were a poor proxy of the actual quality care received for many measurements.

CONCLUSION

To measure QoC consistently and accurately in LMICs, standardized tools and measures are needed along with an established method of combining them for a comprehensive picture of quality care. Data on how different tools proxy quality client care will inform these guidelines. Despite the small number of studies found during the review, we described important differences on how tools measure quality of care.

Survey of women's report for 33 maternal and newborn indicators: EN-BIRTH multi-country validation study

BACKGROUND

Population-based household surveys, notably the Demographic and Health Surveys (DHS) and Multiple Indicator Cluster Surveys (MICS), remain the main source of maternal and newborn health data for many low- and middle-income countries. As part of the Every Newborn Birth Indicators Research Tracking in Hospitals (EN-BIRTH) study, this paper focuses on testing validity of measurement of maternal and newborn indicators around the time of birth (intrapartum and postnatal) in survey-report.

METHODS

EN-BIRTH was an observational study testing the validity of measurement for selected maternal and newborn indicators in five secondary/tertiary hospitals in Bangladesh, Nepal and Tanzania, conducted from July 2017 to July 2018. We compared women's report at exit survey with the gold standard of direct observation or verification from clinical records for women with vaginal births. Population-level validity was assessed by validity ratios (survey-reported coverage: observer-assessed coverage). Individual-level accuracy was assessed by sensitivity, specificity and percent agreement. We tested indicators already in DHS/MICS as well as indicators with potential to be included in population-based surveys, notably the first validation for small and sick newborn care indicators.

RESULTS

33 maternal and newborn indicators were evaluated. Amongst nine indicators already present in DHS/MICS, validity ratios for baby dried or wiped, birthweight measured, low birthweight, and sex of baby (female) were between 0.90-1.10. Instrumental birth, skin-to-skin contact, and early initiation of breastfeeding were highly overestimated by survey-report (2.04-4.83) while umbilical cord care indicators were massively underestimated (0.14-0.22). Amongst 24 indicators not currently in DHS/MICS, two newborn contact indicators (kangaroo mother care 1.00, admission to neonatal unit 1.01) had high survey-reported coverage amongst admitted newborns and high sensitivity. The remaining indicators did not perform well and some had very high "don't know" responses.

CONCLUSIONS

Our study revealed low validity for collecting many maternal and newborn indicators through an exit survey instrument, even with short recall periods among women with vaginal births. Household surveys are already at risk of overload, and some specific clinical care indicators do not perform well and may be under-powered. Given that approximately 80% of births worldwide occur in facilities, routine registers should also be explored to track coverage of key maternal and newborn health interventions, particularly for clinical care.

Kangaroo mother care: EN-BIRTH multi-country validation study

BACKGROUND

Kangaroo mother care (KMC) reduces mortality among stable neonates ≤2000 g. Lack of data tracking coverage and quality of KMC in both surveys and routine information systems impedes scale-up. This paper evaluates KMC measurement as part of the Every Newborn Birth Indicators Research Tracking in Hospitals (EN-BIRTH) study.

METHODS

The EN-BIRTH observational mixed-methods study was conducted in five hospitals in Bangladesh, Nepal and Tanzania from 2017 to 2018. Clinical observers collected time-stamped data as gold standard for mother-baby pairs in KMC wards/corners. To assess accuracy, we compared routine register-recorded and women's exit survey-reported coverage to observed data, using different recommended denominator options (≤2000 g and ≤ 2499 g). We analysed gaps in quality of provision and experience of KMC. In the Tanzanian hospitals, we assessed daily skin-to-skin duration/dose and feeding frequency. Qualitative data were collected from health workers and data collectors regarding barriers and enablers to routine register design, filling and use.

RESULTS

Among 840 mother-baby pairs, compared to observed 100% coverage, both exit-survey reported (99.9%) and register-recorded coverage (92.9%) were highly valid measures with high sensitivity. KMC specific registers outperformed general registers. Enablers to register recording included perceptions of data usefulness, while barriers included duplication of data elements and overburdened health workers. Gaps in KMC quality were identified for position components including wearing a hat. In Temeke Tanzania, 10.6% of babies received daily KMC skin-to-skin duration/dose of ≥20 h and a further 75.3% received 12-19 h. Regular feeding ≥8 times/day was observed for 36.5% babies in Temeke Tanzania and 14.6% in Muhimbili Tanzania. Cup-feeding was the predominant assisted feeding method. Family support during admission was variable, grandmothers co-provided KMC more often in Bangladesh. No facility arrangements for other family members were reported by 45% of women at exit survey.

CONCLUSIONS

Routine hospital KMC register data have potential to track coverage from hospital KMC wards/corners. Women accurately reported KMC at exit survey and evaluation for population-based surveys could be considered. Measurement of content, quality and experience of KMC need consensus on definitions. Prioritising further KMC measurement research is important so that high quality data can be used to accelerate scale-up of high impact care for the most vulnerable.

Uterotonics for prevention of postpartum haemorrhage: EN-BIRTH multi-country validation study

BACKGROUND

Postpartum haemorrhage (PPH) is a leading cause of preventable maternal mortality worldwide. The World Health Organization (WHO) recommends uterotonic administration for every woman after birth to prevent PPH. There are no standardised data collected in large-scale measurement platforms. The Every Newborn Birth Indicators Research Tracking in Hospitals (EN-BIRTH) is an observational study to assess the validity of measurement of maternal and newborn indicators, and this paper reports findings regarding measurement of coverage and quality for uterotonics.

METHODS

The EN-BIRTH study took place in five hospitals in Bangladesh, Nepal and Tanzania, from July 2017 to July 2018. Clinical observers collected tablet-based, time-stamped data. We compared observation data for uterotonics to routine hospital register-records and women's report at exit-interview survey. We analysed the coverage and quality gap for timing and dose of administration. The register design was evaluated against gap analyses and qualitative interview data assessing the barriers and enablers to data recording and use.

RESULTS

Observed uterotonic coverage was high in all five hospitals (> 99%, 95% CI 98.7-99.8%). Survey-report underestimated coverage (79.5 to 91.7%). "Don't know" replies varied (2.1 to 14.4%) and were higher after caesarean (3.7 to 59.3%). Overall, there was low accuracy in survey data for details of uterotonic administration (type and timing). Register-recorded coverage varied in four hospitals capturing uterotonics in a specific column (21.6, 64.5, 97.6, 99.4%). The average coverage measurement gap was 18.1% for register-recorded and 6.0% for survey-reported coverage. Uterotonics were given to 15.9% of women within the "right time" (1 min) and 69.8% within 3 min. Women's report of knowing the purpose of uterotonics after birth ranged from 0.4 to 64.9% between hospitals. Enabling register design and adequate staffing were reported to improve routine recording.

CONCLUSIONS

Routine registers have potential to track uterotonic coverage - register data were highly accurate in two EN-BIRTH hospitals, compared to consistently underestimated coverage by survey-report. Although uterotonic coverage was high, there were gaps in observed quality for timing and dose. Standardisation of register design and implementation could improve data quality and data flow from registers into health management information reporting systems, and requires further assessment.

A, Ruysen H, El Arifeen S, Mee P, Gladstone ME, Blencowe H, Lawn JE. Birthweight: EN-BIRTH multi-country validation study

BACKGROUND

Accurate birthweight is critical to inform clinical care at the individual level and tracking progress towards national/global targets at the population level. Low birthweight (LBW) < 2500 g affects over 20.5 million newborns annually. However, data are lacking and may be affected by heaping. This paper evaluates birthweight measurement within the Every Newborn Birth Indicators Research Tracking in Hospitals (EN-BIRTH) study.

METHODS

The EN-BIRTH study took place in five hospitals in Bangladesh, Nepal and Tanzania (2017-2018). Clinical observers collected time-stamped data (gold standard) for weighing at birth. We compared accuracy for two data sources: routine hospital registers and women's report at exit interview survey. We calculated absolute differences and individual-level validation metrics. We analysed birthweight coverage and quality gaps including timing and heaping. Qualitative data explored barriers and enablers for routine register data recording.

RESULTS

Among 23,471 observed births, 98.8% were weighed. Exit interview survey-reported weighing coverage was 94.3% (90.2-97.3%), sensitivity 95.0% (91.3-97.8%). Register-reported coverage was 96.6% (93.2-98.9%), sensitivity 97.1% (94.3-99%). Routine registers were complete (> 98% for four hospitals) and legible > 99.9%. Weighing of stillbirths varied by hospital, ranging from 12.5-89.0%. Observed LBW rate was 15.6%; survey-reported rate 14.3% (8.9-20.9%), sensitivity 82.9% (75.1-89.4%), specificity 96.1% (93.5-98.5%); register-recorded rate 14.9%, sensitivity 90.8% (85.9-94.8%), specificity 98.5% (98-99.0%). In surveys, "don't know" responses for birthweight measured were 4.7%, and 2.9% for knowing the actual weight. 95.9% of observed babies were weighed within 1 h of birth, only 14.7% with a digital scale. Weight heaping indices were around two-fold lower using digital scales compared to analogue. Observed heaping was almost 5% higher for births during the night than day. Survey-report further increased observed birthweight heaping, especially for LBW babies. Enablers to register birthweight measurement in qualitative interviews included digital scale availability and adequate staffing.

CONCLUSIONS

Hospital registers captured birthweight and LBW prevalence more accurately than women's survey report. Even in large hospitals, digital scales were not always available and stillborn babies not always weighed. Birthweight data are being captured in hospitals and investment is required to further improve data quality, researching of data flow in routine systems and use of data at every level.

The Validity of Women's Reports of Family Planning Service Quality in Cambodia and Kenya

Population-based indicators of the coverage of key elements of high-quality family planning services are tracked via household surveys with female respondents, yet little work has been done to establish their validity. We take advantage of existing data sets from Cambodia and Kenya to compare women's responses at exit interviews following a health facility visit against the observations of a trained third-party observer during the visit. The results, which treat the observations as the reference standard, show that indicators that measure contraceptive methods received are accurately reported while indicators of whether the woman received her preferred method and whether information was "discussed" or "explained" during counseling are less reliably reported. Studies designed explicitly to assess the validity of family planning questions in household surveys, especially questions in large survey programs critical for monitoring demographic trends and programmatic coverage, are needed.

Assessment of the validity of the measurement of newborn and maternal health-care coverage in hospitals (EN-BIRTH): an observational study

BACKGROUND

Progress in reducing maternal and neonatal deaths and stillbirths is impeded by data gaps, especially regarding coverage and quality of care in hospitals. We aimed to assess the validity of indicators of maternal and newborn health-care coverage around the time of birth in survey data and routine facility register data.

METHODS

Every Newborn-BIRTH Indicators Research Tracking in Hospitals was an observational study in five hospitals in Bangladesh, Nepal, and Tanzania. We included women and their newborn babies who consented on admission to hospital. Exclusion critiera at admission were no fetal heartbeat heard or imminent birth. For coverage of uterotonics to prevent post-partum haemorrhage, early initiation of breastfeeding (within 1 h), neonatal bag-mask ventilation, kangaroo mother care (KMC), and antibiotics for clinically defined neonatal infection (sepsis, pneumonia, or meningitis), we collected time-stamped, direct observation or case note verification data as gold standard. We compared data reported via hospital exit surveys and via hospital registers to the gold standard, pooled using random effects meta-analysis. We calculated population-level validity ratios (measured coverage to observed coverage) plus individual-level validity metrics.

FINDINGS

We observed 23 471 births and 840 mother-baby KMC pairs, and verified the case notes of 1015 admitted newborn babies regarding antibiotic treatment. Exit-survey-reported coverage for KMC was 99·9% (95% CI 98·3-100) compared with observed coverage of 100% (99·9-100), but exit surveys underestimated coverage for uterotonics (84·7% [79·1-89·5]) vs 99·4% [98·7-99·8] observed), bag-mask ventilation (0·8% [0·4-1·4]) vs 4·4% [1·9-8·1]), and antibiotics for neonatal infection (74·7% [55·3-90·1] vs 96·4% [94·0-98·6] observed). Early breastfeeding coverage was overestimated in exit surveys (53·2% [39·4-66·8) vs 10·9% [3·8-21·0] observed). "Don't know" responses concerning clinical interventions were more common in the exit survey after caesarean birth. Register data underestimated coverage of uterotonics (77·9% [37·8-99·5] vs 99·2% [98·6-99·7] observed), bag-mask ventilation (4·3% [2·1-7·3] vs 5·1% [2·0-9·6] observed), KMC (92·9% [84·2-98·5] vs 100% [99·9-100] observed), and overestimated early breastfeeding (85·9% (58·1-99·6) vs 12·5% [4·6-23·6] observed). Inter-hospital heterogeneity was higher for register-recorded coverage than for exit survey report. Even with the same register design, accuracy varied between hospitals.

INTERPRETATION

Coverage indicators for newborn and maternal health care in exit surveys had low accuracy for specific clinical interventions, except for self-report of KMC, which had high sensitivity after admission to a KMC ward or corner and could be considered for further assessment. Hospital register design and completion are less standardised than surveys, resulting in variable data quality, with good validity for the best performing sites. Because approximately 80% of births worldwide take place in facilities, standardising register design and information systems has the potential to sustainably improve the quality of data on care at birth.

FUNDING

Children's Investment Fund Foundation and Swedish Research Council.

A, Salim N, Zaman SB, Shamba D, Blencowe H, Ruysen H, El Arifeen S, Boggs D, Gordeev VS, Rahman QSU, Hossain T, Joshi E, Thapa S, Poudel RP, Poudel D, Chaudhary P, Karki R, Chitrakar B, Mkopi N, Wisiko A, Kitende AP, Shirati MR, Chingalo C, Semhando AO, Mtei C, Mwenisongole V, Bakuza JM, Kombo J, Mbaruku G, Lawn JE. Labour and delivery ward register data availability, quality, and utility - Every Newborn - birth indicators research tracking in hospitals (EN-BIRTH) study baseline analysis in three countries

BACKGROUND

Countries with the highest burden of maternal and newborn deaths and stillbirths often have little information on these deaths. Since over 81% of births worldwide now occur in facilities, using routine facility data could reduce this data gap. We assessed the availability, quality, and utility of routine labour and delivery ward register data in five hospitals in Bangladesh, Nepal, and Tanzania. This paper forms the baseline register assessment for the Every Newborn-Birth Indicators Research Tracking in Hospitals (EN-BIRTH) study.

METHODS

We extracted 21 data elements from routine hospital labour ward registers, useful to calculate selected maternal and newborn health (MNH) indicators. The study sites were five public hospitals during a one-year period (2016-17). We measured 1) availability: completeness of data elements by register design, 2) data quality: implausibility, internal consistency, and heaping of birthweight and explored 3) utility by calculating selected MNH indicators using the available data.

RESULTS

Data were extracted for 20,075 births. Register design was different between the five hospitals with 10-17 of the 21 selected MNH data elements available. More data were available for health outcomes than interventions. Nearly all available data elements were > 95% complete in four of the five hospitals and implausible values were rare. Data elements captured in specific columns were 85.2% highly complete compared to 25.0% captured in non-specific columns. Birthweight data were less complete for stillbirths than live births at two hospitals, and significant heaping was found in all sites, especially at 2500g and 3000g. All five hospitals recorded count data required to calculate impact indicators including; stillbirth rate, low birthweight rate, Caesarean section rate, and mortality rates.

CONCLUSIONS

Data needed to calculate MNH indicators are mostly available and highly complete in EN-BIRTH study hospital routine labour ward registers in Bangladesh, Nepal and Tanzania. Register designs need to include interventions for coverage measurement. There is potential to improve data quality if Health Management Information Systems utilization with feedback loops can be strengthened. Routine health facility data could contribute to reduce the coverage and impact data gap around the time of birth.

Progress towards reducing sociodemographic disparities in breastfeeding outcomes in Indonesia: a trend analysis from 2002 to 2017

Background

Improving breastfeeding practice is important for reducing child health inequalities and achieving several Sustainable Development Goals. Indonesia has enacted legislation to promote optimal breastfeeding practices in recent years. We examined breastfeeding practices among Indonesian women from 2002 to 2017, comparing trends within and across sociodemographic subgroups.

Methods

Data from four waves of the Indonesia Demographic and Health Surveys were used to estimate changes in breastfeeding practices among women from selected sociodemographic groups over time. We examined three breastfeeding outcomes: (1) early initiation of breastfeeding; (2) exclusive breastfeeding; and (3) continued breastfeeding at 1 year. Multivariate logistic regression was used to assess changes in time trends of each outcome across population groups.

Results

The proportion of women reporting early initiation of breastfeeding and exclusive breastfeeding increased significantly between 2002 to 2017 (p < 0.05), with larger increases among women who: were from higher wealth quintiles; worked in professional sectors; and lived in Java and Bali. However, 42.7% of women reported not undertaking early initiation of breastfeeding, and 48.9% of women reported not undertaking exclusive breastfeeding in 2017. Women who were employees had lower exclusive breastfeeding prevalence, compared to unemployed or self-employed women. Women in Java and Bali had higher increase in early initiation of breastfeeding and exclusive breastfeeding compared to women in Sumatra. We did not find statistically significant decline in continued breastfeeding at 1 year over time for the overall population, except among women who: were from the second poorest wealth quintile; lived in rural areas; did not have a health facility birth; and lived in Kalimantan and Sulawesi (p < 0.05).

Conclusions

There were considerable improvements in breastfeeding practices in Indonesia during a period of sustained policy reform to regulate breastfeeding and community support of breastfeeding, but these were not distributed uniformly across socioeconomic, occupation and geographic subgroups. Concerted efforts are needed to further reduce inequities in breastfeeding practice through both targeted and population-based strategies.

What is meant by validity in maternal and newborn health measurement? A conceptual framework for understanding indicator validation

BACKGROUND

Rigorous monitoring supports progress in achieving maternal and newborn mortality and morbidity reductions. Recent work to strengthen measurement for maternal and newborn health highlights the existence of a large number of indicators being used for this purpose. The definitions and data sources used to produce indicator estimates vary and challenges exist with completeness, accuracy, transparency, and timeliness of data. The objective of this study is to create a conceptual overview of how indicator validity is defined and understood by those who develop and use maternal and newborn health indicators.

METHODS

A conceptual framework of validity was developed using mixed methods. We were guided by principles for conceptual frameworks and by a review of the literature and key maternal and newborn health indicator guidance documents. We also conducted qualitative semi-structured interviews with 32 key informants chosen through purposive sampling.

RESULTS

We categorised indicator validity into three main types: criterion, convergent, and construct. Criterion or diagnostic validity, comparing a measure with a gold standard, has predominantly been used to assess indicators of care coverage and content. Studies assessing convergent validity quantify the extent to which two or more indicator measurement approaches, none of which is a gold-standard, relate. Key informants considered construct validity, or the accuracy of the operationalisation of a concept or phenomenon, a critical part of the overall assessment of indicator validity.

CONCLUSION

Given concerns about the large number of maternal and newborn health indicators currently in use, a more consistent understanding of validity can help guide prioritization of key indicators and inform development of new indicators. All three types of validity are relevant for evaluating the performance of maternal and newborn health indicators. We highlight the need to establish a common language and understanding of indicator validity among the various global and local stakeholders working within maternal and newborn health.

Validating women’s reports of antenatal and postnatal care received in Bangladesh, Cambodia and Kenya

Background

Global indicators for monitoring progress in maternal and newborn health have tended to rely on contact coverage indicators rather than the content of services received. As part of the effort to improve measurement of progress in maternal and newborn health, this study examines how accurately women can report on information and health interventions received during an antenatal or postnatal health consultation at health facilities in Bangladesh, Cambodia and Kenya.

Methods

We conducted secondary analysis of matched observation and client interview data to compare women’s reports of care received at exit interview with observation by a trained third-party observer. We assessed indicator accuracy by calculating sensitivity, specificity, area under the receiver operating characteristic curve (AUC) and inflation factor (IF). Indicators considered to have both high individual accuracy (an AUC value of 0.70 or greater) and low population-level bias (0.75<IF<1.25) were considered to have acceptable validity. In addition, we considered the number of countries where both validation criteria were met.

Results

For indicators of antenatal care, we found 16 of 18 indicators in Bangladesh, 3 of 6 in Cambodia and 3 of 8 in Kenya met both validation criteria. For postnatal care, we found evidence of acceptable validity for 6 of 8 indicators in Bangladesh, 5 of 14 in Cambodia and 3 of 16 in Kenya. In general, we documented higher validity for indicators related to concrete, observable actions, as opposed to information or advice given. Women were more likely to recall care received for themselves, rather than for their newborn.

Conclusions

Women reported accurately on multiple aspects of antenatal and postnatal care. While we describe broad patterns in the types of indicators likely to be recalled with accuracy, differences by setting warrant further investigation. Findings inform efforts to better monitor the coverage and quality of maternal and newborn health interventions.

Measuring coverage of infant and young child feeding counselling interventions: A framework and empirical considerations for survey question design

Most countries implement nutrition counselling interventions as part of programmes to support breastfeeding and complementary feeding. However, data to track coverage of counselling interventions are rarely available. As a result, little is known about the coverage of counselling on infant and young child feeding (IYCF). Survey-based data collection systems generally collect data on IYCF practices but do not collect data on coverage of interventions to support IYCF, and those surveys that do collect this information do not do so consistently. We present a framework to guide the design of survey questions to measure IYCF counselling coverage. We provide examples of how large-scale surveys for programme evaluation and national monitoring have included survey questions to address these data gaps. Our review suggests that elements relevant to designing survey questions to capture coverage of counselling interventions include timing of contact, target behaviour and message content, place of contact, type of service provider, frequency of contact and mode of intervention. Application of this framework may help strengthen harmonized measurement of IYCF counselling coverage to enable better tracking of programme investments, document progress in scaling up nutrition services and allow for cross-country comparisons. Thus, improving measurement of counselling coverage may lead to improved reach of programmes to support optimal IYCF practices.

Monitoring childbirth care in primary health facilities: a validity study in Gombe State, northeastern Nigeria

BACKGROUND

Improving the quality of facility-based births is a critical strategy for reducing the high burden of maternal and neonatal mortality and morbidity across all settings. Accurate data on childbirth care is essential for monitoring progress. In northeastern Nigeria, we assessed the validity of childbirth care indicators in a rural primary health care context, as documented by health workers and reported by women at different recall periods.

METHODS

We compared birth observations (gold standard) to: (i) facility exit interviews with observed women; (ii) household follow-up interviews 9-22 months after childbirth; and (iii) health worker documentation in the maternity register. We calculated sensitivity, specificity, and area under the receiver operating curve (AUC) to determine individual-level reporting accuracy. We calculated the inflation factor (IF) to determine population-level validity.

RESULTS

Twenty-five childbirth care indicators were assessed to validate health worker documentation and women's self-reports. During exit interviews, women's recall had high validity (AUC≥0.70 and 0.75

CONCLUSIONS

In addition to standard household surveys, monitoring of facility-based childbirth care should consider drawing from and linking multiple data sources, including routine health facility data and exit interviews with recently delivered women.

Collapse

Key Words Collapse

MESH Headings

• Adolescent
• Adult
• Delivery, Obstetric/standards
• Documentation/methods
• Female
• Humans
• Infant, Newborn
• Medical Records
• Middle Aged
• Nigeria
• Pregnancy
• Primary Health Care
• Reproducibility of Results
• Rural Health Services
• Self Report
• Young Adult

Collapse

Grants Collapse

Affiliation(s)

Antoinette Alas Bhattacharya Department of Disease Control, London School of Hygiene & Tropical Medicine, London, UK
Elizabeth Allen Department of Medical Statistics, London School of Hygiene & Tropical Medicine, London, UK
Nasir Umar Department of Disease Control, London School of Hygiene & Tropical Medicine, London, UK
Adamu Umar Usman Data Research and Mapping Consult, Abuja, Nigeria
Habila Felix State Primary Health Care Development Agency, Gombe, Nigeria
Ahmed Audu State Primary Health Care Development Agency, Gombe, Nigeria
Joanna Rm Schellenberg Department of Disease Control, London School of Hygiene & Tropical Medicine, London, UK
Tanya Marchant Department of Disease Control, London School of Hygiene & Tropical Medicine, London, UK

Collapse

"What gets measured better gets done better": The landscape of validation of global maternal and newborn health indicators through key informant interviews

Background

A large number of indicators are currently used to monitor the state of maternal and newborn health, including those capturing dimensions of health system and input, care access and availability, care quality and safety, coverage and outcomes, and impact. Validity of these indicators is a key issue in the process of assessing indicator performance and suitability. This paper aims to understand the meaning of indicator validity in the field of maternal and newborn health, and to identify key recommendations for future research.

Methods

This qualitative study used purposive sampling to identify key informants until thematic saturation was achieved. We interviewed 32 respondents from a variety of backgrounds using semi-structured interviews covering five themes: the meaning of indicator validity, methodological approaches to assessing validity, acceptable levels of indicator validity, gaps in validation research, and recommendations for addressing these gaps. Interview transcripts were analysed data using thematic content approach.

Results

Three conceptually different definitions of indicator validity were described by respondents. They considered indicator validity to encompass meaning and potential to spur action, going beyond diagnostic validity. Indicator validation was seen as an ongoing process of building and synthesising a wide range of evidence rather than a one-size-fits-all cut-off in diagnostic validity tests. Gaps identified included assessing validity of indicators of quality of care and indicators based on facility-level data, as well as expanding studies to a broader range of global settings. The key recommendation was to develop a coordinated approach to summarising and evaluating research on indicator validity, including capacity building in appraising and communicating the available evidence for country-specific needs.

Conclusion

The findings will inform future recommendations around indicator testing and validation.

Validity of self-reported receipt of iron supplements during pregnancy: implications for coverage measurement

Background

Iron-deficiency anemia during pregnancy is an underlying cause of maternal deaths, and reducing risk through routine iron supplementation is a key component of antenatal care (ANC) programs in most low- and middle income countries. Supplementation coverage during pregnancy is estimated from maternal self-reports in population-based household surveys, yet recall bias and social desirability bias lead to errors of unknown magnitude.

Methods

We linked data from household and health facility surveys from 16 countries to estimate input-adjusted coverage of iron supplementation during pregnancy. We assessed the validity of reported receipt of iron supplements in client exit interviews using direct observation as the gold standard across 9 countries with a recent Service Provision Assessment (SPA). Using a sample of 227 women who participated in the Nepal Oil Massage Study (NOMS), we also assessed the validity of self-reported receipt of iron folic acid (IFA) supplements. We used Poisson regression models to explore the association between client and health facility characteristics and agreement of self-reported receipt of iron supplements compared to direct observation.

Results

Across the 16 countries, iron supplements were in supply at most of the 9215 sampled health facilities offering ANC services (91%). We estimated that between 48 and 93% of women attended at least one ANC visit at a health facility with iron supplements available. The specificity of recall of receipt of iron supplementation immediately following a visit was 79.3% and the sensitivity was 88.7% for the entire sample. Individual-level accuracy was high (Area under the curve > 0.7) and population bias low (0.75 < inflation factor < 1.25) across all countries. By contrast, in the NOMS sub-study, the accuracy of self-reported receipt of IFA supplements after 1–2 years was poor (sensitivity 86.1%, specificity 34.3%). Adjusted regression analyses indicated that older age and higher level of education were associated with poorer agreement between self-reports and direct observation.

Conclusions

These findings suggest the need for caution when using self-reported measures with an extended recall period. Further validation studies using conditions similar to widely used population-based household surveys are warranted.

"Every Newborn-BIRTH" protocol: observational study validating indicators for coverage and quality of maternal and newborn health care in Bangladesh, Nepal and Tanzania

Background

To achieve Sustainable Development Goals and Universal Health Coverage, programmatic data are essential. The Every Newborn Action Plan, agreed by all United Nations member states and >80 development partners, includes an ambitious Measurement Improvement Roadmap. Quality of care at birth is prioritised by both Every Newborn and Ending Preventable Maternal Mortality strategies, hence metrics need to advance from health service contact alone, to content of care. As facility births increase, monitoring using routine facility data in DHIS2 has potential, yet validation research has mainly focussed on maternal recall surveys. The Every Newborn – Birth Indicators Research Tracking in Hospitals (EN-BIRTH) study aims to validate selected newborn and maternal indicators for routine tracking of coverage and quality of facility-based care for use at district, national and global levels.

Methods

EN-BIRTH is an observational study including >20 000 facility births in three countries (Tanzania, Bangladesh and Nepal) to validate selected indicators. Direct clinical observation will be compared with facility register data and a pre-discharge maternal recall survey for indicators including: uterotonic administration, immediate newborn care, neonatal resuscitation and Kangaroo mother care. Indicators including neonatal infection management and antenatal corticosteroid administration, which cannot be easily observed, will be validated using inpatient records. Trained clinical observers in Labour/Delivery ward, Operation theatre, and Kangaroo mother care ward/areas will collect data using a tablet-based customised data capturing application. Sensitivity will be calculated for numerators of all indicators and specificity for those numerators with adequate information. Other objectives include comparison of denominator options (ie, true target population or surrogates) and quality of care analyses, especially regarding intervention timing. Barriers and enablers to routine recording and data usage will be assessed by data flow assessments, quantitative and qualitative analyses.

Conclusions

To our knowledge, this is the first large, multi-country study validating facility-based routine data compared to direct observation for maternal and newborn care, designed to provide evidence to inform selection of a core list of indicators recommended for inclusion in national DHIS2. Availability and use of such data are fundamental to drive progress towards ending the annual 5.5 million preventable stillbirths, maternal and newborn deaths.

Measuring malaria diagnosis and treatment coverage in population-based surveys: a recall validation study in Mali among caregivers of febrile children under 5 years

BACKGROUND

Nationally-representative household surveys are the standard approach to monitor access to and treatment with artemisinin-based combination therapy (ACT) among children under 5 years (U5), however these indicators are dependent on caregivers' recall of the treatment received.

METHODS

A prospective case-control study was performed in Mali to validate caregivers' recall of treatment received by U5s when seeking care for fever from rural and urban public health facilities, community health workers and urban private facilities. Clinician-recorded consultation details were the gold standard. Consenting caregivers were followed-up for interview at home within 2 weeks using standard questions from Demographic and Health Surveys and Malaria Indicator Surveys.

RESULTS

Among 1602 caregivers, sensitivity of recalling that the child received a finger/heel prick was 91.5%, with specificity 85.7%. Caregivers' recall of a positive malaria test result had sensitivity 96.2% with specificity 59.7%. Irrespective of diagnostic test result, the sensitivity and specificity of caregivers' recalling a malaria diagnosis made by the health worker were 74.3% and 74.9%, respectively. Caregivers' recall of ACT being given had sensitivity of 43.2% and specificity 90.2%, while recall that any anti-malarial was given had sensitivity 59.0% and specificity 82.7%. Correcting caregivers' response of treatment received using a combination of a visual aid with photographs of common drugs for fever, prescription documents and retained packaging changed ACT recall sensitivity and specificity to 91.5% and 71.1%, respectively.

CONCLUSIONS

These findings indicate that caregivers' responses during household surveys are valid when assessing if a child received a finger/heel prick during a consultation in the previous 2 weeks, and if the malaria test result was positive. Recall of ACT treatment received by U5s was poor when based on interview response only, but was substantially improved when incorporating visual aids, prescriptions and drug packaging review.

A prospective validation study in South-West Nigeria on caregiver report of childhood pneumonia and antibiotic treatment using Demographic and Health Survey (DHS) and Multiple Indicator Cluster Survey (MICS) questions

Background

Childhood pneumonia is the single largest infectious cause of death in children under five worldwide. Demographic and Health Surveys (DHS) and Multiple Indicator Cluster Surveys (MICS) provide health information on care sought for sick children in resource poor settings. Despite not being primarily designed to identify childhood pneumonia, there are concerns that reported episodes of "symptoms of acute respiratory infection" in DHS and MICS are often interpreted by other groups as a "proxy" for childhood pneumonia. Using DHS5 and MICS5 survey tools, this study aimed to assess how accurately caregivers report of "symptoms of acute respiratory infection" reflect pneumonia episodes and antibiotic use in children under five.

Methods

Children aged 0 to 59 months presenting with cough and/or difficult breathing were recruited from four study hospitals in Ibadan, Nigeria from August 2015 to March 2017. Children were assessed using World Health Organization (WHO) standard criteria by study physicians to identify whether they had pneumonia. Three hundred and two matched children in each category of 'pneumonia' and "no pneumonia" were followed up at home, either two or eight weeks later, using questions from DHS5 and MICS5 surveys to assess the accuracy of caregiver recall of pneumonia.

Results

The specificity of DHS5 and MICS5 questions for identifying childhood pneumonia were 87.4 (95% confidence interval (CI) = 83.1-91.0) and 86.1 (95% CI = 81.7-89.8) respectively and the sensitivity of questions were 37.1 (95% CI = 31.6-42.8) and 37.1 (95% CI = 31.6-42.8). Correct recall of antibiotic treatment was poor (kappa statistic = 0.064) but improved with the use of medicine pill boards (kappa statistic = 0.235).

Conclusions

DHS5 and MICS5 survey questions are not designed to identify childhood pneumonia and this study confirms that they do not accurately discern episodes of childhood pneumonia from cough/cold in children under five. The proportion of pneumonia episodes appropriately treated with antibiotics cannot be accurately assessed using current DHS and MICS surveys. If these results are used to guide programmatic decisions, it is likely to encourage overuse and inappropriate prescribing of antibiotics for episodes of cough/cold. International agencies who continue to use these household data to monitor the proportion of children with pneumonia who receive antibiotic treatment should be discouraged from doing this as these data are likely to mislead national and global programmes. Medicine pill boards are used in a number of DHS surveys and should be promoted for wider use in national population surveys to improve the accuracy of antibiotic recall.

A prospective validation study in South-West Nigeria on caregiver report of childhood pneumonia and antibiotic treatment using Demographic and Health Survey (DHS) and Multiple Indicator Cluster Survey (MICS) questions

Background

Childhood pneumonia is the single largest infectious cause of death in children under five worldwide. Demographic and Health Surveys (DHS) and Multiple Indicator Cluster Surveys (MICS) provide health information on care sought for sick children in resource poor settings. Despite not being primarily designed to identify childhood pneumonia, there are concerns that reported episodes of “symptoms of acute respiratory infection” in DHS and MICS are often interpreted by other groups as a “proxy” for childhood pneumonia. Using DHS5 and MICS5 survey tools, this study aimed to assess how accurately caregivers report of “symptoms of acute respiratory infection” reflect pneumonia episodes and antibiotic use in children under five.

Methods

Children aged 0 to 59 months presenting with cough and/or difficult breathing were recruited from four study hospitals in Ibadan, Nigeria from August 2015 to March 2017. Children were assessed using World Health Organization (WHO) standard criteria by study physicians to identify whether they had pneumonia. Three hundred and two matched children in each category of ‘pneumonia’ and “no pneumonia” were followed up at home, either two or eight weeks later, using questions from DHS5 and MICS5 surveys to assess the accuracy of caregiver recall of pneumonia.

Results

The specificity of DHS5 and MICS5 questions for identifying childhood pneumonia were 87.4 (95% confidence interval (CI) = 83.1-91.0) and 86.1 (95% CI = 81.7–89.8) respectively and the sensitivity of questions were 37.1 (95% CI = 31.6-42.8) and 37.1 (95% CI = 31.6-42.8). Correct recall of antibiotic treatment was poor (kappa statistic = 0.064) but improved with the use of medicine pill boards (kappa statistic = 0.235).

Conclusions

DHS5 and MICS5 survey questions are not designed to identify childhood pneumonia and this study confirms that they do not accurately discern episodes of childhood pneumonia from cough/cold in children under five. The proportion of pneumonia episodes appropriately treated with antibiotics cannot be accurately assessed using current DHS and MICS surveys. If these results are used to guide programmatic decisions, it is likely to encourage overuse and inappropriate prescribing of antibiotics for episodes of cough/cold. International agencies who continue to use these household data to monitor the proportion of children with pneumonia who receive antibiotic treatment should be discouraged from doing this as these data are likely to mislead national and global programmes. Medicine pill boards are used in a number of DHS surveys and should be promoted for wider use in national population surveys to improve the accuracy of antibiotic recall.