Perspectives and implications of the Improving Coverage Measurement Core Group's validation studies for household surveys

Patterns of antibiotic use for acute respiratory infections in under-three-year-old children in India: A cross-sectional study

Background

Despite its mostly viral etiology, antibiotics are frequently used to treat acute respiratory infections (ARIs) in children. India is one of the largest global consumers of antibiotics and has one of the highest rates of resistance to antimicrobial treatments. However, the epidemiology of antibiotic treatment among young children in India is poorly understood.

Methods

Using nationally representative household survey data from the Indian National Family Health Surveys (NFHS) conducted between 2015 and 2016 and 2019 and 2021, we estimated the prevalence of antibiotic use among 17 472 children under the age of three who reported ARI symptoms within two weeks before their mothers were interviewed. To assess the factors associated with antibiotic use for the treatment of ARI symptoms, we used multivariable logistic regression models that included sociodemographic, child-related, household, and health care related characteristics, with results reported on the prevalence difference (PD) scale.

Results

We estimated that 18.7% (95% CI = 17.8-19.6) of under-three-year-old (U3) children who exhibited ARI symptoms in the two weeks prior to the survey were given antibiotics as a treatment. The highest prevalence was observed in the southern and northern geographic zones of India. Furthermore, multivariable regression models indicated that children with greater access to health services were more likely to receive antibiotics for ARI treatment, regardless of the type of health care facility (public, private or pharmacy/unregulated). Additionally, the prevalence of antibiotic consumption was higher among children from families with religious affiliations other than Muslim and Hindu backgrounds (i.e. Christian, Sikh, Buddhist/neo-Buddhist, Jain, Jewish, Parsi, no religion and other) (PD = 11.7 (95% CI = 6.3-16.7)) compared to Hindu families and among mothers with a secondary or higher education (PD = 5.8 (95% CI = 1.7-9.9)) compared to mothers lacking formal education.

Conclusions

Our findings provide an important baseline for monitoring the use of antibiotics for the treatment of acute respiratory infections, and for designing interventions to mitigate potential misuse among young children in India.

Exploring the accuracy of self-reported maternal and newborn care in select studies from low and middle-income country settings: do respondent and facility characteristics affect measurement?

BACKGROUND

Accurate data on the receipt of essential maternal and newborn health interventions is necessary to interpret and address gaps in effective coverage. Validation results of commonly used content and quality of care indicators routinely implemented in international survey programs vary across settings. We assessed how respondent and facility characteristics influenced the accuracy of women's recall of interventions received in the antenatal and postnatal periods.

METHODS

We synthesized reporting accuracy using data from a known sample of validation studies conducted in Sub-Saharan Africa and Southeast Asia, which assessed the validity of women's self-report of received antenatal care (ANC) (N = 3 studies, 3,169 participants) and postnatal care (PNC) (N = 5 studies, 2,462 participants) compared to direct observation. For each study, indicator sensitivity and specificity are presented with 95% confidence intervals. Univariate fixed effects and bivariate random effects models were used to examine whether respondent characteristics (e.g., age group, parity, education level), facility quality, or intervention coverage level influenced the accuracy of women's recall of whether interventions were received.

RESULTS

Intervention coverage was associated with reporting accuracy across studies for the majority (9 of 12) of PNC indicators. Increasing intervention coverage was associated with poorer specificity for 8 indicators and improved sensitivity for 6 indicators. Reporting accuracy for ANC or PNC indicators did not consistently differ by any other respondent or facility characteristic.

CONCLUSIONS

High intervention coverage may contribute to higher false positive reporting (poorer specificity) among women who receive facility-based maternal and newborn care while low intervention coverage may contribute to false negative reporting (lower sensitivity). While replication in other country and facility settings is warranted, results suggest that monitoring efforts should consider the context of care when interpreting national estimates of intervention coverage.

Trends of HIV/AIDS knowledge and attitudes among Nigerian women between 2007 and 2017 using Multiple Indicator Cluster Survey data

Background

Globally, Nigeria ranks third among the countries with the highest number of People Living with HIV (PLHIV). Given that HIV/AIDS knowledge is a key factor that determines the risk of transmission and certain attitudes towards PLHIV, there is a need to understand the trend of HIV knowledge within the population for the purpose of assessing the progress and outcome of HIV prevention strategies. The aim of the study was to understand the trends of HIV/AIDS knowledge and attitude towards PLHIV between 2007 to 2017 among Nigerian women, and to investigate change in the factors associated with HIV/AIDS knowledge and attitude towards PLHIV over years.

Methods

Data were derived from three Nigerian Multiple Indicator Cluster Surveys (2007, 2011 and 2016–2017) among women aged 15–49 years old from each geo-political zone (South South, South East, South West, North East, North West, North Central) in Nigeria. Participants who did not answer questions related to HIV/AIDS knowledge and attitude were excluded from the study. The final sample sizes were 17,733 for 2007, 26,532 for 2011 and 23,530 for 2017. In descriptive statistics, frequencies represented the study sample, while percentages represented weighted estimates for the population parameters. Rao-Scott chi-square test for complex survey design studies was used to assess bivariable associations. Factors associated with outcome variables were examined using the survey-weighted multivariable logistic regression models for the complex survey design while controlling for potential confounding variables.

Results

There was a relatively high level of HIV/AIDS knowledge level in 2007 and 2016–2017 surveys (64.6 and 64.1%, respectively), however a decrease in HIV/AIDS knowledge trend was observed in 2011 (45.6%). The positive attitude towards PLHIV progressively increased across the years (from 40.5 to 47.0% to 53.5%). Multivariable analysis revealed that women who had a higher educational level, higher wealth index, and lived in urban areas had higher odds for HIV/AIDS knowledge and positive attitude towards PLHIV across the years. In addition, the Northern zones had predominantly higher knowledge and attitude levels.

Conclusions

Our study found increasing tendency for high HIV/AIDS knowledge and positive attitude towards PLHIV over the years. Women’s age, wealth index, education level and residence were consistently associated with knowledge and attitude over the years. There is a need for more pragmatic HIV/AIDS-related knowledge action plan to target to cover all age groups, all geo-political zones while paying close attention to the rural areas and the less educated women. In addition, more replicative studies of HIV/AIDS knowledge and attitude trends is crucial in monitoring of the progress of HIV interventions in the country in the coming years.

Using cognitive interviewing to bridge the intent-interpretation gap for nutrition coverage survey questions in India

Designing survey questions that clearly and precisely communicate the question's intent and elicit responses based on the intended interpretation is critical but often undervalued. We used cognitive interviewing to qualitatively assess respondents' interpretation of and responses to questions pertaining to maternal and child nutrition intervention coverage. We conducted interviews to cognitively test 25 survey questions with mothers (N = 21) with children less than 1 year in Madhya Pradesh, India. Each question was followed by probes to capture information on four cognitive stages—comprehension, retrieval, judgement, and response. Data were analysed for common and unique patterns across the survey questions. We identified four types of cognitive challenges: (1) retention of multiple concepts in long questions: difficulty in comprehending and retaining questions with three or more key concepts; (2) temporal confusion: difficulty in conceptualizing recall periods such as “in the last 6 months” as compared to life stages such as pregnancy; (3) interpretation of concepts: mismatch of information being asked, meaning of certain terms and intervention scope; and (4) understanding of technical terms: difficulty in understanding commonly used technical words such as “breastfeeding” and “antenatal care” and requiring use of simple alternative language. Findings from this study will be useful for stakeholders involved in survey design and implementation, especially those conducting large‐scale household surveys to measure coverage of essential nutrition interventions.

A Cross-Sectional Analysis of the Association between Domestic Cooking Energy Source Type and Respiratory Infections among Children Aged under Five Years: Evidence from Demographic and Household Surveys in 37 Low-Middle Income Countries

BACKGROUND

In low- and middle-income countries (LMICs), household air pollution as a result of using solid biomass for cooking, lighting and heating (HAP) is associated with respiratory infections, accounting for approximately 4 million early deaths each year worldwide. The majority of deaths are among children under five years. This population-based cross-sectional study investigates the association between solid biomass usage and risk of acute respiratory infections (ARI) and acute lower respiratory infections (ALRI) in 37 LMICs within Africa, Americas, Southeast Asia, European, Eastern Mediterranean and Western Pacific regions.

MATERIALS AND METHODS

Using population-based data obtained from Demographic and Health surveys (2010-2018), domestic cooking energy sources were classified solid biomass (wood, charcoal/dung, agricultural crop) and cleaner energy sources (e.g., Liquid Petroleum Gas (LPG), electricity, biogas and natural gas). Composite measures of ARI (shortness of breath, cough) and ALRI (shortness of breath, cough and fever) were composed using maternally reported respiratory symptoms over the two-week period prior to the interview. Multivariable logistic regression was used to identify the association between biomass fuel usage with ARI and ALRI, accounting for relevant individual, household and situational confounders, including stratification by context (urban/rural).

RESULTS

After adjustment, in the pooled analysis, children residing in solid biomass cooking households had an observed increased adjusted odds ratio of ARI (AOR: 1.17; 95% CI: 1.09-1.25) and ALRI (AOR: 1.16; 95% CI 1.07-1.25) compared to cleaner energy sources. In stratified analyses, a comparable association was observed in urban areas (ARI: 1.16 [1.06-1.28]; ALRI: 1.14 [1.02-1.27]), but only significant for ARI among those living in rural areas (ARI: 1.14 [1.03-1.26]).

CONCLUSION

Switching domestic cooking energy sources from solid biomass to cleaner alternatives would achieve a respiratory health benefit in children under five years worldwide. High quality mixed-methods research is required to improve acceptability and sustained uptake of clean cooking energy source interventions in LMIC settings.

A Systematic Review and Appraisal of Epidemiological Studies on Household Fuel Use and Its Health Effects Using Demographic and Health Surveys

The domestic combustion of polluting fuels is associated with an estimated 3 million premature deaths each year and contributes to climate change. In many low- and middle-income countries (LMICs), valid and representative estimates of people exposed to household air pollution (HAP) are scarce. The Demographic and Health Survey (DHS) is an important and consistent source of data on household fuel use for cooking and has facilitated studies of health effects. However, the body of research based on DHS data has not been systematically identified, nor its strengths and limitations critically assessed as a whole. We aimed to systematically review epidemiological studies using DHS data that considered cooking fuel type as the main exposure, including the assessment of the extent and key drivers of bias. Following PRISMA guidelines, we searched PubMed, Web of Science, Scopus and the DHS publication portal. We assessed the quality and risk of bias (RoB) of studies using a novel tool. Of 2748 records remaining after removing duplicates, 63 were read in full. A total of 45 out of 63 studies were included in our review, spanning 11 different health outcomes and representing 50 unique analyses. In total, 41 of 45 (91%) studies analysed health outcomes in children <5 years of age, including respiratory infections (n = 17), death (all-cause) (n = 14), low birthweight (n = 5), stunting and anaemia (n = 5). Inconsistencies were observed between studies in how cooking fuels were classified into relatively high- and low-polluting. Overall, 36/50 (80%) studies reported statistically significant adverse associations between polluting fuels and health outcomes. In total, 18/50 (36%) of the analyses were scored as having moderate RoB, while 16/50 (32%) analyses were scored as having serious or critical RoB. Although HAP exposure assessment is not the main focus of the DHS, it is the main, often only, source of information in many LMICs. An appreciable proportion of studies using it to analyse the association between cooking fuel use and health have potential for high RoB, mostly related to confounder control, exposure assessment and misclassification, and outcome ascertainment. Based on our findings, we provide some suggestions for ways in which revising the information collected by the DHS could make it even more amenable to studies of household fuel use and health, and reduce the RoB, without being onerous to collect and analyse.

Anthropometric data quality assessment in multisurvey studies of child growth

BACKGROUND

Population-based surveys collect crucial data on anthropometric measures to track trends in stunting [height-for-age z score (HAZ) < -2SD] and wasting [weight-for-height z score (WHZ) < -2SD] prevalence among young children globally. However, the quality of the anthropometric data varies between surveys, which may affect population-based estimates of malnutrition.

OBJECTIVES

We aimed to develop composite indices of anthropometric data quality for use in multisurvey analysis of child health and nutritional status.

METHODS

We used anthropometric data for children 0-59 mo of age from all publicly available Demographic and Health Surveys (DHS) from 2000 onwards. We derived 6 indicators of anthropometric data quality at the survey level, including 1) date of birth completeness, 2) anthropometric measure completeness, 3) digit preference for height and age, 4) difference in mean HAZ by month of birth, 5) proportion of biologically implausible values, and 6) dispersion of HAZ and WHZ distribution. Principal component factor analysis was used to generate a composite index of anthropometric data quality for HAZ and WHZ separately. Surveys were ranked from the highest (best) to the lowest (worst) index values in anthropometric quality across countries and over time.

RESULTS

Of the 145 DHS included, the majority (83 of 145; 57%) were conducted in Sub-Saharan Africa. Surveys were ranked from highest to lowest anthropometric data quality relative to other surveys using the composite index for HAZ. Although slightly higher values in recent DHS suggest potential improvements in anthropometric data quality over time, there continues to be substantial heterogeneity in the quality of anthropometric data across surveys. Results were similar for the WHZ data quality index.

CONCLUSIONS

A composite index of anthropometric data quality using a parsimonious set of individual indicators can effectively discriminate among surveys with excellent and poor data quality. Such indices can be used to account for variations in anthropometric data quality in multisurvey epidemiologic analyses of child health.

What is meant by validity in maternal and newborn health measurement? A conceptual framework for understanding indicator validation

BACKGROUND

Rigorous monitoring supports progress in achieving maternal and newborn mortality and morbidity reductions. Recent work to strengthen measurement for maternal and newborn health highlights the existence of a large number of indicators being used for this purpose. The definitions and data sources used to produce indicator estimates vary and challenges exist with completeness, accuracy, transparency, and timeliness of data. The objective of this study is to create a conceptual overview of how indicator validity is defined and understood by those who develop and use maternal and newborn health indicators.

METHODS

A conceptual framework of validity was developed using mixed methods. We were guided by principles for conceptual frameworks and by a review of the literature and key maternal and newborn health indicator guidance documents. We also conducted qualitative semi-structured interviews with 32 key informants chosen through purposive sampling.

RESULTS

We categorised indicator validity into three main types: criterion, convergent, and construct. Criterion or diagnostic validity, comparing a measure with a gold standard, has predominantly been used to assess indicators of care coverage and content. Studies assessing convergent validity quantify the extent to which two or more indicator measurement approaches, none of which is a gold-standard, relate. Key informants considered construct validity, or the accuracy of the operationalisation of a concept or phenomenon, a critical part of the overall assessment of indicator validity.

CONCLUSION

Given concerns about the large number of maternal and newborn health indicators currently in use, a more consistent understanding of validity can help guide prioritization of key indicators and inform development of new indicators. All three types of validity are relevant for evaluating the performance of maternal and newborn health indicators. We highlight the need to establish a common language and understanding of indicator validity among the various global and local stakeholders working within maternal and newborn health.

"What gets measured better gets done better": The landscape of validation of global maternal and newborn health indicators through key informant interviews

Background

A large number of indicators are currently used to monitor the state of maternal and newborn health, including those capturing dimensions of health system and input, care access and availability, care quality and safety, coverage and outcomes, and impact. Validity of these indicators is a key issue in the process of assessing indicator performance and suitability. This paper aims to understand the meaning of indicator validity in the field of maternal and newborn health, and to identify key recommendations for future research.

Methods

This qualitative study used purposive sampling to identify key informants until thematic saturation was achieved. We interviewed 32 respondents from a variety of backgrounds using semi-structured interviews covering five themes: the meaning of indicator validity, methodological approaches to assessing validity, acceptable levels of indicator validity, gaps in validation research, and recommendations for addressing these gaps. Interview transcripts were analysed data using thematic content approach.

Results

Three conceptually different definitions of indicator validity were described by respondents. They considered indicator validity to encompass meaning and potential to spur action, going beyond diagnostic validity. Indicator validation was seen as an ongoing process of building and synthesising a wide range of evidence rather than a one-size-fits-all cut-off in diagnostic validity tests. Gaps identified included assessing validity of indicators of quality of care and indicators based on facility-level data, as well as expanding studies to a broader range of global settings. The key recommendation was to develop a coordinated approach to summarising and evaluating research on indicator validity, including capacity building in appraising and communicating the available evidence for country-specific needs.

Conclusion

The findings will inform future recommendations around indicator testing and validation.

Exploring the association between sick child healthcare utilisation and health facility quality in Malawi: a cross-sectional study

OBJECTIVE

Increasing the availability of basic healthcare services in low-and middle-income countries is not sufficient to meet the Sustainable Development Goal target for child survival in high-mortality settings, where healthcare utilisation is often inconsistent and quality of care can be poor. We assessed whether poor quality of sick child healthcare in Malawi is associated with low utilisation of sick child healthcare.

DESIGN

We measured two elements of quality of sick child healthcare: facility structural readiness and process of care using data from the 2013 Malawi Service Provision Assessment. Overall quality was defined as the average of these metrics. We extracted demographic data from the 2013-2014 Malawi Multiple Indicator Cluster Survey and linked households to nearby facilities using geocodes. We used logistic regression to examine the association of facility quality with utilisation of formal health services for children under 5 years of age suffering diarrhoea, fever or cough/acute respiratory illness, controlling for demographic and socioeconomic characteristics. We conducted sensitivity analyses (SAs), modifying the travel distance and population-facility matching criteria.

SETTING AND POPULATION

568 facilities were linked with 9701 children with recent illness symptoms in Malawi, of whom 69% had been brought to a health facility.

RESULTS

Overall, facilities showed gaps in structural quality (62% readiness) and major deficiencies in process quality (33%), for an overall quality score of 48%. Better facility quality was associated with higher odds of utilisation of sick child healthcare services (adjusted ORs (AOR): 1.66, 95% CI: 1.04 to 2.63), as was structural quality alone (AOR: 1.33, 95% CI: 0.95 to 1.87). SAs supported the main finding.

CONCLUSION

Although Malawi's health facilities for curative child care are widely available, quality and utilisation of sick child healthcare services are in short supply. Improving facility quality may provide a way to encourage higher utilisation of healthcare, thereby decreasing preventable childhood morbidity and mortality.

'Who assisted with the delivery of (NAME)?' Issues in estimating skilled birth attendant coverage through population-based surveys and implications for improving global tracking

The percentage of live births attended by a skilled birth attendant (SBA) is a key global indicator and proxy for monitoring progress in maternal and newborn health. Yet, the discrepancy between rising SBA coverage and non-commensurate declines in maternal and neonatal mortality in many low-income and middle-income countries has brought increasing attention to the challenge of what the indicator of SBA coverage actually measures, and whether the indicator can be improved. In response to the 2018 revised definition of SBA and the push for improved measurement of progress in maternal and newborn health, this paper examines the evidence on what women can tell us about who assisted them during childbirth and methodological issues in estimating SBA coverage via population-based surveys. We present analyses based on Demographic and Health Surveys and Multiple Indicator Cluster Surveys conducted since 2015 for 23 countries. Our findings show SBA coverage can be reasonably estimated from population-based surveys in settings of high coverage, though women have difficulty reporting specific cadres. We propose improvements in how skilled cadres are classified and documented, how linkages can be made to facility-based data to examine the enabling environment and further ways data can be disaggregated to understand the complexity of delivery care. We also reflect on the limitations of what SBA coverage reveals about the quality and circumstances of childbirth care. While improvements to the indicator are possible, we call for the use of multiple indicators to inform local efforts to improve the health of women and newborns.

Validation studies for population-based intervention coverage indicators: design, analysis, and interpretation

Background

Population-based intervention coverage indicators are widely used to track country and program progress in improving health and to evaluate health programs. Indicator validation studies that compare survey responses to a “gold standard” measure are useful to understand whether the indicator provides accurate information. The Improving Coverage Measurement (ICM) Core Group has developed and implemented a standard approach to validating coverage indicators measured in household surveys, described in this paper.

Methods

The general design of these studies includes measurement of true health status and intervention receipt (gold standard), followed by interviews with the individuals observed, and a comparison of the observations (gold standard) to the responses to survey questions. The gold standard should use a data source external to the respondent to document need for and receipt of an intervention. Most frequently, this is accomplished through direct observation of clinical care, and/or use of a study-trained clinician to obtain a gold standard diagnosis. Follow-up interviews with respondents should employ standard survey questions, where they exist, as well as alternative or additional questions that can be compared against the standard household survey questions.

Results

Indicator validation studies should report on participation at every stage, and provide data on reasons for non-participation. Metrics of individual validity (sensitivity, specificity, area under the receiver operating characteristic curve) and population-level validity (inflation factor) should be reported, as well as the percent of survey responses that are “don’t know” or missing. Associations between interviewer and participant characteristics and measures of validity should be assessed and reported.

Conclusions

These methods allow respondent-reported coverage measures to be validated against more objective measures of need for and receipt of an intervention, and should be considered together with cognitive interviewing, discriminative validity, or reliability testing to inform decisions about which indicators to include in household surveys. Public health researchers should assess the evidence for validity of existing and proposed household survey coverage indicators and consider validation studies to fill evidence gaps.