Yes, no, maybe so: the importance of cognitive interviewing to enhance structured surveys on respectful maternity care in northern India

Translating the consent form is the tip of the iceberg: using cognitive interviews to assess the barriers to informed consent in South African health facilities

The increasing digitisation of personal health data has led to an increase in the demand for onward health data. This study sought to develop local language scripts for use in public sector maternity clinics to capture informed consent for onward health data use. The script considered five possible health data uses: 1. Sending of general health information content via mobile phones; 2. Delivery of personalised health information via mobile phones; 3. Use of women's anonymised health data; 4. Use of child's anonymised health data; and 5. Use of data for recontact. Qualitative interviews (n = 54) were conducted among women attending maternity services in three public health facilities in Gauteng and Western Cape, South Africa. Using cognitive interviewing techniques, interviews sought to:(1) explore understanding of the consent script in five South African languages, (2) assess women's understanding of what they were consenting to, and (3) improve the consent script. Multiple rounds of interviews were conducted, each followed by revisions to the consent script, until saturation was reached, and no additional cognitive failures identified. Cognitive failures were a result of: (1) words and phrases that did not translate easily in some languages, (2) cognitive mismatches that arose as a result of different world views and contexts, (3) linguistic gaps, and (4) asymmetrical power relations that influence how consent is understood and interpreted. Study activities resulted in the development of an informed consent script for onward health data use in five South African languages for use in maternity clinics.

Out-of-Pocket expenditure and patient experience of care under-Indonesia's national health insurance: A cross-sectional facility-based study in six provinces

BACKGROUND

Low- and middle-income countries worldwide are striving to achieve universal health coverage (UHC), frequently through expansion of statutory health insurance schemes. However, oftentimes evidence is lacking on progress towards quality patient-centred care and out-of-pocket expenditure (OOPE), particularly for poor population groups. We contribute patient-centred evidence examining patient experience and OOPE under JKN, the Indonesian social health insurance.

METHODS

Using data from 2526 patient exit interviews conducted among JKN beneficiaries in 2015, we computed a summative patient experience measure from 14 experience items. We used descriptive statistics to assess patient experience and the probability, amount and components of OOPE. We applied a two-part model to examine the relationships between socio-demographics, facility types, and OOPE and an OLS regression on patient experience determinants.

RESULTS

The mean patient experience measure was 11.7 out of 14 maximal points. Differences were observed between single items, with highest ratings on ease of understanding providers' language (97%) and lowest on waiting time (54%). OOPE were reported by 20% of patients with a mean equivalent to US$40, the most prevalent reason being medicines (61% of all OOPE). Considerable OOPE heterogeneity occurred by province and facility type. We found differentials in OOPE by gender (females paying more likely, but less) and subsidised JKN membership (same likelihood as non-subsidised, but paying less).

CONCLUSION

Our findings suggest that during its early implementation, patients under JKN reported mostly positive patient experience yet a fifth incurred OOPE, mostly on medicines. Further patient-centred research is needed to ensure JKN's progress towards UHC.

Revising the Diabetes Distress Scale for Use Among Adults in the Dominican Republic: Findings From Cognitive Interviews

PURPOSE

The purpose of this study is to assess the content validity of the Diabetes Distress Scale (DDS) among adults with type 2 diabetes mellitus (T2DM) living in rural Dominican Republic communities.

METHODS

Researchers conducted cognitive interviews with 20 adults with T2DM to assess how they answered a Spanish version of the 17-item DDS, a commonly used scale to measure diabetes distress. Interviews were done iteratively to allow for revisions and testing of those revisions with the participants. Analysis involved field notes, text summaries, and cognitive coding.

RESULTS

The sample was 55% women, had a mean age of 55 years, and came from 10 rural communities. The cognitive interviews highlighted needed changes across comprehension, judgment (clarity), recall, response process, and logical/structural issue domains. Participants generally understood the DDS; however, 4 items, the introduction, and response options were revised to improve participant response. The items were revised using wording from the participants themselves. By changing certain terms and splitting a couple of items, these items improved comprehension and judgment. The introduction was simplified from 2 paragraphs to 1 to reduce structural issues (ie, scale's features), and the response options were reduced from 6 options to 5 options to improve the response process.

CONCLUSIONS

Based on iterative study findings, the researchers propose expanding the 17-item DDS to 19 items to improve participant response. Revising the DDS to account for cultural and structural changes will improve clinical and public health understanding of the role of diabetes distress on T2DM management among Dominican adults.

“How Are My Age and Cows Related?” Cognitive Interviewing as a Tool to Pretest Survey Questions in Two Limited Resource Settings

Antimicrobial resistance is a complex topic requiring interdisciplinary solutions embedded in One Health thinking. Currently, many surveys are underway in low- and middle-income countries to study how antimicrobial use in the livestock sector is driving resistance. In a survey, the respondents must understand and answer the questions correctly to produce accurate and valuable results. Pretesting survey questions is therefore important but sometimes not performed due to limited time and resources. Cognitive interviewing is a pretesting method to give insights into the respondent's way of interpreting and mentally processing the survey questions to identify problems and finding ways to improve the questions. It has previously been suggested that cognitive interviews may be difficult to use in some cultural settings. This study aimed to use cognitive interviews in a respondent-adjusted way to study how survey questions related to antimicrobial use are understood and answered by 12 small-scale farmers in Kenya and Uganda. The results show that even a small number of interviews and using interviewers with limited knowledge of cognitive interviewing can identify many problems in survey questions and the survey tool. Cognitive interviews may provide a feasible and affordable way of pretesting questionnaires in situations where time and resources are limited, for example, during a disease outbreak.

The impact of a direct to beneficiary mobile communication program on reproductive and child health outcomes: a randomised controlled trial in India

Background

Direct-to-beneficiary communication mobile programmes are among the few examples of digital health programmes to have scaled widely in low-resource settings. Yet, evidence on their impact at scale is limited. This study aims to assess whether exposure to mobile health information calls during pregnancy and postpartum improved infant feeding and family planning practices.

Methods

We conducted an individually randomised controlled trial in four districts of Madhya Pradesh, India. Study participants included Hindi speaking women 4–7 months pregnant (n=5095) with access to a mobile phone and their husbands (n=3842). Women were randomised to either an intervention group where they received up to 72 Kilkari messages or a control group where they received none. Intention-to-treat (ITT) and instrumental variable (IV) analyses are presented.

Results

An average of 65% of the 2695 women randomised to receive Kilkari listened to ≥50% of the cumulative content of calls answered. Kilkari was not observed to have a significant impact on the primary outcome of exclusive breast feeding (ITT, relative risk (RR): 1.04, 95% CI 0.88 to 1.23, p=0.64; IV, RR: 1.10, 95% CI 0.67 to 1.81, p=0.71). Across study arms, Kilkari was associated with a 3.7% higher use of modern reversible contraceptives (RR: 1.12, 95% CI 1.03 to 1.21, p=0.007), and a 2.0% lower proportion of men or women sterilised since the birth of the child (RR: 0.85, 95% CI 0.74 to 0.97, p=0.016). Higher reversible method use was driven by increases in condom use and greatest among those women exposed to Kilkari with any male child (9.9% increase), in the poorest socioeconomic strata (15.8% increase), and in disadvantaged castes (12.0% increase). Immunisation at 10 weeks was higher among the children of Kilkari listeners (2.8% higher; RR: 1.03, 95% CI 1.00 to 1.06, p=0.048). Significant differences were not observed for other maternal, newborn and child health outcomes assessed.

Conclusion

Study findings provide evidence to date on the effectiveness of the largest mobile health messaging programme in the world.

Trial registration number

Trial registration clinicaltrials.gov; ID 90075552, NCT03576157.

Integrated delivery of family planning and childhood immunisation services: a mixed methods assessment of service responsiveness

Background

Postpartum women represent a considerable share of the global unmet need for modern contraceptives. Evidence suggests that the integration of family planning (FP) with childhood immunisation services could help reduce this unmet need by providing repeat opportunities for timely contact with FP services. However, little is known about the clients’ experiences of FP services that are integrated with childhood immunisations, despite being crucial to contraceptive uptake and repeat service utilisation.

Methods

The responsiveness of FP services that were integrated with childhood immunisations in Malawi was assessed using cross-sectional convergent mixed methods. Exit interviews with clients (n=146) and audits (n=15) were conducted in routine outreach clinics. Responsiveness scores across eight domains were determined according to the proportion of clients who rated each domain positively. Text summary analyses of qualitative data from cognitive interviewing probes were also conducted to explain responsiveness scores. Additionally, Spearman rank correlation and Pearson’s chi-squared test were used to identify correlations between domain ratings and to examine associations between domain ratings and client, service and clinic characteristics.

Results

Responsiveness scores varied across domains: dignity (97.9%); service continuity (90.9%); communication (88.7%); ease of access (77.2%); counselling (66.4%); confidentiality (62.0%); environment (53.9%) and choice of provider (28.4%). Despite some low performing domains, 98.6% of clients said they would recommend the clinic to a friend or family member interested in FP. The choice of provider, communication, confidentiality and counselling ratings were positively associated with clients’ exclusive use of one clinic for FP services. Also, the organisation of services in the clinics and the providers’ individual behaviours were found to be critical to service responsiveness.

Conclusions

This study establishes that in routine outreach clinics, FP services can be responsive when integrated with childhood immunisations, particularly in terms of the dignity and service continuity afforded to clients, though less so in terms of the choice of provider, environment, and confidentiality experienced. Additionally, it demonstrates the value of combining cognitive interviewing techniques with Likert questions to assess service responsiveness.

Assessing the reliability of phone surveys to measure reproductive, maternal and child health knowledge among pregnant women in rural India: a feasibility study

OBJECTIVES

Efforts to understand the factors influencing the uptake of reproductive, maternal, newborn, child health and nutrition (RMNCH&N) services in high disease burden low-resource settings have often focused on face-to-face surveys or direct observations of service delivery. Increasing access to mobile phones has led to growing interest in phone surveys as a rapid, low-cost alternatives to face-to-face surveys. We assess determinants of RMNCH&N knowledge among pregnant women with access to phones and examine the reliability of alternative modalities of survey delivery.

PARTICIPANTS

Women 5-7 months pregnant with access to a phone.

SETTING

Four districts of Madhya Pradesh, India.

DESIGN

Cross-sectional surveys administered face-to-face and within 2 weeks, the same surveys were repeated among two random subsamples of the original sample: face-to-face (n=205) and caller-attended telephone interviews (n=375). Bivariate analyses, multivariable linear regression, and prevalence and bias-adjusted kappa scores are presented.

RESULTS

Knowledge scores were low across domains: 52% for maternal nutrition and pregnancy danger signs, 58% for family planning, 47% for essential newborn care, 56% infant and young child feeding, and 58% for infant and young child care. Higher knowledge (≥1 composite score) was associated with older age; higher levels of education and literacy; living in a nuclear family; primary health decision-making; greater attendance in antenatal care and satisfaction with accredited social health activist services. Survey questions had low inter-rater and intermodal reliability (kappa<0.70) with a few exceptions. Questions with the lowest reliability included true/false questions and those with unprompted, multiple response options. Reliability may have been hampered by the sensitivity of the content, lack of privacy, enumerators' and respondents' profile differences, rapport, social desirability bias, and/or enumerator's ability to adequately convey concepts or probe.

CONCLUSIONS

Phone surveys are a reliable modality for generating population-level estimates data about pregnant women's knowledge, however, should not be used for individual-level tracking.

TRIAL REGISTRATION NUMBER

NCT03576157.

Cognitive testing of a survey instrument to assess sexual practices, behaviours, and health outcomes: a multi-country study protocol

BACKGROUND

Population level data on sexual practices, behaviours and health-related outcomes can ensure that responsive, relevant health services are available for all people of all ages. However, while billions of dollars have been invested in attempting to improve sexual and reproductive health (including HIV) outcomes, far less is understood about associated sexual practices and behaviours. Therefore, the World Health Organization embarked on a global consultative process to develop a short survey instrument to assess sexual health practices, behaviours and health outcomes. In order for the resulting draft survey instrument to be published as a 'global' standard instrument, it is important to first determine that the proposed measures are globally comprehensible and applicable. This paper describes a multi-country study protocol to assess the interpretability and comparability of the survey instrument in a number of diverse countries.

METHODS

This study will use cognitive interviewing, a qualitative data collection method that uses semi-structured interviews to explore how participants process and respond to survey instruments. We aim to include study sites in up to 20 countries. The study procedures consist of: (1) localizing the instrument using forward and back-translation; (2) using a series of cognitive interviews to understand how participants engage with each survey question; (3) revising the core instrument based on interview findings; and (4) conducting an optional second round of cognitive interviews. Data generated from interviews will be summarised into a predeveloped analysis matrix. The entire process (a 'wave' of data collection) will be completed simultaneously by 5+ countries, with a total of three waves. This stepwise approach facilitates iterative improvements and sharing across countries.

DISCUSSION

An important output from this research will be a revised survey instrument, which when subsequently published, can contribute to improving the comparability across contexts of measures of sexual practices, behaviours and health-related outcomes. Site-specific results of the feasibility of conducting this research may help shift perceptions of who and what can be included in sexual health-related research.

Health Related Quality of Life in Interstitial Lung Disease: Can We Use the Same Concepts Around the World?

Health-Related Quality of Life (HRQOL) is increasingly viewed as an important patient-centered outcome by leading health organizations, clinicians, and patients alike. This is especially true in the interstitial lung disease community where patients often struggle with progressive and debilitating disease with few therapeutic options. In order to test the effectiveness of new pharmacologic therapies and non-pharmacologic interventions globally in ILD, this will require expansion of clinical research studies to a multinational level and HRQOL will be an important endpoint to many. In order to successfully expand trials across multiple nations and compare the results of studies between different communities we must recognize that there are differences in the concepts of HRQOL across the world and have strategies to address these differences. In this review, we will describe the different global influences on HRQOL both generally and in the context of ILD, discuss the processes of linguistic translation and cross-cultural adaptation of HRQOL Patient Reported Outcome Measures (PROMs), and highlight the gaps and opportunities for improving HRQOL measurement in ILD across the world.

The devil is in the detail: reflections on the value and application of cognitive interviewing to strengthen quantitative surveys in global health

Cognitive interviewing is a qualitative research method for improving the validity of quantitative surveys, which has been underused by academic researchers and monitoring and evaluation teams in global health. Draft survey questions are administered to participants drawn from the same population as the respondent group for the survey itself. The interviewer facilitates a detailed discussion with the participant to assess how the participant interpreted each question and how they formulated their response. Draft survey questions are revised and undergo additional rounds of cognitive interviewing until they achieve high comprehension and cognitive match between the research team's intent and the target population's interpretation. This methodology is particularly important in global health when surveys involve translation or are developed by researchers who differ from the population being surveyed in terms of socio-demographic characteristics, worldview, or other aspects of identity. Without cognitive interviewing, surveys risk measurement error by including questions that respondents find incomprehensible, that respondents are unable to accurately answer, or that respondents interpret in unintended ways. This methodological musing seeks to encourage a wider uptake of cognitive interviewing in global public health research, provide practical guidance on its application, and prompt discussion on its value and practice. To this end, we define cognitive interviewing, discuss how cognitive interviewing compares to other forms of survey tool development and validation, and present practical steps for its application. These steps cover defining the scope of cognitive interviews, selecting and training researchers to conduct cognitive interviews, sampling participants, collecting data, debriefing, analysing the emerging findings, and ultimately generating revised, validated survey questions. We close by presenting recommendations to ensure quality in cognitive interviewing.

Measuring experiences of facility-based care for pregnant women and newborns: a scoping review

BACKGROUND

Access to high-quality, person-centred care during pregnancy and childbirth is a global priority. Positive experience of care is key in particular, because it is both a fundamental right and can influence health outcomes and future healthcare utilisation. Despite its importance for accountability and action, systematic guidance on measuring experience of care is limited.

METHODS

We conducted a scoping review of published literature to identify measures/instruments for experience of facility-based pregnancy and childbirth (abortion, antenatal, intrapartum, postnatal and newborn) care. We systematically searched five bibliographic databases from 1 January 2007 through 1 February 2019. Using a predefined evidence template, we extracted data on study design, data collection method, study population and care type as reported in primary quantitative articles. We report results narratively.

RESULTS

We retrieved 16 528 unique citations, including 171 eligible articles representing, 157 unique instruments and 144 unique parent instruments across 56 countries. Half of the articles (90/171) did not use a validated instrument. While 82% (n=141) of articles reported on labour and childbirth care, only one reported on early pregnancy/abortion care. The most commonly reported sub-domains of user experience were communication (84%, 132/157) and respect and dignity (71%, 111/157). The primary purpose of most papers was measurement (70%, 119/171), largely through cross-sectional surveys.

CONCLUSION

There are alarming gaps in measurement of user experience for abortion, antenatal, postnatal and newborn care, including lack of validated instruments to measure the effects of interventions and policies on user experience.

PROTOCOL REGISTRATION DETAILS

This review was registered and published on PROSPERO (CRD42017070867). PROSPERO is an international database of prospectively registered systematic reviews in health and social care.

Using Machine Learning to Optimize the Quality of Survey Data: Protocol for a Use Case in India

BACKGROUND

Data quality is vital for ensuring the accuracy, reliability, and validity of survey findings. Strategies for ensuring survey data quality have traditionally used quality assurance procedures. Data analytics is an increasingly vital part of survey quality assurance, particularly in light of the increasing use of tablets and other electronic tools, which enable rapid, if not real-time, data access. Routine data analytics are most often concerned with outlier analyses that monitor a series of data quality indicators, including response rates, missing data, and reliability of coefficients for test-retest interviews. Machine learning is emerging as a possible tool for enhancing real-time data monitoring by identifying trends in the data collection, which could compromise quality.

OBJECTIVE

This study aimed to describe methods for the quality assessment of a household survey using both traditional methods as well as machine learning analytics.

METHODS

In the Kilkari impact evaluation's end-line survey amongst postpartum women (n=5095) in Madhya Pradesh, India, we plan to use both traditional and machine learning-based quality assurance procedures to improve the quality of survey data captured on maternal and child health knowledge, care-seeking, and practices. The quality assurance strategy aims to identify biases and other impediments to data quality and includes seven main components: (1) tool development, (2) enumerator recruitment and training, (3) field coordination, (4) field monitoring, (5) data analytics, (6) feedback loops for decision making, and (7) outcomes assessment. Analyses will include basic descriptive and outlier analyses using machine learning algorithms, which will involve creating features from time-stamps, "don't know" rates, and skip rates. We will also obtain labeled data from self-filled surveys, and build models using k-folds cross-validation on a training data set using both supervised and unsupervised learning algorithms. Based on these models, results will be fed back to the field through various feedback loops.

RESULTS

Data collection began in late October 2019 and will span through March 2020. We expect to submit quality assurance results by August 2020.

CONCLUSIONS

Machine learning is underutilized as a tool to improve survey data quality in low resource settings. Study findings are anticipated to improve the overall quality of Kilkari survey data and, in turn, enhance the robustness of the impact evaluation. More broadly, the proposed quality assurance approach has implications for data capture applications used for special surveys as well as in the routine collection of health information by health workers.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/17619.

Novel approaches to measuring knowledge among frontline health workers in India: Are phone surveys a reliable option?

Background

In 2017, India was home to nearly 20% of maternal and child deaths occurring globally. Accredited social health activists (ASHAs) act as the frontline for health services delivery in India, providing a range of reproductive, maternal, newborn, child health, and nutrition (RMNCH&N) services. Empirical evidence on ASHAs’ knowledge is limited, yet is a critical determinant of the quality of health services provided. We assessed the determinants of RMNCH&N knowledge among ASHAs and examined the reliability of alternative modalities of survey delivery, including face-to-face and caller attended telephone interviews (phone surveys) in 4 districts of Madhya Pradesh, India.

Methods

We carried out face-to-face surveys among a random cross-sectional sample of ASHAs (n = 1,552), and administered a follow-up test-retest survey within 2 weeks of the initial survey to a subsample of ASHAs (n = 173). We interviewed a separate sub-sample of ASHAs 2 weeks of the face-to-face interview over the phone (n = 155). Analyses included bivariate analyses, multivariable linear regression, and prevalence and bias adjusted kappa analyses.

Findings

The average ASHA knowledge score was 64% and ranged across sub-domains from 71% for essential newborn care, 71% for WASH/ diarrhea, 64% for infant feeding, 61% for family planning, and 60% for maternal health. Leading determinants of knowledge included geographic location, age <30 years of age, education, experience as an ASHA, completion of seven or more client visits weekly, phone ownership and use as a communication tool for work, as well as the ability to navigate interactive voice response prompts (a measure of digital literacy). Efforts to develop a phone survey tool for measuring knowledge suggest that findings on inter-rater and inter-modal reliability were similar. Reliability was higher for shorter, widely known questions, including those about timing of exclusive breastfeeding or number of tetanus shots during pregnancy. Questions with lower reliability included those on sensitive topics such as family planning; questions with multiple response options; or which were difficult for the enumerator to convey.

Conclusions

Overall results highlight important gaps in the knowledge of ASHAs. Findings on the reliability of phone surveys led to the development of a tool, which can be widely used for the routine, low cost measurement of ASHA RMNCH&N knowledge in India.