The devil is in the detail: reflections on the value and application of cognitive interviewing to strengthen quantitative surveys in global health

Public service motivation, public sector preference and employment of Kenyan medical doctor interns: a cross-sectional and prospective study

BACKGROUND

Kenya grapples with a paradox; severe public sector workforce shortages co-exist with rising unemployment among healthcare professionals. Medical schools have increased trainee outputs, but only 45% of newly qualified/registered doctors were absorbed by the public sector during 2015-2018. In such a context, we explore what influences doctors' career choices at labour market entry, specifically understanding the role of public service motivation (PSM).

METHODS

We conducted a cross-sectional and prospective study of interns and recently graduated doctors to examine PSM, their intention to work in the public sector and their final employment sector and status. We surveyed them on their PSM and job intentions and conducted a prospective follow-up survey of the interns, around one year later, to understand their employment status.

FINDINGS

We recruited 356 baseline participants and followed up 76 out of 129 eligible interns. The overall PSM score was high among all participants (rated 4.50/5.00) irrespective of sector preferences. 48% (171/356) of the participants preferred to work in the public sector immediately after internship, alongside 16% (57/356) preferring direct entry into specialist training-commonly in the public sector. Only 13% (46/356) and 7% (25/365) preferred to work in the private or faith-based sector. Despite the high proportion of interns preferring public sector jobs, only 17% (13/76) were employed in the public sector at follow-up and 13% (10/76) were unemployed, due to lack of job availability.

CONCLUSION

High PSM scores irrespective of sector preferences suggest that doctors are generally committed to serving the 'public good'. Many intended to work in the public sector but were unable to due to lack of job opportunities. Policymakers have an opportunity to tackle workforce gaps in the public sector as young doctors continue to express a preference for such work. To do this they should prioritise creating adequate and sustainable job opportunities.

Conducting cross-cultural, multi-lingual or multi-country scale development and validation in health care research: A 10-step framework based on a scoping review

Background

Valid, reliable and cross-cultural equivalent scales and measurement instruments that enable comparisons across diverse populations in different countries are important for global health research and practice. We developed a 10-step framework through a scoping review of the common strategies and techniques used for scale development and validation in a cross-cultural, multi-lingual, or multi-country setting, especially in health care research.

Methods

We searched MEDLINE, Embase, and PsycINFO for peer-reviewed studies that collected data from two or more countries or in two or more languages at any stages of scale development or validation and published between 2010-22. We categorised the techniques into three commonly used scale development and validation stages (item generation, scale development, and scale evaluation) as well as during the translation stage. We described the most commonly used techniques at each stage.

Results

We identified 141 studies that were included in the analysis. We summarised 14 common techniques and strategies, including focus groups or interviews with diverse target populations, and involvement of measurement experts and linguists for item content validity expert panel at the item generation stage; back-and-forth translation, collaborative team approach for the translation stage; cognitive interviews and different recruitment strategies and incentives in different settings for scale development stage; and three approaches for measurement invariance (multigroup confirmatory factor analysis, differential item functioning and multiple indicator multiple causes) for scale evaluation stage.

Conclusions

We provided a 10-step framework for cross-cultural, multi-lingual or multi-country scale development and validation based on these techniques and strategies. More research and synthesis are needed to make scale development more culturally competent and enable scale application to better meet local health and development needs.

Pilot Testing Transcreated Spanish-Language Study Materials for Symptom Research With Infants and Parents in the Neonatal Intensive Care Unit

BACKGROUND

Rising admission rates of Hispanic/Latinx families to the neonatal intensive care unit (NICU) have increased the number of non-English-speaking individuals who may wish to participate in research studies. However, a lack of appropriately translated research study materials may limit the opportunity for these families to be involved in research that could impact the care that infants and families receive in the NICU.

PURPOSE

The primary purpose was to pilot test study materials that were transcreated from English to Spanish with the assistance of a bilingual community advisory board with Spanish-speaking parents of NICU infants.

METHODS

A total of 19 Spanish-speaking parents (15 mothers and 4 fathers) who were representative of the population of interest completed paper-and-pencil surveys, along with a cognitive interview. Preliminary data related to decision-making and goals of care, infant symptoms, and their experiences in the NICU were also collected.

RESULTS

The internal reliability of the transcreated study instruments ranged from good to excellent (α= 0.82-0.99). Participants reported that study materials were not offensive and did not make them feel uncomfortable; however, they found some words/phrases to be confusing. Parents had the opportunity to provide suggested wording changes.

IMPLICATIONS FOR PRACTICE AND RESEARCH

Language barriers and a lack of cultural responsiveness can affect the care that infants and their families receive. More accurate and culturally appropriate transcreation of study materials can remove barriers to research participation and facilitate better communication with non-English-speaking families, which may lead to the development of better-informed evidence-based interventions and clinical practices in the NICU.

"It's a 50/50 Thing You Know": Exploring the Multileveled Intersections of Gender and Power Within the Relationships of Young South African Men and Women

Efforts to advance sexual health globally require greater understandings of youth intimate relationship dynamics. Among 38 South African youth (21 women/17 men aged 21-30 from Durban and Soweto) we conducted qualitative cognitive interviews to explore how gender and power intersect to shape intimate relationship dynamics (October 2019-March 2021). Participants discussed perceptions and relevance of each of 13 items comprising the Sexual Relationship Power (SRP) scale, a widely used measure of gender equity, and the influence of SRP on youth sexual health. Data analysis was guided by constructivist grounded theory. The findings were organized using the socio-ecological model, revealing how gender and power intersected at multiple levels to influence youth intimate relationships. Key influencing factors included individual-level gender attitudes, male partner expectations, and women's resistance to dominance; intimate relationship-level power dynamics, consent, and intimacy; family-level household configurations and parental monitoring of daughters; and societal-level traditional gender norms. At all levels, women discussed resisting power inequities through communication and rejection of inequitable relationships. While men also displayed resistance to inequitable power structures, most upheld traditional gender norms through institutional affiliation (e.g. church) and deep-rooted socialized beliefs and attitudes. Efforts to improve youth sexual health require multileveled approaches that address inequitable power dynamics.

Translating the consent form is the tip of the iceberg: using cognitive interviews to assess the barriers to informed consent in South African health facilities

The increasing digitisation of personal health data has led to an increase in the demand for onward health data. This study sought to develop local language scripts for use in public sector maternity clinics to capture informed consent for onward health data use. The script considered five possible health data uses: 1. Sending of general health information content via mobile phones; 2. Delivery of personalised health information via mobile phones; 3. Use of women's anonymised health data; 4. Use of child's anonymised health data; and 5. Use of data for recontact. Qualitative interviews (n = 54) were conducted among women attending maternity services in three public health facilities in Gauteng and Western Cape, South Africa. Using cognitive interviewing techniques, interviews sought to:(1) explore understanding of the consent script in five South African languages, (2) assess women's understanding of what they were consenting to, and (3) improve the consent script. Multiple rounds of interviews were conducted, each followed by revisions to the consent script, until saturation was reached, and no additional cognitive failures identified. Cognitive failures were a result of: (1) words and phrases that did not translate easily in some languages, (2) cognitive mismatches that arose as a result of different world views and contexts, (3) linguistic gaps, and (4) asymmetrical power relations that influence how consent is understood and interpreted. Study activities resulted in the development of an informed consent script for onward health data use in five South African languages for use in maternity clinics.

Development and validation of a new measurement instrument to assess internship experience of medical doctors in low-income and middle-income countries

Routine surveys are used to understand the training quality and experiences of junior doctors but there are lack of tools designed to evaluate the training experiences of interns in low-income and middle-income countries (LMICs) where working conditions and resource constraints are challenging. We describe our process developing and validating a 'medical internship experience scale' to address this gap, work involving nine LMICs that varied in geographical locations, income-level and internship training models. We used a scoping review of existing tools, content validity discussions with target populations and an expert panel, back-and-forth translations into four language versions and cognitive interviews to develop and test the tool. Using data collected from 1646 interns and junior medical doctors, we assessed factor structure and assessed its reliability and validity. Fifty items about experiences of medical internship were retained from an initial pool of 102 items. These 50 items represent 6 major factors (constructs): (1) clinical learning and supervision, (2) patient safety, (3) job satisfaction, (4) stress and burnout, (5) mental well-being, and (6) fairness and discrimination. We reflect on the process of multicountry scale development and highlight some considerations for others who may use our scale, using preliminary analyses of the 1646 responses to illustrate that the tool may produce useful data to identify priorities for action. We suggest this tool could enable LMICs to assess key metrics regarding intern straining and initial work experiences and possibly allow comparison across countries and over time, to inform better internship planning and management.

'We were treated like we are nobody': a mixed-methods study of medical doctors' internship experiences in Kenya and Uganda

OBJECTIVE

Medical interns are an important workforce providing first-line healthcare services in hospitals. The internship year is important for doctors as they transition from theoretical learning with minimal hands-on work under supervision to clinical practice roles with considerable responsibility. However, this transition is considered stressful and commonly leads to burn-out due to challenging working conditions and an ongoing need for learning and assessment, which is worse in countries with resource constraints. In this study, we provide an overview of medical doctors' internship experiences in Kenya and Uganda.

METHODS

Using a convergent mixed-methods approach, we collected data from a survey of 854 medical interns and junior doctors and semistructured interviews with 54 junior doctors and 14 consultants. Data collection and analysis were guided by major themes identified from a previous global scoping review (well-being, educational environment and working environment and condition), using descriptive analysis and thematic analysis respectively for quantitative and qualitative data.

FINDINGS

Most medical interns are satisfied with their job but many reported suffering from stress, depression and burn-out, and working unreasonable hours due to staff shortages. They are also being affected by the challenging working environment characterised by a lack of adequate resources and a poor safety climate. Although the survey data suggested that most interns were satisfied with the supervision received, interviews revealed nuances where many interns faced challenging scenarios, for example, poor supervision, insufficient support due to consultants not being available or being 'treated like we are nobody'.

CONCLUSION

We highlight challenges experienced by Kenyan and Ugandan medical interns spanning from burn-out, stress, challenging working environment, inadequate support and poor quality of supervision. We recommend that regulators, educators and hospital administrators should improve the resource availability and capacity of internship hospitals, prioritise individual doctors' well-being and provide standardised supervision, support systems and conducive learning environments.

Development and validation of the activities and participation children and adolescents -neck (APCAN) measure

BACKGROUND

Neck pain is the fourth leading cause of years lost to disability in children warranting a comprehensive assessment of neck pain and its impact on activities and participation. Hence, the purpose of this study was to develop a new measure (i.e., Activities and Participation Children and Adolescents -neck [APCAN]) specific to evaluating activity limitation and participation restrictions in children and adolescents and to establish its content validity.

METHODS

Development and content validation of the APCAN was completed in four steps: (1) item development, (2) item evaluation by content experts, (3) content validity calculation, and (4) cognitive testing via interviews to ensure readability and comprehension of the items on the APCAN.

RESULTS

An initial pool of 52 items was created that was revised to 20 items after modified Delphi process and cognitive interviews. Each item was rated on a 0-10 numeric rating scale (0 = not difficult at all, 10 = extremely difficult) with higher scores indicating higher perceived disability secondary to neck pain. All 20 items retained the content validity ratio critical value and the overall content validity index was 0.88 indicating excellent content validity.

CONCLUSION

The APCAN provides an easy to use, comprehensive assessment of functional limitations associated with neck pain in children.

Spare the rod, spoil the child: measurement and learning from an intervention to shift corporal punishment attitudes and behaviors in Grenada, West Indies

Childrearing practices in the Caribbean and other postcolonial states have long been associated with corporal punishment and are influenced by expectations of children for respectfulness and obedience. Evidence across settings shows that physical punishment of young children is both ineffective and detrimental. Saving Brains Grenada (SBG) implemented a pilot study of an intervention based on the Conscious Discipline curriculum that aimed to build adult caregivers' skills around non-violent child discipline. We hypothesized that attitudes towards corporal punishment would shift to be negative as adults learned more positive discipline methods, and that child neurodevelopment would correspondingly improve. This report reviews the impact of monitoring and evaluation on the design and implementation of the intervention. Study 1 presents findings from the pilot study. Despite positive gains in neurodevelopmental outcomes among children in the intervention compared to controls, attitudes towards corporal punishment and reported use of it did not change. Additionally, several internal conflicts in the measures used to assess corporal punishment behaviors and attitudes were identified. Study 2 is a response to learning from Study 1 and highlights the importance for monitoring and evaluation to be data-informed, adaptive, and culturally appropriate. In Study 2, the SBG research team conducted cognitive interviews and group discussions with stakeholders to assess the content and comprehensibility of the Attitudes Towards Corporal Punishment Scale (ACP). This yielded insights into the measurement of attitudes towards corporal punishment and related parenting behavior, and prompted several revisions to the ACP. To accurately evaluate the intervention's theory of change and its goal to reduce violence against children, reliable and appropriate measures of attitudes towards corporal punishment and punishment behaviors are needed. Together, these two studies emphasize the value of continuous monitoring, evaluation, and learning in the implementation, adaptation, evaluation, and scaling of SBG and similar early childhood development interventions.

Development and validation of the Self-Efficacy in Addressing Menstrual Needs Scale (SAMNS-26) in Bangladeshi schools: A measure of girls' menstrual care confidence

OBJECTIVE

Qualitative studies have described girls' varying levels of confidence in managing their menstruation, with greater confidence hypothesized to positively impact health, education, and social participation outcomes. Yet, measurement of this and other psychosocial components of adolescent girls' menstrual experiences has been weak in global health research, in part due to a dearth of appropriate psychometric tools. We describe the development and validation of the Self-Efficacy in Addressing Menstrual Needs Scale (SAMNS-26).

METHODS

We conducted nine focus group discussions with girls in schools in rural and urban Bangladesh to identify tasks involved in menstrual self-care. This informed our creation of an initial pool of 50 items, which were reviewed by menstrual health experts and refined through 21 cognitive interviews with schoolgirls. Using a self-administered survey, we administered 34 refined items plus additional validation measures to a random sample of 381 post-menarcheal girls (ages 9-17) and retested a subsample of 42 girls two weeks later. We examined the measure's dimensionality using exploratory factor analysis and assessed internal consistency, temporal stability, and construct validity.

RESULTS

Exploratory factor analysis suggested a 26-item scale comprising three correlated sub-scales: the 17-item Menstrual Hygiene Preparation and Maintenance (α = 0.86), the 5-item Menstrual Pain Management (α = 0.87), and the 4-item Executing Stigmatized Tasks (α = 0.77). Sub-scales exhibited good temporal stability. SAMNS-26 scores correlated negatively with measures of anxiety, and girls who preferred to stay at home during their periods had lower SAMNS-26 scores than those who did not.

CONCLUSION

The SAMNS-26 provides a reliable measure of a schoolgirl's confidence in her capabilities to address her menstrual needs. There is initial evidence to support the measure's construct validity in the Bangladesh context as indicated by its relationships with other factors in its theorized nomological network. The tool enables incorporation of self-efficacy into multivariate models for exploring the relationships among antecedents to menstrual experiences and hypothesized impacts on health, wellbeing, and education attainment. Further testing of the tool is recommended to strengthen evidence of its validity in additional contexts.

“How Are My Age and Cows Related?” Cognitive Interviewing as a Tool to Pretest Survey Questions in Two Limited Resource Settings

Antimicrobial resistance is a complex topic requiring interdisciplinary solutions embedded in One Health thinking. Currently, many surveys are underway in low- and middle-income countries to study how antimicrobial use in the livestock sector is driving resistance. In a survey, the respondents must understand and answer the questions correctly to produce accurate and valuable results. Pretesting survey questions is therefore important but sometimes not performed due to limited time and resources. Cognitive interviewing is a pretesting method to give insights into the respondent's way of interpreting and mentally processing the survey questions to identify problems and finding ways to improve the questions. It has previously been suggested that cognitive interviews may be difficult to use in some cultural settings. This study aimed to use cognitive interviews in a respondent-adjusted way to study how survey questions related to antimicrobial use are understood and answered by 12 small-scale farmers in Kenya and Uganda. The results show that even a small number of interviews and using interviewers with limited knowledge of cognitive interviewing can identify many problems in survey questions and the survey tool. Cognitive interviews may provide a feasible and affordable way of pretesting questionnaires in situations where time and resources are limited, for example, during a disease outbreak.

The impact of a direct to beneficiary mobile communication program on reproductive and child health outcomes: a randomised controlled trial in India

BACKGROUND

Direct-to-beneficiary communication mobile programmes are among the few examples of digital health programmes to have scaled widely in low-resource settings. Yet, evidence on their impact at scale is limited. This study aims to assess whether exposure to mobile health information calls during pregnancy and postpartum improved infant feeding and family planning practices.

METHODS

We conducted an individually randomised controlled trial in four districts of Madhya Pradesh, India. Study participants included Hindi speaking women 4-7 months pregnant (n=5095) with access to a mobile phone and their husbands (n=3842). Women were randomised to either an intervention group where they received up to 72 Kilkari messages or a control group where they received none. Intention-to-treat (ITT) and instrumental variable (IV) analyses are presented.

RESULTS

An average of 65% of the 2695 women randomised to receive Kilkari listened to ≥50% of the cumulative content of calls answered. Kilkari was not observed to have a significant impact on the primary outcome of exclusive breast feeding (ITT, relative risk (RR): 1.04, 95% CI 0.88 to 1.23, p=0.64; IV, RR: 1.10, 95% CI 0.67 to 1.81, p=0.71). Across study arms, Kilkari was associated with a 3.7% higher use of modern reversible contraceptives (RR: 1.12, 95% CI 1.03 to 1.21, p=0.007), and a 2.0% lower proportion of men or women sterilised since the birth of the child (RR: 0.85, 95% CI 0.74 to 0.97, p=0.016). Higher reversible method use was driven by increases in condom use and greatest among those women exposed to Kilkari with any male child (9.9% increase), in the poorest socioeconomic strata (15.8% increase), and in disadvantaged castes (12.0% increase). Immunisation at 10 weeks was higher among the children of Kilkari listeners (2.8% higher; RR: 1.03, 95% CI 1.00 to 1.06, p=0.048). Significant differences were not observed for other maternal, newborn and child health outcomes assessed.

CONCLUSION

Study findings provide evidence to date on the effectiveness of the largest mobile health messaging programme in the world.

TRIAL REGISTRATION NUMBER

Trial registration clinicaltrials.gov; ID 90075552, NCT03576157.

Integrated delivery of family planning and childhood immunisation services: a mixed methods assessment of service responsiveness

Background

Postpartum women represent a considerable share of the global unmet need for modern contraceptives. Evidence suggests that the integration of family planning (FP) with childhood immunisation services could help reduce this unmet need by providing repeat opportunities for timely contact with FP services. However, little is known about the clients’ experiences of FP services that are integrated with childhood immunisations, despite being crucial to contraceptive uptake and repeat service utilisation.

Methods

The responsiveness of FP services that were integrated with childhood immunisations in Malawi was assessed using cross-sectional convergent mixed methods. Exit interviews with clients (n=146) and audits (n=15) were conducted in routine outreach clinics. Responsiveness scores across eight domains were determined according to the proportion of clients who rated each domain positively. Text summary analyses of qualitative data from cognitive interviewing probes were also conducted to explain responsiveness scores. Additionally, Spearman rank correlation and Pearson’s chi-squared test were used to identify correlations between domain ratings and to examine associations between domain ratings and client, service and clinic characteristics.

Results

Responsiveness scores varied across domains: dignity (97.9%); service continuity (90.9%); communication (88.7%); ease of access (77.2%); counselling (66.4%); confidentiality (62.0%); environment (53.9%) and choice of provider (28.4%). Despite some low performing domains, 98.6% of clients said they would recommend the clinic to a friend or family member interested in FP. The choice of provider, communication, confidentiality and counselling ratings were positively associated with clients’ exclusive use of one clinic for FP services. Also, the organisation of services in the clinics and the providers’ individual behaviours were found to be critical to service responsiveness.

Conclusions

This study establishes that in routine outreach clinics, FP services can be responsive when integrated with childhood immunisations, particularly in terms of the dignity and service continuity afforded to clients, though less so in terms of the choice of provider, environment, and confidentiality experienced. Additionally, it demonstrates the value of combining cognitive interviewing techniques with Likert questions to assess service responsiveness.

Assessing the reliability of phone surveys to measure reproductive, maternal and child health knowledge among pregnant women in rural India: a feasibility study

OBJECTIVES

Efforts to understand the factors influencing the uptake of reproductive, maternal, newborn, child health and nutrition (RMNCH&N) services in high disease burden low-resource settings have often focused on face-to-face surveys or direct observations of service delivery. Increasing access to mobile phones has led to growing interest in phone surveys as a rapid, low-cost alternatives to face-to-face surveys. We assess determinants of RMNCH&N knowledge among pregnant women with access to phones and examine the reliability of alternative modalities of survey delivery.

PARTICIPANTS

Women 5-7 months pregnant with access to a phone.

SETTING

Four districts of Madhya Pradesh, India.

DESIGN

Cross-sectional surveys administered face-to-face and within 2 weeks, the same surveys were repeated among two random subsamples of the original sample: face-to-face (n=205) and caller-attended telephone interviews (n=375). Bivariate analyses, multivariable linear regression, and prevalence and bias-adjusted kappa scores are presented.

RESULTS

Knowledge scores were low across domains: 52% for maternal nutrition and pregnancy danger signs, 58% for family planning, 47% for essential newborn care, 56% infant and young child feeding, and 58% for infant and young child care. Higher knowledge (≥1 composite score) was associated with older age; higher levels of education and literacy; living in a nuclear family; primary health decision-making; greater attendance in antenatal care and satisfaction with accredited social health activist services. Survey questions had low inter-rater and intermodal reliability (kappa<0.70) with a few exceptions. Questions with the lowest reliability included true/false questions and those with unprompted, multiple response options. Reliability may have been hampered by the sensitivity of the content, lack of privacy, enumerators' and respondents' profile differences, rapport, social desirability bias, and/or enumerator's ability to adequately convey concepts or probe.

CONCLUSIONS

Phone surveys are a reliable modality for generating population-level estimates data about pregnant women's knowledge, however, should not be used for individual-level tracking.

TRIAL REGISTRATION NUMBER

NCT03576157.

Comparison of clinicians' perceptions of the Recommended Summary Plan for Emergency Care and Treatment (ReSPECT) before and during the COVID-19 pandemic

INTRODUCTION

The Recommended Summary Plan for Emergency Care and Treatment (ReSPECT) process encourages collaboration between clinicians, patients, and relatives on emergency care wishes and resuscitation decisions. The impact of the COVID-19 pandemic on clinicians' views of the ReSPECT process was unknown. We examined whether there were changes in clinicians' knowledge, skills, and attitudes regarding ReSPECT during the pandemic.

METHODS

We conducted a cross-sectional survey of clinicians at one acute hospital in the UK. We developed a questionnaire with a defined 5-point Likert scale and asked clinicians to recall their pre-pandemic views on ReSPECT and report their current views at the time of survey distribution (May 2020, end of the first COVID-19 wave in the UK). We compared their self-reported views before and during the pandemic.

RESULTS

We analysed 171 questionnaire responses. Clinicians reported ReSPECT telephone discussions with relatives were more challenging (pre-pandemic median 4, IQR 3-4; during pandemic median 4, IQR 4-5; p < 0.001) and negative emotions whilst conducting these discussions with relatives increased during the pandemic (pre-pandemic median 3, IQR 2-3.5; during pandemic median 3, IQR 2-4; p < 0.001). Clinicians also reported an increase in the importance of reaching a shared understanding of decisions with patients and relatives (pre-pandemic median 4, IQR 4-5; during pandemic median 5, IQR 4-5; p < 0.001).

CONCLUSIONS

There were differences in clinicians' knowledge, skills, and attitudes scores before and during the pandemic. Our findings highlighted that clinicians could benefit from training in remote ReSPECT conversations with relatives.