Use of most significant change (MSC) technique to evaluate health promotion training of maternal community health workers in Cianjur district, Indonesia

Remotely and collaboratively evaluating a campus-based therapy dog program during the COVID-19 pandemic

The COVID-19 pandemic adversely affected the wellbeing of university students and adults in general, emphasizing the need for mental health programming that was compliant with physical distancing mandates. The present evaluation investigated mental health and social connection within the context of COVID-19 by remotely evaluating a virtual animal-assisted activity at the University of Saskatchewan - PAWS Your Stress. The purpose of this article is to outline our evaluation methods and findings, while calling specific attention to the collaborative strategies that were implemented within a remote, time-sensitive context. The evaluation findings revealed that remote animal-assisted programming can facilitate connections with humans and animals, and promote multiple mental health benefits, despite the lack of physical interaction with the animals. Our lessons learned indicate that remote program logic modelling workshops are feasible when suited to audience demographics. Further, our experience suggests that the Most Significant Change technique (a qualitative, participatory, storytelling method that elicits outcome data) can be useful in time-restricted evaluations, and the necessity of central steps in the process may vary depending on evaluation goals. This project has implications for future evaluation work, by demonstrating the effective use of remote methods that allowed for successful stakeholder collaboration.

Does social capital influence community health worker knowledge, attitude and practices towards COVID-19? Findings from a cross-sectional study in Malang district, Indonesia

Community health workers (CHWs) are the first point of contact with the primary health care system in many low- and middle-income countries and are situated to play a critical role in the public health response to the COVID-19 pandemic. The knowledge, attitude and practices of CHWs regarding COVID-19 may be influenced by their level of trust and participation in the community, collectively defined as their level of social capital. To assess whether social capital influences CHWs’ knowledge, attitude and practices related to COVID-19, we conducted a web-based survey of CHWs (n = 478) in Malang district, Indonesia between October 2020 and January 2021. CHW social capital was measured using the Shortened Adapted Social Capital Assessment Tool. Multiple logistic regression results show that cognitive social capital was associated with higher self-reported knowledge of COVID-19, more confidence in answering COVID-related questions from the community and feeling safe from COVID-19 when working. Membership of community organisations was associated with a higher number of COVID-related tasks conducted. Thus, CHWs in Malang district with higher levels of cognitive social capital were more likely to be confident in their knowledge and ability to respond to COVID-19, and CHWs embedded in their community were more likely to be engaged in pandemic response duties. Our findings suggest that policies aimed at promoting CHW embeddedness, targeted recruitment and addressing training needs hold promise in strengthening the positive contribution of the community health workforce to the COVID-19 response.

•

We conducted a survey to examine how social capital influences CHW knowledge, attitudes and practices re COVID-19 in Malang district, Indonesia.

•

Structural social capital was associated with a higher number of COVID-related tasks conducted.

•

Cognitive social capital was associated with higher knowledge of COVID-19 and higher confidence in answering COVID-related questions.

•

Promoting CHW embeddedness and targeted recruitment may strengthen the contribution of CHWs to future public health crises in Malang district.

Understanding community health worker employment preferences in Malang district, Indonesia, using a discrete choice experiment

Background

Community health workers (CHWs) play a critical role in supporting health systems, and in improving accessibility to primary healthcare. In many settings CHW programmes do not have formalised employment models and face issues of high attrition and poor performance. This study aims to determine the employment preferences of CHWs in Malang district, Indonesia, to inform policy interventions.

Methods

A discrete choice experiment was conducted with 471 CHWs across 28 villages. Attributes relevant to CHW employment were identified through a multistage process including literature review, focus group discussions and expert consultation. Respondents’ choices were analysed with a mixed multinomial logit model and latent class analyses.

Results

Five attributes were identified: (1) supervision; (2) training; (3) monthly financial benefit; (4) recognition; and (5) employment structure. The most important influence on choice of job was a low monthly financial benefit (US$~2) (β=0.53, 95% CI=0.43 to 0.63), followed by recognition in the form of a performance feedback report (β=0.13, 95% CI=0.07 to 0.20). A large monthly financial benefit (US$~20) was most unappealing to respondents (β=−0.13, 95% CI=−0.23 to −0.03). Latent class analysis identified two groups of CHWs who differed in their willingness to accept either job presented and preferences over specific attributes. Preferences diverged based on respondent characteristics including experience, hours’ worked per week and income.

Conclusion

CHWs in Malang district, Indonesia, favour a small monthly financial benefit which likely reflects the unique cultural values underpinning the programme and a desire for remuneration that is commensurate with the limited number of hours worked. CHWs also desire enhanced methods of performance feedback and greater structure around training and their rights and responsibilities. Fulfilling these conditions may become increasingly important should CHWs work longer hours.

Evaluating a peer-led wellbeing programme for doctors-in-training during the COVID-19 pandemic in Victoria, Australia, using the Most Significant Change technique

This article discusses the use of the Most Significant Change (MSC) technique in a mixed-methods evaluation of a pilot wellbeing programme for obstetrics and gynaecology doctors-in-training introduced at a large public hospital during Melbourne, Australia's second coronavirus (COVID-19) lockdown, which occurred from 7 July to 26 October 2020. The evaluation was conducted remotely using videoconferencing technology, to conform with pandemic restrictions. MSC complemented the program's participatory principles and was chosen because it seeks to learn about participants' perceptions of programme impacts by evaluating their stories of significant change. Stakeholders select one story exemplifying the most significant change resulting from the evaluated program. Inductive thematic analysis of all stories is combined with reasons for making the selection, to inform learnings (Dart & Davies, 2003; Tonkin et al., 2021). Nine stories of change were included in the selection. The most significant change was a more supportive workplace culture brought about by enabling basic needs to be met and breaking down hierarchical barriers. This was linked to five interconnected themes - connection, caring, communication, confidence and cooperation. The evaluation learnings are explored and reflections on remotely conducting MSC evaluation are shared.

Sustainability of the Effects and Impacts of Using Digital Technology to Extend Maternal Health Services to Rural and Hard-to-Reach Populations: Experience From Southwest Nigeria

Background

Nigeria has one of the worst health and development profiles globally. A weak health system, poor infrastructure, and varied socio-cultural factors are cited as inhibitors to optimal health system performance and improved maternal and child health status. eHealth has become a major solution to closing these gaps in health care delivery in low- and middle-income countries (LMICs). This research reports the use of satellite communication (SatCom) technology and the existing 3G mobile network for providing video training (VTR) for health workers and improving the digitization of healthcare data.

Objective

To evaluate whether the expected project outcomes that were achieved at the end-line evaluation of 2019 were sustained 12 months after the project ended.

Methods

From March 2017 to March 2019, digital innovations including VTR and data digitization interventions were delivered in 62 healthcare facilities in Ondo State, southwest Nigeria, most of which lacked access to a 3G mobile network. Data collection for the evaluation combined documents' review with quantitative data extracted from health facility registers, and 24 of the most significant change stories to assess the longevity of the outcomes and impacts of digital innovation in the four domains of healthcare: use of eHealth technology for data management, utilization of health facilities by patients, the standard of care, and staff attitude. Stories of the most significant changes were audio-recorded, transcribed for analysis, and categorized by the above domains to identify the most significant changes 12 months after the project closedown.

Results

Findings showed that four project outcomes which were achieved at end-line evaluation were sustained 12 months after project closedown namely: staff motivation and satisfaction; increased staff confidence to perform healthcare roles; improved standard of healthcare delivery; and increased adoption of eHealth innovations beyond the health sector. Conversely, an outcome that was reversed following the discontinuation of SatCom from health facilities is the availability of accurate and reliable data for decision-making.

Conclusion

Digital technology can have lasting impacts on health workers, patients, and the health system, through improving data management for decision-making, the standard of maternity service delivery, boosting attendance at health facilities, and utilization of services. Locally driven investment is essential for ensuring the long-term survival of eHealth projects to achieve sustainable development goals (SDGs) in LMICs.

Measuring the Impact of Patient Engagement From the Perspective of Health Professionals Leading Quality Improvement Projects

INTRODUCTION

The value of engaging patients and families in health care quality improvement (QI) initiatives is to help align health care system efforts with patient priorities. Meaningful evaluation of engaging with patients and families within QI may promote future collaboration. The aim of this study was to identify the experiential impact of patient engagement from the perspective of health professionals who were leading health care QI projects.

METHODS

Point-of-care health professionals who completed a fellowship capacity building program between 2014 and 2018 that provided an opportunity to learn about patient engagement concepts and to engage patients, families, and caregivers in their QI projects were invited to participate in the study. The Most Significant Change technique was used as a participatory approach to obtain qualitative evaluative data from semistructured interviews with health professional fellows. Significant change stories were curated from self-narratives grounded in the experiences of health professional fellows.

RESULTS

The stories demonstrated that gaining new knowledge on concepts related to patient engagement as part of a structured curriculum is effective in both supporting engagement in practice and cultivating the importance of patient engagement among health professionals. The early and ongoing involvement of patients was a key factor in shaping the project while fostering a patient-centered focus. Seeking out the patient voice throughout the QI project led to improvements in patient care experiences.

DISCUSSION

The findings of this study can inform programs seeking to promote patient engagement in health care QI. The positive changes that stem from aligning capacity building programs with patient-oriented priorities support the vision that patient engagement should be at the foundation of health care QI.

How beneficiaries see complex health interventions: a practice review of the Most Significant Change in ten countries

Background

The Most Significant Change is a story-based evaluation approach used in many international development programs. This practice review summarises practical experience with the approach in complex health interventions in ten countries, with the objective of making it more accessible in evaluation of other complex health interventions.

Results

Participatory research practitioners and trainees discussed five themes following brief presentations by each of the seven attendees who led the exercise: (i) sampling and recruitment; (ii) phrasing the questions to elicit stories; (iii) story collection strategies; (iv) quality assurance; and (v) analysis. Notes taken during the meeting provided the framework for this article.

Recruitment strategies in small studies included universal engagement and, in larger studies, a purposive, systematic or random sampling. Meeting attendees recommended careful phrasing and piloting of the question(s) as this affects the quality and focus of the stories generated. They stressed the importance of careful training and monitoring of fieldworkers collecting stories to ensure full stories are elicited and recorded. For recording, in most settings they preferred note taking with back-checking or self-writing of stories by story tellers, rather than audio-recording. Analysis can combine participatory selection of a small number of stories, deductive or inductive thematic analysis and discourse analysis. Meeting attendees noted that involvement in collection of the stories and their analysis and discussion had a positive impact for research team members.

Conclusions

Our review confirms the plasticity, feasibility and acceptability of the Most Significant Change technique across different sociopolitical, cultural and environmental contexts of complex interventions. Although the approach can surface unexpected impacts, it is not a 360-degree evaluation. Its strength lies in characterising the changes, where these happen, in the words of the beneficiaries. We hope this distillation of our practice makes the technique more readily available to health sector researchers.

Evaluation of global health capacity building initiatives in low-and middle-income countries: A systematic review

BACKGROUND

Low- and middle-income countries (LMICs) are in dire need to improve their health outcomes. Although Global Health Capacity Building (GHCB) initiatives are recommended approaches, they risk being ineffective in the absence of standardized evaluation methods. This study systematically reviews evaluation approaches for GHCB initiatives in LMICs.

METHODS

We searched the Medline (OVID), PubMed, Scopus, and Embase.com databases for studies reporting evaluation of a GHCB initiative in a LMIC from January 1, 2009 until August 15, 2019. To differentiate them from intervention, prevention, and awareness initiatives, included articles reported at least one approach to evaluate their learning modality. We excluded cross-sectional studies, reviews, and book chapters that only assessed the effect of interventions. Data identifying the learning modality, and evaluation method, level, time interval, and approach were extracted from articles as primary outcomes.

RESULTS

Of 8324 identified studies, 63 articles were eligible for analysis. Most studies stemmed from Africa and Asia (69.8%), were delivered and evaluated face-to-face (74.6% and 76.2%), mainly to professionals (57.1%) and community workers (20.6%). Although the use of online and blended modalities showed an increase over the past 4 years, only face-to-face initiatives were evaluated long-term beyond individual-level. GHCB evaluations in general lacked standardization especially regarding the tools.

CONCLUSION

This is an important resource for evaluating GHCB initiatives in LMICs. It synthesizes evaluation approaches, offers recommendations for improvement, and calls for the standardization of evaluations, especially for long-term and wider impact assessment of online and blended modalities.