Lessons learned in evaluating system interdependencies using qualitative methods

This article shares lessons learned while evaluating the system interdependencies for a clinical and translational research centre (CTR). It explores the methodological challenge of discussing system concepts (e.g., cascading failures, feedback loops, and reflex arcs) with layperson participants during evaluation interviews. The article discusses the iterative process of moving from structured interview approaches in favor of an open narrative approach for data collection and lessons learned.

Using the Homeland Security Exercise and Evaluation Program (HSEEP) Building Block Approach to Implement System Evaluation Theory (SET)

This article shares lessons learned in applying system evaluation theory (SET) to evaluate a Clinical and Translational Research Center (CTR) funded by the National Institutes of Health. After describing how CTR support cores are intended to work interdependently as a system, the case is made for SET as the best fit for evaluating this evaluand. The article then details how the evaluation was also challenged to facilitate a CTR culture shift, helping support cores to move from working autonomously to working together and understanding how the cores' individual operating processes impact each other. This was achieved by incorporating the Homeland Security Exercise and Evaluation Program (HSEEP) building block approach to implement SET. Each of the seven HSEEP building blocks is examined for alignment with each of SET's three steps and the ability to systematically support the goal of moving CTR cores toward working interdependently. The implications of using HSEEP to support SET implementation for future evaluations are discussed.

Evaluating health service coverage in Ghana's Volta Region using a modified Tanahashi model

Background: The United Nations 2030 Sustainable Development Goals have reaffirmed the international community’s commitment to maternal, newborn, and child health, with further investments in achieving quality essential service coverage and financial protection for all.

Objective: Using a modified version of the 1978 Tanahashi model as an analytical framework for measuring and assessing health service coverage, this paper aims to examine the system of care at the community level in Ghana’s Volta Region to highlight the continued reforms needed to achieve Universal Health Coverage.

Methods: The Tanahashi model evaluates health system coverage through five key measures that reflect different stages along the service provision continuum: availability of services; accessibility; initial contact with the health system; continued utilization; and quality coverage. Data from cross-sectional household and health facility surveys were used in this study. Immunization and antenatal care services were selected as tracer interventions to serve as proxies to assess systems bottlenecks.

Results: Financial access and quality coverage were identified as the biggest bottlenecks for both tracer indicators. Financial accessibility, measured by enrollment in Ghana’s National Health Insurance Scheme was poor with 16.94% presenting valid membership cards. Childhood immunization was high but dropped modestly from 93.8% at initial contact to 76.7% quality coverage. For antenatal care, estimates ranged from 65.9% at initial visit to 25.1% quality coverage.

Conclusion: Results highlight the difficulty in achieving high levels of quality service coverage and the large variations that exist within services provided at the primary care level. While vertical investments have been prioritized to benefit specific health services, a comprehensive systems approach to primary health care needs to be further strengthened to reach Ghana’s Universal Health Coverage objectives.

A Closer Look at the "Right" Format for Clinical Decision Support: Methods for Evaluating a Storyboard BestPractice Advisory

(1) Background: The five rights of clinical decision support (CDS) are a well-known framework for planning the nuances of CDS, but recent advancements have given us more options to modify the format of the alert. One-size-fits-all assessments fail to capture the nuance of different BestPractice Advisory (BPA) formats. To demonstrate a tailored evaluation methodology, we assessed a BPA after implementation of Storyboard for changes in alert fatigue, behavior influence, and task completion; (2) Methods: Data from 19 weeks before and after implementation were used to evaluate differences in each domain. Individual clinics were evaluated for task completion and compared for changes pre- and post-redesign; (3) Results: The change in format was correlated with an increase in alert fatigue, a decrease in erroneous free text answers, and worsened task completion at a system level. At a local level, however, 14% of clinics had improved task completion; (4) Conclusions: While the change in BPA format was correlated with decreased performance, the changes may have been driven primarily by the COVID-19 pandemic. The framework and metrics proposed can be used in future studies to assess the impact of new CDS formats. Although the changes in this study seemed undesirable in aggregate, some positive changes were observed at the level of individual clinics. Personalized implementations of CDS tools based on local need should be considered.