Implementation Outcomes and Indicators as a New Challenge in Health Services Research: A Systematic Scoping Review

Availability of published evidence on coverage, cost components, and funding support for digitalisation of infectious disease surveillance in Africa, 2003-2022: a systematic review

BACKGROUND

The implementation of digital disease surveillance systems at national levels in Africa have been challenged by many factors. These include user applicability, utility of IT features but also stable financial support. Funding closely intertwines with implementations in terms of geographical reach, disease focus, and sustainability. However, the practice of evidence sharing on geographical and disease coverage, costs, and funding sources for improving the implementation of these systems on the continent is unclear.

OBJECTIVES

To analyse the key characteristics and availability of evidence for implementing digital infectious disease surveillance systems in Africa namely their disease focus, geographical reach, cost reporting, and external funding support.

METHODS

We conducted a systematic review of peer-reviewed and grey literature for the period 2003 to 2022 (PROSPERO registration number: CRD42022300849). We searched five databases (PubMed, MEDLINE over Ovid, EMBASE, Web of Science, and Google Scholar) and websites of WHO, Africa CDC, and public health institutes of African countries. We mapped the distribution of projects by country; identified reported implementation cost components; categorised the availability of data on cost components; and identified supporting funding institutions outside Africa.

RESULTS

A total of 29 reports from 2,033 search results were eligible for analysis. We identified 27 projects implemented in 13 countries, across 32 sites. Of these, 24 (75%) were pilot projects with a median duration of 16 months, (IQR: 5-40). Of the 27 projects, 5 (19%) were implemented for HIV/AIDs and tuberculosis, 4 (15%) for malaria, 4 (15%) for all notifiable diseases, and 4 (15%) for One Health. We identified 17 cost components across the 29 reports. Of these, 11 (38%) reported quantified costs for start-up capital, 10 (34%) for health personnel compensation, 9 (31%) for training and capacity building, 8 (28%) for software maintenance, and 7(24%) for surveillance data transmission. Of 65 counts of external funding sources, 35 (54%) were governmental agencies, 15 (23%) foundations, and 7 (11%) UN agencies.

CONCLUSIONS

The evidence on costing data for the digitalisation of surveillance and outbreak response in the published literature is sparse in quantity, limited in detail, and without a standardised reporting format. Most initial direct project costs are substantially donor dependent, short lived, and thus unsustainable.

Implementation and dissemination of home- and community-based interventions for informal caregivers of people living with dementia: a systematic scoping review

BACKGROUND

Informal caregivers of people with dementia (PwD) living at home are often the primary source of care, and, in their role, they often experience loss of quality of life. Implementation science knowledge is needed to optimize the real-world outcomes of evidence-based interventions (EBIs) for informal caregivers. This scoping review aims to systematically synthesize the literature that reports implementation strategies employed to deliver home- and community-based EBIs for informal caregivers of PwD, implementation outcomes, and the barriers and facilitators to implementation of these EBIs.

METHODS

Embase, MEDLINE, Web of Science, and Cochrane Library were searched from inception to March 2021; included studies focused on "implementation science," "home- and community-based interventions," and "informal caregivers of people with dementia." Titles and abstracts were screened using ASReview (an innovative AI-based tool for evidence reviews), and data extraction was guided by the ERIC taxonomy, the Implementation Outcome Framework, and the Consolidated Framework for Implementation Science Research; each framework was used to examine a unique element of implementation.

RESULTS

Sixty-seven studies were included in the review. Multicomponent (26.9%) and eHealth (22.3%) interventions were most commonly reported, and 31.3% of included studies were guided by an implementation science framework. Training and education-related strategies and provision of interactive assistance were the implementation strategy clusters of the ERIC taxonomy where most implementation strategies were reported across the reviewed studies. Acceptability (82.1%), penetration (77.6%), and appropriateness (73.1%) were the most frequently reported implementation outcomes. Design quality and packaging (intervention component suitability) and cosmopolitanism (partnerships) constructs, and patient's needs and resources and available resources (infrastructure) constructs as per the CFIR framework, reflected the most frequently reported barriers and facilitators to implementation.

CONCLUSION

Included studies focused largely on intervention outcomes rather than implementation outcomes and lacked detailed insights on inner and outer setting determinants of implementation success or failure. Recent publications suggest implementation science in dementia research is developing but remains in nascent stages, requiring future studies to apply implementation science knowledge to obtain more contextually relevant findings and to structurally examine the mechanisms through which implementation partners can strategically leverage existing resources and regional networks to streamline local implementation. Mapping local evidence ecosystems will facilitate structured implementation planning and support implementation-focused theory building.

TRIAL REGISTRATION

Not applicable.

Development of a conceptual framework to scale up co-managed care for older patients with hip fracture in China: a qualitative study

BACKGROUND

Hip fracture creates a major burden on society due to high mortality, loss of independence and excess medical costs for older people. A multidisciplinary co-managed model of care is widely considered as the best practice for the management of older patients with hip fracture. The study aims to develop a conceptual framework to inform the future scale-up of this model of care through the identification of barriers and enablers that may influence successful uptake.

METHODS

This qualitative study was conducted within an interventional study, which aimed to test the effectiveness of co-managed model of care for older patients with hip fracture. Health providers and health administrators from three hospitals were purposively selected and interviewed in-depth. The Consolidated Framework for Implementation Research (CFIR) was used to develop interview guides, collect and analyse data. Inductive and deductive approaches were used to generate enablers or barriers, aligned with the CFIR constructs. All barriers or enablers were inductively summarised to a conceptual framework with essential components to guide the implementation of co-managed model of care in other hospitals.

RESULTS

A total of 13 health providers and 3 health administrators were recruited. The main barriers to co-managed care implementation included perceived complexity of implementation, insufficient international collaboration and incentives, the absence of national guideline support and lack of digital health applications for communication between health providers, insufficient number of health providers and beds, and poor understanding about the effectiveness of this care model. A conceptual framework for future scale-up was then developed, consisting of the following essential components: hospital authority support, enabling environment, adequate number of beds, sufficient and skilled health providers, use of digital health technology, regular quality supervision, evaluation and feedback, and external collaborations.

CONCLUSIONS

Despite the complexity of the intervention, the co-managed model of care has the potential to be implemented and promoted in China and in similar settings, although there is a need to demonstrate feasibility in different settings.

Evaluating change in a pressured healthcare system: a cross-sectional study of implementation outcomes using routine data indicators and proxies

BACKGROUND

Implementation evaluation should focus on implementation success, guided by theories and frameworks. With high staff vacancies in the health services, it is important to consider pragmatic methods of data collection for implementation evaluation. This paper presents a cross-sectional rapid evaluation of a handheld medical device designed for remote examinations, piloted in Northern England. By using downloaded device data and administrative records mapped to domains from the implementation outcomes framework, this evaluation offers a pragmatic example of assessing implementation success.

METHODS

The pilot design was pragmatic: sites volunteered, decided which services to use the device in, and launched when ready. The pilot and evaluation together lasted 1 year. Data was downloaded from the devices, and administrative records for the pilot accessed. Variables were mapped to five of the implementation outcomes, after reviewing with the device manufacturer and pilot team to assess robustness.

RESULTS

N=352 care episodes were recorded using the device with 223 patients. Out of 19 sites 'signed up' to the pilot, 5 launched and delivered 10 of 35 proposed projects: a site and project adoption rate of 26 and 29%, respectively. Six sites signed up to an extension period; three had launched and three had not during the original timelines, indicating some sustainability. Feasibility was high, with only one in seven care episodes needing to be repeated due to poor device quality or error (sound/audio/internet). Fidelity of device usage was low for two of the eight available device examinations. Device and staffing costs were high but potential cost savings were attributable to fewer in-person appointments.

CONCLUSIONS

Through using device and administrative data, this evaluation minimised burden on busy healthcare staff yet was still guided by an evaluation framework. Five out of the eight implementation outcomes were measured, including sustainability and costs. The findings give insight into implementation challenges, particularly around adoption. For future research, it is recommended to engage with staff to prioritise outcome measurements and to focus on meaningful interpretation of indicators.

Parsley Health: Feasibility and acceptability of a large-scale holistic telehealth program for chronic disease care

Background

A holistic, personalized approach to medicine can be used to prevent and manage a variety of chronic diseases. However, effectively managing chronic diseases can be difficult due to barriers related to insufficient provider time, staffing, and lack of patient engagement. To address these challenges telehealth strategies are being increasingly adopted, yet few studies have explored how to evaluate the feasibility and implementation success of large-scale holistic telehealth models for chronic disease care. The aim of this study is to assess the feasibility and acceptability of a large-scale holistic telehealth program for the management of chronic diseases. Our study findings can inform the future development and assessment of chronic disease programs delivered through telehealth strategies.

Methods

Data was collected from participants enrolled in a Parsley Health membership from June 1, 2021 to June 1, 2022, a subscription-based holistic medicine practice designed to help people prevent or manage chronic diseases. Implementation outcome frameworks were used to understand engagement with services, participant satisfaction, and preliminary effectiveness of the program via a patient-reported symptom severity tool.

Results

Data from 10,205 participants with a range of chronic diseases were included in our analysis. Participants averaged 4.8 visits with their clinical team and reported high levels of satisfaction with their care (average NPS score of 81.35%). Preliminary evidence also showed substantial reduction in patient reported symptom severity.

Conclusion

Our findings suggest the Parsley Health program is a feasible and acceptable large-scale holistic telehealth program for chronic disease care. Successful implementation was due, in part, to services that promoted participant engagement along with tools and interfaces that were helpful and easy to use. These findings can be used to develop future holistic-focused telehealth programs for the management and prevention of chronic diseases.

Monitoring the Implementation of Tobacco Cessation Support Tools: Using Novel Electronic Health Record Activity Metrics

BACKGROUND

Clinical decision support (CDS) tools in electronic health records (EHRs) are often used as core strategies to support quality improvement programs in the clinical setting. Monitoring the impact (intended and unintended) of these tools is crucial for program evaluation and adaptation. Existing approaches for monitoring typically rely on health care providers' self-reports or direct observation of clinical workflows, which require substantial data collection efforts and are prone to reporting bias.

OBJECTIVE

This study aims to develop a novel monitoring method leveraging EHR activity data and demonstrate its use in monitoring the CDS tools implemented by a tobacco cessation program sponsored by the National Cancer Institute's Cancer Center Cessation Initiative (C3I).

METHODS

We developed EHR-based metrics to monitor the implementation of two CDS tools: (1) a screening alert reminding clinic staff to complete the smoking assessment and (2) a support alert prompting health care providers to discuss support and treatment options, including referral to a cessation clinic. Using EHR activity data, we measured the completion (encounter-level alert completion rate) and burden (the number of times an alert was fired before completion and time spent handling the alert) of the CDS tools. We report metrics tracked for 12 months post implementation, comparing 7 cancer clinics (2 clinics implemented the screening alert and 5 implemented both alerts) within a C3I center, and identify areas to improve alert design and adoption.

RESULTS

The screening alert fired in 5121 encounters during the 12 months post implementation. The encounter-level alert completion rate (clinic staff acknowledged completion of screening in EHR: 0.55; clinic staff completed EHR documentation of screening results: 0.32) remained stable over time but varied considerably across clinics. The support alert fired in 1074 encounters during the 12 months. Providers acted upon (ie, not postponed) the support alert in 87.3% (n=938) of encounters, identified a patient ready to quit in 12% (n=129) of encounters, and ordered a referral to the cessation clinic in 2% (n=22) of encounters. With respect to alert burden, on average, both alerts fired over 2 times (screening alert: 2.7; support alert: 2.1) before completion; time spent postponing the screening alert was similar to completing (52 vs 53 seconds) the alert, and time spent postponing the support alert was more than completing (67 vs 50 seconds) the alert per encounter. These findings inform four areas where the alert design and use can be improved: (1) improving alert adoption and completion through local adaptation, (2) improving support alert efficacy by additional strategies including training in provider-patient communication, (3) improving the accuracy of tracking for alert completion, and (4) balancing alert efficacy with the burden.

CONCLUSIONS

EHR activity metrics were able to monitor the success and burden of tobacco cessation alerts, allowing for a more nuanced understanding of potential trade-offs associated with alert implementation. These metrics can be used to guide implementation adaptation and are scalable across diverse settings.

Quality criteria/key components for high quality of diabetes management to avoid diabetes-related complications

Aim

The aim of this review shoud be to map data and to identify quality indicators for good practices for diabetes management and secondary prevention, specifically of type 2 diabetes.

Methods

To achieve this aim we performed the following steps: (i) a literature review on evaluation criteria, (ii) selection of quality criteria and key components for high quality of care, (iii) creation of a checklist to identify the best practice of diabetes management based on the detected criteria.

Results

The literature search about the quality indicators for diabetes care resulted in the following: identifying of key components and quality indicators for structure, process and outcome quality.

Conclusions

The set of quality criteria will be discussed and used to identify the best practice diabetes management programs for secondary prevention of type 2 diabetes.

Acceptability of employment readiness measures in youth and young adults on the autism spectrum: a mixed-methods study

PURPOSE

Reliable, valid, and pragmatic measures are essential for monitoring and evaluating employment readiness and comparing the effectiveness of alternative implementation strategies. The Work Readiness Inventory (WRI) and Ansell-Casey Life Skills Assessment (ACLSA) are valid measures of employment readiness in neurotypical populations; however, their acceptability (i.e., user perception of measure as agreeable/satisfactory) for persons on the autism spectrum is not yet known. This investigation assesses the acceptability of the WRI and a modified ACLSA (ACLSA-M) in measuring employment readiness in youth/young adults on the spectrum.

METHODS

A concurrent triangulation mixed-methods study design utilizing quantitative pre-post measurement of a community-based employment readiness program alongside qualitative survey assessment was employed to determine concurrent acceptability. For robustness, further explication through peer debriefing of experts evaluated the retrospective acceptability via interview and acceptability-rate assessment.

RESULTS

Findings indicated that both measures are acceptable, although individual- and job-specific item modifications are advised, particularly due to disability-specific needs. Significant change in employment readiness in youth/young adults on the spectrum supports concurrent acceptability. Peer debriefing provided rich data on retrospective acceptability. Acceptability-rates of 0.84 and 0.91 confirm broad acceptability of these measures.

CONCLUSIONS

Implications are presented for clinicians and researchers, highlighting the relevance for autism-specific measurement development and acceptability.Implications for rehabilitationGiven the lower labor force participation of persons on the autism spectrum, a combination of measures should be used in the assessment of an individual's employment readiness.In youth and young adults on the spectrum, employment readiness can be measured using the Work Readiness Inventory (WRI) and a modified version of the Ansell-Casey Life Skills Assessment (ACLSA-M).In clinical practice and research, modifying the contents of these measures may be advised to minimize language complexity, and maximize ease in self report.When designing, developing, and testing new measures in rehabilitation practice or research, the intent should be broadened by involving diverse representation from the project outset, by engaging both those on the spectrum and neurotypical populations.

Implementation outcome instruments for use in physical healthcare settings: a systematic review

BACKGROUND

Implementation research aims to facilitate the timely and routine implementation and sustainment of evidence-based interventions and services. A glaring gap in this endeavour is the capability of researchers, healthcare practitioners and managers to quantitatively evaluate implementation efforts using psychometrically sound instruments. To encourage and support the use of precise and accurate implementation outcome measures, this systematic review aimed to identify and appraise studies that assess the measurement properties of quantitative implementation outcome instruments used in physical healthcare settings.

METHOD

The following data sources were searched from inception to March 2019, with no language restrictions: MEDLINE, EMBASE, PsycINFO, HMIC, CINAHL and the Cochrane library. Studies that evaluated the measurement properties of implementation outcome instruments in physical healthcare settings were eligible for inclusion. Proctor et al.'s taxonomy of implementation outcomes was used to guide the inclusion of implementation outcomes: acceptability, appropriateness, feasibility, adoption, penetration, implementation cost and sustainability. Methodological quality of the included studies was assessed using the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) checklist. Psychometric quality of the included instruments was assessed using the Contemporary Psychometrics checklist (ConPsy). Usability was determined by number of items per instrument.

RESULTS

Fifty-eight publications reporting on the measurement properties of 55 implementation outcome instruments (65 scales) were identified. The majority of instruments assessed acceptability (n = 33), followed by appropriateness (n = 7), adoption (n = 4), feasibility (n = 4), penetration (n = 4) and sustainability (n = 3) of evidence-based practice. The methodological quality of individual scales was low, with few studies rated as 'excellent' for reliability (6/62) and validity (7/63), and both studies that assessed responsiveness rated as 'poor' (2/2). The psychometric quality of the scales was also low, with 12/65 scales scoring 7 or more out of 22, indicating greater psychometric strength. Six scales (6/65) rated as 'excellent' for usability.

CONCLUSION

Investigators assessing implementation outcomes quantitatively should select instruments based on their methodological and psychometric quality to promote consistent and comparable implementation evaluations. Rather than developing ad hoc instruments, we encourage further psychometric testing of instruments with promising methodological and psychometric evidence.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO 2017 CRD42017065348.

A pragmatic method for costing implementation strategies using time-driven activity-based costing

BACKGROUND

Implementation strategies increase the adoption of evidence-based practices, but they require resources. Although information about implementation costs is critical for decision-makers with budget constraints, cost information is not typically reported in the literature. This is at least partly due to a need for clearly defined, standardized costing methods that can be integrated into implementation effectiveness evaluation efforts.

METHODS

We present a pragmatic approach to systematically estimating detailed, specific resource use and costs of implementation strategies that combine time-driven activity-based costing (TDABC), a business accounting method based on process mapping and known for its practicality, with a leading implementation science framework developed by Proctor and colleagues, which guides specification and reporting of implementation strategies. We illustrate the application of this method using a case study with synthetic data.

RESULTS

This step-by-step method produces a clear map of the implementation process by specifying the names, actions, actors, and temporality of each implementation strategy; determining the frequency and duration of each action associated with individual strategies; and assigning a dollar value to the resources that each action consumes. The method provides transparent and granular cost estimation, allowing a cost comparison of different implementation strategies. The resulting data allow researchers and stakeholders to understand how specific components of an implementation strategy influence its overall cost.

CONCLUSION

TDABC can serve as a pragmatic method for estimating resource use and costs associated with distinct implementation strategies and their individual components. Our use of the Proctor framework for the process mapping stage of the TDABC provides a way to incorporate cost estimation into implementation evaluation and may reduce the burden associated with economic evaluations in implementation science.