Usability Issues in Evidence-Based Psychosocial Interventions and Implementation Strategies: Cross-project Analysis

Development of a method for Making Optimal Decisions for Intervention Flexibility during Implementation (MODIFI): a modified Delphi study

BACKGROUND

Intervention adaptation is often necessary to improve the fit between evidence-based practices/programs and implementation contexts. Existing frameworks describe intervention adaptation processes but do not provide detailed steps for prospectively designing adaptations, are designed for researchers, and require substantial time and resources to complete. A pragmatic approach to guide implementers through developing and assessing adaptations in local contexts is needed. The goal of this project was to develop Making Optimal Decisions for Intervention Flexibility during Implementation (MODIFI), a method for intervention adaptation that leverages human centered design methods and is tailored to the needs of intervention implementers working in applied settings with limited time and resources.

METHOD

MODIFI was iteratively developed via a mixed-methods modified Delphi process. Feedback was collected from 43 implementation research and practice experts. Two rounds of data collection gathered quantitative ratings of acceptability and inclusion (Round 1) and feasibility (Round 2), as well as qualitative feedback regarding MODIFI revisions analyzed using conventional content analysis.

RESULTS

In Round 1, most participants rated all proposed components as essential but identified important avenues for revision which were incorporated into MODIFI prior to Round 2. Round 2 emphasized feasibility, where ratings were generally high and fewer substantive revisions were recommended. Round 2 changes largely surrounded operationalization of terms/processes and sequencing of content. Results include a detailed presentation of the final version of the three-step MODIFI method (Step 1: Learn about the users, local context, and intervention; Step 2: Adapt the intervention; Step 3: Evaluate the adaptation) along with a case example of its application.

DISCUSSION

MODIFI is a pragmatic method that was developed to extend the contributions of other research-based adaptation theories, models, and frameworks while integrating methods that are tailored to the needs of intervention implementers. Guiding teams to tailor evidence-based interventions to their local context may extend for whom, where, and under what conditions an intervention can be effective.

HCI Contributions in Mental Health: A Modular Framework to Guide Psychosocial Intervention Design

Many people prefer psychosocial interventions for mental health care or other concerns, but these interventions are often complex and unavailable in settings where people seek care. Intervention designers use technology to improve user experience or reach of interventions, and HCI researchers have made many contributions toward this goal. Both HCI and mental health researchers must navigate tensions between innovating on and adhering to the theories of change that guide intervention design. In this paper, we propose a framework that describes design briefs and evaluation approaches for HCI contributions at the scopes of capabilities, components, intervention systems, and intervention implementations. We show how theories of change (from mental health) can be translated into design briefs (in HCI), and that these translations can bridge and coordinate efforts across fields. It is our hope that this framework can support researchers in motivating, planning, conducting, and communicating work that advances psychosocial intervention design.

Formative Evaluation of a Comprehensive Self-Management Intervention for Irritable Bowel Syndrome, Comorbid Anxiety, and Depression: Mixed Methods Study

BACKGROUND

Irritable bowel syndrome (IBS) is a disorder of the gut-brain interaction that is associated with abdominal pain, altered bowel patterns, and reduced quality of life. Up to 50% of patients with IBS also report anxiety or depressive symptoms. Although effective self-management interventions exist for individuals with IBS, few have been effectively implemented, and most do not consider the unique needs of patients with comorbid IBS and anxiety or depression.

OBJECTIVE

This study aimed to determine the anticipated acceptability, appropriateness, feasibility, and usability of a comprehensive self-management intervention using an implementation science and human-centered design approach among individuals with comorbid IBS and anxiety or depression and health care providers.

METHODS

A convergent mixed methods design was used to elicit feedback on the comprehensive self-management intervention outline and content to identify refinement needs before testing. Patients with IBS and moderate to severe anxiety or depression and health care providers were purposefully sampled from primary care and gastroenterology settings. Participants completed semistructured interviews and surveys on anticipated acceptability, appropriateness, feasibility, and usability.

RESULTS

Patient participants (n=12) were on average 36.8 (SD 12.2) years of age, and 42% (5/12) were currently receiving psychological therapy. Health care providers (n=14) were from primary care (n=7) and gastroenterology (n=7) settings. The mean usability scores (out of 100) were 52.5 (SD 14.5) for patients and 45.6 (SD 11.6) for providers. For patients and providers, qualitative data expanded the quantitative findings for acceptability and appropriateness. Acceptability findings were the comprehensive nature of the intervention and discussion of the gut-brain interaction. For appropriateness, participants reported that the intervention provided structure, accountability, and support. Feasibility was confirmed for patients, but there was a divergence of findings between quantitative and qualitative measures for providers. Patients focused on intervention feasibility, while providers focused on implementation feasibility in the clinic. Identified usability issues to address before implementation included the intervention delivery format, length, and lack of integration into health care settings that, if not addressed, may limit the reach of the intervention.

CONCLUSIONS

Patients and health care providers found the intervention acceptable and appropriate. Several feasibility and usability issues were identified, including intervention delivery methods, length of intervention, and the best methods to implement in the clinic setting. The next steps are to refine the intervention to address the identified issues and test in a pilot study whether addressing usability issues leads to the anticipated improvements in implementation and uptake.

Digital and precision clinical trials: innovations for testing mental health medications, devices, and psychosocial treatments

Mental health treatment advances - including neuropsychiatric medications and devices, psychotherapies, and cognitive treatments - lag behind other fields of clinical medicine such as cardiovascular care. One reason for this gap is the traditional techniques used in mental health clinical trials, which slow the pace of progress, produce inequities in care, and undermine precision medicine goals. Newer techniques and methodologies, which we term digital and precision trials, offer solutions. These techniques consist of (1) decentralized (i.e., fully-remote) trials which improve the speed and quality of clinical trials and increase equity of access to research, (2) precision measurement which improves success rate and is essential for precision medicine, and (3) digital interventions, which offer increased reach of, and equity of access to, evidence-based treatments. These techniques and their rationales are described in detail, along with challenges and solutions for their utilization. We conclude with a vignette of a depression clinical trial using these techniques.

Helping Educational Leaders Mobilize Evidence (HELM): The iterative redesign of the Leadership and Organizational Change for Implementation (LOCI) intervention for use in schools

Background

Few "intervention agnostic" strategies have been developed that can be applied to the broad array of evidence-based practices (EBPs) in schools. This paper describes two studies that reflect the initial iterative redesign phases of an effective leadership-focused implementation strategy-Leadership and Organizational Change for Implementation (LOCI)-to ensure its acceptability, feasibility, contextual appropriateness, and usability when used in elementary schools. Our redesigned strategy-Helping Educational Leaders Mobilize Evidence (HELM)-is designed to improve principals' use of strategic implementation leadership to support the adoption and high-fidelity delivery of a universal EBP to improve student outcomes.

Method

In Study 1, focus groups were conducted (n = 6) with 54 district administrators, principals, and teachers. Stakeholders provided input on the appropriateness of original LOCI components to maximize relevance and utility in schools. Transcripts were coded using conventional content analysis. Key themes referencing low appropriateness were summarized to inform LOCI adaptations. We then held a National Expert Summit (Study 2) with 15 research and practice experts. Participants provided feedback via a nominal group process (NGP; n = 6 groups) and hackathon (n = 4 groups). The research team rated each NGP suggestion for how actionable, impactful/effective, and feasible it was. We also coded hackathon notes for novel ideas or alignment with LOCI components.

Results

Study 1 suggestions included modifications to LOCI content and delivery. Study 2's NGP results revealed most recommendations to be actionable, impactful/effective, and feasible. Hackathon results surfaced two novel ideas (distributed leadership teams and leaders' knowledge to support educators EBP use) and several areas of alignment with LOCI components.

Conclusion

Use of these iterative methods informed the redesign of LOCI and the development of HELM. Because it was collaboratively constructed, HELM has the potential to be an effective implementation strategy to support the use of universal EBP in schools.

The oracle of D-A-Ch - Results of a Delphi Survey for the development of the evidence- and consensus-based tool "My Logbook"

OBJECTIVE

The present report describes the results of four delphi surveys conducted within the quality improvement project "My Logbook" which aims to translate evidence-based standards for psychosocial care in pediatric oncology into a practical consensus-based tool.

METHODS

In four consecutive delphi surveys a total of n=153 international, multi-disciplinary experts rated the content, method, and design of the different booklets of "My Logbook" which a local expert group had conceptualized. After each survey, the feedback was incorporated, and the changes were evaluated in a final consensus vote by the quality assurance panel of the PSAPOH.

RESULTS

While some surveys led to a review on a page level, most booklets as a whole reached the consensus-level of approval. Over the course of the surveys, any revisions and comments were incorporated in the booklets, and approval rates increased steadily.

DISCUSSION

The delphi surveys ensured the integration of multi-disciplinary, international expertise, uncovering issues such as language barriers and the need for a user manual that would not have been evident in first line . The incorporation of the input led to a continuous improvement of the tool, reflected in steadily increasing acceptance rates in the consecutive survey rounds.

CONCLUSION

The incorporation of the expert input as well as the additional development of a user manual resulted in a final version of the "My Logbook" apt for the interdisciplinary application in pediatric oncology in the entire DACH-region.

Development of a method for Making Optimal Decisions for Intervention Flexibility during Implementation (MODIFI): A modified Delphi study

Background

Intervention adaptation is often necessary to improve the fit between evidence-based practices/programs and implementation contexts. Existing frameworks describe intervention adaptation processes but do not provide detailed steps for prospectively designing adaptations, are designed for researchers, and require substantial time and resources to complete. A pragmatic approach to guide implementers through developing and assessing adaptations in local contexts is needed. The goal of this project was to develop Making Optimal Decisions for Intervention Flexibility during Implementation (MODIFI), a method for intervention adaptation that leverages human centered design methods and is tailored to the needs of intervention implementers working in applied settings with limited time and resources.

Method

MODIFI was iteratively developed via a mixed-methods modified Delphi process. Feedback was collected from 43 implementation research and practice experts. Two rounds of data collection gathered quantitative ratings of acceptability (Round 1) and feasibility (Round 2), as well as qualitative feedback regarding MODIFI revisions analyzed using conventional content analysis.

Results

In Round 1, most participants rated all proposed components as essential but identified important avenues for revision which were incorporated into MODIFI prior to Round 2. Round 2 emphasized feasibility, where ratings were generally high and fewer substantive revisions were recommended. Round 2 changes largely surrounded operationalization of terms/processes and sequencing of content. Results include a detailed presentation of the final version of the three-step MODIFI method (Step 1: Learn about the users, local context, and intervention; Step 2: Adapt the intervention; Step 3: Evaluate the adaptation) along with a case example of its application.

Discussion

MODIFI is a pragmatic method that was developed to extend the contributions of other research-based adaptation theories, models, and frameworks while integrating methods that are tailored to the needs of intervention implementers. Guiding teams to tailor evidence-based interventions to their local context may extend for whom, where, and under what conditions an intervention can be effective.

Bridging the gap: A quality improvement project to implement psychosocial care standards into clinical practice in pediatric oncology

BACKGROUND

Psychosocial guidelines and standards systematically describe stressors and resources in particularly challenging situations and hence serve as a basis for interventions to achieve defined psychosocial goals. Despite fundamental principles and guidelines for psychosocial methods, the quality of provided care varies considerably depending on setting, provision, and profession. The purpose of the present protocol is to illustrate the development and evaluation of the standardized psychological intervention "My Logbook", a practical guide accompanying children through all stages of treatment by directly translating current quality standards of psychosocial care into practice.

METHODS

In an evidence-based set-up, using face-to-face discussions and telephone conferences, a multi-professional team of local experts decide on critical disease-related issues, structure, content (information and intervention elements) and design of the quality improvement tool. Via delphi surveys an extended expert team is asked to rate the content, method, and design of all booklets which is concluded by a final agreement by the specialist group for quality assurance of the psychosocial working group in the Society for Pediatric Oncology (PSAPOH). The developed tools are piloted in an international multicenter study to evaluate the patient-reported outcome and feasibility and to integrate practical views of patients, as well as psychosocial and interdisciplinary professionals into the further development of the "My Logbook".

DISCUSSION

The iterative development of the "My Logbook" including local and international experts as well as the patient and practical perspective allow for the design of a process-oriented, consensus - and evidence-based tool directly translating the S3-Guideline into clinical practice. Feasibility and applicability are fostered through an iterative process of constant evaluation and adaptation of the tool by international experts and through the clinical experience gathered in the multi-centered pilot study. Furthermore, the systematic evaluation of the tool by patients, psychosocial, and interdisciplinary professionals enables the identification of persisting gaps between evidence-based standards and clinical practice, discrepancies between the various stakeholders' perspectives as well as regional differences in feasibility, thereby directly linking practice and research. The preliminary results emphasize that psychological support can be standardized, enabling an evaluation and optimization of psychosocial care which future studies need to assess in multicenter clinical randomized controlled trials.

Clinical insights: Preventing psychosis treatment disengagement

Many patients with schizophrenia spectrum disorders disengage from treatment, with negative impacts on prognosis and recovery. This commentary provides an overview of common reasons underlying treatment dropout and strategies for overcoming obstacles and enhancing patient engagement. Basic clinical skills such as open-ended questions and reflective listening can be powerful tools in unmasking structural and attitudinal barriers to psychosis care.