Applying User Input to the Design and Testing of an Electronic Behavioral Health Information System for Wraparound Care Coordination

Preliminary Implementation Outcomes of a Free Online Toolkit to Support Exposure Therapy Implementation for Youth

Background

Exposure therapy ("exposure") for youth anxiety is highly underutilized in clinical practice. Asynchronous, online implementation strategies such as online toolkits hold promise as pragmatic approaches for extending the sustainability of evidence-based interventions, but their long-term usage, perceived utility, and impact are rarely studied.

Objective

This study presents three-year preliminary implementation outcomes for a free, online toolkit to support exposure therapy use with youth: the Resource for Exposure for Anxiety Disordered Youth (READY; www.bravepracticeforkids.com). Implementation outcomes of interest included READY usage statistics, adoption, perceived utility, and clinician exposure use.

Methods

Web analytics characterized usage patterns. A survey of READY users (N = 49; M age = 34.2, 82.9% female, 71% White) assessed adoption, perceived utility, clinician exposure use, and persistent barriers to exposure use.

Results

In its first three years, READY had 13,543 page views across 1,731 unique users; 442 (25.6%) registered as a site user to access specialized content. Survey data suggested variability in usage and perceived utility across toolkit components. Qualitative analyses highlighted persistent exposure barriers that pointed to potential READY refinements.

Conculsions

Overall, READY has been accessed by hundreds of clinicians, but its impact was limited by low return to the site. This study highlights strengths and limitations of standalone online implementation supports and identifies additional steps needed to optimally support clinicians to deliver exposure to youth in need.

Moving from development to implementation of digital innovations within the NHS: myHealthE, a remote monitoring system for tracking patient outcomes in child and adolescent mental health services

Objective

This paper aims to report our experience of developing, implementing, and evaluating myHealthE (MHE), a digital innovation for Child and Adolescents Mental Health Services (CAMHS), which automates the remote collection and reporting of Patient-Reported Outcome Measures (PROMs) into National Health Services (NHS) electronic healthcare records.

Methods

We describe the logistical and governance issues encountered in developing the MHE interface with patient-identifiable information, and the steps taken to overcome these development barriers. We describe the application's architecture and hosting environment to enable its operability within the NHS, as well as the capabilities needed within the technical team to bridge the gap between academic development and NHS operational teams.

Results

We present evidence on the feasibility and acceptability of this system within clinical services and the process of iterative development, highlighting additional functions that were incorporated to increase system utility.

Conclusion

This article provides a framework with which to plan, develop, and implement automated PROM collection from remote devices back to NHS infrastructure. The challenges and solutions described in this paper will be pertinent to other digital health innovation researchers aspiring to deploy interoperable systems within NHS clinical systems.

Ensuring patient and public involvement in the transition to AI-assisted mental health care: A systematic scoping review and agenda for design justice

Background

Machine‐learning algorithms and big data analytics, popularly known as ‘artificial intelligence’ (AI), are being developed and taken up globally. Patient and public involvement (PPI) in the transition to AI‐assisted health care is essential for design justice based on diverse patient needs.

Objective

To inform the future development of PPI in AI‐assisted health care by exploring public engagement in the conceptualization, design, development, testing, implementation, use and evaluation of AI technologies for mental health.

Methods

Systematic scoping review drawing on design justice principles, and (i) structured searches of Web of Science (all databases) and Ovid (MEDLINE, PsycINFO, Global Health and Embase); (ii) handsearching (reference and citation tracking); (iii) grey literature; and (iv) inductive thematic analysis, tested at a workshop with health researchers.

Results

The review identified 144 articles that met inclusion criteria. Three main themes reflect the challenges and opportunities associated with PPI in AI‐assisted mental health care: (a) applications of AI technologies in mental health care; (b) ethics of public engagement in AI‐assisted care; and (c) public engagement in the planning, development, implementation, evaluation and diffusion of AI technologies.

Conclusion

The new data‐rich health landscape creates multiple ethical issues and opportunities for the development of PPI in relation to AI technologies. Further research is needed to understand effective modes of public engagement in the context of AI technologies, to examine pressing ethical and safety issues and to develop new methods of PPI at every stage, from concept design to the final review of technology in practice. Principles of design justice can guide this agenda.

Designing the Future of Children's Mental Health Services

Advancements in evidence-based psychosocial interventions, digital technologies, and implementation strategies (i.e., health services research products) for youth mental health services have yet to yield significant improvement in public health outcomes. Achieving such impact will require that these research products are easy to use, useful, and contextually appropriate. This paper describes how human-centered design (HCD), an approach that aligns product development with the needs of the people and settings that use those products, can be leveraged to improve youth mental health services. We articulate how HCD can advance accessibility, effectiveness, and equity, with specific consideration of unique aspects of youth mental health services.

Multidisciplinary Software Design for the Routine Monitoring and Assessment of Pain in Palliative Care Services: The Development of PainCheck

PURPOSE

The use of health information technology (HIT) to support patient and health professional communication is emerging as a core component of modern cancer care. Approaches to HIT development for cancer care are often underreported, despite their implementation in complex, multidisciplinary environments, typically supporting patients with multifaceted needs. We describe the development and evaluation of an e-health tool for pain management in patients with advanced cancer, arising from collaboration between health researchers and a commercial software development company.

METHODS

We adopted a research-led development process, involving patients with advanced cancer and their health professionals, focusing on use within real clinical settings. A software development approach (disciplined agile delivery) was combined with health science research methods (ie, diary studies, face-to-face interviews, questionnaires, prototyping, think aloud, process reviews, and pilots). Three software iterations were managed through three disciplined agile delivery phases to develop PainCheck and prepare it for use in a clinical trial.

RESULTS

Findings from development phases (inception, elaboration, and construction) informed the design and implementation of PainCheck. During the transition phase, where PainCheck was evaluated in a randomized clinical trial, there was variation in the extent of engagement by patients and health professionals. Prior personal experience and confidence with HIT led to a gatekeeping effect among health professionals, who were reluctant to introduce PainCheck to patients. Patients who did use PainCheck seemed to benefit, and no usability issues were reported.

CONCLUSION

Health science research methods seemed to help in the development of PainCheck, although a more rigorous application of implementation science methodologies might help to elucidate further the barriers and facilitators to adoption and inform an evidence-based plan for future implementation.

Automated analysis of free-text comments and dashboard representations in patient experience surveys: a multimethod co-design study

Background

Patient experience surveys (PESs) often include informative free-text comments, but with no way of systematically, efficiently and usefully analysing and reporting these. The National Cancer Patient Experience Survey (CPES), used to model the approach reported here, generates > 70,000 free-text comments annually.

Main aim

To improve the use and usefulness of PES free-text comments in driving health service changes that improve the patient experience.

Secondary aims

(1) To structure CPES free-text comments using rule-based information retrieval (IR) (‘text engineering’), drawing on health-care domain-specific gazetteers of terms, with in-built transferability to other surveys and conditions; (2) to display the results usefully for health-care professionals, in a digital toolkit dashboard display that drills down to the original free text; (3) to explore the usefulness of interdisciplinary mixed stakeholder co-design and consensus-forming approaches in technology development, ensuring that outputs have meaning for all; and (4) to explore the usefulness of Normalisation Process Theory (NPT) in structuring outputs for implementation and sustainability.

Design

A scoping review, rapid review and surveys with stakeholders in health care (patients, carers, health-care providers, commissioners, policy-makers and charities) explored clinical dashboard design/patient experience themes. The findings informed the rules for the draft rule-based IR [developed using half of the 2013 Wales CPES (WCPES) data set] and prototype toolkit dashboards summarising PES data. These were refined following mixed stakeholder, concept-mapping workshops and interviews, which were structured to enable consensus-forming ‘co-design’ work. IR validation used the second half of the WCPES, with comparison against its manual analysis; transferability was tested using further health-care data sets. A discrete choice experiment (DCE) explored which toolkit features were preferred by health-care professionals, with a simple cost–benefit analysis. Structured walk-throughs with NHS managers in Wessex, London and Leeds explored usability and general implementation into practice.

Key outcomes

A taxonomy of ranked PES themes, a checklist of key features recommended for digital clinical toolkits, rule-based IR validation and transferability scores, usability, and goal-oriented, cost–benefit and marketability results. The secondary outputs were a survey, scoping and rapid review findings, and concordance and discordance between stakeholders and methods.

Results

(1) The surveys, rapid review and workshops showed that stakeholders differed in their understandings of the patient experience and priorities for change, but that they reached consensus on a shortlist of 19 themes; six were considered to be core; (2) the scoping review and one survey explored the clinical toolkit design, emphasising that such toolkits should be quick and easy to use, and embedded in workflows; the workshop discussions, the DCE and the walk-throughs confirmed this and foregrounded other features to form the toolkit design checklist; and (3) the rule-based IR, developed using noun and verb phrases and lookup gazetteers, was 86% accurate on the WCPES, but needs modification to improve this and to be accurate with other data sets. The DCE and the walk-through suggest that the toolkit would be well accepted, with a favourable cost–benefit ratio, if implemented into practice with appropriate infrastructure support.

Limitations

Small participant numbers and sampling bias across component studies. The scoping review studies mostly used top-down approaches and focused on professional dashboards. The rapid review of themes had limited scope, with no second reviewer. The IR needs further refinement, especially for transferability. New governance restrictions further limit immediate use.

Conclusions

Using a multidisciplinary, mixed stakeholder, use of co-design, proof of concept was shown for an automated display of patient experience free-text comments in a way that could drive health-care improvements in real time. The approach is easily modified for transferable application.

Future work

Further exploration is needed of implementation into practice, transferable uses and technology development co-design approaches.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

Multilevel Predictors of Case Managers' Assessment Administration Behavior in a Precursor to a Measurement Feedback System

This study examines factors associated with administration of a treatment monitoring assessment measure (the Ohio Scales) in the context of a precursor to a measurement feedback system in a youth public mental health setting. 82% of all state case managers (N = 46) completed interviews and administered at least one Ohio Scale over a 12-month period. A multi-level model accounting for variance between both case managers and their administrative offices indicated that case manager characteristics (lower self-reported burnout, more experience) and monthly caseload characteristics (fewer active cases, younger average age of youth, and increased time since initial administration) predicted increased monthly administration proportions.

Impact of a Web-Based Electronic Health Record on Behavioral Health Service Delivery for Children and Adolescents: Randomized Controlled Trial

Background

Electronic health records (EHRs) have been widely proposed as a mechanism for improving health care quality. However, rigorous research on the impact of EHR systems on behavioral health service delivery is scant, especially for children and adolescents.

Objective

The current study evaluated the usability of an EHR developed to support the implementation of the Wraparound care coordination model for children and youth with complex behavioral health needs, and impact of the EHR on service processes, fidelity, and proximal outcomes.

Methods

Thirty-four Wraparound facilitators working in two programs in two states were randomized to either use the new EHR (19/34, 56%) or to continue to implement Wraparound services as usual (SAU) using paper-based documentation (15/34, 44%). Key functions of the EHR included standard fields such as youth and family information, diagnoses, assessment data, and progress notes. In addition, there was the maintenance of a coordinated plan of care, progress measurement on strategies and services, communication among team members, and reporting on services, expenditures, and outcomes. All children and youth referred to services for eight months (N=211) were eligible for the study. After excluding those who were ineligible (69/211, 33%) and who declined to participate (59/211, 28%), a total of 83/211 (39%) children and youth were enrolled in the study with 49/211 (23%) in the EHR condition and 34/211 (16%) in the SAU condition. Facilitators serving these youth and families and their supervisors completed measures of EHR usability and appropriateness, supervision processes and activities, work satisfaction, and use of and attitudes toward standardized assessments. Data from facilitators were collected by web survey and, where necessary, by phone interviews. Parents and caregivers completed measures via phone interviews. Related to fidelity and quality of behavioral health care, including Wraparound team climate, working alliance with providers, fidelity to the Wraparound model, and satisfaction with services.

Results

EHR-assigned facilitators from both sites demonstrated the robust use of the system. Facilitators in the EHR group reported spending significantly more time reviewing client progress (P=.03) in supervision, and less time overall sending reminders to youth/families (P=.04). A trend toward less time on administrative tasks (P=.098) in supervision was also found. Facilitators in both groups reported significantly increased use of measurement-based care strategies overall, which may reflect cross-group contamination (given that randomization of staff to the EHR occurred within agencies and supervisors supervised both types of staff). Although not significant at P<.05, there was a trend (P=.10) toward caregivers in the EHR group reporting poorer shared agreement on tasks on the measure of working alliance with providers. No other significant between-group differences were found.

Conclusions

Results support the proposal that use of EHR systems can promote the use of client progress data and promote efficiency; however, there was little evidence of any impact (positive or negative) on overall service quality, fidelity, or client satisfaction. The field of children’s behavioral health services would benefit from additional research on EHR systems using designs that include larger sample sizes and longer follow-up periods.

Trial Registration

ClinicalTrials.gov NCT02421874; https://clinicaltrials.gov/ct2/show/NCT02421874 (Archived by WebCite at http://www.webcitation.org/6yyGPJ3NA)

Facilitators and Barriers of Implementing a Measurement Feedback System in Public Youth Mental Health

This study examines implementation facilitators and barriers of a statewide roll-out of a measurement feedback system (MFS) in a youth public mental health system. 76 % of all state care coordinators (N = 47) completed interviews, which were coded via content analysis until saturation. Facilitators (e.g., recognition of the MFS's clinical utility) and barriers (e.g., MFS's reliability and validity) emerged paralleling the Exploration, Adoption/Preparation, Implementation, and Sustainment framework outlined by Aarons et al. (Adm Policy Mental Health Mental Health Serv Res, 38:4-23, 2011). Sustainment efforts may leverage innovation fit, individual adopter, and system related facilitators.

Feasibility and acceptability of a toolkit to facilitate clinician use of exposure therapy for youth

Exposure therapy is a key element of cognitive-behavioral therapy for youth with anxiety. However, few clinicians outside of specialty anxiety clinics routinely use exposures with anxious youth. This study reports on the acceptability and feasibility of a clinician support toolkit for exposures with anxious youth in community settings. The toolkit was developed through an extensive literature review of exposure science and cited barriers to exposure therapy, with input from multiple experts in pediatric anxiety (n = 10). Acceptability and feasibility data was gathered via survey from 70 community clinicians attending a training on exposure therapy for youth. Qualitative data was collected from a subset of participants who used the toolkit with youth on their caseload for approximately one month. Survey data suggested that the toolkit was viewed highly favorably. Qualitative interviews indicated that the toolkit was viewed as a feasible therapeutic tool that positively impacted clinician motivation to use exposure. Primary challenges related to exposure use more generally, and included client and service setting barriers. Results suggested initial toolkit acceptability and feasibility. Future work to refine the toolkit and test its efficacy as an implementation strategy to increase clinician use of exposure therapy is warranted. Next steps include development of additional content in partnership with key stakeholders, and testing the toolkit's impact on increasing clinician use of exposure therapy.

Digital technology and clinical decision making in depression treatment: Current findings and future opportunities

Clinical decision making encompasses a broad set of processes that contribute to the effectiveness of depression treatments. There is emerging interest in using digital technologies to support effective and efficient clinical decision making. In this paper, we provide "snapshots" of research and current directions on ways that digital technologies can support clinical decision making in depression treatment. Practical facets of clinical decision making are reviewed, then research, design, and implementation opportunities where technology can potentially enhance clinical decision making are outlined. Discussions of these opportunities are organized around three established movements designed to enhance clinical decision making for depression treatment, including measurement-based care, integrated care, and personalized medicine. Research, design, and implementation efforts may support clinical decision making for depression by (1) improving tools to incorporate depression symptom data into existing electronic health record systems, (2) enhancing measurement of treatment fidelity and treatment processes, (3) harnessing smartphone and biosensor data to inform clinical decision making, (4) enhancing tools that support communication and care coordination between patients and providers and within provider teams, and (5) leveraging treatment and outcome data from electronic health record systems to support personalized depression treatment. The current climate of rapid changes in both healthcare and digital technologies facilitates an urgent need for research, design, and implementation of digital technologies that explicitly support clinical decision making. Ensuring that such tools are efficient, effective, and usable in frontline treatment settings will be essential for their success and will require engagement of stakeholders from multiple domains.

Capabilities and Characteristics of Digital Measurement Feedback Systems: Results from a Comprehensive Review

Measurement feedback systems (MFS) are a class of health information technology (HIT) that function as an implementation support strategy for integrating measurement based care or routine outcome monitoring into clinical practice. Although many MFS have been developed, little is known about their functions. This paper reports findings from an application of health information technology-academic and commercial evaluation (HIT-ACE), a systematic and consolidated evaluation method, to MFS designed for use in behavioral healthcare settings. Forty-nine MFS were identified and subjected to systematic characteristic and capability coding. Results are presented with respect to the representation of characteristics and capabilities across MFS.

Exploring Mental Health Providers' Interest in Using Web and Mobile-Based Tools in their Practices

A growing number of Internet sites and mobile applications are being developed intended for use in clinical practice. However, during the development process (e.g., creating features and determining use cases), the needs and interests of providers are often overlooked. We explored providers' interests using a mixed-methods approach incorporating both qualitative and quantitative research methods. A first study used an interview approach to identify the challenges providers faced, tools they used, and any use of computers and apps specifically. Fifteen providers from both the United States and Canada completed the interview and recordings were transcribed and analyzed using a constructivist grounded theory approach. Four primary themes were identified including challenges, potential tools, access and usability. A second study used a brief survey completed by 132 providers at a large healthcare system to explore current use of and potential interest in Internet and mobile technologies. Although many providers (80.9%) reported recommending some form of technology to patients, this was mostly Internet websites that were predominantly informational/psychoeducational in nature. Overall, these studies combine to suggest a strong interest in websites and apps for use in clinical settings while highlighting potential areas (ease of use, patient security and privacy) that should be considered in the design and deployment of these tools.

Challenges in the Implementation of Measurement Feedback Systems

This commentary on the articles published in the special section on the development and implementation of measurement feedback systems (MFSs) discusses three challenging themes in the process of MFS implementation: design and planning, organizational context, and sustainability and unintended consequences. It is argued that the implementation of MFSs is complex, but is an important step in improving outcomes in routine care for children and young persons.