Health Information Technologies-Academic and Commercial Evaluation (HIT-ACE) methodology: description and application to clinical feedback systems

Implementation of the Wolverine Mental Health Program, Part 2: Implementation Phase

To enhance mental health care for youth in a midwestern residential treatment facility, Wolverine Human Services partnered with the Beck Institute (an intermediary) and an implementation research team to implement cognitive-behavioral therapy (CBT). CBT has strong evidence supporting effectiveness for treating youth internalizing and externalizing problems, but it is a complex psychosocial intervention that demands a thoughtful implementation approach. This study outlines the implementation phase (2.5 years) of a 5-year collaborative effort. The implementation phase focused on (a) adapting CBT to fit the complex youth needs and the roles of the multidisciplinary team members resulting in a new comprehensive and coordinated care model, and (b) the strategies utilized to support its competent integration by all team members. Six blended implementation strategies were deployed in this phase: forging implementation teams, installing progress monitoring, adapting CBT, training, providing supervision and consultation, and training the trainers. A components-based approach to CBT yielded six core skills: active listening, problem solving, mood monitoring and intervention mapping, activity scheduling, distress tolerance, and cognitive restructuring. By the end of this phase, all staff had robust exposure to and experience with the adapted form of CBT. The work of our academic-community partnership has both research and clinical implications, with respect to integrating an adapted version of CBT for residential environments (CBT-RE).

Middle and High School Student Perspectives on Digitally-Delivered Mental Health Assessments and Measurement Feedback Systems

Implementation of measurement-based care (MBC) by child-serving community mental health providers, particularly school-based providers, is low. To inform user-centered design of measurement feedback systems (MFSs) and MBC implementation more broadly, semi-structured interviews were conducted with 61 middle and high school students. Interviews explored student preferences for and perceived helpfulness of different assessment methods and use of MFS in counseling. Results indicate that student preference for digitally-administered assessment is equivocal, with preferences being influenced by student perceptions of the ease of use, impersonalization, and confidentiality. Students with exposure to the MFS found it helpful when used by their provider to share assessment feedback.

Improving measurement feedback systems for measurement-based care

Abstract Objectives: Measurement feedback systems (MFS) are a class of health information technologies developed to facilitate measurement-based care. The individual clinical decision support features within MFS are diverse and their influence on clinicians is largely unknown. This study tested the impact of MFS features on clinicians' progress assessments and treatment decisions in different scenarios. Method: Clinicians (n = 299) were randomly assigned to view one of six combinations of the following MFS features: graph, expected change trajectory line, alert, and treatment suggestions. The assigned feature combination was paired with three vignettes and clinical data representing three clinical scenarios: patient deterioration, no progress, and approaching remission. Clinicians answered questions after each vignette, and at the conclusion. Results: MFS features differentially impacted clinicians' progress assessment accuracy, their likelihood of making a treatment change, and their treatment choices. Which feature was most impactful varied depending on the clinical scenario. Clinicians reported graphs influenced their assessments and choices significantly more than the other features, and the majority stated they would prefer to use all of the features. Conclusions: Specific MFS features impact clinicians' assessments and choices to greater degrees, and the impact of those features can be influenced by the clinical state of the patient.

Designing the Future of Children's Mental Health Services

Advancements in evidence-based psychosocial interventions, digital technologies, and implementation strategies (i.e., health services research products) for youth mental health services have yet to yield significant improvement in public health outcomes. Achieving such impact will require that these research products are easy to use, useful, and contextually appropriate. This paper describes how human-centered design (HCD), an approach that aligns product development with the needs of the people and settings that use those products, can be leveraged to improve youth mental health services. We articulate how HCD can advance accessibility, effectiveness, and equity, with specific consideration of unique aspects of youth mental health services.

Adapting a Compilation of Implementation Strategies to Advance School-Based Implementation Research and Practice

Schools, like other service sectors, are confronted with an implementation gap, with the slow adoption and uneven implementation of evidence-based practices (EBP) as part of routine service delivery, undermining efforts to promote better youth behavioral health outcomes. Implementation researchers have undertaken systematic efforts to publish taxonomies of implementation strategies (i.e., methods or techniques that are used to facilitate the uptake, use, and sustainment of EBP), such as the Expert Recommendations for Implementing Change (ERIC) Project. The 73-strategy ERIC compilation was developed in the context of healthcare and largely informed by research and practice experts who operate in that service sector. Thus, the comprehensibility, contextual appropriateness, and utility of the existing compilation to other service sectors, such as the educational setting, remain unknown. The purpose of this study was to initiate the School Implementation Strategies, Translating ERIC Resources (SISTER) Project to iteratively adapt the ERIC compilation to the educational sector. The results of a seven-step adaptation process resulted in 75 school-adapted strategies. Surface-level changes were made to the majority of the original ERIC strategies (52 out of 73), while five of the strategies required deeper modifications for adaptation to the school context. Six strategies were deleted and seven new strategies were added based on existing school-based research. The implications of this study's findings for prevention scientists engaged in implementation research (e.g., creating a common nomenclature for implementation strategies) and limitations are discussed.

Fuzzy-Based Symmetrical Multi-Criteria Decision-Making Procedure for Evaluating the Impact of Harmful Factors of Healthcare Information Security

Growing concern about healthcare information security in the wake of alarmingly rising cyber-attacks is being given symmetrical priority by current researchers and cyber security experts. Intruders are penetrating symmetrical mechanisms of healthcare information security continuously. In the same league, the paper presents an overview on the current situation of healthcare information and presents a layered model of healthcare information management in organizations. The paper also evaluates the various factors that have a key contribution in healthcare information security breaches through a hybrid fuzzy-based symmetrical methodology of AHP-TOPSIS. Furthermore, for assessing the effect of the calculated results, the authors have tested the results on local hospital software of Varanasi. Tested results of the factors are validated through the comparison and sensitivity analysis in this study. Tabulated results of the proposed study propose a symmetrical mechanism as the most conversant technique which can be employed by the experts and researchers for preparing security guidelines and strategies.

Advancing the state-level tracking of evidence-based practices: a case study

Background

Despite a sustained focus by policymakers and researchers on improving the standard of clinical care in public mental health services, the use of evidence-based practice remains low. Among other challenges, this reflects the difficulty of translating clinical research into useable policy that can be feasibly funded and monitored by state or large healthcare systems.

Case presentation

In this paper we present a case study of Washington State’s strategy for monitoring the use of clinical elements at the session level for all Medicaid-funded children’s mental health services. The implementation of this strategy reflects policy actions to promote effective practice while also actively influencing multiple other levels of the implementation ecology. The approach is informed by the Policy Ecology Framework, the Consolidated Framework for Implementation Research, the evidence-based policymaking literature, and common ontology and clinical elements models.

Conclusions

We found the strategy developed in Washington State to be a feasible method of collecting session level information about the use of effective clinical mental health practices. In addition, the approach appears to be having influence on multiple layers of the implementation ecology that could be explored through further study.

Mind the Gap: Considering Disparities in Implementing Measurement-Based Care

Synergistic advancements in evidence-based practice, implementation science, health care policy, and health information technology (HIT) have led to increasing efforts to broadly implement measurement-based care (MBC)-the systematic use of repeated outcome measures to monitor treatment progress and support clinical decision making¹-in psychiatric services. Much of this work has been done with adult populations, and more efforts are needed for MBC to gain traction in child and adolescent psychiatry. In this Letter to the Editor, we describe a quality improvement (QI) project that leveraged HIT to implement MBC in the child and adolescent psychiatry department of a regional pediatric tertiary-care center and report long-term (5-year) implementation outcomes (ie, adoption and penetration). Although a myriad of implementation strategies was used, here we focus on the most complex strategy-integrating a digital measurement-feedback system (MFS) into standard workflow. Then, we discuss pitfalls and lessons learned with special attention to potential unintended effects of QI efforts on existing health disparities.

Will E-Monitoring of Policy and Program Implementation Stifle or Enhance Practice? How Would We Know?

Electronic or digital monitoring systems could promote the visibility of health promotion and disease prevention programs by providing new tools to support the collection, analysis, and reporting of data. In clinical settings however, the benefits of e-monitoring of service delivery remain contested. While there are some examples of e-monitoring systems improving patient outcomes, the smooth introduction into clinical practice has not occurred. Expected efficiencies have not been realized. The restructuring of team work has been problematic. Most particularly, knowledge from research has not advanced sufficiently because the meaning of e-monitoring has not been well theorized in the first place. As enthusiasm for e-monitoring in health promotion grows, it behooves us to ensure that health promotion practice learns from these insights. We outline the history of program monitoring in health promotion and the development of large-scale e-monitoring systems to track policy and program delivery. We interrogate how these technologies can be understood, noticing how they inevitably elevate some parts of practice over others. We suggest that progress in e-monitoring research and development could benefit from the insights and methods of improvement science (the science that underpins how practitioners attempt to solve problems and promote quality) as conceptually distinct from implementation science (the science of getting particular evidence-based programs into practice). To fully appreciate whether e-monitoring of program implementation will act as an aid or barrier to health promotion practice we canvass a wide range of theoretical perspectives. We illustrate how different theories draw attention to different aspects of the role of e-monitoring, and its impact on practice.

Impact of a Web-Based Electronic Health Record on Behavioral Health Service Delivery for Children and Adolescents: Randomized Controlled Trial

BACKGROUND

Electronic health records (EHRs) have been widely proposed as a mechanism for improving health care quality. However, rigorous research on the impact of EHR systems on behavioral health service delivery is scant, especially for children and adolescents.

OBJECTIVE

The current study evaluated the usability of an EHR developed to support the implementation of the Wraparound care coordination model for children and youth with complex behavioral health needs, and impact of the EHR on service processes, fidelity, and proximal outcomes.

METHODS

Thirty-four Wraparound facilitators working in two programs in two states were randomized to either use the new EHR (19/34, 56%) or to continue to implement Wraparound services as usual (SAU) using paper-based documentation (15/34, 44%). Key functions of the EHR included standard fields such as youth and family information, diagnoses, assessment data, and progress notes. In addition, there was the maintenance of a coordinated plan of care, progress measurement on strategies and services, communication among team members, and reporting on services, expenditures, and outcomes. All children and youth referred to services for eight months (N=211) were eligible for the study. After excluding those who were ineligible (69/211, 33%) and who declined to participate (59/211, 28%), a total of 83/211 (39%) children and youth were enrolled in the study with 49/211 (23%) in the EHR condition and 34/211 (16%) in the SAU condition. Facilitators serving these youth and families and their supervisors completed measures of EHR usability and appropriateness, supervision processes and activities, work satisfaction, and use of and attitudes toward standardized assessments. Data from facilitators were collected by web survey and, where necessary, by phone interviews. Parents and caregivers completed measures via phone interviews. Related to fidelity and quality of behavioral health care, including Wraparound team climate, working alliance with providers, fidelity to the Wraparound model, and satisfaction with services.

RESULTS

EHR-assigned facilitators from both sites demonstrated the robust use of the system. Facilitators in the EHR group reported spending significantly more time reviewing client progress (P=.03) in supervision, and less time overall sending reminders to youth/families (P=.04). A trend toward less time on administrative tasks (P=.098) in supervision was also found. Facilitators in both groups reported significantly increased use of measurement-based care strategies overall, which may reflect cross-group contamination (given that randomization of staff to the EHR occurred within agencies and supervisors supervised both types of staff). Although not significant at P<.05, there was a trend (P=.10) toward caregivers in the EHR group reporting poorer shared agreement on tasks on the measure of working alliance with providers. No other significant between-group differences were found.

CONCLUSIONS

Results support the proposal that use of EHR systems can promote the use of client progress data and promote efficiency; however, there was little evidence of any impact (positive or negative) on overall service quality, fidelity, or client satisfaction. The field of children's behavioral health services would benefit from additional research on EHR systems using designs that include larger sample sizes and longer follow-up periods.

TRIAL REGISTRATION

ClinicalTrials.gov NCT02421874; https://clinicaltrials.gov/ct2/show/NCT02421874 (Archived by WebCite at http://www.webcitation.org/6yyGPJ3NA).

Dynamics behind the scale up of evidence-based obesity prevention: protocol for a multi-site case study of an electronic implementation monitoring system in health promotion practice

BACKGROUND

The effectiveness of many interventions to promote health and prevent disease has been well established. The imperative has therefore shifted from amassing evidence about efficacy to scale-up to maximise population-level health gains. Electronic implementation monitoring, or 'e-monitoring', systems have been designed to assist and track the delivery of preventive policies and programs. However, there is little evidence on whether e-monitoring systems improve the dissemination, adoption, and ongoing delivery of evidence-based preventive programs. Also, given considerable difficulties with e-monitoring systems in the clinical sector, scholars have called for a more sophisticated re-examination of e-monitoring's role in enhancing implementation.

METHODS

In the state of New South Wales (NSW), Australia, the Population Health Information Management System (PHIMS) was created to support the dissemination of obesity prevention programs to 6000 childcare centres and elementary schools across all 15 local health districts. We have established a three-way university-policymaker-practice research partnership to investigate the impact of PHIMS on practice, how PHIMS is used, and how achievement of key performance indicators of program adoption may be associated with local contextual factors. Our methods encompass ethnographic observation, key informant interviews and participatory workshops for data interpretation at a state and local level. We use an on-line social network analysis of the collaborative relationships across local health district health promotion teams to explore the relationship between PHIMS use and the organisational structure of practice.

DISCUSSION

Insights will be sensitised by institutional theory, practice theory and complex adaptive system thinking, among other theories which make sense of socio-technical action. Our working hypothesis is that the science of getting evidence-based programs into practice rests on an in-depth understanding of the role they play in the on-going system of local relationships and multiple accountabilities. Data will be synthesised to produce a typology to characterise local context, PHIMS use and key performance indicator achievement (of program implementation) across the 15 local health districts. Results could be used to continuously align e-monitoring technologies within quality improvement processes to ensure that such technologies enhance practice and innovation. A partnership approach to knowledge production increases the likelihood that findings will be put into practice.

Are We There Yet? Human Factors Knowledge and Health Information Technology - the Challenges of Implementation and Impact

Objective: To review the developments in human factors (HF) research on the challenges of health information technology (HIT) implementation and impact given the continuing incidence of usability problems and unintended consequences from HIT development and use. Methods: A search of PubMed/Medline and Web of Science® identified HF research published in 2015 and 2016. Electronic health records (EHRs) and patient-centred HIT emerged as significant foci of recent HF research. The authors selected prominent papers highlighting ongoing HF and usability challenges in these areas. This selective rather than systematic review of recent HF research highlights these key challenges and reflects on their implications on the future impact of HF research on HIT. Results: Research provides evidence of continued poor design, implementation, and usability of HIT, as well as technology-induced errors and unintended consequences. The paper highlights support for: (i) strengthening the evidence base on the benefits of HF approaches; (ii) improving knowledge translation in the implementation of HF approaches during HIT design, implementation, and evaluation; (iii) increasing transparency, governance, and enforcement of HF best practices at all stages of the HIT system development life cycle. Discussion and Conclusion: HF and usability approaches are yet to become embedded as integral components of HIT development, implementation, and impact assessment. As HIT becomes ever-more pervasive including with patients as end-users, there is a need to expand our conceptualisation of the problems to be addressed and the suite of tactics and strategies to be used to calibrate our pro-active involvement in its improvement.

Pediatric eMental healthcare technologies: a systematic review of implementation foci in research studies, and government and organizational documents

Background

Researchers, healthcare planners, and policymakers convey a sense of urgency in using eMental healthcare technologies to improve pediatric mental healthcare availability and access. Yet, different stakeholders may focus on different aspects of implementation. We conducted a systematic review to identify implementation foci in research studies and government/organizational documents for eMental healthcare technologies for pediatric mental healthcare.

Methods

A search of eleven electronic databases and grey literature was conducted. We included research studies and documents from organization and government websites if the focus included eMental healthcare technology for children/adolescents (0–18 years), and implementation was studied and reported (research studies) or goals/recommendations regarding implementation were made (documents). We assessed study quality using the Mixed Methods Appraisal Tool and document quality using the Appraisal of Guidelines for Research & Evaluation II. Implementation information was grouped according to Proctor and colleagues’ implementation outcomes—acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability—and grouped separately for studies and documents.

Results

Twenty research studies and nine government/organizational documents met eligibility criteria. These articles represented implementation of eMental healthcare technologies in the USA (14 studies), United Kingdom (2 documents, 3 studies), Canada (2 documents, 1 study), Australia (4 documents, 1 study), New Zealand (1 study), and the Netherlands (1 document). The quality of research studies was excellent (n = 11), good (n = 6), and poor (n = 1). These eMental health studies focused on the acceptability (70%, n = 14) and appropriateness (50%, n = 10) of eMental healthcare technologies to users and mental healthcare professionals. The quality of government and organizational documents was high (n = 2), medium (n = 6), and low (n = 1). These documents focused on cost (100%, n = 9), penetration (89%, n = 8), feasibility (78%, n = 7), and sustainability (67%, n = 6) of implementing eMental healthcare technology.

Conclusion

To date, research studies have largely focused on acceptability and appropriateness, while government/organizational documents state goals and recommendations regarding costs, feasibility, and sustainability of eMental healthcare technologies. These differences suggest that the research evidence available for pediatric eMental healthcare technologies does not reflect the focus of governments and organizations. Partnerships between researchers, healthcare planners, and policymakers may help to align implementation research with policy development, decision-making, and funding foci.

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-017-0608-6) contains supplementary material, which is available to authorized users.

Study of the uses of Information and Communication Technologies by Pain Treatment Unit Physicians

Adequate use of Information and Communication Technologies (ICTs) in health has been shown to save the patient and caregiver time, improve access to the health system, improve diagnosis and control of disease or treatment. All this results in cost savings, and more importantly, they help improve the quality of service and the lives of patients. The purpose of this study is to analyse the differences in the uses of this ICTs between those physicians that belong to Pain Treatment Units (PU) and other physicians that work in pain not linked to these PUs. An online survey, generated by Netquest online survey tool, was sent to both groups of professionals and the data collected was statistical analysed through a logistic regression methodology which is the Logit binomial model. Our results show that those physicians that belong to PUs use ICTs more frequently and consider it more relevant to their clinical practice.

Capabilities and Characteristics of Digital Measurement Feedback Systems: Results from a Comprehensive Review

Measurement feedback systems (MFS) are a class of health information technology (HIT) that function as an implementation support strategy for integrating measurement based care or routine outcome monitoring into clinical practice. Although many MFS have been developed, little is known about their functions. This paper reports findings from an application of health information technology-academic and commercial evaluation (HIT-ACE), a systematic and consolidated evaluation method, to MFS designed for use in behavioral healthcare settings. Forty-nine MFS were identified and subjected to systematic characteristic and capability coding. Results are presented with respect to the representation of characteristics and capabilities across MFS.