Technology-Enabled Solutions for Australian Mental Health Services Reform: Impact Evaluation

Barriers and facilitators to the implementation of digital technologies in mental health systems: a qualitative systematic review to inform a policy framework

BACKGROUND

Despite the potential for improved population mental health and wellbeing, the integration of mental health digital interventions has been difficult to achieve. In this qualitative systematic review, we aimed to identify barriers and facilitators to the implementation of digital technologies in mental healthcare systems, and map these to an implementation framework to inform policy development.

METHODS

We searched Medline, Embase, Scopus, PsycInfo, Web of Science, and Google Scholar for primary research articles published between January 2010 and 2022. Studies were considered eligible if they reported barriers and/or facilitators to the integration of any digital mental healthcare technologies. Data were extracted using EPPI-Reviewer Web and analysed thematically via inductive and deductive cycles.

RESULTS

Of 12,525 references identified initially, 81 studies were included in the final analysis. Barriers and facilitators were grouped within an implementation (evidence-practice gap) framework across six domains, organised by four levels of mental healthcare systems. Broadly, implementation was hindered by the perception of digital technologies as impersonal tools that add additional burden of care onto both providers and patients, and change relational power asymmetries; an absence of resources; and regulatory complexities that impede access to universal coverage. Facilitators included person-cantered approaches that consider patients' intersectional features e.g., gender, class, disability, illness severity; evidence-based training for providers; collaboration among colleagues; appropriate investment in human and financial resources; and policy reforms that tackle universal access to digital health.

CONCLUSION

It is important to consider the complex and interrelated nature of barriers across different domains and levels of the mental health system. To facilitate the equitable, sustainable, and long-term digital transition of mental health systems, policymakers should consider a systemic approach to collaboration between public and private sectors to inform evidence-based planning and strengthen mental health systems.

PROTOCOL REGISTRATION

The protocol is registered on PROSPERO, CRD42021276838.

What have been the clinical outcomes of the Project Synergy/InnoWell digital health platform?

Project Synergy is a digital mental health tool for assessment, referral and follow-up of people with mental health problems. The Australian federal government Department of Health entered an AUD33 million formal funding arrangement with InnoWell, a proprietary company vehicle (primarily the consultancy firm PwC and University of Sydney) to continue development of Project Synergy. This followed an initial federal National Health and Medical Research Council grant of AUD5.5 million over the previous 3 years. However, based on the assessment of peer-reviewed research data, the Project Synergy/InnoWell platform does not seem to have demonstrated clinical outcomes of healthcare value to date.

The Polarization of Clinician and Service Staff Perspectives After the Use of Health Information Technology in Youth Mental Health Services: Implementation and Evaluation Study

BACKGROUND

Highly personalized care is substantially improved by technology platforms that assess and track patient outcomes. However, evidence regarding how to successfully implement technology in real-world mental health settings is limited.

OBJECTIVE

This study aimed to naturalistically monitor how a health information technology (HIT) platform was used within 2 real-world mental health service settings to gain practical insights into how HIT can be implemented and sustained to improve mental health service delivery.

METHODS

An HIT (The Innowell Platform) was naturally implemented in 2 youth mental health services in Sydney, Australia. Web-based surveys (n=19) and implementation logs were used to investigate staff attitudes toward technology before and after implementation. Descriptive statistics were used to track staff attitudes over time, whereas qualitative thematic analysis was used to explore implementation log data to gain practical insights into useful implementation strategies in real-world settings.

RESULTS

After the implementation, the staff were nearly 3 times more likely to agree that the HIT would improve care for their clients (3/12, 25% agreed before the implementation compared with 7/10, 70% after the implementation). Despite this, there was also an increase in the number of staff who disagreed that the HIT would improve care (from 1/12, 8% to 2/10, 20%). There was also decreased uncertainty (from 6/12, 50% to 3/10, 30%) about the willingness of the service to implement the technology for its intended purpose, with similar increases in the number of staff who agreed and disagreed with this statement. Staff were more likely to be uncertain about whether colleagues in my service are receptive to changes in clinical processes (not sure rose from 5/12, 42% to 7/10, 70%). They were also more likely to report that their service already provides the best mental health care (agreement rose from 7/12, 58% to 8/10, 80%). After the implementation, a greater proportion of participants reported that the HIT enabled shared or collaborative decision-making with young people (2/10, 20%, compared with 1/12, 8%), enabled clients to proactively work on their mental health care through digital technologies (3/10, 30%, compared with 2/12, 16%), and improved their response to suicidal risk (4/10, 40% compared with 3/12, 25%).

CONCLUSIONS

This study raises important questions about why clinicians, who have the same training and support in using technology, develop more polarized opinions on its usefulness after implementation. It seems that the uptake of HIT is heavily influenced by a clinician's underlying beliefs and attitudes toward clinical practice in general as well as the role of technology, rather than their knowledge or the ease of use of the HIT in question.

Evaluation of the Use of Digital Mental Health Platforms and Interventions: Scoping Review

BACKGROUND

The increasing use of digital mental health (DMH) platforms and digital mental health interventions (DMHIs) is hindered by uncertainty over effectiveness, quality and usability. There is a need to identify the types of available evidence in this domain.

AIM

This study is a scoping review identifying evaluation of the (1) DMH platform/s used; and (2) DMHI/s applied on the DMH platform/s.

METHODS

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guided the review process. Empirical studies that focused on evaluation of the use and application of DMH platforms were included from journal articles (published 2012-2022). A literature search was conducted using four electronic databases (Scopus, ScienceDirect, Sage and ACM Digital Library) and two search engines (PubMed and Google Scholar).

RESULTS

A total of 6874 nonduplicate records were identified, of which 144 were analyzed and 22 met the inclusion criteria. The review included general/unspecified mental health and/or suicidality indications (n = 9, 40.9%), followed by depression (n = 5, 22.7%), psychosis (n = 3, 13.6%), anxiety and depression (n = 2, 9.1%), as well as anxiety, depression and suicidality (n = 1, 4.5%), loneliness (n = 1, 4.5%), and addiction (n = 1, 4.5%). There were 11 qualitative studies (50%), 8 quantitative studies (36.4%), and 3 mixed-methods studies (n = 3, 13.6%). The results contained 11 studies that evaluated the DMH platform/s and 11 studies that evaluated the DMHI/s. The studies focused on feasibility, usability, engagement, acceptability and effectiveness. There was a small amount of significant evidence (1 in each 11), notably the (cost-)effectiveness of a DMHI with significant long-term impact on anxiety and depression in adults.

CONCLUSION

The empirical research demonstrates the feasibility of DMH platforms and DMHIs. To date, there is mostly heterogeneous, preliminary evidence for their effectiveness, quality and usability. However, a scalable DMHI reported effectiveness in treating adults' anxiety and depression. The scope of effectiveness may be widened through targeted strategies, for example by engaging independent young people.

Informing the Future of Integrated Digital and Clinical Mental Health Care: Synthesis of the Outcomes From Project Synergy

BACKGROUND

Globally, there are fundamental shortcomings in mental health care systems, including restricted access, siloed services, interventions that are poorly matched to service users' needs, underuse of personal outcome monitoring to track progress, exclusion of family and carers, and suboptimal experiences of care. Health information technologies (HITs) hold great potential to improve these aspects that underpin the enhanced quality of mental health care.

OBJECTIVE

Project Synergy aimed to co-design, implement, and evaluate novel HITs, as exemplified by the InnoWell Platform, to work with standard health care organizations. The goals were to deliver improved outcomes for specific populations under focus and support organizations to enact significant system-level reforms.

METHODS

Participating health care organizations included the following: Open Arms-Veterans & Families Counselling (in Sydney and Lismore, New South Wales [NSW]); NSW North Coast headspace centers for youth (Port Macquarie, Coffs Harbour, Grafton, Lismore, and Tweed Heads); the Butterfly Foundation's National Helpline for eating disorders; Kildare Road Medical Centre for enhanced primary care; and Connect to Wellbeing North Coast NSW (administered by Neami National), for population-based intake and assessment. Service users, families and carers, health professionals, and administrators of services across Australia were actively engaged in the configuration of the InnoWell Platform to meet service needs, identify barriers to and facilitators of quality mental health care, and highlight potentially the best points in the service pathway to integrate the InnoWell Platform. The locally configured InnoWell Platform was then implemented within the respective services. A mixed methods approach, including surveys, semistructured interviews, and workshops, was used to evaluate the impact of the InnoWell Platform. A participatory systems modeling approach involving co-design with local stakeholders was also undertaken to simulate the likely impact of the platform in combination with other services being considered for implementation within the North Coast Primary Health Network to explore resulting impacts on mental health outcomes, including suicide prevention.

RESULTS

Despite overwhelming support for integrating digital health solutions into mental health service settings and promising impacts of the platform simulated under idealized implementation conditions, our results emphasized that successful implementation is dependent on health professional and service readiness for change, leadership at the local service level, the appropriateness and responsiveness of the technology for the target end users, and, critically, funding models being available to support implementation. The key places of interoperability of digital solutions and a willingness to use technology to coordinate health care system use were also highlighted.

CONCLUSIONS

Although the COVID-19 pandemic has resulted in the widespread acceptance of very basic digital health solutions, Project Synergy highlights the critical need to support equity of access to HITs, provide funding for digital infrastructure and digital mental health care, and actively promote the use of technology-enabled, coordinated systems of care.

Telehealth to the Rescue During COVID-19: A Convergent Mixed Methods Study Investigating Patients' Perception

Background: The onset of the pandemic necessitated abrupt transition to telehealth consultations. Although there is a few tools that gauge the patients' perception about their experiences, none of them are contextualized to an emergency in the Middle East and North Africa region. Accordingly, this study aims at developing and validating a tool to address this gap, and deploying it to assess the patients' perception of telehealth services during COVID-19 in Dubai, United Arab Emirates (UAE). Methods: A convergent mixed methods design was adapted. A random selection of 100 patients from Dubai, UAE were invited to participate. Qualitative and quantitative datasets were collected using a tailor-made survey. The qualitative data, collected through open-ended questions, was analyzed using multi-staged thematic analysis. As for the quantitative data, it captured the patients' extent of satisfaction, and was assessed using SPSS (with a series of descriptive and inferential analyses). The qualitative and quantitative findings were then merged via joint display analysis. Results: Out of the 100 patients that were randomly selected, 94 patients participated in this study. The reliability score of Cronbach's Alpha for the instrument was 98.9%. The percentage of the total average of satisfaction was 80.67%. The Principal Component Analysis showed that 88.1% of the variance can be explained by the instrument (p < 0.001). The qualitative data analysis expanded upon the quantitative findings enabling a better understanding of the patients' perception. Three themes, revolving around the quality of the patient telehealth experiences, surfaced: "Factors that worked to the benefit of the patients," "Factors that the patients were not in favor of," and "Opportunities for improvements as perceived by the patients." Discussion: This study introduced a novel patient satisfaction with telehealth consultation survey contextualized to the COVID-19 times in Dubai, UAE. The participants were quite satisfied with the quality of their experience, however they suggested areas for improvement. Regional healthcare decision-makers can leverage the identified advantages and opportunities for improvement of telehealth. This will enable making informed decisions regarding the continuity of telehealth irrespective of how matters unfold in relation to the pandemic. It will also better prepare the healthcare sector for potential resurgence(s) of COVID-19 and/or the occurrence of other similar emergencies.

Implementing a digital health model of care in Australian youth mental health services: protocol for impact evaluation

Background

The World Economic Forum has recently highlighted substantial problems in mental health service provision and called for the rapid deployment of smarter, digitally-enhanced health services as a means to facilitate effective care coordination and address issues of demand. In mental health, the biggest enabler of digital solutions is the implementation of an effective model of care that is facilitated by integrated health information technologies (HITs); the latter ensuring the solution is easily accessible, scalable and sustainable. The University of Sydney’s Brain and Mind Centre (BMC) has developed an innovative digital health solution – delivered through the Youth Mental Health and Technology Program – which incorporates two components: 1) a highly personalised and measurement-based (data-driven) model of youth mental health care; and 2) an industrial grade HIT registered on the Australian Register of Therapeutic Goods. This paper describes a research protocol to evaluate the impact of implementing the BMC’s digital health solution into youth mental health services (i.e. headspace - a highly accessible, youth-friendly integrated service that responds to the mental health, physical health, alcohol or other substance use, and vocational concerns of young people aged 12 to 25 years) within urban and regional areas of Australia.

Methods

The digital health solution will be implemented into participating headspace centres using a naturalistic research design. Quantitative and qualitative data will be collected from headspace health professionals, service managers and administrators, as well as from lead agency and local Primary Health Network (PHN) staff, via service audits, Implementation Officer logs, online surveys, and semi-structured interviews, at baseline and then three-monthly intervals over the course of 12 months.

Discussion

At the time of publication, six headspace centres had been recruited to this study and had commenced implementation and impact evaluation. The first results are expected to be submitted for publication in 2021. This study will focus on the impact of implementing a digital health solution at both a service and staff level, and will evaluate digital readiness of service and staff adoption; quality, usability and acceptability of the solution by staff; staff self-reported clinical competency; overall impact on headspace centres as well as their lead agencies and local PHNs; and social return on investment.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-06394-4.

Evaluating the quality and safety of health-related apps and e-tools: Adapting the Mobile App Rating Scale and developing a quality assurance protocol

BACKGROUND

Whilst apps and e-tools have tremendous potential as low-cost, scalable mental health intervention and prevention tools, it is essential that consumers and health professionals have a means by which to evaluate their quality and safety.

OBJECTIVE

This study aimed to: 1) adapt the original Mobile App Rating Scale (MARS) in order to be appropriate for the evaluation of both mobile phone applications as well as e-tools; 2) test the reliability of the revised scale; and 3) develop a quality assurance protocol for identifying and rating new apps and e-tools to determine appropriateness for use in clinical practice.

METHODS

The MARS was adapted to include items specific to health-related apps and e-tools, such as the availability of resources, strategies for self-management, and quality information. The 41 apps and e-tools in the standard youth configuration of the InnoWell Platform, a digital tool designed to support or enhance mental health service delivery, were independently rated by two expert raters using the A-MARS. Cronbach's alpha was used to calculate the internal consistency and interclass correlation coefficients were used to calculate interrater reliability.

RESULTS

The A-MARS was shown to be a reliable scale with acceptable to excellent internal consistency and moderate to excellent interrater reliability across the subscales. Given the ever-increasing number of health information technologies on the market, a protocol to identify and rate new apps and e-tools for potential clinical use is presented.

CONCLUSIONS

Whilst the A-MARS is a useful tool to guide health professionals as they explore available apps and e-tools for potential clinical use, the training, time, and skill required to use it effectively may be prohibitive. As such, health professionals and services are likely to benefit from including a digital navigator as part of the care team to assist in selecting and rating apps and e-tools, increasing the usability of the data, and technology troubleshooting. When selecting, evaluating and/or recommending apps and e-tools to consumers, it is important to consider: 1) the availability of explicit strategies to set, monitor and review SMART goals; 2) the accessibility of credible, user friendly information and resources from reputable sources; 3) evidence of effectiveness; and 4) interoperability with other health information technologies.

Technology-Enabled Reform in a Nontraditional Mental Health Service for Eating Disorders: Participatory Design Study

BACKGROUND

The recent Australian National Agenda for Eating Disorders highlights the role technology can play in improving accessibility and service development through web-based prevention, early access pathways, self-help, and recovery assistance. However, engagement with the eating disorders community to co-design, build, and evaluate these much-needed technology solutions through participatory design processes has been lacking and, until recently, underresourced.

OBJECTIVE

This study aims to customize and configure a technology solution for a nontraditional (web-based, phone, email) mental health service that provides support for eating disorders and body image issues through the use of participatory design processes.

METHODS

Participants were recruited chiefly through the Butterfly National Helpline 1800 ED HOPE (Butterfly's National Helpline), an Australian-wide helpline supporting anyone concerned by an eating disorder or body image issue. Participants included individuals with lived experience of eating disorders and body image issues, their supportive others (such as family, health professionals, support workers), and staff of the Butterfly Foundation. Participants took part in participatory design workshops, running up to four hours, which were held nationally in urban and regional locations. The workshop agenda followed an established process of discovery, evaluation, and prototyping. Workshop activities included open and prompted discussion, reviewing working prototypes, creating descriptive artifacts, and developing user journeys. Workshop artifacts were used in a knowledge translation process, which identified key learnings to inform user journeys, user personas, and the customization and configuration of the InnoWell Platform for Butterfly's National Helpline. Further, key themes were identified using thematic techniques and coded in NVivo 12 software.

RESULTS

Six participatory design workshops were held, of which 45 participants took part. Participants highlighted that there is a critical need to address some of the barriers to care, particularly in regional and rural areas. The workshops highlighted seven overarching qualitative themes: identified barriers to care within the current system; need for people to be able to access the right care anywhere, anytime; recommendations for the technological solution (ie, InnoWell Platform features and functionality); need for communication, coordination, and integration of a technological solution embedded in Butterfly's National Helpline; need to consider engagement and tone within the technological solution; identified challenges and areas to consider when implementing a technological solution in the Helpline; and potential outcomes of the technological solution embedded in the Helpline relating to system and service reform. Ultimately, this technology solution should ensure that the right care is provided to individuals the first time.

CONCLUSIONS

Our findings highlight the value of actively engaging stakeholders in participatory design processes for the customization and configuration of new technologies. End users can highlight the critical areas of need, which can be used as a catalyst for reform through the implementation of these technologies in nontraditional services.

Optimising the integration of technology-enabled solutions to enhance primary mental health care: a service mapping study

Background

Despite the widely acknowledged potential for health information technologies to improve the accessibility, quality and clinical safety of mental health care, implementation of such technologies in services is frequently unsuccessful due to varying consumer, health professional, and service-level factors. The objective of this co-design study was to use process mapping (i.e. service mapping) to illustrate the current consumer journey through primary mental health services, identify barriers to and facilitators of quality mental health care, and highlight potential points at which to integrate the technology-enabled solution to optimise the provision of care based on key service performance indicators.

Methods

Interactive, discussion-based workshops of up to six hours were conducted with representative stakeholders from each participating service, including health professionals, service managers and administrators from Open Arms – Veterans & Families Counselling Service (Sydney), a counselling service for veterans and their families, and five headspace centres in the North Coast Primary Health Network, primary youth mental health services. Service maps were drafted and refined in real time during the workshops. Through both group discussion and the use of post-it notes, participants worked together to evaluate performance indicators (e.g. safety) at each point in the consumer journey (e.g. intake) to indicate points of impact for the technology-enabled solution, reviewing and evaluating differing opinions in order to reach consensus.

Results

Participants (n=84 across participating services) created service maps illustrating the current consumer journey through the respective services and highlighting barriers to and facilitators of quality mental health care. By consensus, the technology-enabled solution as facilitated by the InnoWell Platform was noted to enable the early identification of risk, reduce or eliminate lengthy intake processes, enable routine outcome monitoring to revise treatment plans in relation to consumer response, and serve as a personal data record for consumers, driving person-centred, coordinated care.

Conclusions

Service mapping was shown to be an effective methodology to understand the consumer’s journey through a service and served to highlight how the co-designed technology-enabled solution can optimise service pathways to improve the accessibility, quality and clinical safety of care relative to key service performance indicators, facilitating the delivery of the right care.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-06069-0.