Toward an Objective Assessment of Implementation Processes for Innovations in Health Care: Psychometric Evaluation of the Normalization Measure Development (NoMAD) Questionnaire Among Mental Health Care Professionals

A pragmatic, stepped-wedge, hybrid type II trial of interoperable clinical decision support to improve venous thromboembolism prophylaxis for patients with traumatic brain injury

BACKGROUND

Venous thromboembolism (VTE) is a preventable medical condition which has substantial impact on patient morbidity, mortality, and disability. Unfortunately, adherence to the published best practices for VTE prevention, based on patient centered outcomes research (PCOR), is highly variable across U.S. hospitals, which represents a gap between current evidence and clinical practice leading to adverse patient outcomes. This gap is especially large in the case of traumatic brain injury (TBI), where reluctance to initiate VTE prevention due to concerns for potentially increasing the rates of intracranial bleeding drives poor rates of VTE prophylaxis. This is despite research which has shown early initiation of VTE prophylaxis to be safe in TBI without increased risk of delayed neurosurgical intervention or death. Clinical decision support (CDS) is an indispensable solution to close this practice gap; however, design and implementation barriers hinder CDS adoption and successful scaling across health systems. Clinical practice guidelines (CPGs) informed by PCOR evidence can be deployed using CDS systems to improve the evidence to practice gap. In the Scaling AcceptabLE cDs (SCALED) study, we will implement a VTE prevention CPG within an interoperable CDS system and evaluate both CPG effectiveness (improved clinical outcomes) and CDS implementation.

METHODS

The SCALED trial is a hybrid type 2 randomized stepped wedge effectiveness-implementation trial to scale the CDS across 4 heterogeneous healthcare systems. Trial outcomes will be assessed using the RE2-AIM planning and evaluation framework. Efforts will be made to ensure implementation consistency. Nonetheless, it is expected that CDS adoption will vary across each site. To assess these differences, we will evaluate implementation processes across trial sites using the Exploration, Preparation, Implementation, and Sustainment (EPIS) implementation framework (a determinant framework) using mixed-methods. Finally, it is critical that PCOR CPGs are maintained as evidence evolves. To date, an accepted process for evidence maintenance does not exist. We will pilot a "Living Guideline" process model for the VTE prevention CDS system.

DISCUSSION

The stepped wedge hybrid type 2 trial will provide evidence regarding the effectiveness of CDS based on the Berne-Norwood criteria for VTE prevention in patients with TBI. Additionally, it will provide evidence regarding a successful strategy to scale interoperable CDS systems across U.S. healthcare systems, advancing both the fields of implementation science and health informatics.

TRIAL REGISTRATION

Clinicaltrials.gov - NCT05628207. Prospectively registered 11/28/2022, https://classic.

CLINICALTRIALS

gov/ct2/show/NCT05628207 .

A deprescribing programme aimed to optimise blood glucose-lowering medication in older people with type 2 diabetes mellitus, the OMED2-study: the study protocol for a randomised controlled trial

BACKGROUND

Older patients with type 2 diabetes mellitus (T2D) have an increased risk of hypoglycaemic episodes when using sulphonylureas or insulin. In the Netherlands, guidelines exist for reducing glucose-lowering medication in older patients. However, evidence is lacking that a medication reduction in older patients can be safely pursued. Here, we will examine if promoting the deprescribing of insulin/sulphonylureas with a deprescribing programme (DPP) in general practice affects T2D-complications in older overtreated patients.

METHODS

We will perform a 1:1 cluster randomised controlled trial in 86 general practices in the Netherlands. The DPP will consist of education sessions with general practitioners and practice nurses about reducing glucose-lowering medication in older patients (≥ 70 years). Topics of the sessions include the necessity of deprescribing, tools to initiate deprescribing and strategies to discuss deprescribing with patients (shared decision making). The DPP further includes a support programme with practice visits. The study will employ a selection tool to identify possibly overtreated older patients from the electronic medical records of the general practitioner. Eligibility for enrolment in the study will be based on HbA1c targets indicated by the Dutch guidelines, which depend on age, diabetes duration, presence of frailty, and life expectancy. The control group will provide usual care. We aim to include 406 patients. The follow-up period will be 2 years. For the primary outcome, the effect of the DPP on T2D-complications will be assessed by counting the cumulative incidence of events related to under- and overtreatment in T2D as registered in the electronic medical records. We shall perform an intention-to-treat analysis and an analysis including only patients for whom deprescribing was initiated. The implementation of the DPP in general practice will be evaluated quantitatively and qualitatively using the Extended Normalisation Process Theory (ENPT) and the Reach, Efficacy - Adoption, Implementation and Maintenance (RE-AIM) model. Other secondary outcomes include quality of life, cognitive functioning, events related to overtreatment or undertreatment, biomarkers of health, amount of blood glucose-lowering medication prescriptions, and cost-effectiveness.

DISCUSSION

This study will provide insight into the safety and feasibility of a programme aimed at deprescribing sulphonylureas/insulin in older people with T2D who are treated in general practice.

TRIAL REGISTRATION

ISRCTN Registry, ISRCTN50008265 , registered 09 March, 2023.

How is tailored implementation undertaken using a self-guided toolkit? Qualitative study of the ItFits-toolkit in the ImpleMentAll project

BACKGROUND

The process of tailored implementation is ill-defined and under-explored. The ItFits-toolkit was developed and subsequently tested as a self-guided online platform to facilitate implementation of tailored strategies for internet-based cognitive behavioural therapy (iCBT) services. In ImpleMentAll, ItFits-toolkit had a small but positive effect on the primary outcome of iCBT normalisation. This paper investigates, from a qualitative perspective, how implementation teams developed and undertook tailored implementation using the toolkit within the trial.

METHODS

Implementation teams in thirteen sites from nine countries (Europe and Australia) used the ItFits-toolkit for six months minimum, consistent with the trial protocol. A qualitative process evaluation was conducted. Descriptive data regarding goals, barriers, strategies, and implementation plans collected within the toolkit informed qualitative data collection in real time. Qualitative data included remote longitudinal interviews (n = 55) with implementation team members (n = 30) and observations of support calls (n = 19) with study sites. Qualitative data were analysed thematically, using a team-based approach.

RESULTS

Implementation teams developed and executed tailored implementation projects across all steps in the toolkit process. Working in a structured way but with room for flexibility, decisions were shaped by team members' ideas and goals, iterative stakeholder engagement, internal and external influences, and the context of the ImpleMentAll project. Although teams reported some positive impacts of their projects, 'time', both for undertaking the work, and for seeing project impacts, was described as a key factor in decisions about implementation strategies and assessments of success.

CONCLUSION

This study responds directly to McHugh et al.'s (2022) call for empirical description of what implementation tailoring looks like in action, in service settings. Self-guided facilitation of tailored implementation enables implementers in service settings to undertake tailoring within their organisations. Implementation tailoring takes considerable time and involves detailed work but can be supported through the provision of implementation science informed guidance and materials, iterative and ongoing stakeholder engagement, and working reflectively in response to external influencing factors. Directions for advancement of tailored implementation are suggested.

Development of an implementation plan for a school-based multimodal approach for depression and suicide prevention in adolescents

Strong Teens and Resilient Minds (STORM) is a multimodal, school-based approach for depression and suicide prevention in adolescents that is currently implemented in a region in the Netherlands. The STORM approach will be implemented in new regions in the coming years. This study used the implementation mapping protocol to report on the development of the STORM implementation plan. First, a needs assessment was conducted through semi-structured interviews with stakeholders and brainstorming sessions with regional programme leaders in the two regions that started implementing STORM in 2023. This led to the identification of six main barriers to implementation: high level of demands for schools, insufficient understanding of the programme content, insufficient network collaboration, no perceived relative advantage of STORM by stakeholders, lack of attention to sustainability, and high work pressure. Second, performance and change objectives were formulated based on these barriers. For example, a performance objective for potential providers was that they felt supported by STORM. Third, implementation strategies were selected from theory and translated into practical applications through brainstorming sessions with programme leaders. The following strategies were included in the implementation plan: collaborate with similar initiatives within the region, free up time for STORM tasks, tailor strategies, identify and prepare STORM champions, and promote network weaving. Last, a plan to evaluate the implementation of STORM and the application of the STORM implementation plan was formulated. Planned evaluation research will provide more insight into the usefulness and impact of the STORM implementation plan.

Geriatric Assessment in CKD Care: An Implementation Study

Rationale & Objective

Older people with progressive chronic kidney disease (CKD) have complex health care needs. Geriatric evaluation preceding decision making for kidney replacement is recommended in guidelines, but implementation is lacking in routine care. We aimed to evaluate implementation of geriatric assessment in CKD care.

Study Design

Mixed methods implementation study.

Setting & Participants

Dutch nephrology centers were approached for implementation of geriatric assessment in patients aged ≥70 years and with an estimated glomerular filtration rate of ≤20 mL/min/1.73 m2.

Quality Improvement Activities/Exposure

We implemented a consensus-based nephrology-tailored geriatric assessment: a patient questionnaire and professionally administered test set comprising 16 instruments covering functional, cognitive, psychosocial, and somatic domains and patient-reported outcome measures.

Outcomes

We aimed for implementation in 10 centers and 200 patients. Implementation was evaluated by (i) perceived enablers and barriers of implementation, including integration in work routines (Normalization Measure Development Tool) and (ii) relevance of the instruments to routine care for the target population.

Analytical Approach

Variations in implementation practices were described based on field notes. The postimplementation survey among health care professionals was analyzed descriptively, using an explanatory qualitative approach for open-ended questions.

Results

Geriatric assessment was implemented in 10 centers among 191 patients. Survey respondents (n = 71, 88% response rate) identified determinants that facilitated implementation, ie, multidisciplinary collaboration (with geriatricians) -meetings and reports and execution of assessments by nurses. Barriers to implementation were patient illiteracy or language barrier, time constraints, and patient burden. Professionals considered geriatric assessment sufficiently integrated into work routines (mean, 6.7/10 ± 2.0 [SD]) but also subject to improvement. Likewise, the relevance of geriatric assessment for routine care was scored as 7.8/10 ± 1.2. The Clinical Frailty Score and Montreal Cognitive Assessment were perceived as the most relevant instruments.

Limitations

Selection bias of interventions' early adopters may limit generalizability.

Conclusions

Geriatric assessment could successfully be integrated in CKD care and was perceived relevant to health care professionals.

Psychometric properties of two implementation measures: Normalization MeAsure Development questionnaire (NoMAD) and organizational readiness for implementing change (ORIC)

Background

Effective interventions need to be implemented successfully to achieve impact. Two theory-based measures exist for measuring the effectiveness of implementation strategies and monitor implementation progress. The Normalization MeAsure Development questionnaire (NoMAD) explores the four core concepts (Coherence, Cognitive Participation, Collective Action, Reflexive Monitoring) of the Normalization Process Theory. The Organizational Readiness for Implementing Change (ORIC) is based on the theory of Organizational Readiness for Change, measuring organization members' psychological and behavioral preparedness for implementing a change. We examined the measurement properties of the NoMAD and ORIC in a multi-national implementation effectiveness study.

Method

Twelve mental health organizations in nine countries implemented Internet-based cognitive behavioral therapy (iCBT) for common mental disorders. Staff involved in iCBT service delivery (n = 318) participated in the study. Both measures were translated into eight languages using a standardized forward-backward translation procedure. Correlations between measures and subscales were estimated to examine convergent validity. The theoretical factor structures of the scales were tested using confirmatory factor analysis (CFA). Test-retest reliability was based on the correlation between scores at two time points 3 months apart. Internal consistency was assessed using Cronbach's alpha. Floor and ceiling effects were quantified using the proportion of zero and maximum scores.

Results

NoMAD and ORIC measure related but distinct latent constructs. The CFA showed that the use of a total score for each measure is appropriate. The theoretical subscales of the NoMAD had adequate internal consistency. The total scale had high internal consistency. The total ORIC scale and subscales demonstrated high internal consistency. Test-retest reliability was suboptimal for both measures and floor and ceiling effects were absent.

Conclusions

This study confirmed the psychometric properties of the NoMAD and ORIC in multi-national mental health care settings. While measuring on different but related aspects of implementation processes, the NoMAD and ORIC prove to be valid and reliable across different language settings.

Validation of the German Normalisation Process Theory Measure G-NoMAD: translation, adaptation, and pilot testing

BACKGROUND

Implementing evidence-based healthcare practices (EBPs) is a complex endeavour and often lags behind research-informed decision processes. Understanding and systematically improving implementation using implementation theory can help bridge the gap between research findings and practice. This study aims to translate, pilot, and validate a German version of the English NoMAD questionnaire (G-NoMAD), an instrument derived from the Normalisation Process Theory, to explore the implementation of EBPs.

METHODS

Survey data has been collected in four German research projects and subsequently combined into a validation data set. Two versions of the G-NoMAD existed, independently translated from the original English version by two research groups. A measurement invariance analysis was conducted, comparing latent scale structures between groups of respondents to both versions. After determining the baseline model, the questionnaire was tested for different degrees of invariance (configural, metric, scalar, and uniqueness) across samples. A confirmatory factor analysis for three models (a four-factor, a unidimensional, and a hierarchical model) was used to examine the theoretical structure of the G-NoMAD. Finally, psychometric results were discussed in a consensus meeting, and the final instructions, items, and scale format were consented to.

RESULTS

A total of 539 health care professionals completed the questionnaire. The results of the measurement invariance analysis showed configural, partial metric, and partial scalar invariance indicating that the questionnaire versions are comparable. Internal consistency ranged from acceptable to good (0.79 ≤ α ≤ 0.85) per subscale. Both the four factor and the hierarchical model achieved a better fit than the unidimensional model, with indices from acceptable (SRMR = 0.08) to good (CFI = 0.97; TLI = 0.96). However, the RMSEA values were only close to acceptable (four-factor model: χ2164 = 1029.84, RMSEA = 0.10; hierarchical model: χ2166 = 1073.43, RMSEA = 0.10).

CONCLUSIONS

The G-NoMAD provides a reliable and promising tool to measure the degree of normalisation among individuals involved in implementation activities. Since the fit was similar in the four-factor and the hierarchical model, priority should be given to the practical relevance of the hierarchical model, including a total score and four subscale scores. The findings of this study support the further usage of the G-NoMAD in German implementation settings.

TRIAL REGISTRATION

Both the AdAM project (No. NCT03430336, 06/02/2018) and the EU-project ImpleMentAll (No. NCT03652883, 29/08/2018) were registered on ClinicalTrials.gov. The ImplementIT study was registered at the German Clinical Trial Registration (No. DRKS00017078, 18/04/2019). The G-NoMAD validation study was registered at the Open Science Framework (No7u9ab, 17/04/2023).

The National Coordinated Citrien eHealth Program to Scale Up Telemonitoring: Protocol for a Before-and-After Evaluation Study

BACKGROUND

Sustainable implementation of telemonitoring in health care is challenging, especially if one aims to scale up telemonitoring initiatives nationwide. The National collaborative eHealth program in the Netherlands is supporting the nationwide upscaling of telemonitoring in 3 clinical domains by implementing telemonitoring in all Dutch university medical centers (UMCs). The chosen telemonitoring concepts are (1) telemonitoring solutions in the domain of cardiology, (2) telemonitoring solutions providing care from a distance in obstetrics, and (3) telemonitoring solutions monitoring vital functions in hospital wards.

OBJECTIVE

The aim of this study is to evaluate the upscaling of telemonitoring in Dutch university hospitals in order to gain a better knowledge of the process, methods, and outcomes of nationwide upscaling strategies. Our hypothesis is that by the completion of the Citrien program's scale-up, telemonitoring will be operational in all UMCs but not normalized in routine care.

METHODS

A before-and-after study will be conducted to assess upscaling. The theoretical frameworks used are the framework for nonadoption, abandonment, scale-up, spread, and sustainability; the Normalization Process Theory; and a project management tool Project Canvas. The primary outcome of the study is the degree of normalization to which health care providers at UMCs consider telemonitoring a part of their routine practice, measured using the Normalization MeAsurement Development tool (NoMAD). Our secondary outcome is the uptake of telemonitoring at the Dutch UMCs, using management data from UMCs' business intelligence systems query.

RESULTS

Data will be collected between May 2020 and December 2022. Results were retrieved in June 2023. UMCs' business intelligence systems are queried for data for the secondary outcome measures. There is a risk that the UMCs will not be able to provide this management information. The laws and regulations governing telemonitoring in the Netherlands are changing, with the Electronic Data Exchange in Health Care Act (Wet elektronische gegevensuitwisseling in de zorg) and the European Health Data Space Act expected to positively influence implementation and upscaling.

CONCLUSIONS

The Citrien program is a nationally coordinated change management program that is scaling up telemonitoring across contexts and settings. This study will produce original data on the uptake and upscaling of telemonitoring at Dutch UMCs. Future initiatives to implement eHealth in the health care sector may be guided by the wide range of success factors, obstacles, and experiences collected through this program. The network itself may be of great value impacting future acceleration of eHealth initiatives.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/45201.

Upscaling telemonitoring in Dutch University Medical Centres: A baseline measurement

INTRODUCTION AND OBJECTIVE

The Dutch university medical centres (UMC's) are on the forefront when it comes to validation, implementation and research of telemonitoring. To aid the UMC's in their effort, the Dutch Government has supported the UMC's by fostering the 'Citrien eHealth program'. This program aims at nationwide implementation and upscaling of telemonitoring via a collaborative network. To quantify the success of this program, this study aims to provide insights into the current adoption of telemonitoring by health care professionals (HCP) within Dutch UMC's.

METHODS

Based on the evaluation framework as adapted from the Normalization Process Theory (NPT) a cross-sectional study was conducted in all Dutch UMC's. Thirty healthcare professionals (HCPs) per UMC were invited to complete the 23-item Normalization MeAsure Development (NoMAD) questionnaire, a tool to assess the degree of normalisation of telemonitoring.

RESULTS

The over-all response rate was 52.4% (124/240). Over 80% of respondents agreed or strongly agreed that they understand how telemonitoring affects the nature of their work, with a mean score of 1.49 (N = 117, SD 0.74). HCPs reported to believe telemonitoring will become a normal part of their work in the near future (N = 124, mean = 8.67, SD = 1.38). Using the Wilcoxon signed-rank test, the difference between current practise and future use of telemonitoring predicts to be statistically significant (Z =  - 7.505, p ≤ 0.001). Mean scores for appropriate training and sufficient resources are relatively low (2.39 and 2.70 respectively), indicating a barrier for collective action.

CONCLUSION

This is the first study to assess the implementation of telemonitoring as standard practise across Dutch UMCs. The HCPs in this study are the frontrunners, believing that telemonitoring will become standard practise in the future despite the fact that it is currently not. Based on the results of this study, both educational and implementation strategies including practical skills training are highly recommended in order to scale up telemonitoring widely.

Evaluating the implementation of the Mayo-Portland Adaptability Inventory-4 (MPAI-4) in three rehabilitation settings in Quebec: a mixed-methods study protocol

INTRODUCTION

Stroke is a leading cause of morbidity and mortality worldwide, placing an immense burden on patients and the health system. Timely access to rehabilitation services can improve stroke survivors' quality of life. The use of standardised outcome measures is endorsed for optimising patient rehabilitation outcomes and improving clinical decision-making. This project results from a provincially mandated recommendation to use the fourth version of the Mayo-Portland Adaptability Inventory (MPAI-4) to measure changes in social participation of stroke survivors and to maintain commitment to evidence-informed practices in stroke care. This protocol outlines the implementation process of the MPAI-4 for three rehabilitation centres. The objectives are to: (a) describe the context of MPAI-4 implementation; (b) determine clinical teams' readiness for change; (c) identify barriers and enablers to implementing the MPAI-4 and match the implementation strategies; (d) evaluate the MPAI-4 implementation outcomes including the degree of integration of the MPAI-4 into clinical practice and (e) explore participants' experiences using the MPAI-4.

METHODS AND ANALYSIS

We will use a multiple case study design within an integrated knowledge translation (iKT) approach with active engagement from key informants. Each case is a rehabilitation centre implementing MPAI-4. We will collect data from clinicians and programme managers using mixed methods guided by several theoretical frameworks. Data sources include surveys, focus groups and patient charts. We will conduct descriptive, correlational and content analyses. Ultimately, we will analyse, integrate data from qualitative and quantitative components and report them within and across participating sites. Results will provide insights about iKT within stroke rehabilitation settings that could be applied to future research projects.

ETHICS AND DISSEMINATION

The project received Institutional Review Board approval from the Centre for Interdisciplinary Research in Rehabilitation of Greater Montreal. We will disseminate results in peer-reviewed publications and at local, national and international scientific conferences.

Exploring readiness for implementing goal-oriented care in primary care using normalization process theory

AIM

To use normalization process theory (NPT) to build a strategy for the implementation of goal-oriented care (GOC) in primary care in Flanders, Belgium.

BACKGROUND

GOC is a possible approach to more coordinated and integrated care and tailors care to patients' personal life goals. The concept has gained interest among policy makers and researchers, but the main drivers for successful implementation are the primary healthcare professionals (PHCPs) who need to see added value of GOC in order to embed it into their daily practice. NPT, developed to understand the processes of implementing new ways of organizing care, offers a useful lens to understand adoption of GOC in primary care practice.

METHOD

PHCPs (n = 131) who participated in a 2-hour community meeting on GOC were asked to complete the Normalization MeAsure Development survey. This 23-item survey is based on NPT and describes participants' views about how an intervention would impact their work, their expectations about it, and whether it could become a routine part of their work.

FINDINGS

The NPT constructs coherence (sense-making work) and cognitive participation (relational work) showed positive tendency toward implementation of GOC. The participants had an initial understanding on GOC and there was much interest in supporting and start working with this approach. The other constructs collective action (operational work) and reflexive monitoring (appraisal work) will need further efforts to trigger implementation. A common ground is needed to integrate GOC as a common practice which can be achieved by intensive interprofessional collaboration.

Effectiveness of Self-guided Tailored Implementation Strategies in Integrating and Embedding Internet-Based Cognitive Behavioral Therapy in Routine Mental Health Care: Results of a Multicenter Stepped-Wedge Cluster Randomized Trial

BACKGROUND

Internet-based cognitive behavioral therapy (iCBT) services for common mental health disorders have been found to be effective. There is a need for strategies that improve implementation in routine practice. One-size-fits-all strategies are likely to be ineffective. Tailored implementation is considered as a promising approach. The self-guided integrated theory-based Framework for intervention tailoring strategies toolkit (ItFits-toolkit) supports local implementers in developing tailored implementation strategies. Tailoring involves identifying local barriers; matching selected barriers to implementation strategies; developing an actionable work plan; and applying, monitoring, and adapting where necessary.

OBJECTIVE

This study aimed to compare the effectiveness of the ItFits-toolkit with implementation-as-usual (IAU) in implementing iCBT services in 12 routine mental health care organizations in 9 countries in Europe and Australia.

METHODS

A stepped-wedge cluster randomized trial design with repeated measures was applied. The trial period lasted 30 months. The primary outcome was the normalization of iCBT delivery by service providers (therapists, referrers, IT developers, and administrators), which was measured with the Normalization Measure Development as a proxy for implementation success. A 3-level linear mixed-effects modeling was applied to estimate the effects. iCBT service uptake (referral and treatment completion rates) and implementation effort (hours) were used as secondary outcomes. The perceived satisfaction (Client Satisfaction Questionnaire), usability (System Usability Scale), and impact of the ItFits-toolkit by implementers were used to assess the acceptability of the ItFits-toolkit.

RESULTS

In total, 456 mental health service providers were included in this study. Compared with IAU, the ItFits-toolkit had a small positive statistically significant effect on normalization levels in service providers (mean 0.09, SD 0.04; P=.02; Cohen d=0.12). The uptake of iCBT by patients was similar to that of IAU. Implementers did not spend more time on implementation work when using the ItFits-toolkit and generally regarded the ItFits-toolkit as usable and were satisfied with it.

CONCLUSIONS

The ItFits-toolkit performed better than the usual implementation activities in implementing iCBT services in routine practice. There is practical utility in the ItFits-toolkit for supporting implementers in developing and applying effective tailored implementation strategies. However, the effect on normalization levels among mental health service providers was small. These findings warrant modesty regarding the effectiveness of self-guided tailored implementation of iCBT services in routine practice.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03652883; https://clinicaltrials.gov/ct2/show/NCT03652883.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.1186/s13063-020-04686-4.

Evaluation of measures of sustainability and sustainability determinants for use in community, public health, and clinical settings: a systematic review

BACKGROUND

Sustainability is concerned with the long-term delivery and subsequent benefits of evidence-based interventions. To further this field, we require a strong understanding and thus measurement of sustainability and what impacts sustainability (i.e., sustainability determinants). This systematic review aimed to evaluate the quality and empirical application of measures of sustainability and sustainability determinants for use in clinical, public health, and community settings.

METHODS

Seven electronic databases, reference lists of relevant reviews, online repositories of implementation measures, and the grey literature were searched. Publications were included if they reported on the development, psychometric evaluation, or empirical use of a multi-item, quantitative measure of sustainability, or sustainability determinants. Eligibility was not restricted by language or date. Eligibility screening and data extraction were conducted independently by two members of the research team. Content coverage of each measure was assessed by mapping measure items to relevant constructs of sustainability and sustainability determinants. The pragmatic and psychometric properties of included measures was assessed using the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). The empirical use of each measure was descriptively analyzed.

RESULTS

A total of 32,782 articles were screened from the database search, of which 37 were eligible. An additional 186 publications were identified from the grey literature search. The 223 included articles represented 28 individual measures, of which two assessed sustainability as an outcome, 25 covered sustainability determinants and one explicitly assessed both. The psychometric and pragmatic quality was variable, with PAPERS scores ranging from 14 to 35, out of a possible 56 points. The Provider Report of Sustainment Scale had the highest PAPERS score and measured sustainability as an outcome. The School-wide Universal Behaviour Sustainability Index-School Teams had the highest PAPERS score (score=29) of the measure of sustainability determinants.

CONCLUSIONS

This review can be used to guide selection of the most psychometrically robust, pragmatic, and relevant measure of sustainability and sustainability determinants. It also highlights that future research is needed to improve the psychometric and pragmatic quality of current measures in this field.

TRIAL REGISTRATION

This review was prospectively registered with Research Registry (reviewregistry1097), March 2021.

Validity and reliability of the Chinese version of the Normalization MeAsure Development(NoMAD)

Background

The Normalization MeAsure Development (NoMAD) is a brief quantitative tool based on the Normalization Process Theory (NPT), which can measure the implementation process of new technologies and complex interventions. The aim of our study was to translate and culturally adapt the NoMAD into Chinese, and to evaluate the psychometric properties of the Chinese version of NoMAD.

Methods

According to the NoMAD translation guideline, we undertook forward translation, backward translation, and compared these translations to get a satisfactory result, then we performed cognitive interviews to achieve cross-culture adaptation. And the psychometric properties of the final version were evaluated among clinical nurses who used the pressure injuries management system via WeChat mini-program at a tertiary hospital in northwestern China.

Results

A total of 258 nurses were enrolled in our study, and the response rate was 92.1%. The Cronbach’s alpha of four dimensions were as follow: Coherence (0.768), Cognitive Participation (0.904), Collective Action (0.820), and Reflexive Monitoring (0.808). The overall internal consistency was 0.941. The confirmatory factor analysis results showed a good fit for its theoretical structure (CFI = 0.924, TLI = 0.910, RMSEA = 0.0079, SRMSR = 0.046, χ2/df = 2.61). The item-level content validity index ranged from 0.857 to 1, and the scale-level content validity index was 0.95. There were positive correlations between four constructs scores and three general normalization scores.

Conclusions

The Chinese version of NoMAD is a reliable and valid tool to evaluate the implementation process of innovations.

Implementation, mechanisms of change and contextual factors of a complex intervention to improve interprofessional collaboration and the quality of medical care for nursing home residents: study protocol of the process evaluation of the interprof ACT intervention package

BACKGROUND

To improve interprofessional collaboration between registered nurses (RNs) and general practitioners (GPs) for nursing home residents (NHRs), the interprof ACT intervention package was developed. This complex intervention includes six components (e.g., shared goal setting, standardized procedures for GPs' nursing home visits) that can be locally adapted. The cluster-randomized interprof ACT trial evaluates the effects of this intervention on the cumulative incidence of hospital admissions (primary outcome) and secondary outcomes (e.g., length of hospital stays, utilization of emergency care services, and quality of life) within 12 months. It also includes a process evaluation which is subject of this protocol. The objectives of this evaluation are to assess the implementation of the interprof ACT intervention package and downstream effects on nurse-physician collaboration as well as preconditions and prospects for successive implementation into routine care.

METHODS

This study uses a mixed methods triangulation design involving all 34 participating nursing homes (clusters). The quantitative part comprises paper-based surveys among RNs, GPs, NHRs, and nursing home directors at baseline and 12 months. In the intervention group (17 clusters), data on the implementation of preplanned implementation strategies (training and supervision of nominated IPAVs, interprofessional kick-off meetings) and local implementation activities will be recorded. Major outcome domains are the dose, reach and fidelity of the implementation of the intervention package, changes in interprofessional collaboration, and contextual factors. The qualitative part will be conducted in a subsample of 8 nursing homes (4 per study group) and includes repeated non-participating observations and semistructured interviews on the interaction between involved health professionals and their work processes. Quantitative and qualitative data will be descriptively analyzed and then triangulated by means of joint displays and mixed methods informed regression models.

DISCUSSION

By integrating a variety of qualitative and quantitative data sources, this process evaluation will allow comprehensive assessment of the implementation of the interprof ACT intervention package, the changes induced in interprofessional collaboration, and the influence of contextual factors. These data will reveal expected and unexpected changes in the procedures of interprofessional care delivery and thus facilitate accurate conclusions for the further design of routine care services for NHRs.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03426475 . Registered on 07/02/2018.

The implementation and validation of the NoMAD during a complex primary care intervention

Background

Normalization process theory (NPT) has been widely used to better understand how new interventions are implemented and embedded. The NoMAD (Normalization Measurement Development questionnaire) is a 23-item NPT instrument based on NPT. As the NoMAD is a relatively new instrument, the objectives of this paper are: to describe the experience of implementing the NoMAD, to describe it being used as a feedback mechanism to gain insight into the normalization process of a complex health intervention, and to further explore the psychometric properties of the instrument.

Methods

Health TAPESTRY was implemented in six Family Health Teams (total of seven sites) across Ontario. Healthcare team members at each site were invited to complete the NoMAD, and three general questions about normalization, six times over a 12-month period. Each site was then provided a visual traffic light summary (TLS) reflecting the implementation of the Health TAPESTRY. The internal consistency of each sub-scale and validity of the NoMAD were assessed. Learnings from the implementation of the NoMAD and subsequent feedback mechanism (TLS) are reported descriptively.

Results

In total, 56 diverse health care team members from six implementation sites completed the NoMAD. Each used it at least once during the 12-month study period. The implementation of the NoMAD and TLS was time consuming to do with multiple collection (and feedback) points. Most (60%) internal consistency values of the four subscales (pooled across site) across each collection point were satisfactory. All correlations were positive, and most (86%) were statistically significant among NoMAD subscales. All but one correlation between the NoMAD subscales and the general questions were positive, and most (72%) were significant. Generally, scores on the subscales were higher at 12-month than baseline, albeit did not follow a linear pattern of change across implementation. Generally, scores were higher for experienced sites compared to first-time implementors.

Conclusion

Our experience would suggest fewer collection points; three timepoints spaced out by several months are adequate, if repeated administration of the NoMAD is used for feedback loops. We provide additional evidence of the psychometric properties of the NoMAD.

Trial Registration

Registered at ClinicalTrials.gov: NCT03397836.

Evaluation of Normalization After Implementation of the Digital Dutch Obstetric Telephone Triage System: Mixed Methods Study With a Questionnaire Survey and Focus Group Discussion

Background

The Dutch Obstetric Telephone Triage System (DOTTS) was developed to improve the quality of acute obstetric care. To achieve optimal effect, the DOTTS should be adopted in the daily care process by triage staff.

Objective

The primary aim was to evaluate the degree of implementation (ie, normalization) of the DOTTS, and the secondary aim was to evaluate which lessons can be learned from its current implementation in Dutch hospitals.

Methods

An evaluation study with a mixed methods design was performed. All triage staff in 9 Dutch hospitals that implemented the DOTTS before September 1, 2019, were invited to complete the Normalization Measure Development (NoMAD) questionnaire between December 2019 and July 2020. The questionnaire is based on the Normalization Process Theory (NPT). This self-reported questionnaire provides insights into the work people do in order to integrate and embed new practice in routine care. The NPT is based on the following 4 constructs: coherence, cognitive participation, collective action, and reflexive monitoring. Within the questionnaire, each construct is represented by 4-7 questions. Questions are scored on a 5-point normalization process scale. Descriptive statistics were used for analysis of questionnaire scores. Subsequently, the questionnaire findings were discussed during a focus group. Template analysis following the 4 constructs was used for analyzing the results of the focus group.

Results

Overall, 173 of 294 (58.8%) triage staff members completed the NoMAD questionnaire, and 90.2% (156/173) of the participants had used the DOTTS for over 6 months. The digital application was used as much as possible or always by 137 of 173 (79.2%) participants. The overall normalization process score was 3.77 (SD 0.36). The constructs coherence, cognitive participation, collective action, and reflexive monitoring scored 4.01 (SD 0.47), 4.05 (SD 0.45), 3.5 (SD 0.45), and 3.72 (SD 0.47), respectively. Analysis of the focus group discussion showed that the added value of the DOTTS was seen as a quality improvement for the care of pregnant women. Dedication of the complete multidisciplinary implementation team was important for facilitating normalization. Support from the medical staff and proper use by all disciplines involved in the triage were seen as facilitating factors. Participants appreciated training and evaluation, and indicated a need for ongoing training and evaluation in relation to goal achievement.

Conclusions

The DOTTS has been integrated into normal care in daily practice. Evaluation by the NoMAD questionnaire provided a positive overall score. These results are in line with or, in some aspects, better than the results of other evaluation studies. Key factors in the normalization process of the DOTTS in obstetric triage are the shared added value for stakeholders, the dedication of the complete multidisciplinary implementation team, and implementation plans that are tailor made in the practical context of the hospital.

Prior to Implementation of Digital Pathology-Assessment of Expectations among Staff by Means of Normalization Process Theory

The Region of Southern Denmark is the first in Denmark to implement digital pathology (DIPA), starting at the end of 2020. The DIPA process involves changes in workflow, and the pathologist will have to diagnose based on digital whole slide imaging instead of through the traditional use of the conventional light microscope and glass slides. In addition, in the laboratory, the employees will have to implement one more step to their workflow—scanning of tissue. The aim of our study was to assess the expectations and readiness among employees and management towards the implementation of DIPA, including their thoughts and motivations for starting to use DIPA. We used a mixed-method approach. Based on the findings derived from 18 semi-structured interviews with employees from the region’s departments of pathology, we designed a questionnaire, including questions from the normalization measure development tool. The questionnaires were e-mailed to 181 employees. Of these employees, 131 responded to the survey. Overall, they reported feeling sufficiently tech-savvy to be able to use DIPA, and they had high expectations as well as motivation and readiness for the upcoming changes. However, the employees were skeptical regarding the allocation of resources, and few were aware of reports about the effects of DIPA. Based on the findings, it seems to be important to provide not only a thorough introduction to the new intervention and the changes it will entail, but also to continue to ensure that the staff know how it works and why it is necessary to implement.

Adapting and Implementing Apps for Mental Healthcare

PURPOSE OF REVIEW

To describe examples of adapting apps for use in mental healthcare and to formulate recommendations for successful adaptation in mental healthcare settings.

RECENT FINDINGS

International examples are given to explore implementation procedures to address this multitude of challenges. There are only few published examples of adapting apps for use in mental healthcare. From these examples and from results of studies in implementation science in general clinical settings, it can be concluded that the process of adapting apps for mental healthcare needs to address clinician training and information needs, user needs which include cultural adaptation and go beyond mere translation, and organizational needs for blending app use into everyday clinical mental healthcare workflows.

A randomized controlled trial of cognitive control training (CCT) as an add-on treatment for late-life depression: a study protocol

BACKGROUND

Already a major health concern, late-life depression (LLD) is expected to form an increasing problem in the aging population. Moreover, despite current treatments, LLD is associated with a poor long-term prognosis and high rate of chronicity. Treatment provision and treatment accordingly warrant improvement, where add-on treatments might contribute to the efficacy of conventional therapies. Although it is known that impaired cognitive control contributes to LDD, it is not targeted sufficiently by current interventions. Research on cognitive control training (CCT) shows promising results on depressive symptoms, cognitive performance, and overall functioning. However, further research is needed to determine the long-term effects of CCT on LLD, its cost-effectiveness, and mechanisms of change.

METHODS

In the current multicenter randomized controlled trial (RCT) with a between-subjects design participants aged 60 years and over with a current LLD receiving treatment as usual (TAU) are randomized to add-on CCT or placebo training. Randomization is stratified by depression severity. Participants will receive eight online CCT or placebo sessions spread across four consecutive weeks. They will complete a post-training assessment after 1 month and three follow-up assessments scheduled three, six and 12 months after completing the training. We expect CCT and TAU to be more (cost-)effective in reducing depressive symptoms than placebo training and TAU. Additionally, we will be looking at secondary clinical, cognitive and global functioning outcomes and likely mechanisms of change (e.g., improved cognitive functioning, reduced rumination, and improved inhibition of negative stimuli).

DISCUSSION

The proposed RCT aims to contribute to the clinical and scientific knowledge on the long-term effects of CCT as an add-on treatment for LLD. Cost-effectiveness is particularly relevant considering the expected volume of the target demographic. The study will be a pragmatic trial with few inclusion restrictions, providing information on feasibility of web-based trainings in clinical settings. The outcomes are potentially generalizable to guidelines for treatment of LLD.

TRIAL REGISTRATION

This trial is registered in the Netherlands Trial Register (code: NL7639 ). Registered 3 april 2019.

Mental health specialist video consultations versus treatment as usual in patients with depression or anxiety disorders in primary care: study protocol for an individually randomised superiority trial (the PROVIDE-C trial)

BACKGROUND

Most people with mental disorders, including those with severe and chronic disorders, are treated solely by their general practitioner (GP). Nevertheless, specialised mental health care may be required for specific patients. Notably, the accessibility of mental health specialist care is mainly complicated by (a) long waiting times for an appointment with specialists, (b) long travel distances to specialists, particularly in rural and remote areas, and (c) patients' reservations about mental health specialist care (including fear of being stigmatised by seeking such care). To mitigate those barriers, technology-based integrated care models have been proposed. The purpose of this study is to examine the effectiveness and cost-effectiveness of a mental health specialist video consultations model versus treatment as usual in patients with depression or anxiety disorders in primary care.

METHODS

In an individually randomised, prospective, two-arm superiority trial with parallel group design, N = 320 patients with anxiety and/or depressive disorder will be recruited in general practices in Germany. The intervention includes a newly developed treatment model based on video consultations with focus on diagnostics, treatment planning, and short-term intervention by mental health specialists. We will systematically compare the effectiveness, cost-effectiveness, and adverse effects of this new model with usual care by the GP: the primary outcome is the absolute change in the mean depressive and anxiety symptom severity measured on the Patient Health Questionnaire Anxiety and Depression Scale (PHQ-ADS) from baseline to 6 months after baseline assessment. Follow-up in both groups will be conducted by blinded outcome assessors at 6 months and 12 months after baseline. The main analysis will be based on the intention-to-treat principle. We will optimise the likelihood of treatment effectiveness by strict inclusion criteria for patients, enhanced intervention integrity, and conducting a process evaluation.

DISCUSSION

To the best of our knowledge, this is the first confirmatory study on a video-based, integrated care model for the treatment of anxiety and depressive disorders in GP patients in Germany.

TRIAL REGISTRATION

ClinicalTrials.gov, United States National Institutes of Health NCT04316572 . Prospectively registered on 20 March 2020.

Implementation of a first-trimester prognostic model to improve screening for gestational diabetes mellitus

Background

Improvement in the accuracy of identifying women who are at risk to develop gestational diabetes mellitus (GDM) is warranted, since timely diagnosis and treatment improves the outcomes of this common pregnancy disorder. Although prognostic models for GDM are externally validated and outperform current risk factor based selective approaches, there is little known about the impact of such models in day-to-day obstetric care.

Methods

A prognostic model was implemented as a directive clinical prediction rule, classifying women as low- or high-risk for GDM, with subsequent distinctive care pathways including selective midpregnancy testing for GDM in high-risk women in a prospective multicenter birth cohort comprising 1073 pregnant women without pre-existing diabetes and 60 obstetric healthcare professionals included in nine independent midwifery practices and three hospitals in the Netherlands (effectiveness-implementation hybrid type 2 study). Model performance (c-statistic) and implementation outcomes (acceptability, adoption, appropriateness, feasibility, fidelity, penetration, sustainability) were evaluated after 6 months by indicators and implementation instruments (NoMAD; MIDI).

Results

The adherence to the prognostic model (c-statistic 0.85 (95%CI 0.81–0.90)) was 95% (n = 1021). Healthcare professionals scored 3.7 (IQR 3.3–4.0) on implementation instruments on a 5-point Likert scale. Important facilitators were knowledge, willingness and confidence to use the model, client cooperation and opportunities for reconfiguration. Identified barriers mostly related to operational and organizational issues. Regardless of risk-status, pregnant women appreciated first-trimester information on GDM risk-status and lifestyle advice to achieve risk reduction, respectively 89% (n = 556) and 90% (n = 564)).

Conclusions

The prognostic model was successfully implemented and well received by healthcare professionals and pregnant women. Prognostic models should be recommended for adoption in guidelines.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12884-021-03749-x.

Tailored implementation of internet-based cognitive behavioural therapy in the multinational context of the ImpleMentAll project: a study protocol for a stepped wedge cluster randomized trial

BACKGROUND

Internet-based Cognitive Behavioural Therapy (iCBT) is found effective in treating common mental disorders. However, the use of these interventions in routine care is limited. The international ImpleMentAll study is funded by the European Union's Horizon 2020 programme. It is concerned with studying and improving methods for implementing evidence-based iCBT services for common mental disorders in routine mental health care. A digitally accessible implementation toolkit (ItFits-toolkit) will be introduced to mental health care organizations with the aim to facilitate the ongoing implementation of iCBT services within local contexts. This study investigates the effectiveness of the ItFits-toolkit by comparing it to implementation-as-usual activities.

METHODS

A stepped wedge cluster randomized controlled trial (SWT) design will be applied. Over a trial period of 30 months, the ItFits-toolkit will be introduced sequentially in twelve routine mental health care organizations in primary and specialist care across nine countries in Europe and Australia. Repeated measures are applied to assess change over time in the outcome variables. The effectiveness of the ItFits-toolkit will be assessed in terms of the degree of normalization of the use of the iCBT services. Several exploratory outcomes including uptake of the iCBT services will be measured to feed the interpretation of the primary outcome. Data will be collected via a centralized data collection system and analysed using generalized linear mixed modelling. A qualitative process evaluation of routine implementation activities and the use of the ItFits-toolkit will be conducted within this study.

DISCUSSION

The ImpleMentAll study is a large-scale international research project designed to study the effectiveness of tailored implementation. Using a SWT design that allows to examine change over time, this study will investigate the effect of tailored implementation on the normalization of the use of iCBT services and their uptake. It will provide a better understanding of the process and methods of tailoring implementation strategies. If found effective, the ItFits-toolkit will be made accessible for mental health care service providers, to help them overcome their context-specific implementation challenges.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03652883 . Retrospectively registered on 29 August 2018.

Community mental healthcare: new developments and innovative strategies

PURPOSE OF REVIEW

The aim of this scoping review was to identify and map the available evidence on recent innovations in community mental healthcare across the globe.

RECENT FINDINGS

This review highlights the different innovative approaches and strategies being currently used in the field of community mental health. Key approaches found in the reviewed studies include collaborative care with the inclusion of peer workers, growing use of e-health and telepsychiatry, improved reforms on national mental health policies and de-institutionalization, modification of outreach models and mental health promotion in the community. The studies reviewed here suggest that continued innovation and implementation of new models and strategies have the potential to reduce the burden of disease and increase the quality of life for patients with mental health issues.

SUMMARY

Growing body of evidence shows that integrative care is the new standard of care for people with mental illnesses, with necessity of continuity of care from emergency department to community mental health services. Social determinants of rehabilitation and recovery, and peers support remain a new main topic of research in area of treatment of people with severe mental illnesses. E-health tools are becoming prevalent in the processes of promotion, prevention and treatment in mental healthcare.

Optimizing the implementation of a population panel management intervention in safety-net clinics for pediatric hypertension (The OpTIMISe-Pediatric Hypertension Study)

BACKGROUND

Though clinical practice guidelines are available, the diagnosis of pediatric hypertension (HTN) is often missed. Management may not follow guidelines due to the measurement challenges in children, complexity of interpreting youth blood pressure standards that are dependent on height, age, and sex, familiarity with diagnostic criteria, and variable comfort with management of pediatric HTN among providers. Evidence suggests that wide adoption and adherence to pediatric HTN guidelines would result in lower cardiovascular disease and kidney damage in adulthood. The proposed project will develop an implementation strategy package to increase adherence to clinical practice guidelines for pediatric HTN within safety-net community health centers (CHCs). The centerpiece of which is a provider-facing population panel management (PPM) tool and point-of-care clinical decision support (CDS). Prior research indicates that multiple discrete implementation strategies (e.g., stakeholder involvement, readiness planning, training, ongoing audit and feedback) are needed to institute practice- and provider-level adoption of such tools.

METHODS

Using participatory research methods involving stakeholders from a practice-based research network of CHCs, with input from scientific advisors, the project aims to (1) employ user-centered design methods to tailor an existing CDS tool for use at the point of care and optimize cohort management with a PPM tool to support adherence to the latest pediatric HTN guidelines, and (2) use a stakeholder-driven method for selecting implementation strategies that support tool adoption and increase guideline-adherent physician behaviors. Multilevel process evaluation using surveys and key informant interview data will assess the acceptability, adoption, appropriateness, cost, and feasibility of the PPM tool and its multicomponent implementation strategy package. Usability testing will be conducted with the PPM tool to iteratively refine features and ensure proper functionality.

DISCUSSION

The proposed research has the potential to improve identification, diagnosis, and management of HTN in primary care settings for high-risk youth by assisting healthcare providers in implementing the American Academy of Pediatrics' 2017 guidelines using an EHR-integrated PPM tool with CDS. Should the strategy package for PPM tool adoption be successful for pediatric HTN, findings will be translatable to other settings and PPM of other chronic cardiovascular conditions affecting overall population health.

Implementation of e-mental health interventions for informal caregivers of adults with chronic diseases: a protocol for a mixed-methods systematic review with a qualitative comparative analysis

INTRODUCTION

Informal caregivers provide the majority of care to individuals with chronic health conditions, benefiting the care recipient and reducing use of formal care services. However, providing informal care negatively impacts the mental health of many caregivers. E-mental health interventions have emerged as a way to provide accessible mental healthcare to caregivers. Much attention has been given to reviewing the effectiveness and efficacy of such interventions, however, factors related to implementation have received less consideration. Therefore, this mixed-methods systematic review will aim to examine factors associated with the effectiveness and implementation of e-mental health interventions for caregivers.

METHODS AND ANALYSIS

Eligible studies published since 1 January 2007 will be searched for in several electronic databases (CINAHL Plus with Full Text, the Cochrane Library, EMBASE, PsycINFO, PubMed and Web of Science), clinical trial registries and OpenGrey, with all screening steps conducted by two independent reviewers. Studies will be included if they focus on the implementation or effectiveness of e-mental health interventions designed for informal adult caregivers of adults with cancer, heart disease, stroke, diabetes, dementia or chronic obstructive pulmonary disease. Pragmatic randomised controlled trials quantitatively reporting on caregiver anxiety, depression, psychological distress or stress will be used for a qualitative comparative analysis to identify combinations of conditions that result in effective interventions. Qualitative and quantitative data on implementation of e-mental health interventions for caregivers will be integrated in a thematic synthesis to identify barriers and facilitators to implementation. These results will inform future development and implementation planning of e-mental health interventions for caregivers.

ETHICS AND DISSEMINATION

Ethical approval is not required for this study as no primary data will be collected. Results will be disseminated in the form of a scientific publication and presentations at academic conferences and plain language summaries for various stakeholders.

PROSPERO REGISTRATION NUMBER

CRD42020155727.

Evaluation of The Practitioner Online Referral and Treatment Service (PORTS): the first 18 months of a state-wide digital service for adults with anxiety, depression, or substance use problems

The Practitioner Online Referral and Treatment Service (PORTS) is a new digital mental health service (DMHS) providing assessment, treatment, and consultation across Western Australia, for adults with anxiety, depression, or substance use problems, and experiencing financial hardship or geographical disadvantage. From July 2017 to December 2018, a total of 2,527 individuals were referred to PORTS. Of these, 150 (6%) did not give consent for their results to be analysed. Of the remaining 2,377 patients, 615 (26%) could not be contacted to confirm the referral, 596 (25%) received assessment or information from PORTS, 427 (18%) were referred to another service, and 739 (31%) commenced treatment at PORTS. Almost half (47%) of patients were from areas with significant socio-economic disadvantage. Those referred by another mental health service were more likely to engage in treatment than those referred by a General Practitioner (GP). Overall outcomes were excellent, with large effect sizes (Cohen's d: 1.1-1.4), from assessment to post-treatment and 3-month follow-up, reliable deterioration was low, and GP and patient satisfaction was high. These results indicate that the PORTS DMHS model is a promising method for engaging primary care patients with anxiety and depression, including those experiencing financial and geographical disadvantage.

From Research to Practice: Ten Lessons in Delivering Digital Mental Health Services

There is a large body of research showing that psychological treatment can be effectively delivered via the internet, and Digital Mental Health Services (DMHS) are now delivering those interventions in routine care. However, not all attempts to translate these research outcomes into routine care have been successful. This paper draws on the experience of successful DMHS in Australia and Canada to describe ten lessons learned while establishing and delivering internet-delivered cognitive behavioural therapy (ICBT) and other mental health services as part of routine care. These lessons include learnings at four levels of analysis, including lessons learned working with (1) consumers, (2) therapists, (3) when operating DMHS, and (4) working within healthcare systems. Key themes include recognising that DMHS should provide not only treatment but also information and assessment services, that DMHS require robust systems for training and supervising therapists, that specialist skills are required to operate DMHS, and that the outcome data from DMHS can inform future mental health policy. We also confirm that operating such clinics is particularly challenging in the evolving funding, policy, and regulatory context, as well as increasing expectations from consumers about DMHS. Notwithstanding the difficulties of delivering DMHS, we conclude that the benefits of such services for the broader community significantly outweigh the challenges.