The merits of implementing routine outcome monitoring in clinical practice

Developing the Open Psychedelic Evaluation Nexus consensus measures for assessment of supervised psilocybin services: An e-Delphi study

BACKGROUND

Voter initiatives in Oregon and Colorado authorize legal frameworks for supervised psilocybin services, but no measures monitor safety or outcomes.

AIMS

To develop core measures of best practices.

METHODS

A three-phase e-Delphi process recruited 36 experts with 5 or more years' experience facilitating psilocybin experiences in various contexts (e.g., ceremonial settings, indigenous practices, clinical trials), or other pertinent psilocybin expertise. Phase I, an on-line survey with qualitative, open-ended text responses, generated potential measures to assess processes, outcomes, and structure reflecting high quality psilocybin services. In Phase II, experts used seven-point Likert scales to rate the importance and feasibility of the Phase I measures. Measures were priority ranked. Qualitative interviews and analysis in Phase III refined top-rated measures.

RESULTS

Experts (n = 36; 53% female; 71% white; 56% heterosexual) reported currently providing psilocybin services (64%) for a mean of 15.2 [SD 13.1] years, experience with indigenous psychedelic practices (67%), and/or conducting clinical trials (36%). Thematic analysis of Phase I responses yielded 55 candidate process measures (e.g., preparatory hours with client, total dose of psilocybin administered, documentation of touch/sexual boundaries), outcome measures (e.g., adverse events, well-being, anxiety/depression symptoms), and structure measures (e.g., facilitator training in trauma informed care, referral capacity for medical/psychiatric issues). In Phase II and III, experts prioritized a core set of 11 process, 11 outcome, and 17 structure measures that balanced importance and feasibility.

CONCLUSION

Service providers and policy makers should consider standardizing core measures developed in this study to monitor the safety, quality, and outcomes of community-based psilocybin services.

The Patient-Perceived Helpfulness of Measures Scale: Development and Validation of a Scale to Assess the Helpfulness of Using Measures in Psychological Treatment

In response to the increase in Routine Outcome Monitoring and Clinical Feedback, the Patient-Perceived Helpfulness of Measures Scale (ppHMS) was developed to assess the helpfulness-as perceived by patients-of using measures in psychological treatment. Study 1: The construct of patient-perceived helpfulness of measures was explored using thematic analysis with 15 patients. Six helpful and three unhelpful themes were identified and informed item development. Study 2: 28 items were formulated and rated by experts. Ten items were taken forward for psychometric shortening in a sample of 76 patients. Confirmatory factor analysis (CFA) led to an adequately fitting six-item model with excellent internal consistency, and convergence with the Delighted-Terrible single item of product satisfaction and a single item of measure helpfulness. Study 3: In a stratified online sample of 514 U.K. psychotherapy patients, a five-item model constituted the best fit. The final ppHMS had excellent internal consistency (McDonald's ω = .90), convergent validity with psychotherapy satisfaction (r = .5; p < .001), divergence from social desirability (r = .1), and metric and scalar invariance across measures. Study 4: Analyses were replicated and confirmed in a stratified U.S. sample (n = 602). The ppHMS is a reliable and valid scale that can be used to assess and compare patients' perceptions of the helpfulness of different measures as part of their psychological treatment.

Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluation

Large language models (LLMs) such as Open AI's GPT-4 (which power ChatGPT) and Google's Gemini, built on artificial intelligence, hold immense potential to support, augment, or even eventually automate psychotherapy. Enthusiasm about such applications is mounting in the field as well as industry. These developments promise to address insufficient mental healthcare system capacity and scale individual access to personalized treatments. However, clinical psychology is an uncommonly high stakes application domain for AI systems, as responsible and evidence-based therapy requires nuanced expertise. This paper provides a roadmap for the ambitious yet responsible application of clinical LLMs in psychotherapy. First, a technical overview of clinical LLMs is presented. Second, the stages of integration of LLMs into psychotherapy are discussed while highlighting parallels to the development of autonomous vehicle technology. Third, potential applications of LLMs in clinical care, training, and research are discussed, highlighting areas of risk given the complex nature of psychotherapy. Fourth, recommendations for the responsible development and evaluation of clinical LLMs are provided, which include centering clinical science, involving robust interdisciplinary collaboration, and attending to issues like assessment, risk detection, transparency, and bias. Lastly, a vision is outlined for how LLMs might enable a new generation of studies of evidence-based interventions at scale, and how these studies may challenge assumptions about psychotherapy.

Leveraging Single-Case Experimental Designs to Promote Personalized Psychological Treatment: Step-by-Step Implementation Protocol with Stakeholder Involvement of an Outpatient Clinic for Personalized Psychotherapy

Our objective is to implement a single-case experimental design (SCED) infrastructure in combination with experience-sampling methods (ESM) into the standard diagnostic procedure of a German outpatient research and training clinic. Building on the idea of routine outcome monitoring, the SCED infrastructure introduces intensive longitudinal data collection, individual effectiveness measures, and the opportunity for systematic manipulation to push personalization efforts further. It aims to empower psychotherapists and patients to evaluate their own treatment (idiographic perspective) and to enable researchers to analyze open questions of personalized psychotherapy (nomothetic perspective). Organized around the principles of agile research, we plan to develop, implement, and evaluate the SCED infrastructure in six successive studies with continuous stakeholder involvement: In the project development phase, the business model for the SCED infrastructure is developed that describes its vision in consideration of the context (Study 1). Also, the infrastructure's prototype is specified, encompassing the SCED procedure, ESM protocol, and ESM survey (Study 2 and 3). During the optimization phase, feasibility and acceptability are tested and the infrastructure is adapted accordingly (Study 4). The evaluation phase includes a pilot implementation study to assess implementation outcomes (Study 5), followed by actual implementation using a within-institution A-B design (Study 6). The sustainability phase involves continuous monitoring and improvement. We discuss to what extent the generated data could be used to address current questions of personalized psychotherapy research. Anticipated barriers and limitations during the implementation processes are outlined.

Culture matters: Chinese mental health professionals' fear of losing face in routine outcome monitoring

OBJECTIVE

The culturally salient fear of losing face might influence Chinese therapists' attitudes toward and use of routine outcome monitoring (ROM). We tested a model wherein self-face concern is associated with ROM use by way of attitudes toward ROM, and whether this process is weakened when therapists report high counseling self-efficacy and perspective-taking.

METHOD

A national sample of Chinese mental health professionals (N = 371) completed questionnaires on their fear of losing face, attitudes toward ROM, ROM use, counseling self-efficacy, and perspective-taking.

RESULTS

Regression-based analyses showed that fear of losing face was linked to greater negative attitudes toward ROM and lower ROM use. Greater negative attitudes mediated the relationship between fear of losing face and ROM use. However, neither counseling self-efficacy nor perspective-taking mitigated the relationship between self-face concern and ROM use; instead, they exacerbated this relationship through different paths. In the mediated pathway, counseling self-efficacy in coping with clients with difficult problems interacted with self-face concern to predict negative attitudes toward ROM. Perspective-taking served as a moderator that exacerbated the direct relationship between self-face concern and ROM use.

CONCLUSIONS

Findings suggest the importance of considering culturally salient factors in implementing ROM in China and other non-Western contexts.

Effects and experiences of idiographic patient-reported outcome measures for feedback in psychotherapy: A systematic review and secondary analysis of the empirical literature

Objective: There is a growing interest in idiographic patient-reported outcome measures (I-PROMs) for routine outcome monitoring (ROM) and clinical feedback (CF) in psychotherapy, but to our knowledge, no systematic reviews of the empirical literature. Method: We conducted a systematic literature search for empirical papers investigating effects and experiences of I-PROMs for ROM/CF and found a total of 13 papers. There was only one experimental controlled effectiveness study. Results: We formulated a narrative summary of the data set as a whole. We conducted a secondary analysis of nine papers containing qualitative data on stakeholder experiences with I-PROMs and found three superordinate themes and eight subthemes, summarized as "I-PROMs can facilitate self-reflection for patients, assist in identifying and tracking therapeutic topics, and make patients more committed to therapy by giving them a greater sense of responsibility and empowerment. Formulating goals and problems can be difficult, and lack of progress can be demotivating. Time in therapy is precious and must be spent wisely. Effective use of I-PROMs is facilitated by flexibility and therapists' use of clinical skills." Conclusion: We discuss the limitations of the study and provide recommendations for future research and clinical practice.

From symptoms to subjective and bodily experiences: the contribution of the Psychodynamic Diagnostic Manual (PDM-2) to diagnosis and treatment monitoring in eating disorders

PURPOSE

Atheoretical and descriptive conceptualizations of eating disorders (EDs) have faced substantial criticism due to their limited ability to assess patients' subjective characteristics and experiences, as needed to determine the most appropriate treatment options. The present article provides an overview of the clinical and empirical literature supporting the potential contribution of the Psychodynamic Diagnostic Manual (PDM-2) to both diagnostic assessment and treatment monitoring.

METHODS

Following a discussion of the most relevant shortcomings of current diagnostic models of EDs and a description of the rationale and structure of the PDM-2, evidence supporting the core PDM-2 dimensions of ED patients' subjective experiences (i.e., affective states, cognitive processes, relational patterns, somatic/bodily experiences and states) are examined, alongside their relevance to ED diagnosis and treatment.

RESULTS

Overall, the reviewed studies support the diagnostic importance of these patterns of subjective experiences in EDs, highlighting their potential role as either predisposing or maintaining factors to target in psychotherapy. A growing body of multidisciplinary evidence also shows that bodily and somatic experiences are central to the diagnosis and clinical management of ED patients. Moreover, there is evidence that a PDM-based assessment may enable closer monitoring of patient progress during treatment, with regard to both subjective experiences and symptom patterns.

CONCLUSIONS

The study suggests that current diagnostic frameworks for EDs would benefit from the addition of a person-centered perspective that considers not only symptoms, but also patients' full range of functioning-including their deep and surface-level emotional, cognitive, interpersonal, and social patterns-to improve patient-tailored interventions.

LEVEL OF EVIDENCE

Level V, narrative review.

The Building Emotional Awareness and Mental health (BEAM) program developed with a community partner for mothers of infants: protocol for a feasibility randomized controlled trial

BACKGROUND

Drastic increases in the rates of maternal depression and anxiety have been reported since the COVID-19 pandemic began. Most programs aim to improve maternal mental health or parenting skills separately, despite it being more effective to target both concurrently. The Building Emotional Awareness and Mental health (BEAM) program was developed to address this gap. BEAM is a mobile health program aiming to mitigate the impacts of pandemic stress on family well-being. Since many family agencies lack infrastructure and personnel to adequately treat maternal mental health concerns, a partnership will occur with Family Dynamics (a local family agency) to address this unmet need. The study's objective is to examine the feasibility of the BEAM program when delivered with a community partner to inform a larger randomized controlled trial (RCT).

METHODS

A pilot RCT will be conducted with mothers who have depression and/or anxiety with a child 6-18 months old living in Manitoba, Canada. Mothers will be randomized to the 10 weeks of the BEAM program or a standard of care (i.e., MoodMission). Back-end App data (collected via Google Analytics and Firebase) will be used to examine feasibility, engagement, and accessibility of the BEAM program; cost-effectiveness will also be examined. Implementation elements (e.g., maternal depression [Patient Health Questionnaire-9] and anxiety [Generalized Anxiety Disorder-7]) will be piloted to estimate the effect size and variance for future sample size calculations.

DISCUSSION

In partnership with a local family agency, BEAM holds the potential to promote maternal-child health via a cost-effective and an easily accessible program designed to scale. Results will provide insight into the feasibility of the BEAM program and will inform future RCTs. TRIAL REGISTRATION {2A}: This trial was retrospectively registered with ClinicalTrial.gov ( NCT05398107 ) on May 31st, 2022.

Implementation of a routine outcome monitoring and feedback system for psychotherapy in Argentina: A pilot study

Introduction

Routine Outcome Monitoring (ROM) has emerged as a strong candidate to improve psychotherapy processes and outcome. However, its use and implementation are greatly understudied in Latin-America. Therefore, the aim of the present pilot study conducted in Argentina was to implement a ROM and feedback system grounded on a psychometrically sound instrument to measure session by session outcome in psychotherapy.

Methods

The sample consisted of 40 patients and 13 therapists. At baseline, the patients completed the Patient Health Questionnaire-9 and the Generalized Anxiety Disorder-7, and they also completed the Hopkins Symptom Checklist-11 before each of the first five sessions. To estimate patient change during the first sessions, we conducted a quantitative analysis using Hierarchical Linear Models. Furthermore, we conducted a qualitative analysis using Consensual Qualitative Research to analyze therapist perception regarding the ROM and feedback system.

Results

Results showed a significant reduction in patients' symptomatic severity during the first five sessions. Additionally, baseline depression significantly predicted the estimated severity at the end of the fifth session. Feedback was given to the therapists after the first four sessions based on these analyses. With regard to the perception of the feedback system, clinicians underlined its usefulness and user-friendly nature. They also mentioned that there was a match between the information provided and their clinical judgment. Furthermore, they provided suggestions to enhance the system that was incorporated in a new and improved version.

Discussion

Limitations and clinical implications are discussed.

A Common Measurement Scale for Self-Report Instruments in Mental Health Care

Abstract. The diversity of measures in clinical psychology hampers a straightforward interpretation of test results, complicates communication with the patient, and constitutes a challenge to the implementation of measurement-based care. In educational research and assessment, it is common practice to convert test scores to a common metric, such as T scores. We recommend applying this also in clinical psychology and propose and test a procedure to arrive at T scores approximating a normal distribution that can be applied to individual test scores. We established formulas to estimate normalized T scores from raw scale scores by regressing IRT-based θ scores on raw scores. With data from a large population and clinical samples, we established crosswalk formulas. Their validity was investigated by comparing calculated T scores with IRT-based T scores. IRT and formulas yielded very similar T scores, supporting the validity of the latter approach. Theoretical and practical advantages and disadvantages of both approaches to convert scores to common metrics and alternative approaches are discussed. Provided that scale characteristics allow for their computation, T scores will help to better understand measurement results, which makes it easier for patients and practitioners to use test results in joint decision-making about the course of treatment.

Digital Intervention Barriers Scale–7 (DIBS-7): Development, Evaluation, and Preliminary Validation (Preprint)

BACKGROUND

The translation of mental health services into digital formats, deemed digital mental health interventions (DMHIs), has the potential to address long-standing obstacles to accessing care. However, DMHIs have barriers of their own that impact enrollment, adherence, and attrition in these programs. Unlike in traditional face-to-face therapy, there is a paucity of standardized and validated measures of barriers in DMHIs.

OBJECTIVE

In this study, we describe the preliminary development and evaluation of such a scale, the Digital Intervention Barriers Scale-7 (DIBS-7).

METHODS

Following an iterative QUAN → QUAL mixed methods approach, item generation was guided by qualitative analysis of feedback from participants (n=259) who completed a DMHI trial for anxiety and depression and identified barriers related to self-motivation, ease of use, acceptability, and comprehension of tasks. Item refinement was achieved through DMHI expert review. A final item pool was administered to 559 treatment completers (mean age 23.02 years; 438/559, 78.4% female; 374/559, 69.9% racially or ethnically minoritized). Exploratory factor analyses and confirmatory factor analyses were estimated to determine the psychometric properties of the measure. Finally, criterion-related validity was examined by estimating partial correlations between the DIBS-7 mean score and constructs related to treatment engagement in DMHIs.

RESULTS

Statistical analyses estimated a 7-item unidimensional scale with high internal consistency (α=.82, ω=0.89). Preliminary criterion-related validity was supported by significant partial correlations between the DIBS-7 mean score and treatment expectations (pr=-0.25), number of modules with activity (pr=-0.55), number of weekly check-ins (pr=-0.28), and treatment satisfaction (pr=-0.71).

CONCLUSIONS

Overall, these results provide preliminary support for the use of the DIBS-7 as a potentially useful short scale for clinicians and researchers interested in measuring an important variable often associated with treatment adherence and outcomes in DMHIs.

Praxisorientierte Forschung in der Psychotherapie

Zusammenfassung. In den letzten Jahrzehnten hat sich durch randomisiert-kontrollierte Studien (RCTs) eine breite Evidenzbasis von Psychotherapie mit mittleren bis großen Effekten für verschiedene psychische Störungen gebildet. Neben der Bestimmung dieser Wirksamkeit („Efficacy“) ebneten Studien zur Wirksamkeit unter alltäglichen Routinebedingungen („Effectiveness“) historisch den Weg zur Entwicklung eines praxisorientierten Forschungsparadigmas. Im Beitrag wird argumentiert, dass im Rahmen dieses Paradigmas praxisbasierte Studien eine wertvolle Ergänzung zu RCTs darstellen, da sie existierende Probleme in der Psychotherapieforschung adressieren können. In der gegenwärtigen praxisorientierten Forschung liefern dabei neue Ansätze aus der personalisierten Medizin und Methoden aus der ‚Computational Psychiatry‘ wichtige Anhaltspunkte zur Optimierung von Effekten in der Psychotherapie. Im Kontext der Personalisierung werden bspw. klinische multivariable Prädiktionsmodelle entwickelt, welche durch Rückmeldeschleifen an Praktiker_innen kurzfristig ein evidenzbasiertes Outcome-Monitoring ermöglicht und langfristig das Praxis-Forschungsnetzwerk in Deutschland stärkt. Am Ende des Beitrags werden zukünftige Richtungen für die praxisorientierte Forschung im Sinne des ‘Precision Mental Health Care’ -Paradigmas abgeleitet und diskutiert.

Performance of a Supervisor Observational Coding System and an Audit and Feedback Intervention

Workplace-based clinical supervision is common in community based mental health care for youth and families and could be a leveraged to scale and improve the implementation of evidence-based treatment (EBTs). Accurate methods are needed to measure, monitor, and support supervisor performance with limited disruption to workflow. Audit and Feedback (A&F) interventions may offer some promise in this regard. The study-a randomized controlled trial with 60 clinical supervisors measured longitudinally for 7 months-had two parts: (1) psychometric evaluation of an observational coding system for measuring adherence and competence of EBT supervision and (2) evaluation of an experimental Supervisor Audit and Feedback (SAF) intervention on outcomes of supervisor adherence and competence. All supervisors recorded and uploaded weekly supervision sessions for 7 months, and those in the experimental condition were provided a single, monthly web-based feedback report. Psychometric performance was evaluated using measurement models based in Item Response Theory, and the effect of the SAF intervention was evaluated using mixed-effects regression models. The observational instrument performed well across psychometric indicators of dimensionality, rating scale functionality, and item fit; however, coder reliability was lower for competence than for adherence. Statistically significant A&F effects were largely in the expected directions and consistent with hypotheses. The observational coding system performed well, and a monthly electronic feedback report showed promise in maintaining or improving community-based clinical supervisors' adherence and, to a lesser extent, competence. Limitations discussed include unknown generalizability to the supervision of other EBTs.

Distress tolerance as a moderator of the dynamic associations between interpersonal needs and suicidal thoughts

BACKGROUND

Identifying the interaction between dispositional and dynamic risk factors is necessary in understanding, predicting, and managing suicide risk. Interpersonal factors have consistently been linked to suicidal ideation over short-term periods. Additionally, distress tolerance may be a relevant dispositional protective factor against stressful events.

METHODS

Seven hundred and seventeen psychiatric inpatients (Male = 30.31%, Average Age = 40.71 years, Min = 14, Max = 82) self-reported their distress tolerance at hospital admission, and interpersonal needs and suicidal ideation on a daily basis. Dynamic structural equation modelling was used to examine whether within-level dynamics were moderated by distress tolerance.

RESULTS

Both perceived burdensomeness and thwarted belongingness were significantly associated with same-day suicidal ideation. Higher distress tolerance was associated with weaker daily associations between suicidal ideation and both perceived burdensomeness and thwarted belongingness. Distress tolerance was also associated with lower variability in suicidal ideation. Moderating effects were also evident when lifetime suicide attempts were added as a covariate, which was associated with stronger associations between interpersonal dysfunction and suicidal ideation, and higher variability in ideation.

CONCLUSIONS

Distress tolerance may be important to consider when examining the dynamic relationships between suicidal ideation and proximal factors. Psychotherapy that specifically targets distress tolerance may be effective in reducing reactivity to interpersonal stressors.

Common measures or common metrics? A plea to harmonize measurement results

OBJECTIVE

There is a great variety of measurement instruments to assess similar constructs in clinical research and practice. This complicates the interpretation of test results and hampers the implementation of measurement-based care.

METHOD

For reporting and discussing test results with patients, we suggest converting test results into universally applicable common metrics. Two well-established metrics are reviewed: T scores and percentile ranks. Their calculation is explained, their merits and drawbacks are discussed, and recommendations for the most convenient reference group are provided.

RESULTS

We propose to express test results as T scores with the general population as reference group. To elucidate test results to patients, T scores may be supplemented with percentile ranks, based on data from a clinical sample. The practical benefits are demonstrated using the published data of four frequently used instruments for measuring depression: the CES-D, PHQ-9, BDI-II and the PROMIS depression measure.

DISCUSSION

Recent initiatives have proposed to mandate a limited set of outcome measures to harmonize clinical measurement. However, the selected instruments are not without flaws and, potentially, this directive may hamper future instrument development. We recommend using common metrics as an alternative approach to harmonize test results in clinical practice, as this will facilitate the integration of measures in day-to-day practice.

Determining prognostic variables of treatment outcome in obsessive-compulsive disorder: effectiveness and its predictors in routine clinical care

The objectives of this study were to investigate the naturalistic effectiveness of routine inpatient treatment for patients with obsessive-compulsive disorder (OCD) and to identify predictors of treatment outcome. A routinely collected data set of 1,596 OCD inpatients (M = 33.9 years, SD = 11.7; 60.4% female) having received evidence-based psychotherapy based on the cognitive-behavioral therapy (CBT) in five German psychotherapeutic clinics was analyzed. Effect sizes (Hedges' g) were calculated for several outcome variables to determine effectiveness. Predictor analyses were performed on a subsample (N = 514; M = 34.3 years, SD = 12.2; 60.3% female). For this purpose, the number of potential predictors was reduced using factor analysis, followed by multiple regression analysis to identify robust predictors. Effect sizes of various outcome variables could be classified as large (g = 1.34 of OCD-symptom change). Predictors of changes in OCD and depressive symptoms were symptom severity at admission and general psychopathological distress. In addition, patients with higher social support and more washing compulsions benefited more from treatment. Subgroup analyses showed a distinct predictor profile of changes in compulsions and obsessions. The results indicate that an evidence-based psychotherapy program for OCD can be effectively implemented in routine inpatient care. In addition to well-established predictors, social support, and washing compulsions in particular were identified as important positive predictors. Specific predictor profiles for changes in obsessions and compulsions are discussed.

Evaluation of an Online System for Routine Outcome Monitoring: Cross-sectional Survey Study

BACKGROUND

The use of routine outcome monitoring (ROM) in the treatment of mental health has emerged as a method of improving psychotherapy treatment outcomes. Despite this, very few clinicians regularly use ROM in clinical practice. Online ROM has been suggested as a solution to increase adoption.

OBJECTIVE

The aim of this study is to identify the influence of moving ROM online on client completion rates of self-reported outcome measures and to identify implementation and utilization barriers to online ROM by assessing clinicians' views on their experience using the online system over previous paper-based methods.

METHODS

Client completion rates of self-reported outcome measures were compared pre- and postimplementation of an online system of ROM. In addition, a survey questionnaire was administered to 324 mental health service providers regarding their perception of the benefits with an online system of ROM.

RESULTS

Client completion rates of self-reported measures increased from 15.62% (427/2734) to 53.98% (1267/2347) after they were moved online. Furthermore, 57% (56/98) of service providers found the new system less time-consuming than the previous paper-based ROM, and 64% (63/98) found that it helped monitor clients. However, the perceived value of the system remains in doubt as only 23% (23/98) found it helped them identify clients at risk for treatment failure, and only 18% (18/98) found it strengthened the therapeutic alliance.

CONCLUSIONS

Although the current study suggests mixed results regarding service providers' views on their experience using an online system for ROM, it has identified barriers and challenges that are actionable for improvement.

The Session Wants and Need Outcome Measure: The Development of a Brief Outcome Measure for Single-Sessions of Web-Based Support

Single-session, brief interventions in therapy for young people make up a large proportion of service provision, including in digital mental health settings. Current nomothetic mental health measures are not specifically designed to capture the benefit or ‘change’ directly related to these brief interventions. As a consequence, we set out to design an outcome measure to concretely demonstrate the value of single-session interventions. The Session Wants and Needs Outcome Measure (SWAN-OM) aims to capture in-session goals and focuses on being user-centric, elements critical to the success of single-session and brief interventions which typically are asset-based and solution-focused. We describe the 4-stage process that was followed to develop this measure: (I) classical item generation and development, (II) content and (III) face validity pilot testing, and (IV) a user-experience approach with young people using framework analysis. This final stage was critical to ensure the integration of this outcome tool into a web-based digital therapy setting, a context which adds another layer of design complexity to item and measure development. This iterative methodology was used to overcome the challenges encountered and to place the needs of the young people and service practitioners at the centre of the design process, thus ensuring measure usability. To end, we highlight the main lessons learnt from engaging in this design process. Specifically, the needs of a measure for single-session interventions are considered, before outlining the learning associated with integrating the measure into a digital mental health platform. Both of these areas are emerging fields and, as such, this study contributes to our understanding of how an idiographic patient outcome theory driven measure can be created for use in a web-based digital mental health therapy service.

Provider perceived barriers and facilitators to integrating routine outcome monitoring into practice in an urban community psychiatry clinic: A mixed-methods quality improvement project

BACKGROUND AND AIMS

Although mental health clinics are under increasing pressure to demonstrate value and routine outcome monitoring (ROM) has become a mandated component of care, providers have been slow to adopt ROM into practice, with some estimating that less than 20% of mental health clinicians use it consistently in the United States. This article explores perceived barriers and facilitators to integrating ROM into practice among clinicians and administrators in a large urban US community psychiatry clinic.

METHOD

One hundred and thirty-eight clinical and administrative staff were administered an anonymous web-based survey to elicit attitudes towards ROM. Responses were summarized descriptively and qualitatively synthesized into a conceptual model using inductive thematic analysis.

RESULTS

Common barriers to integration included insufficient time to collect and/or use measures, not knowing what measures to use, measures being difficult to access, and insufficient training. Facilitators included increased access/ease of use, training and support, measure relevance/validity, and accountability.

CONCLUSIONS

In order for psychiatry clinics to successfully implement ROM into practice, they must diagnose organization-side barriers and translate this knowledge into actionable quality improvement initiatives ranging from the infrastructural to the cultural.

Applying behaviour change theory to understand the barriers to implementing routine outcome monitoring

BACKGROUND

Routine outcome monitoring (ROM) is a valuable tool for monitoring client progress and pre-empting deterioration, however, there is considerable variation in how data are collected and recorded and uptake in clinical practice remains low. The aim of this study was to develop a self-report measure of practitioner attitudes to ROM in order to better understand the barriers to successful implementation in Child and Adolescent Mental Health Services (CAMHS).

METHODS

An anonymous survey was completed by 184 CAMHS practitioners in the United Kingdom. The survey was designed using the Capability, Opportunity, and Motivation Model of Behaviour (COM-B). Practitioners who reported using ROM frequently in their clinical work (53%) were compared to those who used ROM infrequently (47%) across dimensions of the COM-B survey subscales.

RESULTS

Confirmatory factor analysis confirmed the proposed four-factor structure, showing acceptable model fit, with high factor loadings and good reliability for all subscales. Frequent users of ROM exhibited significantly higher psychological capability, physical opportunity, social opportunity, and motivation, compared to infrequent users F (4, 140) = 14.76, p < .0001; Pillai's Trace = .297, partial η²  = .30. Results highlight several barriers to ROM, including the belief that there is not a strong evidence base for ROM, not receiving external training, and not discussing feedback and outcome data in supervision.

IMPLICATIONS

In the hope of improving the successful implementation of ROM, this research provides an evidence-based tool for assessing practitioners' attitudes to ROM, which map on to intervention functions and represent targets for future implementation efforts.

PRACTITIONER POINTS

The value of routine outcome monitoring (ROM) as a means to measure client progress and to elevate the efficiency and quality of mental health care is well-documented in the research literature, however, uptake in practice remains relatively low. This study applied behaviour change theory to develop a psychometrically sound self-report measure of practitioners' perspectives and practices to understand the barriers to implementation in child and adolescent mental health services in the United Kingdom. The complex and multifaceted nature of the barriers to implementation requires multilevel behaviour change strategies at the client, clinician, and organisational level. Recommendations for practice include the need for integrated, multilevel strategies aimed at improving practitioners' capabilities and motivations, strong organisational leadership and a culture of data gathering and sharing, and implementation interventions, which are tailored to target local barriers.

Matching research and practice: Prediction of individual patient progress and dropout risk for basic routine outcome monitoring

OBJECTIVE

Despite evidence showing that systematic outcome monitoring can prevent treatment failure, the practical conditions that allow for implementation are seldom met in naturalistic psychological services. In the context of limited time and resources, session-by-session evaluation is rare in most clinical settings. This study aimed to validate innovative prediction methods for individual treatment progress and dropout risk based on basic outcome monitoring.

METHODS

Routine data of a naturalistic psychotherapy outpatient sample were analyzed (N = 3902). Patients were treated with cognitive behavioral therapy with up to 95 sessions (M = 39.19, SD = 16.99) and assessment intervals of 5-15 sessions. Treatment progress and dropout risk were predicted in two independent analyses using the nearest neighbor method and least absolute shrinkage and selection operator regression, respectively.

RESULTS

The correlation between observed and predicted patient progress was r = .46. Intrinsic treatment motivation, previous inpatient treatment, university-entrance qualification, baseline impairment, diagnosed personality disorder, and diagnosed eating disorder were identified as significant predictors of dropout, explaining 11% of variance.

CONCLUSIONS

Innovative outcome prediction in naturalistic psychotherapy is not limited to elaborated progress monitoring. This study demonstrates a reasonable approach for tracking patient progress as long as session-by-session assessment is not a valid standard.

How therapists and patients need to develop a clinical feedback system after 18 months of use in a practice-research network: a qualitative study

BACKGROUND

A personalized computer-adaptive system for clinical feedback and routine outcome monitoring in mental health, Norse Feedback aims to bridge the needs for standardized and idiographic measures in ordinary practice.

METHODS

Item response theory analyses of completed treatment processes (n = 800) informed a qualitative study comprising individual in-depth interviews and focus groups with patients (n = 9) and clinicians (n = 10). The research question was: How do clinicians and patients contribute to developing a clinical feedback system in a continuous process aimed at refining its clinical usefulness?

RESULTS

We conducted thematic analyses and found five themes: 1. Added clinical needs, 2. Needs for re-organizing the clinician report, 3.Needs for differentiation of clinical content, 4. User-interface needs, and 5. Item level suggestions.

CONCLUSION

In this article, we detail resulting needs for continuous adaptation to practice, and discuss implications of the concrete experiences with the Norse action research program for the larger field of ROM/CFS implementation.

Attitudes Towards the Use of Routine Outcome Monitoring of Psychological Therapies Among Mental Health Providers: The EBPAS-ROM

Implementation of routine outcome monitoring (ROM) in mental health care is progressing slowly. Knowledge about factors influencing ROM implementation, including health providers' attitudes towards ROM, is necessary. Based on a survey of 662 psychologists and nurses, this article describes (1) the development of a short instrument measuring provider attitudes towards ROM, derived from the Evidence-based Practice Attitude Scale (EBPAS), and (2) how attitudinal domains relate to clinicians' current use of standardized instruments for treatment evaluation. The EBPAS-ROM showed concurrent validity in predicting aspects important for the implementation of ROM, including perceived limitations and the value of organizational support.

Effects of Internet-Based Cognitive Behavioral Therapy in Routine Care for Adults in Treatment for Depression and Anxiety: Systematic Review and Meta-Analysis

BACKGROUND

Although there is evidence for the efficacy of internet-based cognitive behavioral therapy (iCBT), the generalizability of results to routine care is limited.

OBJECTIVE

This study systematically reviews effectiveness studies of guided iCBT interventions for the treatment of depression or anxiety.

METHODS

The acceptability (uptake, participants' characteristics, adherence, and satisfaction), effectiveness, and negative effects (deterioration) of nonrandomized pre-post designs conducted under routine care conditions were synthesized using systematic review and meta-analytic approaches.

RESULTS

A total of 19 studies including 30 groups were included in the analysis. Despite high heterogeneity, individual effect sizes of investigated studies indicate clinically relevant changes, with effect sizes ranging from Hedges' g=0.42-1.88, with a pooled effect of 1.78 for depression and 0.94 for anxiety studies. Uptake, participants' characteristics, adherence, and satisfaction indicate a moderate to high acceptability of the interventions. The average deterioration across studies was 2.9%.

CONCLUSIONS

This study provides evidence supporting the acceptability and effectiveness of guided iCBT for the treatment of depression and anxiety in routine care. Given the high heterogeneity between interventions and contexts, health care providers should select interventions that have been proven in randomized controlled clinical trials. The successful application of iCBT may be an effective way of increasing health care in multiple contexts.

Clinical dilemmas of routine outcome monitoring and clinical feedback: A qualitative study of patient experiences

ABSTRACT Purpose: Routine outcome monitoring (ROM) and clinical feedback systems (CFS) are becoming prevalent in mental health services, but there are several challenges to successful implementation. ROM/CFS seem to be helpful for some patients, but not for others. To investigate this, we explored patients' experiences with ROM/CFS as an interpersonal and psychotherapeutic process, in naturalistic settings. Method: We used video-assisted interpersonal process recall interviews to investigate the experiences of 12 patients using ROM/CFS in a Norwegian mental health outpatient clinic. Data were analyzed through systematic text condensation. Results: Our analysis resulted in three pairs of experiences with ROM/CFS: (1) Explicit vs. implicit use of information, (2) Directing focus towards or away from therapeutic topics, and (3) Giving and receiving feedback. These experiences could be helpful or hindering, depending on participants' needs and preferences. All participants needed to know that the CFS was used in a meaningful way. If not, it could be detrimental to the therapeutic process. Conclusion: In order to be helpful for patients, ROM/CFS should be used in a way that is flexible, meaningful to patients, and sensitive to individual needs and preferences. Future research should further explore this how-to aspect of ROM/CFS with different CFS and populations.

Complexity and potentials of clinical feedback in mental health: an in-depth study of patient processes

Purpose

Routine outcome monitoring (ROM) and clinical feedback systems (CFS) are becoming increasingly prevalent in mental health services. Their overall efficacy is unclear, but quantitative evidence suggests they can be useful tools for preventing treatment failure and enhancing therapeutic outcomes, especially for patients who are not progressing in therapy. The body of qualitative material, however, is smaller and less refined. We need to know more about how ROM/CFS is used in psychotherapy, and why it is helpful for some patients, but not others.

Methods

We recorded therapy sessions of 12 patients who were using a CFS as part of their therapies at an outpatient clinic in Norway. We then conducted video-assisted interviews and follow-up interviews with patients. Data were analyzed with systematic text condensation.

Results

Analysis revealed three themes: (1) triggering reflections, emotions, and self-awareness, (2) Ambivalent and ambiguous self-presentation, and (3) potential for feeling understood and talking about what matters.

Conclusion

Answering questions in a CFS is an interpretative and intentional process of self-presentation and the results from ROM/CFS must be interpreted and explored in conversation to be clinically useful. When they are, they have potential for enhancing the therapeutic process by stimulating self-awareness, reflexivity, and allowing access to new therapeutic topics. Further research should explore this how-to aspect of ROM/CFS with different CFS and different types of patients. Integrating clinical feedback in therapeutic practice can be conceptualized as a clinical skill, which should be a part of training programs for therapists.

Electronic supplementary material

The online version of this article (10.1007/s11136-020-02550-1) contains supplementary material, which is available to authorized users.

Why are some cases not on track? An item analysis of the Assessment for Signal Cases during inpatient psychotherapy

Within the Routine Outcome Monitoring system “OQ‐Analyst,” the questionnaire “Assessment for Signal Cases” (ASC) supports therapists in detecting potential reasons for not‐on‐track trajectories. Factor analysis and a machine learning algorithm (LASSO with 10‐fold cross‐validation) were applied, and potential predictors of not‐on‐track classifications were tested using logistic multilevel modeling methods. The factor analysis revealed a shortened (30 items) version of the ASC with good internal consistency (α = 0.72–0.89) and excellent predictive value (area under the curve = 0.98; positive predictive value = 0.95; negative predictive value = 0.94). Item‐level analyses showed that interpersonal problems captured by specific ASC items (not feeling able to speak about problems with family members; feeling rejected or betrayed) are the most important predictors of not‐on‐track trajectories. It should be considered that our results are based on analyses of ASC items only. Our findings need to be replicated in future studies including other potential predictors of not‐on‐track trajectories (e.g., changes in medication, specific therapeutic techniques, or treatment adherence), which were not measured this study.

Development and validation of a progress monitoring tool tailored for use in intensive eating disorder treatment

OBJECTIVE

Despite calls for routine use of progress and outcome monitoring in private and intensive treatment centres for eating disorders (EDs), existing measures have limited relevance to these supervised treatment settings. This study sought to develop and validate the progress monitoring tool for eating disorders, a multidimensional measure for progress monitoring in the context of intensive ED treatment.

METHOD

Thirty-seven items were generated by a team of content experts, clinicians, and administrative staff from the target treatment setting. Adolescent and adult females (N = 531) seeking residential ED treatment completed the items at admission as part of the clinic's routine assessment battery; 83% were retained for repeat assessment at discharge. Exploratory factor analysis was conducted for preliminary measure development.

RESULTS

Results yielded a five-factor, 26-item structure explaining 50% of total variance. Final construct domains included weight and shape concern, ED behaviours and urges, emotion avoidance, adaptive coping, and relational connection. The measure demonstrated adequate internal consistency, sensitivity to change during treatment, and convergence with validated assessment measures.

CONCLUSIONS

Preliminary data support the progress monitoring tool for eating disorders as a novel and valid multidimensional measure of treatment-relevant constructs. This measure may have utility in measuring treatment progress for patients receiving intensive treatment for EDs.

Assessing the unwanted: A systematic review of instruments used to assess negative effects of psychotherapy

OBJECTIVE

While the efficacy of psychotherapy in the treatment of mental disorders is well examined, systematic research into negative effects of psychotherapy seems comparatively rare. Therefore, this review evaluates instruments for assessing negative effects of psychotherapy in order to create a consensus framework and make recommendations for their assessment.

METHODS

The study selection procedure follows current best-practice guidelines for conducting systematic reviews, with 10 included studies in three databases (PsycINFO, PubMed, and Web of Science). The nine instruments identified were each critically reviewed concerning the theoretical orientation, including the assessed domains of negative effects, psychometric properties, and diagnostic characteristics.

RESULTS

Seventeen domains of negative effects of psychotherapy were identified but inconsistently assessed by the nine instruments. Most instruments provide some initial data on their psychometric properties. Regarding diagnostic characteristics, different item-response formats are used but often with reference to "attribution to therapy."

CONCLUSION

This review indicates that the existing instruments for assessing negative effects of psychotherapy cover a wide range of relevant domains without any consensus on the most important ones and their psychometric properties are usually unsatisfactory. A framework for consensus, building on the definition and conceptualization of negative effects, is synthesized, and recommendations for improving the assessment are derived.