Patients' experiences with routine outcome monitoring and clinical feedback systems: A systematic review and synthesis of qualitative empirical literature

The Patient-Perceived Helpfulness of Measures Scale: Development and Validation of a Scale to Assess the Helpfulness of Using Measures in Psychological Treatment

In response to the increase in Routine Outcome Monitoring and Clinical Feedback, the Patient-Perceived Helpfulness of Measures Scale (ppHMS) was developed to assess the helpfulness-as perceived by patients-of using measures in psychological treatment. Study 1: The construct of patient-perceived helpfulness of measures was explored using thematic analysis with 15 patients. Six helpful and three unhelpful themes were identified and informed item development. Study 2: 28 items were formulated and rated by experts. Ten items were taken forward for psychometric shortening in a sample of 76 patients. Confirmatory factor analysis (CFA) led to an adequately fitting six-item model with excellent internal consistency, and convergence with the Delighted-Terrible single item of product satisfaction and a single item of measure helpfulness. Study 3: In a stratified online sample of 514 U.K. psychotherapy patients, a five-item model constituted the best fit. The final ppHMS had excellent internal consistency (McDonald's ω = .90), convergent validity with psychotherapy satisfaction (r = .5; p < .001), divergence from social desirability (r = .1), and metric and scalar invariance across measures. Study 4: Analyses were replicated and confirmed in a stratified U.S. sample (n = 602). The ppHMS is a reliable and valid scale that can be used to assess and compare patients' perceptions of the helpfulness of different measures as part of their psychological treatment.

Routine Outcome Monitoring and Clinical Feedback in Psychotherapy: Recent Advances and Future Directions

In the past decade, there has been an increase in research related to the routine collection and active use of standardized patient data in psychotherapy. Research has increasingly focused on personalization of care to patients, clinical skills and interventions that modulate treatment outcomes, and implementation strategies, all of which appear to enhance the beneficial effects of ROM and feedback. In this article, we summarize trends and recent advances in the research on this topic and identify several essential directions for the field in the short to medium term. We anticipate a broadening of research from the focus on average effects to greater specificity around what kinds of feedback, provided at what time, to which individuals, in what settings, are most beneficial. We also propose that the field needs to focus on issues of health equity, ensuring that ROM can be a vehicle for increased wellbeing for those who need it most. The complexity of mental healthcare systems means that there may be multiple viable measurement solutions with varying costs and benefits to diverse stakeholders in different treatment contexts, and research is needed to identify the most influential components in each of these contexts.

The Connections Matter: Bi-Directional Learning in Program Evaluation and Practice-Oriented Research

Intended for researchers and clinical leaders, this article suggests that embedded program evaluation is a good fit with the desired features of practice-oriented research. The systematic nature of evaluation that is built into the operational workflow of a practice setting may increase the diversity of methods available to explore processes and outcomes of interest. We propose a novel conceptual framework that uses a human-centered systems lens to foster such embedded evaluation in clinical routine. This approach emphasizes the evaluator-practitioner partnership to build confidence in the bi-directional learning of practice-based evidence with evidence-based practice. The iterative cycles inherent to design thinking are aimed at developing better evaluation questions. The attention to structure and context inherent to systems thinking is intended to support meaningful perspectives in the naturally complex world of health care. Importantly, the combined human-centered systems lens can create greater awareness of the influence of individual and systemic biases that exist in any endeavor or institution that involves people. Recommended tools and strategies include systems mapping, program theory development, and visual facilitation using a logic model to represent the complexity of mental health treatment for communication, shared understanding, and connection to the broader evidence base. To illustrate elements of the proposed conceptual framework, two case examples are drawn from routine outcome monitoring (ROM) and progress feedback. We conclude with questions for future collaboration and research that may strengthen the partnership of evaluators and practitioners as a community of learners in service of local and system-level improvement.

Psychotherapists' Experience with In-Session Use of Routine Outcome Monitoring: A Qualitative Meta-analysis

Routine outcome monitoring (ROM) has become an increasingly utilized tool in therapeutic practice that has the potential to improve therapy outcomes. This study aimed to synthesize the findings of existing qualitative studies investigating how clinicians use ROM in their work with clients. A systematic search of qualitative studies on clinicians' experience with the use of ROM in mental health services was conducted via PsycInfo, PsycArticles, Medline, Web of Science, and Scopus databases. Qualitative meta-analysis was used to synthesize the finding of the primary studies. Forty-seven studies met the inclusion criteria. The analysis resulted in 21 meta-categories organized into six clusters, namely (1) obtaining clinically relevant information, (2) adapting treatment, (3) facilitating communication, (4) enhancing the therapeutic relationship, (5) facilitating change in clients, and (6) personalized usage of ROM. The meta-analysis revealed that clinicians utilized ROM in diverse ways, including both informational and communicational functions. From the clinicians' perspective, ROM was an element that, on the one hand, introduced additional structure and standardization in treatment and, on the other hand, allowed for greater flexibility and tailoring of treatment.

Therapists' and patients' experiences of using patients' self-reported data in ongoing psychotherapy processes-A systematic review and meta-analysis of qualitative studies

OBJECTIVE

Using patient-generated quantitative data in psychotherapy (feedback) appears to enhance treatment outcome, but there is variability in its effect. Different ways and reasons to implement routine outcome measurement might explain such variability. The goal of this review is to address the insufficient knowledge on how these data are used by therapists and patients.

METHODS

The present study is a systematic review and meta-analysis of qualitative reports of therapists' and patients' experiences using patient-generated quantitative data during ongoing psychotherapy.

RESULTS

Four main categories of use were identified: (1) uses of patients' self-reported data as nomothetic/objective markers for assessment, process monitoring, and treatment planning; (2) intrapersonal uses that enhance self-awareness, initiate reflection, and influence patients' mood or responses; (3) uses that prompt interactional processes by facilitating communication, supporting exploration, creating ownership in patients, changing treatment focus, enhancing therapeutic alliance, or disturbing the psychotherapy process; and (4) patients responding for specific purposes due to uncertainty and interpersonal motives, or strategic responding to achieve a desired result.

CONCLUSION

These results demonstrate that patient-reported data, when used in active psychotherapy, is very clearly not just an objective measurement of client functioning: the inclusion of patient-data has the potential to influence psychotherapy in numerous ways.

Feasibility of Recovery Assessment Scale - Domains and Stages (RAS-DS) for everyday mental health practice

Introduction

Routine use of self-rated measures of mental health recovery can support recovery-oriented practice. However, to be widely adopted, outcome measures must be feasible. This study examined the feasibility of Recovery Assessment Scale - Domains and Stages (RAS-DS) from the perspectives of mental health workers.

Method

Mental health workers who had previously sought permission to use RAS-DS (n=58) completed an online survey that explored three aspects of feasibility: practicality, acceptability and applicability.

Results

The highest-rated feasibility items related to applicability, or usefulness in practice, with over 90% of participants reporting that RAS-DS helps "promote discussion" and covers areas that are "meaningful to consumers". Acceptability items indicated that the purpose of RAS-DS is clear but length was an issue for some participants. At a practical level, RAS-DS was seen as easy to access but training was seen by many as necessary to ensure optimal use.

Conclusion

Results suggest potential usefulness of RAS-DS as a routine outcome measure and identify aspects that can be addressed to further enhance feasibility including provision of training materials and opportunities, wide-reaching promotion of its use as a collaborative tool, and further investigation of issues around instrument length.

Effects and experiences of idiographic patient-reported outcome measures for feedback in psychotherapy: A systematic review and secondary analysis of the empirical literature

Objective: There is a growing interest in idiographic patient-reported outcome measures (I-PROMs) for routine outcome monitoring (ROM) and clinical feedback (CF) in psychotherapy, but to our knowledge, no systematic reviews of the empirical literature. Method: We conducted a systematic literature search for empirical papers investigating effects and experiences of I-PROMs for ROM/CF and found a total of 13 papers. There was only one experimental controlled effectiveness study. Results: We formulated a narrative summary of the data set as a whole. We conducted a secondary analysis of nine papers containing qualitative data on stakeholder experiences with I-PROMs and found three superordinate themes and eight subthemes, summarized as "I-PROMs can facilitate self-reflection for patients, assist in identifying and tracking therapeutic topics, and make patients more committed to therapy by giving them a greater sense of responsibility and empowerment. Formulating goals and problems can be difficult, and lack of progress can be demotivating. Time in therapy is precious and must be spent wisely. Effective use of I-PROMs is facilitated by flexibility and therapists' use of clinical skills." Conclusion: We discuss the limitations of the study and provide recommendations for future research and clinical practice.

A Clinical Leadership Lens on Implementing Progress Feedback in Three Countries: Development of a Multidimensional Qualitative Coding Scheme

BACKGROUND

Progress feedback, also known as measurement-based care (MBC), is the routine collection of patient-reported measures to monitor treatment progress and inform clinical decision-making. Although a key ingredient to improving mental health care, sustained use of progress feedback is poor. Integration into everyday workflow is challenging, impacted by a complex interrelated set of factors across patient, clinician, organizational, and health system levels. This study describes the development of a qualitative coding scheme for progress feedback implementation that accounts for the dynamic nature of barriers and facilitators across multiple levels of use in mental health settings. Such a coding scheme may help promote a common language for researchers and implementers to better identify barriers that need to be addressed, as well as facilitators that could be supported in different settings and contexts.

METHODS

Clinical staff, managers, and leaders from two Dutch, three Norwegian, and four mental health organizations in the USA participated in semi-structured interviews on how intra- and extra-organizational characteristics interact to influence the use of progress feedback in clinical practice, supervision, and program improvement. Interviews were conducted in the local language, then translated to English prior to qualitative coding.

RESULTS

A team-based consensus coding approach was used to refine an a priori expert-informed and literature-based qualitative scheme to incorporate new understandings and constructs as they emerged. First, this hermeneutic approach resulted in a multi-level coding scheme with nine superordinate categories and 30 subcategories. Second-order axial coding established contextually sensitive categories for barriers and facilitators.

CONCLUSIONS

The primary outcome is an empirically derived multi-level qualitative coding scheme that can be used in progress feedback implementation research and development. It can be applied across contexts and settings, with expectations for ongoing refinement. Suggestions for future research and application in practice settings are provided. Supplementary materials include the coding scheme and a detailed playbook.

Routine Outcome Monitoring (ROM) and Feedback: Research Review and Recommendations

OBJECTIVE

To provide a research review of the components and outcomes of routine outcome monitoring (ROM) and recommendations for research and therapeutic practice.

METHOD

A narrative review of the three phases of ROM - data collection, feeding back data, and adapting therapy - and an overview of patient outcomes from 11 meta-analytic studies.

RESULTS

Patients support ROM when its purpose is clear and integrated within therapy. Greater frequency of data collection is more important for shorter-term therapies, and use of graphs, greater specificity of feedback, and alerts are helpful. Overall effects on patient outcomes are statistically significant (g ≈ 0.15) and increase when clinical support tools (CSTs) are used for not-on-track cases (g ≈ 0.36-0.53). Effects are additive to standard effects of psychological therapies. Organizational, personnel, and resource issues remain the greatest obstacles to the successful adoption of ROM.

CONCLUSION

ROM offers a low-cost method for enhancing patient outcomes, on average resulting in an ≈ 8% advantage (success rate difference; SRD) over standard care. CSTs are particularly effective for not-on-track patients (SRD between ≈ 20% and 29%), but ROM does not work for all patients and successful implementation is a major challenge, along with securing appropriate cultural adaptations.

This is what I need a clinical feedback system to do for me: A qualitative inquiry into perspectives of adolescents and their therapists

OBJECTIVE

Routine outcome monitoring and clinical feedback systems might be beneficial for adolescent psychotherapy processes.

METHODS

Clinicians (n = 34) and adolescent clients (n = 22) aged 14-19 from seven different outpatient clinics located in Norway participated in the study. Adolescents were interviewed in individual in-depth interviews (n = 7) or in four adolescent-only focus groups (n = 15), clinicians participated in seven clinician-only focus groups.

RESULTS

We report two core domains, (1) feedback about the therapeutic relationship and (2) feedback about the therapeutic work. Seven subthemes specify the functionality that participants need in a feedback system.

CONCLUSION

Adolescents and therapists requested a feedback system that was relationally oriented, supported collaborative action, and was personalized to the needs of the individual adolescent. The research indicates that a clinical feedback system should have idiographic, as well as nomothetic, components. A clinical feedback system for adolescents should monitor experiences of personal autonomy and the quality of the therapeutic relationship.

How patients and clinicians experience the utility of a personalized clinical feedback system in routine practice

OBJECTIVE

The objective was to explore how a person-adaptive clinical feedback system (CFS) effects its users, and how meaning and relevance are negotiated.

METHODS

We conducted a 10-month case-study of the implementation and practice of Norse Feedback, a personalized CFS. The data material consisted of 12 patient interviews, 22 clinician interviews, 23 field notes, and 16 archival documents.

RESULTS

We identified four main categories or themes: (i) patients' use of clinical feedback for enhanced awareness and insight; (ii) patients work to make clinical feedback a communication mode; (iii) patients and clinicians negotiate clinical feedback as a way to influence treatment; and (iv) clinical feedback requires an interactive sense-making effort.

CONCLUSION

Patients and therapists produced the meaning and relevance of the CFS by interpreting the CFS measures to reflect the unique patient experience of the patient-therapist relationship. Patients regarded CFS as a tool to inform therapy with important issues. Patients became more self-aware and prepared for therapy.

Mental health outcome measures in the Australian context: what is the problem represented to be?

BACKGROUND

There is growing interest in the use of routine outcome measures (ROM) in mental health services worldwide. Australia has been at the forefront of introducing ROM in public mental health services, with the aim of improving services and consumer outcomes.

METHODS

An in-depth policy and document analysis was conducted using Carol Bacchi's 'What is the problem represented to be?' approach to critically analyse the use of ROM. This approach was used to identify and analyse the problem representations relating to the need for, and the choice of, outcome measures in Australian public mental health services, and the potential consequences of policy and practice. Data included in the analysis were seven policy documents, four reports on the introduction of outcome measures in Australia, the Australian Mental Health Outcomes and Classifications Network website, and the content of the outcome measures themselves.

RESULTS

Two dominant representations of the 'problem' were identified: 1) the 'problem' of mental health service quality and accountability, relating to the need for mental health outcome measures; and 2) the 'problem' of addressing deficits in biopsychosocial functioning of mental health consumers, which relates to the choice of outcome measures. Framing the 'problem' of mental health outcomes in these ways locates the problem within individual health providers, services, and consumers, ignoring the broader socioeconomic conditions underpinning mental health and effective service provision.

CONCLUSIONS

This critical analysis of the introduction and use of ROM in public mental health services in Australia highlights the need to consider the role of the social determinants of mental health, mental health service funding, and recovery-oriented care in ensuring services are meeting consumer needs and expectations. Broader governmental engagement is central to genuine change and opportunities.

Implementation of a routine outcome monitoring and feedback system for psychotherapy in Argentina: A pilot study

Introduction

Routine Outcome Monitoring (ROM) has emerged as a strong candidate to improve psychotherapy processes and outcome. However, its use and implementation are greatly understudied in Latin-America. Therefore, the aim of the present pilot study conducted in Argentina was to implement a ROM and feedback system grounded on a psychometrically sound instrument to measure session by session outcome in psychotherapy.

Methods

The sample consisted of 40 patients and 13 therapists. At baseline, the patients completed the Patient Health Questionnaire-9 and the Generalized Anxiety Disorder-7, and they also completed the Hopkins Symptom Checklist-11 before each of the first five sessions. To estimate patient change during the first sessions, we conducted a quantitative analysis using Hierarchical Linear Models. Furthermore, we conducted a qualitative analysis using Consensual Qualitative Research to analyze therapist perception regarding the ROM and feedback system.

Results

Results showed a significant reduction in patients' symptomatic severity during the first five sessions. Additionally, baseline depression significantly predicted the estimated severity at the end of the fifth session. Feedback was given to the therapists after the first four sessions based on these analyses. With regard to the perception of the feedback system, clinicians underlined its usefulness and user-friendly nature. They also mentioned that there was a match between the information provided and their clinical judgment. Furthermore, they provided suggestions to enhance the system that was incorporated in a new and improved version.

Discussion

Limitations and clinical implications are discussed.

The Implementation of Measurement-Based Care in the Context of Telemedicine: Qualitative Study

BACKGROUND

The Measurement Based Care in Mental Health Initiative launched by the Department of Veterans Affairs in 2016 is an example of an evidence-based practice that uses patient-reported outcome measures (PROMs) to improve patient outcomes. The acceptance of measurement-based care (MBC) among Veterans Affairs providers is relatively high. However, there are barriers to MBC for telehealth providers. Health information technologies might afford opportunities to address some of the barriers related to the uptake of MBC.

OBJECTIVE

This paper reports on an implementation effort to integrate MBC into mental health care telehealth practice using eHealth solutions.

METHODS

Qualitative data were generated from 22 semistructured interviews with psychiatrists (n=4), psychologists (n=3), social workers (n=3), nurses (n=6), a pharmacist (n=1), and administrative staff (n=5) who provide telemental health care through a community-based outpatient clinic in the rural Midwestern United States. The interviews were conducted during the pilot phase of an implementation initiative to increase the adoption of MBC by revising clinic workflows to integrate the use of eHealth technologies. Data were analyzed using thematic analysis.

RESULTS

Time burden and workflow issues were the most common barrier to provider adoption of MBC; sharing and reviewing pencil-and-paper measures and results in the same room was no longer possible in novel telehealth workflows necessitated by the COVID-19 pandemic. Providers voiced concerns about how long it would take to collect, adequately score, interpret, share, and document the PROMs during the telehealth visit. Concerns about time might also correspond to a gap in providers' familiarity with these assessments, greater comfort in assessing symptoms through clinical interviews, and being accustomed to using the assessments as screening tools more so than longitudinal outcome measures. Capacities associated with eHealth technologies may address workflow concerns and promote providers' understanding and use of the measures as tracking tools.

CONCLUSIONS

The need to use limited appointment time well was a top priority for telemental health providers. eHealth technologies provided operative supports that protect time in appointments by shifting when and how PROMs are collected. Bolstering providers' familiarity with how to use PROMs in the course of treatment may impact providers' buy-in by encouraging them to reconsider how sharing and acting on PROMs could be time well spent.

Using routine outcome measures as clinical process tools: Maximising the therapeutic yield in the IAPT programme when working remotely

OBJECTIVES

The objective of the study was to investigate the administration and use of routine outcome monitoring session by session in the context of improving guided-self-help interventions when delivered remotely at Step 2 care in the English Improving Access to Psychological Therapies (IAPT) services.

DESIGN

Qualitative research using recordings of telephone-treatment sessions.

METHOD

Participants (11 patients and 11 practitioners) were recruited from four nationally funded IAPT services and one-third sector organisation commissioned to deliver Step 2 IAPT services, in England. Data collection took place prior to the COVID-19 pandemic. Transcripts of telephone-treatment sessions were analysed using thematic analysis.

RESULTS

Four themes were identified: (1) lack of consistency in the administration of outcome measures (e.g. inconsistent wording); (2) outcome measures administered as a stand-alone inflexible task (e.g. mechanical administration); (3) outcome measures as impersonal numbers (e.g. summarising, categorising and comparing total scores); and (4) missed opportunities to use outcome measures therapeutically (e.g. lack of therapeutic use of item and total scores).

CONCLUSIONS

The administration of outcome measures needs to ensure validity and reliability. Therapeutic yield from session-by-session outcome measures could be enhanced by focusing on three main areas: (1) adopting a collaborative conversational approach, (2) maximising the use of total and items scores and (3) integrating outcome measures with in-session treatment decisions. Shifting the perception of outcome measures as impersonal numbers to being process clinical tools ensures a personalised delivery of psychological interventions and has the potential to enhance engagement from practitioners and patients what may reduce drop-out rates and improve clinical outcomes.

Feedback-informed Group Treatment: A Qualitative Study of the Experiences and Needs of Patients and Therapists

Therapists, including group therapists, can systematically gather feedback from patients about how their group members are responding to treatment. However, results of research on using feedback-informed group treatment (FIGT) are mixed, and the underlying mechanisms responsible for positive patient changes remain unclear. Therefore, the present qualitative study examined the perceptions and experiences of both (a) group therapists and (b) group members regarding using feedback in their therapy groups to gauge treatment progress, across five different therapy groups. Specifically, three interpersonal psychotherapy groups and two cognitive-behavioral therapy groups used a FIGT tool in which treatment progress updates were provided to patients and therapists. Observational data were collected in the form of feedback discussions in these therapy groups, as well as during interviews conducted with patients and therapists. Data were analyzed using thematic analysis and a grounded theory approach. Overall, patients were mostly positive about their experiences with FIGT, but therapists also expressed concerns about FIGT. Results indicated that FIGT is useful for gaining insight and strengthening the working alliance. In addition, specific group processes were also found to be important, especially interpersonal learning, cohesion, and social comparison. Practical implications are discussed.

Drug and alcohol practitioners' attitudes toward the use of standardized assessment

Despite their importance to evidence-based assessment, standardized assessments remain underutilized by mental health practitioners in practice. The underutilization has been attributed to a lack of appreciation of the importance of standardized assessments, lack of knowledge of standardized assessments, and practical barriers to implementation. This study sought to gather the first descriptive data on alcohol and other drug (AOD) practitioners' attitudes toward, and knowledge and self-reported use of, standardized assessments. Practical barriers to implementation in initial assessment and progress monitoring were also assessed. Ninety-nine Australian AOD practitioners recruited via newsletters of national representative bodies and practitioner networks completed an online survey. While practitioners' attitudes towards using standardized assessments for initial assessment and progress monitoring were generally positive and consistent with other populations of health practitioners, assessments remained underutilized in practice. Most AOD practitioners did not consider standardized assessments to be feasible to implement. The current findings highlight the importance of practical barriers, particularly organization-level barriers, in the underutilization of standardized assessments in AOD practice. Findings support an extension of dialogue surrounding evidence-based practice beyond treatment selection to include assessment practices at a more general level. The present study offers a starting point from which efforts to improve practitioner compliance with evidence-based best practices can be conceived, designed, and implemented.

Patients’ and Psychologists’ Preferences for Feedback Reports on Expected Mental Health Treatment Outcomes: A Discrete-Choice Experiment

In recent years, there has been an increasing focus on routine outcome monitoring (ROM) to provide feedback on patient progress during mental health treatment, with some systems also predicting the expected treatment outcome. The aim of this study was to elicit patients’ and psychologists’ preferences regarding how ROM system-generated feedback reports should display predicted treatment outcomes. In a discrete-choice experiment, participants were asked 12–13 times to choose between two ways of displaying an expected treatment outcome. The choices varied in four different attributes: representation, outcome, predictors, and advice. A conditional logistic regression was used to estimate participants’ preferences. A total of 104 participants (68 patients and 36 psychologists) completed the questionnaire. Participants preferred feedback reports on expected treatment outcome that included: (a) both text and images, (b) a continuous outcome or an outcome that is expressed in terms of a probability, (c) specific predictors, and (d) specific advice. For both patients and psychologists, specific predictors appeared to be most important, specific advice was second most important, a continuous outcome or a probability was third most important, and feedback that includes both text and images was fourth in importance. The ranking in importance of both the attributes and the attribute levels was identical for patients and psychologists. This suggests that, as long as the report is understandable to the patient, psychologists and patients can use the same ROM feedback report, eliminating the need for ROM administrators to develop different versions.

Monitoring and Measurement in Child and Adolescent Mental Health: It's about More than Just Symptoms

Routine outcome monitoring (ROM) provides information to practitioners and others providing healthcare support to demonstrate the impact of interventions and for service evaluation [...].

Feasibility of a Mobile Health App for Routine Outcome Monitoring and Feedback in SMART Recovery Mutual Support Groups: Stage 1 Mixed Methods Pilot Study

Background

Mutual support groups are an important source of long-term help for people impacted by addictive behaviors. Routine outcome monitoring (ROM) and feedback are yet to be implemented in these settings. SMART Recovery mutual support groups focus on self-empowerment and use evidence-based techniques (eg, motivational and behavioral strategies). Trained facilitators lead all SMART Recovery groups, providing an opportunity to implement ROM.

Objective

The aim of this stage 1 pilot study is to explore the feasibility, acceptability, and preliminary outcomes of a novel, purpose-built mobile health ROM and feedback app (SMART Track) in mutual support groups coordinated by SMART Recovery Australia (SRAU) over 8 weeks.

Methods

SMART Track was developed during phase 1 of this study using participatory design methods and an iterative development process. During phase 2, 72 SRAU group participants were recruited to a nonrandomized, prospective, single-arm trial of the SMART Track app. Four modes of data collection were used: ROM data directly entered by participants into the app; app data analytics captured by Amplitude Analytics (number of visits, number of unique users, visit duration, time of visit, and user retention); baseline, 2-, and 8-week follow-up assessments conducted through telephone; and qualitative telephone interviews with a convenience sample of study participants (20/72, 28%) and facilitators (n=8).

Results

Of the 72 study participants, 68 (94%) created a SMART Track account, 64 (88%) used SMART Track at least once, and 42 (58%) used the app for more than 5 weeks. During week 1, 83% (60/72) of participants entered ROM data for one or more outcomes, decreasing to 31% (22/72) by the end of 8 weeks. The two main screens designed to provide personal feedback data (Urges screen and Overall Progress screen) were the most frequently visited sections of the app. Qualitative feedback from participants and facilitators supported the acceptability of SMART Track and the need for improved integration into the SRAU groups. Participants reported significant reductions between the baseline and 8- week scores on the Severity of Dependence Scale (mean difference 1.93, SD 3.02; 95% CI 1.12-2.73) and the Kessler Psychological Distress Scale-10 (mean difference 3.96, SD 8.31; 95% CI 1.75-6.17), but no change on the Substance Use Recovery Evaluator (mean difference 0.11, SD 7.97; 95% CI –2.02 to 2.24) was reported.

Conclusions

Findings support the feasibility, acceptability, and utility of SMART Track. Given that sustained engagement with mobile health apps is notoriously difficult to achieve, our findings are promising. SMART Track offers a potential solution for ROM and personal feedback, particularly for people with substance use disorders who attend mutual support groups.

Trial Registration

Australian New Zealand Clinical Trials Registry ACTRN12619000686101; https://anzctr.org.au/Trial/Registration/TrialReview.aspx?id=377336

International Registered Report Identifier (IRRID)

RR2-10.2196/15113

Perceived benefits and limitations of using patient-reported outcome measures in clinical practice with individual patients: a systematic review of qualitative studies

PURPOSE

Patient-reported outcome measures (PROMs) are increasingly used in clinical settings to inform individual patient care. In-depth understanding of end-users' experiences may help identify factors that promote or hinder their use in clinical decision-making. We aimed to examine stakeholder perceptions of the utility of using PROMs in clinical practice based on real-life experience.

METHODS

Systematic review searching Medline, Embase and PsychINFO from inception to May 2021. Qualitative studies examining patients' and/or clinicians' experiences of using PROMs in clinical settings were included. Study screening and data extraction were performed by two independent reviewers. Qualitative data from included studies was analysed thematically.

RESULTS

Of 2388 abstracts retrieved, 52 articles reporting 50 studies met eligibility. Five key benefits were identified: (1) promotes active patient involvement (enables goal setting and discussion of sensitive topics); (2) enhances the focus of consultations (prioritizes patient needs); (3) improves quality of care (enables tailored, holistic care and prompts action); (4) enables standardized monitoring of patient outcomes; and (5) enhances the patient-clinician relationship (provides reassurance). Perceived limitations included the capacity of PROMs to shift the focus of consultations; inaccurately estimate problems; raise unrealistic expectations for care; inhibit patient-clinician interaction; lack clinically meaningful information; and not be suitable for all patients.

CONCLUSION

Both patients and clinicians reported benefits of using PROMs across diverse health conditions and clinical settings, but also highlighted several limitations. These limitations shed some light on why PROM use may not always improve patient outcomes and provide considerations for the design and implementation of future PROM initiatives.

Patient, family and provider views of measurement-based care in an early-psychosis intervention programme

Background

Measurement-based care (MBC) in mental health improves patient outcomes and is a component of many national guidelines for mental healthcare delivery. Nevertheless, MBC is not routinely integrated into clinical practice. Several known reasons for the lack of integration exist but one lesser explored variable is the subjective perspectives of providers and patients about MBC. Such perspectives are critical to understand facilitators and barriers to improve the integration of MBC into routine clinical practice.

Aims

This study aimed to uncover the perspectives of various stakeholders towards MBC within a single treatment centre.

Method

Researchers conducted qualitative semi-structured interviews with patients (n = 15), family members (n = 7), case managers (n = 8) and psychiatrists (n = 6) engaged in an early-psychosis intervention programme. Data were analysed using thematic analysis, informed by critical realist theory.

Results

Analysis converged on several themes. These include (a) implicit negative assumptions; (b) relevance and utility to practice; (c) equity versus flexibility; and (d) shared decision-making. Providers assumed patients’ perspectives of MBC were negative. Patients’ perspectives of MBC were actually favourable, particularly if MBC was used as an instrument to engage patients in shared decision-making and communication rather than as a dogmatic and rigid clinical decision tool.

Conclusions

This qualitative study presents the views of various stakeholders towards MBC, providing an in-depth examination of the barriers and facilitators to MBC through qualitative investigation. The findings from this study should be used to address the challenges organisations have experienced in implementing MBC.

Improving the Quality of Children's Mental Health Care with Progress Measures: A Mixed-Methods Study of PCIT Therapist Attitudes

Progress measures are an evidence-based technique for improving the quality of mental health care, however, clinicians rarely incorporate them into treatment. Research into how measure type impacts clinician preference has been recommended to help improve measure implementation. Parent–Child Interaction Therapy (PCIT) is an assessment-driven treatment that serves as an ideal intervention through which to investigate measure preferences given its routine use of two types of assessments, a behavioral observation (the Dyadic Parent–Child Interaction Coding System) and a parent-report measure (the Eyberg Child Behavior Inventory). This study investigated PCIT therapist attitudes towards progress measures used within PCIT and children’s mental health treatment generally. A mixed-method (QUAN + QUAL) study design examined PCIT therapist attitudes towards two types of progress measures and measures used in two contexts (PCIT and general practice). Multi-level modeling of a survey distributed to 324 PCIT therapists identified predictors of therapist attitudes towards measures, while qualitative interviews with 23 therapists expanded and clarified the rationale for differing perceptions. PCIT therapists reported more positive attitudes towards a behavioral observation measure, the DPICS, than a parent-report measure, the ECBI, and towards measures used in PCIT than in general practice. Clinician race/ethnicity was significantly related to measure-specific attitudes. Qualitative interviews highlighted how perceptions of measure reliability, type of data offered, ease of use, utility in guiding sessions and motivating clients, and embeddedness in treatment protocol impact therapist preferences. Efforts to implement progress monitoring should consider preferences for particular types of measures, as well as how therapists are trained to embed measures in treatment.

Applying behaviour change theory to understand the barriers to implementing routine outcome monitoring

BACKGROUND

Routine outcome monitoring (ROM) is a valuable tool for monitoring client progress and pre-empting deterioration, however, there is considerable variation in how data are collected and recorded and uptake in clinical practice remains low. The aim of this study was to develop a self-report measure of practitioner attitudes to ROM in order to better understand the barriers to successful implementation in Child and Adolescent Mental Health Services (CAMHS).

METHODS

An anonymous survey was completed by 184 CAMHS practitioners in the United Kingdom. The survey was designed using the Capability, Opportunity, and Motivation Model of Behaviour (COM-B). Practitioners who reported using ROM frequently in their clinical work (53%) were compared to those who used ROM infrequently (47%) across dimensions of the COM-B survey subscales.

RESULTS

Confirmatory factor analysis confirmed the proposed four-factor structure, showing acceptable model fit, with high factor loadings and good reliability for all subscales. Frequent users of ROM exhibited significantly higher psychological capability, physical opportunity, social opportunity, and motivation, compared to infrequent users F (4, 140) = 14.76, p < .0001; Pillai's Trace = .297, partial η²  = .30. Results highlight several barriers to ROM, including the belief that there is not a strong evidence base for ROM, not receiving external training, and not discussing feedback and outcome data in supervision.

IMPLICATIONS

In the hope of improving the successful implementation of ROM, this research provides an evidence-based tool for assessing practitioners' attitudes to ROM, which map on to intervention functions and represent targets for future implementation efforts.

PRACTITIONER POINTS

The value of routine outcome monitoring (ROM) as a means to measure client progress and to elevate the efficiency and quality of mental health care is well-documented in the research literature, however, uptake in practice remains relatively low. This study applied behaviour change theory to develop a psychometrically sound self-report measure of practitioners' perspectives and practices to understand the barriers to implementation in child and adolescent mental health services in the United Kingdom. The complex and multifaceted nature of the barriers to implementation requires multilevel behaviour change strategies at the client, clinician, and organisational level. Recommendations for practice include the need for integrated, multilevel strategies aimed at improving practitioners' capabilities and motivations, strong organisational leadership and a culture of data gathering and sharing, and implementation interventions, which are tailored to target local barriers.

Inside the Clockwork of the ECHO Factorial Trial: A Conceptual Model With Proposed Mediators for Prevention of Emotional Problems in Children

Having interventions that are not only evidence-based and effective but also cost-effective and efficient is important for the prevention and treatment of child and adolescent emotional problems. A randomized clinical trial (RCT) tests the total interventions effect but does not address specific components of the intervention. In this article the hypothesis and a conceptual model of the ECHO study are presented and discussed. The ECHO intervention consists of three different components each containing two levels of intervention. By using a cluster randomized factorial design, children aged 8-12 at 40 schools across Norway will be randomized to eight different experimental conditions investigating the optimal balance between effect, cost-effectiveness, and efficiency. The article presents the design and the different components being tested and discusses how optimalization can be reached through this innovative design. The article also discusses how interventions can be improved by investigating and understanding the mechanisms of change within psychological interventions. For each of the three components in the study we consider the mediators that could be active within the intervention and how the study investigates such mediation. The results will contribute to a better understanding of how psychological interventions work and how we intend to optimize the EMOTION intervention.

Developing a mHealth Routine Outcome Monitoring and Feedback App ("SMART Track") to Support Self-Management of Addictive Behaviours

Background: Routine outcome monitoring (ROM) has been implemented across a range of addiction treatment services, settings and organisations. Mutual support groups are a notable exception. Innovative solutions are needed. SMART Track is a purpose built smartphone app designed to capture ROM data and provide tailored feedback to adults attending Australian SMART Recovery groups for addictive behaviour(s). Objective: Details regarding the formative stage of app development is essential, but often neglected. Improved consideration of the end-user is vital for curtailing app attrition and enhancing engagement. This paper provides a pragmatic example of how principles embedded in published frameworks can be operationalised to address these priorities during the design and development of the SMART Track app. Methods: Three published frameworks for creating digital health technologies ("Person-Based Approach," "BIT" Model and IDEAS framework) were integrated and applied across two stages of research to inform the development, design and content of SMART Track. These frameworks were chosen to ensure that SMART Track was informed by the needs and preferences of the end-user ("Person-Based"); best practise recommendations for mHealth development ("BIT" Model) and a collaborative, iterative development process between the multi-disciplinary research team, app developers and end-users (IDEAS framework). Results: Stage one of the research process generated in-depth knowledge to inform app development, including a comprehensive set of aims (clinical, research/organisation, and usage); clear articulation of the target behaviour (self-monitoring of recovery related behaviours and experiences); relevant theory (self-determination and social control); appropriate behavioural strategies (e.g., behaviour change taxonomy and process motivators) and key factors that may influence engagement (e.g., transparency, relevance and trust). These findings were synthesised into guiding principles that were applied during stage two in an iterative approach to app design, content and development. Conclusions: This paper contributes new knowledge on important person-centred and theoretical considerations that underpin a novel ROM and feedback app for people with addictive behaviour(s). Although person-centred design and best-practise recommendations were employed, further research is needed to determine whether this leads to improved usage outcomes. Clinical Trial Registration: Pilot Trial: http://anzctr.org.au/Trial/Registration/TrialReview.aspx?id=377336.

Psychological recovery in a step 4 service: a qualitative study exploring the views of service users and clinicians

BACKGROUND

Psychological recovery is typically measured in improving access to psychological therapies (IAPT) services as the reduction of symptoms on routine outcome measures (ROMs). However, conceptualisations of psychological recovery vary, and there are concerns that ROMs may not be suitable for measuring recovery from complex mental health difficulties.

AIM

To gain insight into psychological recovery and measurement in clinical practice.

METHOD

A qualitative study at a step-4 IAPT service in the North of England. Eight clinicians contributed to focus groups, and 10 service users were later interviewed individually.

RESULTS

Factors that informed four congruent and interacting themes of recovery were identified among participants as an ongoing personal process, of intra-therapeutic and extra-therapeutic factors impacting on recovery in helpful and unhelpful ways, and the need for idiographic long-term recovery measures. The functional and long-term nature of recovery and the socio-economic context for recovery were considered particularly crucial.

CONCLUSIONS

The need for idiographic long-term recovery measures to support the ongoing process of recovery was reported in the majority of instances. Study findings mirrored existing recovery and outcome measurement literature, provided new evidence in relation to recovery from complex mental health difficulties, and made recommendations for future research and clinical practice.

How therapists and patients need to develop a clinical feedback system after 18 months of use in a practice-research network: a qualitative study

BACKGROUND

A personalized computer-adaptive system for clinical feedback and routine outcome monitoring in mental health, Norse Feedback aims to bridge the needs for standardized and idiographic measures in ordinary practice.

METHODS

Item response theory analyses of completed treatment processes (n = 800) informed a qualitative study comprising individual in-depth interviews and focus groups with patients (n = 9) and clinicians (n = 10). The research question was: How do clinicians and patients contribute to developing a clinical feedback system in a continuous process aimed at refining its clinical usefulness?

RESULTS

We conducted thematic analyses and found five themes: 1. Added clinical needs, 2. Needs for re-organizing the clinician report, 3.Needs for differentiation of clinical content, 4. User-interface needs, and 5. Item level suggestions.

CONCLUSION

In this article, we detail resulting needs for continuous adaptation to practice, and discuss implications of the concrete experiences with the Norse action research program for the larger field of ROM/CFS implementation.

Initial quantitative development of the Norse Feedback system: a novel clinical feedback system for routine mental healthcare

Purpose

As routine outcome monitoring has become prevalent in psychological practice, there is need for measurement tools covering diverse symptoms, treatment processes, patient strengths, and risks. Here we describe the development and initial tests of the psychometric properties of a multi-scale system for use in mental healthcare, Norse Feedback.

Methods

In Study 1, we present the item-generation process and structure of the Norse Feedback, a 17-scale digital-first measurement tool for psychopathology and treatment-relevant variables. In Study 2, we present analyses of this initial measure in a nonclinical sample of 794 healthy controls and a sample of 222 mental health patients. In Study 3, we present the analysis of a revised 20-scale system in two separate samples of patients. In each analysis, we investigate item and test information in particular, including analysis of differential item functioning on gender, age, site, and sample differences where applicable.

Results

Scales performed variably. Changes to items and scales are described. Several scales appeared to reliably discriminate individuals entering mental health treatment on severity, and others are less reliable. Marked improvements in scale internal consistency and measurement precision were observed between the first and second implemented versions.

Conclusion

This system includes some scales with reasonable structural validity, though several areas for future development are identified. The system was developed to be iteratively re-evaluated, to strengthen the validity of its scales over time. There are currently a number of limitations on inferences from these scores, which future developments should address.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11136-021-02825-1.

Barriers and Facilitators in Adolescent Psychotherapy Initiated by Adults-Experiences That Differentiate Adolescents' Trajectories Through Mental Health Care

Mental health problems start early in life. However, the majority of adolescents fulfilling the criteria for mental health disorders do not receive treatment, and half of those who do get treatment drop out. This begs the question of what differentiates helpful from unhelpful treatment processes from the perspective of young clients. In this study, we interviewed 12 young people who entered mental health care reluctantly at the initiative of others before the age of 18. Their journeys through mental health care varied significantly despite sharing the same starting point. Our analyses resulted in a model of three trajectories. We describe relational and structural facilitators and obstacles within each trajectory and have formulated narratives highlighting core experiences differentiating them. Trajectory 1 (I never saw the point - Being met as a case) was characterized by a rapid loss of hope, leading the adolescents to conclude that mental health care was not worth the investment. Trajectory 2 (I gave it a go, but nothing came of it - Being met by a therapist representing a rigid and unhelpful system) was characterized by a lingering hope that never materialized into a constructive therapeutic process despite prevailing efforts by both therapists and adolescents. Trajectory 3 (Something good came of it - Being met by a therapist who cares and wants to help) was characterized by genuine meetings, allowing the therapist to transform from an unsafe stranger into a safe, competent, and benevolent adult. We discuss how our results have implications for understanding agency displayed by adolescent clients in therapy, therapist flexibility and authenticity, service organization, and attributional processes influencing clinical judgment and therapeutic processes when adolescent psychotherapy has a difficult starting point (i.e., initiated by adults).

Discussing measurement-based care with patients: An analysis of clinician-patient dyads

Abstract Measurement-based care (MBC) refers to the use of three integrated strategies to improve effectiveness of behavioral health care: routine outcomes monitoring using symptom measures; regularly sharing these data with patients; and using these data to inform treatment decisions. This study examined how clinicians discuss MBC data with patients, including identifying what aspects of these discussions contribute to clinician-patient agreement on the value of MBC, and how clinicians use MBC data to inform treatment decisions. Twenty-six clinician-patient dyads participated in semi-structured interviews and provided a treatment session recording in which MBC data were discussed. Qualitative data analyses revealed four subtypes of dyads: clinician and patient both valued MBC; clinician valued MBC, patient passively participated in MBC; clinician valued MBC, patient had mixed perceptions of MBC; clinician and patient reported moderate or low value for MBC. In dyads for whom both the clinician and patient valued MBC, the clinician provided clear and repeated rationale for MBC, discussed data with patients at every administration, and connected observed scores to patient skills or strategies. Emerging best practices for discussing MBC include providing a strong rationale, discussing results frequently, actively engaging patients in discussions, and using graphs to visualize progress.

Clinical dilemmas of routine outcome monitoring and clinical feedback: A qualitative study of patient experiences

ABSTRACT Purpose: Routine outcome monitoring (ROM) and clinical feedback systems (CFS) are becoming prevalent in mental health services, but there are several challenges to successful implementation. ROM/CFS seem to be helpful for some patients, but not for others. To investigate this, we explored patients' experiences with ROM/CFS as an interpersonal and psychotherapeutic process, in naturalistic settings. Method: We used video-assisted interpersonal process recall interviews to investigate the experiences of 12 patients using ROM/CFS in a Norwegian mental health outpatient clinic. Data were analyzed through systematic text condensation. Results: Our analysis resulted in three pairs of experiences with ROM/CFS: (1) Explicit vs. implicit use of information, (2) Directing focus towards or away from therapeutic topics, and (3) Giving and receiving feedback. These experiences could be helpful or hindering, depending on participants' needs and preferences. All participants needed to know that the CFS was used in a meaningful way. If not, it could be detrimental to the therapeutic process. Conclusion: In order to be helpful for patients, ROM/CFS should be used in a way that is flexible, meaningful to patients, and sensitive to individual needs and preferences. Future research should further explore this how-to aspect of ROM/CFS with different CFS and populations.

Complexity and potentials of clinical feedback in mental health: an in-depth study of patient processes

Purpose

Routine outcome monitoring (ROM) and clinical feedback systems (CFS) are becoming increasingly prevalent in mental health services. Their overall efficacy is unclear, but quantitative evidence suggests they can be useful tools for preventing treatment failure and enhancing therapeutic outcomes, especially for patients who are not progressing in therapy. The body of qualitative material, however, is smaller and less refined. We need to know more about how ROM/CFS is used in psychotherapy, and why it is helpful for some patients, but not others.

Methods

We recorded therapy sessions of 12 patients who were using a CFS as part of their therapies at an outpatient clinic in Norway. We then conducted video-assisted interviews and follow-up interviews with patients. Data were analyzed with systematic text condensation.

Results

Analysis revealed three themes: (1) triggering reflections, emotions, and self-awareness, (2) Ambivalent and ambiguous self-presentation, and (3) potential for feeling understood and talking about what matters.

Conclusion

Answering questions in a CFS is an interpretative and intentional process of self-presentation and the results from ROM/CFS must be interpreted and explored in conversation to be clinically useful. When they are, they have potential for enhancing the therapeutic process by stimulating self-awareness, reflexivity, and allowing access to new therapeutic topics. Further research should explore this how-to aspect of ROM/CFS with different CFS and different types of patients. Integrating clinical feedback in therapeutic practice can be conceptualized as a clinical skill, which should be a part of training programs for therapists.

Electronic supplementary material

The online version of this article (10.1007/s11136-020-02550-1) contains supplementary material, which is available to authorized users.

"Becoming myself": how participants in a longitudinal substance use disorder recovery study experienced receiving continuous feedback on their results

BACKGROUND

Being a participant in longitudinal follow-up studies is not commonly a factor considered when investigating useful self-change aspects for individuals attempting recovery from substance use disorder (SUD). This study reports on how ongoing monitoring, and feedback on data results in a longitudinal follow-up study of SUD recovery were perceived by individuals who had achieved long-term abstinence and social recovery.

METHODS

Interviewers with first-hand experience with the topic conducted interviews with 30 participants and analysed the data using a thematic analytic approach within an interpretative-phenomenological framework.

RESULTS

Analyses resulted in the following themes. 1) Ongoing short text messaging (SMS) monitoring: helped participants by offering recovery milestones and reminders of the past. 2) Feedback on data results helped participants track physical and cognitive recovery: "I am more like myself". 3) Using feedback in treatment: understanding the importance of a functional brain to participants may help with long-term retention in treatment.

CONCLUSIONS

Self-changes that were challenging to detect on a day-to-day basis were available for reflection through longitudinal study participation, including ongoing monitoring and feedback on the results, allowing personal consolidation of change processes. Clinical services could benefit from continuing development and implementation of such technology for ongoing monitoring and feedback on assessments to motivate self-change in SUD recovery. The development of guidelines for providing the results of research assessments to individuals could help reduce attrition in research projects and support recovery and healthy choices for study participants.

What are "good outcomes" in public mental health settings? A qualitative exploration of clients' and therapists' experiences

BACKGROUND

The mental health field sees a surge of interest in Routine Outcome Monitoring, mandated by a wish to help better those not-on-track to recovery. What constitutes positive outcomes for these patients is not fully understood.

AIMS

To contribute knowledge into what constitutes meaningful outcome concepts in the experiences of patients with long and complex mental health suffering and treatment, and the clinicians who work to help them.

METHODS

A qualitative in-depth study of 50 participants' experiences. Data are collected through focus groups and individual interviews, and analyzed using a team based structured thematic analytic approach.

RESULTS

We found an overarching theme of outcome as an ongoing process of recovery, with the four constituent themes: (1) strengthening approach patterns for new coping; (2) embodying change reflected by others; (3) using new understandings developed in dialogue; and (4) integrating collaborative acceptance.

CONCLUSIONS

We discuss our findings in light of existing empirical studies and different recovery concepts, and suggest that if outcomes monitoring is to become an integral part of routine practice, it might be beneficial to integrate an understanding of outcomes as ongoing processes of recovery within mental health suffering into these systems.