Methodological background of decision rules and feedback tools for outcomes management in psychotherapy

Prediction of adolescent psychotherapy outcomes using youth- and caregiver-reported symptoms data

OBJECTIVE

We used longitudinal youth- and caregiver-reports of adolescent psychological symptoms from three samples of youth receiving mental health services in routine treatment settings to derive expected change trajectories and identify cases at risk for treatment failure.

METHOD

Participants were 1906 youth (1053 caregivers) receiving treatment in community mental health settings, merged across three samples. The Symptoms and Functioning Severity Scale (SFSS) was used as an indicator of weekly clinical change. Multilevel modeling methods were used to develop expected change trajectories and identify cases at risk for treatment failure (not on track; NOT). Logistic regression was used to predict client improvement as a function of NOT status.

RESULTS

The SFSS was a reliable indicator of therapeutic change according to youth-reported symptoms. Caregiver reports were not as robust. Whereas predictive accuracy of NOT status yielded moderately high sensitivity in detecting improvement according to youth report, caregiver reports were not as predictive.

CONCLUSIONS

The youth-reported version of the SFSS-based algorithm seems appropriate for implementation in clinical care. Future studies should search for similarly predictive measures for caregivers.

Bridging the feedback implementation gap: A comparison of empirical and rational decision rules in naturalistic psychotherapy

Objective: Previous research indicates positive effects of feedback based on rational or empirical decision rules in psychotherapy. The implementation of these usually session-to-session-based feedback systems into clinical practice, however, remains challenging. This study aims to evaluate decision rules based on routine outcome monitoring with reduced assessment frequency. Method: Data routinely collected every 5-20 sessions of N = 3758 patients treated with CBT in an outpatient clinic (Msessions = 42.8, SD = 15.4) were used to develop feedback decision rules based on the expected treatment response and nearest neighbors approach, the reliable change index, and method of percental improvement. The detection of patients at risk of treatment failure served as primary endpoint. Results: Significantly lower reliable improvement, higher reliable deterioration rates, and smaller effect sizes were found for patients identified at risk of treatment failure by all rules. The nearest neighbors-based approach showed the highest sensitivity regarding the detection of reliably deteriorated cases. Conclusion: Consistent with previous research, the empirical models outperformed the rational rules. Still, the first-time used percental improvement-based rule also showed satisfactory results. Overall, the results point to the potential of basic feedback systems that might be easier to implement in practice than session-to-session based systems.

Natural language processing for mental health interventions: a systematic review and research framework

Neuropsychiatric disorders pose a high societal cost, but their treatment is hindered by lack of objective outcomes and fidelity metrics. AI technologies and specifically Natural Language Processing (NLP) have emerged as tools to study mental health interventions (MHI) at the level of their constituent conversations. However, NLP's potential to address clinical and research challenges remains unclear. We therefore conducted a pre-registered systematic review of NLP-MHI studies using PRISMA guidelines (osf.io/s52jh) to evaluate their models, clinical applications, and to identify biases and gaps. Candidate studies (n = 19,756), including peer-reviewed AI conference manuscripts, were collected up to January 2023 through PubMed, PsycINFO, Scopus, Google Scholar, and ArXiv. A total of 102 articles were included to investigate their computational characteristics (NLP algorithms, audio features, machine learning pipelines, outcome metrics), clinical characteristics (clinical ground truths, study samples, clinical focus), and limitations. Results indicate a rapid growth of NLP MHI studies since 2019, characterized by increased sample sizes and use of large language models. Digital health platforms were the largest providers of MHI data. Ground truth for supervised learning models was based on clinician ratings (n = 31), patient self-report (n = 29) and annotations by raters (n = 26). Text-based features contributed more to model accuracy than audio markers. Patients' clinical presentation (n = 34), response to intervention (n = 11), intervention monitoring (n = 20), providers' characteristics (n = 12), relational dynamics (n = 14), and data preparation (n = 4) were commonly investigated clinical categories. Limitations of reviewed studies included lack of linguistic diversity, limited reproducibility, and population bias. A research framework is developed and validated (NLPxMHI) to assist computational and clinical researchers in addressing the remaining gaps in applying NLP to MHI, with the goal of improving clinical utility, data access, and fairness.

Identification of psychiatric inpatient recovery trajectories using routine outcome monitoring with emerging adults

Early intervention for emerging adults with addiction and mental health disorders is beneficial for long-term recovery. The present study investigated the utility of routine outcome monitoring during acute inpatient hospitalization for identifying emerging adults at risk of poor outcomes. This is a retrospective study using latent class growth analysis (LCGA) to identify patient groups with different recovery trajectories, with additional analyses to clarify the characteristics of these trajectory groups. The results identified four patient groups: Rapid responders (38%), gradual responders (34%), high distress non-responders (9%), and low distress non-responders (19%). The high distress non-responding group is characterized by behaviours and disorders associated with ambivalent care seeking: Voluntary admission, longer length of stay, lower service satisfaction, higher outpatient service utilization, elevated risk of emergency department presentation and hospital readmission, and depression/personality disorder diagnosis. The low distress group is characterized by behaviours and disorders associated with treatment rejection: Involuntary admission, shorter length of stay, reduced post-discharge service utilization, and psychotic disorder diagnosis. The results have implications for identifying at-risk youth and developing stepped-care models for more effective and efficient inpatient care.

How Routine Patient-Centered Monitoring Relates to Therapeutic Gains in Family Therapy: A Single-Case Study

As routine outcome monitoring systems develop, questions emerge about how therapists incorporate feedback into their practice, and how this relates to therapeutic gains. A case of covert grief was monitored in each session with the Personal Questionnaire and the Helpful Aspects of Therapy instruments. At 4 months follow-up, the Change Interview was administered. Individualized items facilitated access to the private views and needs of each member, which was useful for case formulation and ongoing personalization of the intervention. Qualitative feedback of treatment experiences helped therapists confirm the impacts of interventions, monitor therapeutic alliance, reformulate clinical hypothesis, and plan sessions. Therapists followed a critical triangulation process to decide the clinical meaning of feedback, according to his/her intervention model, expertise, and case-specific context.

What Is a Clinically Relevant Improvement in Quality of Life in Adults With ADHD?

OBJECTIVE

To estimate a minimal clinically important difference (MCID) on the adult ADHD Quality of Life (AAQoL) scale.

METHOD

The MCID was determined from data from short-term ( N = 537) and long-term ( N = 440), placebo-controlled atomoxetine trials in adults with ADHD. For the anchor-based approach, change in clinician-rated Clinical Global Impressions-ADHD-Severity (CGI-ADHD-S) scores was used to derive MCID. For the distribution-based approach, baseline-to-endpoint mean ( SD) changes in AAQoL scores corresponding to 0.5 SD were computed.

RESULTS

The MCID was similar (approximately 8-point difference) between the short-term and the long-term treatment groups when either the anchor-based or distribution-based approach was used.

CONCLUSION

These results suggest that approximately 8 points in the change from baseline on the AAQoL is a MCID.

Treatment results for severe psychiatric illness: which method is best suited to denote the outcome of mental health care?

BACKGROUND

The present study investigates the suitability of various treatment outcome indicators to evaluate performance of mental health institutions that provide care to patients with severe mental illness. Several categorical approaches are compared to a reference indicator (continuous outcome) using pretest-posttest data of the Health of Nation Outcome Scales (HoNOS).

METHODS

Data from 10 institutions and 3189 patients were used, comprising outcomes of the first year of treatment by teams providing long-term care.

RESULTS

Findings revealed differences between continuous indicators (standardized pre-post difference score ES and ΔT) and categorical indicators (SEM, JTRCI, JTCS, JTRCI&CS, JTrevised) on their ranking of institutions, as well as substantial differences among categorical indicators; the outcome according to the traditional JT approach was most concordant with the continuous outcome indicators.

CONCLUSIONS

For research comparing group averages, a continuous outcome indicator such as ES or ΔT is preferred, as this best preserves information from the original variable. Categorical outcomes can be used to illustrate what is accomplished in clinical terms. For categorical outcome, the classical Jacobson-Truax approach is preferred over the more complex method of Parabiaghi et al. with eight outcome categories. The latter may be valuable in clinical practice as it allows for a more detailed characterization of individual patients.

Trauma-related emotions and radical acceptance in dialectical behavior therapy for posttraumatic stress disorder after childhood sexual abuse

BACKGROUND

Posttraumatic Stress Disorder (PTSD) related to childhood sexual abuse (CSA) is often associated with a wide range of trauma-related aversive emotions such as fear, disgust, sadness, shame, guilt, and anger. Intense experience of aversive emotions in particular has been linked to higher psychopathology in trauma survivors. Most established psychosocial treatments aim to reduce avoidance of trauma-related memories and associated emotions. Interventions based on Dialectical Behavior Therapy (DBT) also foster radical acceptance of the traumatic event.

METHODS

This study compares individual ratings of trauma-related emotions and radical acceptance between the start and the end of DBT for PTSD (DBT-PTSD) related to CSA. We expected a decrease in trauma-related emotions and an increase in acceptance. In addition, we tested whether therapy response according to the Clinician Administered PTSD-Scale (CAPS) for the DSM-IV was associated with changes in trauma-related emotions and acceptance. The data was collected within a randomized controlled trial testing the efficacy of DBT-PTSD, and a subsample of 23 women was included in this secondary data analysis.

RESULTS

In a multilevel model, shame, guilt, disgust, distress, and fear decreased significantly from the start to the end of the therapy whereas radical acceptance increased. Therapy response measured with the CAPS was associated with change in trauma-related emotions.

CONCLUSIONS

Trauma-related emotions and radical acceptance showed significant changes from the start to the end of DBT-PTSD. Future studies with larger sample sizes and control group designs are needed to test whether these changes are due to the treatment.

TRIAL REGISTRATION

ClinicalTrials.gov, number NCT00481000.

Improving the efficiency of cognitive-behavioural therapy by using formal client feedback

OBJECTIVE

Feedback from clients on their view of progress and the therapeutic relationship can improve effectiveness and efficiency of psychological treatments in general. However, what the added value is of client feedback specifically within cognitive-behavioural therapy (CBT), is not known. Therefore, the extent to which the outcome of CBT can be improved is investigated by providing feedback from clients to therapists using the Outcome Rating Scale (ORS) and Session Rating Scale (SRS).

METHOD

Outpatients (n = 1006) of a Dutch mental health organization either participated in the "treatment as usual" (TAU) condition, or in Feedback condition of the study. Clients were invited to fill in the ORS and SRS and in the Feedback condition therapists were asked to frequently discuss client feedback.

RESULTS

Outcome on the SCL-90 was only improved specifically with mood disorders in the Feedback condition. Also, in the Feedback condition, in terms of process, the total number of required treatment sessions was on average two sessions fewer.

CONCLUSION

Frequently asking feedback from clients using the ORS/SRS does not necessarily result in a better treatment outcome in CBT. However, for an equal treatment outcome significantly fewer sessions are needed within the Feedback condition, thus improving efficiency of CBT.

Alliance: a common factor of psychotherapy modeled by structural theory

There is broad consensus that the therapeutic alliance constitutes a core common factor for all modalities of psychotherapy. Meta-analyses corroborated that alliance, as it emerges from therapeutic process, is a significant predictor of therapy outcome. Psychotherapy process is traditionally described and explored using two categorically different approaches, the experiential (first-person) perspective and the behavioral (third-person) perspective. We propose to add to this duality a third, structural approach. Dynamical systems theory and synergetics on the one hand and enactivist theory on the other together can provide this structural approach, which contributes in specific ways to a clarification of the alliance factor. Systems theory offers concepts and tools for the modeling of the individual self and, building on this, of alliance processes. In the enactive perspective, the self is conceived as a socially enacted autonomous system that strives to maintain identity by observing a two-fold goal: to exist as an individual self in its own right (distinction) while also being open to others (participation). Using this conceptualization, we formalized the therapeutic alliance as a phase space whose potential minima (attractors) can be shifted by the therapist to approximate therapy goals. This mathematical formalization is derived from probability theory and synergetics. We draw the conclusion that structural theory provides powerful tools for the modeling of how therapeutic change is staged by the formation, utilization, and dissolution of the therapeutic alliance. In addition, we point out novel testable hypotheses and future applications.

Factors of psychological distress: clinical value, measurement substance, and methodological artefacts

PURPOSE

Psychometric models and statistical techniques are cornerstones of research into latent structures of specific psychopathology and general mental health. We discuss "pivot points" for future research efforts from a psychometric epidemiology perspective, emphasising sampling and selection processes of both indicators that guide data collection as well as samples that are confronted with them.

METHOD

First, we discuss how a theoretical model of psychopathology determines which empirical indicators (questions, diagnoses, etc.) and modelling methods are appropriate to test its implications. Second, we deal with how different research designs introduce different (co-)variances between indicators, potentially leading to a different understanding of latent structures. Third, we discuss widening the range of statistical models available within the "psychometrics class": the inclusion of categorical approaches can help to enlighten the debate on the structure of psychopathology and agreement on a minimal set of models might lead to greater convergence between studies. Fourth, we deal with aspects of methodology that introduce spurious (co-)variance in latent structure analysis (response styles, clustered data) and differential item functioning to gather more detailed information and to guard against over-generalisation of results, which renders assessments unfair.

CONCLUSIONS

Building on established insights, future research efforts should be more explicit about their theoretical understanding of psychopathology and how the analysis of a given indicator-respondent set informs this theoretical model. A coherent treatment of theoretical assumptions, indicators, and samples holds the key to building a comprehensive account of the latent structures of different types of psychopathology and mental health in general.

Advancing psychotherapy and evidence-based psychological interventions

Psychological models of mental disorders guide research into psychological and environmental factors that elicit and maintain mental disorders as well as interventions to reduce them. This paper addresses four areas. (1) Psychological models of mental disorders have become increasingly transdiagnostic, focusing on core cognitive endophenotypes of psychopathology from an integrative cognitive psychology perspective rather than offering explanations for unitary mental disorders. It is argued that psychological interventions for mental disorders will increasingly target specific cognitive dysfunctions rather than symptom-based mental disorders as a result. (2) Psychotherapy research still lacks a comprehensive conceptual framework that brings together the wide variety of findings, models and perspectives. Analysing the state-of-the-art in psychotherapy treatment research, "component analyses" aiming at an optimal identification of core ingredients and the mechanisms of change is highlighted as the core need towards improved efficacy and effectiveness of psychotherapy, and improved translation to routine care. (3) In order to provide more effective psychological interventions to children and adolescents, there is a need to develop new and/or improved psychotherapeutic interventions on the basis of developmental psychopathology research taking into account knowledge of mediators and moderators. Developmental neuroscience research might be instrumental to uncover associated aberrant brain processes in children and adolescents with mental health problems and to better examine mechanisms of their correction by means of psychotherapy and psychological interventions. (4) Psychotherapy research needs to broaden in terms of adoption of large-scale public health strategies and treatments that can be applied to more patients in a simpler and cost-effective way. Increased research on efficacy and moderators of Internet-based treatments and e-mental health tools (e.g. to support "real time" clinical decision-making to prevent treatment failure or relapse) might be one promising way forward.

Diagnostik und psychometrische Verlaufsrückmeldungen im Rahmen eines Modellprojektes zur Qualitätssicherung in der ambulanten Psychotherapie

Zusammenfassung. Als erstes dieser Art in Deutschland zielt das Modellprojekt „Qualitätsmonitoring in der ambulanten Psychotherapie” darauf ab, ein modifiziertes Qualitätssicherungs- und Rückmeldeverfahren in der Regelversorgung der gesetzlichen Krankenkassen hinsichtlich seiner Einsetzbarkeit zu untersuchen. Im vorliegenden Beitrag werden die Aspekte der Eingangsdiagnostik, des Rückmeldesystems, der Patientenzufriedenheit, der Therapiedauer und der Evaluation des Projektes durch die Patienten vorgestellt. In der Interventionsgruppe (IG) wurden Checklisten zur Diagnosestellung genutzt und im Gegensatz zur Kontrollgruppe (KG) ebenfalls kontinuierliche Rückmeldungen zum Therapiefortschritt im Verlauf gegeben. Daten zu Therapiebeginn lagen von 1708 Patienten (IG:1031; KG:677) und 245 Therapeuten vor. Die ambulante Psychotherapie stellte sich als effektives Verfahren bei klar psychisch belasteten Personen heraus. In der IG (Einsatz Diagnosechecklisten) wurden im Vergleich zur Kontrollgruppe (KG) insgesamt mehr Diagnosen pro Patient durch die Therapeuten vergeben. Der überwiegende Teil der Therapeuten nutzte die Rückmeldungen für die Therapie und besprach die Ergebnisse mit den Patienten. Konsistent mit der Literatur zu Rückmeldungen in die Psychotherapiepraxis ließen sich frühzeitig im Therapieverlauf durch das Feedbacksystem prädiktive Verlaufsmuster identifizieren. Ferner war die Patientenzufriedenheit in der IG höher (Effektstärke: d = 0.27). Die Behandlungsdauer der IG lag im Rahmen der sich in den unterschiedlichen Kontrollgruppen der Versorgungspraxis ergebenden Behandlungsdauer. Die Zufriedenheit mit dem Modellprojekt ist bei den Patienten am höchsten. Die Patienten erlebten den Einsatz der Rückmeldungen bei vertretbarem Zeitaufwand als wichtig und hilfreich und mit der eigenen Einschätzung übereinstimmend. Die Befunde unterstreichen die Bedeutung von Qualitätssicherungs- und Rückmeldesystemen in der ambulanten Psychotherapie.

Lending an ear to feedback systems: evaluation of recovery and non-response in psychotherapy in a German outpatient setting

Systems providing feedback on treatment progress have been implemented in outpatient psychotherapy. They are recognized as a helpful tool to identify possible treatment failures. This report presents the ideas underlying the planning of feedback interventions and the implementation of such programs into practice settings. Strategies to identify patients at risk for treatment failure (rationally- and empirically-derived decision rules) are presented. Additionally, evidence for the usefulness of feedback systems is discussed. The report ends with the description of an ongoing feedback intervention study in private practices in Germany (aimed at gathering information on 400 therapists with 2,000 patients).