1
|
Youn SJ, Boswell JF, Douglas S, Harris BA, Aajmain S, Arnold KT, Creed TA, Gutner CA, Orengo-Aguayo R, Oswald JM, Stirman SW. Implementation Science and Practice-Oriented Research: Convergence and Complementarity. ADMINISTRATION AND POLICY IN MENTAL HEALTH AND MENTAL HEALTH SERVICES RESEARCH 2024; 51:336-347. [PMID: 37646966 DOI: 10.1007/s10488-023-01296-5] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 08/16/2023] [Indexed: 09/01/2023]
Abstract
Implementation science is the scientific study of methods to promote the uptake of research findings and other evidence-based practices in routine care, with the goal of improving the quality and effectiveness of health services (Bauer et al., 2015). In addition to this common goal, practice-oriented psychotherapy research (and researchers) and implementation science (and scientists) share a common focus on the people and the places where treatment happens. Thus, there exists strong potential for combining these two approaches. In this article, we provide a primer on implementation science for psychotherapy researchers and highlight important areas and examples of convergence and complementarity between implementation science and practice-oriented psychotherapy research. Specifically, we (a) define and describe the core features of implementation science; (b) discuss similarities and areas of complementarity between implementation science and practice-oriented psychotherapy research; (c) discuss a case example that exemplifies the integration of implementation science and practice-oriented research; and (d) propose directions for future research and collaborations that leverage both implementation science and practice-oriented research.
Collapse
Affiliation(s)
- Soo Jeong Youn
- Reliant Medical Group, OptumCare, Harvard Medical School, Worcester, MA, USA.
| | - James F Boswell
- Department of Psychology, University at Albany, State University of New York, Albany, NY, USA
| | - Susan Douglas
- Department of Leadership, Policy and Organizations, Vanderbilt University, Nashville, TN, USA
| | - Bethany A Harris
- Department of Psychology, University at Albany, State University of New York, Albany, NY, USA
| | - Syed Aajmain
- Department of Psychology, University at Albany, State University of New York, Albany, NY, USA
| | - Kimberly T Arnold
- Whole Health Equity Lab, Department of Family Medicine and Community Health, University of Pennsylvania Perelman School of Medicine, Philadelphia, USA
| | - Torrey A Creed
- Penn Collaborative for CBT and Implementation Science, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA, USA
| | - Cassidy A Gutner
- Boston University Chobanian & Avedisian School of Medicine, Boston, MA, USA
| | - Rosaura Orengo-Aguayo
- Medical University of South Carolina (MUSC), National Crime Victims Research & Treatment Center, Charleston, SC, USA
| | | | - Shannon Wiltsey Stirman
- Dissemination and Training Division, National Center for PTSD, VA Palo Alto Healthcare System, Department of Psychiatry and Behavioral Sciences, Stanford University, Menlo Park, CA, USA
| |
Collapse
|
2
|
Stade EC, Stirman SW, Ungar LH, Boland CL, Schwartz HA, Yaden DB, Sedoc J, DeRubeis RJ, Willer R, Eichstaedt JC. Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluation. NPJ MENTAL HEALTH RESEARCH 2024; 3:12. [PMID: 38609507 PMCID: PMC10987499 DOI: 10.1038/s44184-024-00056-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/24/2023] [Accepted: 01/30/2024] [Indexed: 04/14/2024]
Abstract
Large language models (LLMs) such as Open AI's GPT-4 (which power ChatGPT) and Google's Gemini, built on artificial intelligence, hold immense potential to support, augment, or even eventually automate psychotherapy. Enthusiasm about such applications is mounting in the field as well as industry. These developments promise to address insufficient mental healthcare system capacity and scale individual access to personalized treatments. However, clinical psychology is an uncommonly high stakes application domain for AI systems, as responsible and evidence-based therapy requires nuanced expertise. This paper provides a roadmap for the ambitious yet responsible application of clinical LLMs in psychotherapy. First, a technical overview of clinical LLMs is presented. Second, the stages of integration of LLMs into psychotherapy are discussed while highlighting parallels to the development of autonomous vehicle technology. Third, potential applications of LLMs in clinical care, training, and research are discussed, highlighting areas of risk given the complex nature of psychotherapy. Fourth, recommendations for the responsible development and evaluation of clinical LLMs are provided, which include centering clinical science, involving robust interdisciplinary collaboration, and attending to issues like assessment, risk detection, transparency, and bias. Lastly, a vision is outlined for how LLMs might enable a new generation of studies of evidence-based interventions at scale, and how these studies may challenge assumptions about psychotherapy.
Collapse
Affiliation(s)
- Elizabeth C Stade
- Dissemination and Training Division, National Center for PTSD, VA Palo Alto Health Care System, Palo Alto, CA, USA.
- Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA.
- Institute for Human-Centered Artificial Intelligence & Department of Psychology, Stanford University, Stanford, CA, USA.
| | - Shannon Wiltsey Stirman
- Dissemination and Training Division, National Center for PTSD, VA Palo Alto Health Care System, Palo Alto, CA, USA
- Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA
| | - Lyle H Ungar
- Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, USA
| | - Cody L Boland
- Dissemination and Training Division, National Center for PTSD, VA Palo Alto Health Care System, Palo Alto, CA, USA
| | - H Andrew Schwartz
- Department of Computer Science, Stony Brook University, Stony Brook, NY, USA
| | - David B Yaden
- Department of Psychiatry and Behavioral Sciences, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - João Sedoc
- Department of Technology, Operations, and Statistics, New York University, New York, NY, USA
| | - Robert J DeRubeis
- Department of Psychology, University of Pennsylvania, Philadelphia, PA, USA
| | - Robb Willer
- Department of Sociology, Stanford University, Stanford, CA, USA
| | - Johannes C Eichstaedt
- Institute for Human-Centered Artificial Intelligence & Department of Psychology, Stanford University, Stanford, CA, USA.
| |
Collapse
|
3
|
Higa-McMillan CK, Park AL, Daleiden EL, Becker KD, Bernstein A, Chorpita BF. Getting More Out of Clinical Documentation: Can Clinical Dashboards Yield Clinically Useful Information? ADMINISTRATION AND POLICY IN MENTAL HEALTH AND MENTAL HEALTH SERVICES RESEARCH 2024; 51:268-285. [PMID: 38261119 DOI: 10.1007/s10488-023-01329-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/29/2023] [Indexed: 01/24/2024]
Abstract
This study investigated coded data retrieved from clinical dashboards, which are decision-support tools that include a graphical display of clinical progress and clinical activities. Data were extracted from clinical dashboards representing 256 youth (M age = 11.9) from 128 practitioners who were trained in the Managing and Adapting Practice (MAP) system (Chorpita & Daleiden in BF Chorpita EL Daleiden 2014 Structuring the collaboration of science and service in pursuit of a shared vision. 43(2):323 338. 2014, Chorpita & Daleiden in BF Chorpita EL Daleiden 2018 Coordinated strategic action: Aspiring to wisdom in mental health service systems. 25(4):e12264. 2018) in 55 agencies across 5 regional mental health systems. Practitioners labeled up to 35 fields (i.e., descriptions of clinical activities), with the options of drawing from a controlled vocabulary or writing in a client-specific activity. Practitioners then noted when certain activities occurred during the episode of care. Fields from the extracted data were coded and reliability was assessed for Field Type, Practice Element Type, Target Area, and Audience (e.g., Caregiver Psychoeducation: Anxiety would be coded as Field Type = Practice Element; Practice Element Type = Psychoeducation; Target Area = Anxiety; Audience = Caregiver). Coders demonstrated moderate to almost perfect interrater reliability. On average, practitioners recorded two activities per session, and clients had 10 unique activities across all their sessions. Results from multilevel models showed that clinical activity characteristics and sessions accounted for the most variance in the occurrence, recurrence, and co-occurrence of clinical activities, with relatively less variance accounted for by practitioners, clients, and regional systems. Findings are consistent with patterns of practice reported in other studies and suggest that clinical dashboards may be a useful source of clinical information. More generally, the use of a controlled vocabulary for clinical activities appears to increase the retrievability and actionability of healthcare information and thus sets the stage for advancing the utility of clinical documentation.
Collapse
|
4
|
Harned MS, Schmidt SC, Korslund KE, Gallop RJ. Development and Evaluation of a Pragmatic Measure of Adherence to Dialectical Behavior Therapy: The DBT Adherence Checklist for Individual Therapy. ADMINISTRATION AND POLICY IN MENTAL HEALTH AND MENTAL HEALTH SERVICES RESEARCH 2023; 50:734-749. [PMID: 37284966 PMCID: PMC10246542 DOI: 10.1007/s10488-023-01274-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 05/08/2023] [Indexed: 06/08/2023]
Abstract
This paper presents two studies conducted to develop and evaluate a new pragmatic measure of therapist adherence to Dialectical Behavior Therapy (DBT): the DBT Adherence Checklist for Individual Therapy (DBT AC-I). Study 1 used item response analysis to select items from the gold standard DBT Adherence Coding Scale (DBT ACS) using archival data from 1271 DBT sessions. Items were then iteratively refined based on feedback from 33 target end-users to ensure relevance, usability, and understandability. Study 2 examined the psychometric properties of the DBT AC-I as a therapist self-report and observer-rated measure in 100 sessions from 50 therapist-client dyads, while also evaluating predictors of therapist accuracy in self-rated adherence. When used as a therapist self-report measure, concordance between therapist and observer ratings was at least moderate (AC1 ≥ 0.41) for all DBT AC-I items but overall concordance (ICC = 0.09) as well as convergent (r = 0.05) and criterion validity (AUC = 0.54) with the DBT ACS were poor. Higher therapist accuracy was predicted by greater DBT knowledge and adherence as well as more severe client suicidal ideation. When used by trained observers, the DBT AC-I had excellent interrater reliability (ICC = 0.93), convergent validity (r = 0.90), and criterion validity (AUC = 0.94). While therapists' self-rated adherence on the DBT AC-I should not be assumed to reflect their actual adherence, some therapists may self-rate accurately. The DBT AC-I offers an effective and relatively efficient method of evaluating adherence to DBT when used by trained observers.
Collapse
Affiliation(s)
- Melanie S Harned
- VA Puget Sound Health Care System, 1660 South Columbian Way, Mailstop S-116-MHC, Seattle, WA, 98108, USA.
- Department of Psychiatry & Behavioral Sciences, University of Washington, Seattle, WA, USA.
- Department of Psychology, University of Washington, Seattle, WA, USA.
| | - Sara C Schmidt
- VA Puget Sound Health Care System, 1660 South Columbian Way, Mailstop S-116-MHC, Seattle, WA, 98108, USA
| | | | - Robert J Gallop
- Department of Mathematics, West Chester University, West Chester, PA, USA
| |
Collapse
|
5
|
Cohen ZD, Barnes-Horowitz NM, Forbes CN, Craske MG. Measuring the active elements of cognitive-behavioral therapies. Behav Res Ther 2023; 167:104364. [PMID: 37429044 DOI: 10.1016/j.brat.2023.104364] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/06/2022] [Revised: 06/09/2023] [Accepted: 07/02/2023] [Indexed: 07/12/2023]
Abstract
Understanding how and for whom cognitive-behavioral therapies work is central to the development and improvement of mental health interventions. Suboptimal quantification of the active elements of cognitive-behavioral therapies has hampered progress in elucidating mechanisms of change. To advance process research on cognitive-behavioral therapies, we describe a theoretical measurement framework that focuses on the delivery, receipt, and application of the active elements of these interventions. We then provide recommendations for measuring the active elements of cognitive-behavioral therapies aligned with this framework. Finally, to support measurement harmonization and improve study comparability, we propose the development of a publicly available repository of assessment tools: the Active Elements of Cognitive-Behavioral Therapies Measurement Kit.
Collapse
Affiliation(s)
- Zachary D Cohen
- Department of Psychiatry and Biobehavioral Sciences, University of California, Los Angeles, United States.
| | | | - Courtney N Forbes
- Department of Psychology, University of California, Los Angeles, United States
| | - Michelle G Craske
- Department of Psychiatry and Biobehavioral Sciences, University of California, Los Angeles, United States; Department of Psychology, University of California, Los Angeles, United States
| |
Collapse
|
6
|
McLeod BD, Porter N, Hogue A, Becker-Haimes EM, Jensen-Doss A. What is the Status of Multi-Informant Treatment Fidelity Research? JOURNAL OF CLINICAL CHILD AND ADOLESCENT PSYCHOLOGY : THE OFFICIAL JOURNAL FOR THE SOCIETY OF CLINICAL CHILD AND ADOLESCENT PSYCHOLOGY, AMERICAN PSYCHOLOGICAL ASSOCIATION, DIVISION 53 2023; 52:74-94. [PMID: 36480728 DOI: 10.1080/15374416.2022.2151713] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
Abstract
OBJECTIVE The precise measurement of treatment fidelity (quantity and quality in the delivery of treatment strategies in an intervention) is essential for intervention development, evaluation, and implementation. Various informants are used in fidelity assessment (e.g., observers, practitioners [clinicians, teachers], clients), but these informants often do not agree on ratings. This scoping review aims to ascertain the state of science around multi-informant assessment of treatment fidelity. METHOD A literature search of articles published through December 2021 identified 673 articles. Screening reduced the number of articles to 44, and the final study set included 35 articles. RESULTS There was substantial variability across studies regarding study design, how fidelity was operationalized, and how reliability was defined and assessed. Most studies evaluated the agreement between independent observers and practitioner-report, though several other informant pairs were assessed. Overall, findings suggest that concordance across fidelity informants was low to moderate, with a few key exceptions. CONCLUSIONS It is difficult to draw clear conclusions about the degree to which single versus multiple informant assessment is needed to produce an accurate and complete picture of treatment fidelity. The field needs to take steps to determine how to leverage multi-informant assessment to accurately assess treatment fidelity.
Collapse
Affiliation(s)
- Bryce D McLeod
- Department of Psychology, Virginia Commonwealth University
| | | | | | | | | |
Collapse
|
7
|
LoSavio ST, Holder N, Wells SY, Resick PA. Clinician Concerns About Cognitive Processing Therapy: A Review of the Evidence. COGNITIVE AND BEHAVIORAL PRACTICE 2022. [DOI: 10.1016/j.cbpra.2022.08.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
8
|
Wiltsey Stirman S. Implementing Evidence-Based Mental-Health Treatments: Attending to Training, Fidelity, Adaptation, and Context. CURRENT DIRECTIONS IN PSYCHOLOGICAL SCIENCE 2022. [DOI: 10.1177/09637214221109601] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
This article provides a very high-level overview of some key considerations in the field of implementation science as it relates to clinical psychology. The article reviews recent findings regarding treatment fidelity, adaptation of treatments, and clinical outcomes. It then details some recent findings on training therapists to provide evidence-based therapies as they were designed to be delivered, with sufficient skill and adaptations if needed to ensure better fit for clients who receive treatments in different treatment settings. Finally, the article considers implications for implementing and sustaining new treatments and supporting their fidelity and adaptation, as well as key directions for future research.
Collapse
Affiliation(s)
- Shannon Wiltsey Stirman
- Dissemination and Training Division, National Center for PTSD, VA Palo Alto Healthcare System, Menlo Park, California, and Department of Psychiatry and Behavioral Sciences, Stanford University
| |
Collapse
|
9
|
Ghassemi AE. Reliable and comprehensive assessment of post-traumatic stress disorder is required to provide population-based prevention and treatment of PTSD during the COVID-19 pandemic. Evid Based Nurs 2021; 25:102. [PMID: 34615671 DOI: 10.1136/ebnurs-2020-103388] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 09/20/2021] [Indexed: 11/03/2022]
|
10
|
Dondanville KA, Fina BA, Straud CL, Finley EP, Tyler H, Jacoby V, Blount TH, Moring JC, Pruiksma KE, Blankenship AE, Evans WR, Zaturenskaya M. Launching a Competency-Based Training Program in Evidence-Based Treatments for PTSD: Supporting Veteran-Serving Mental Health Providers in Texas. Community Ment Health J 2021; 57:910-919. [PMID: 32666417 DOI: 10.1007/s10597-020-00676-7] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/07/2020] [Accepted: 07/03/2020] [Indexed: 10/23/2022]
Abstract
Community mental health providers play an essential role in delivering services to veterans who either have limited access to U.S. Department of Veterans Affairs (VA) facilities or who prefer to seek care outside of the VA. However, there are limited training opportunities in evidence-based treatments for posttraumatic stress disorder (PTSD) outside of the VA. In 2017, the STRONG STAR Training Initiative was established to develop competency-based training in two evidence-based therapies for PTSD and to provide that training for mental health providers serving veterans and their families in community settings in Texas. This article describes the program's development and implementation, baseline characteristics of participating clinicians, and lessons learned toward the scale-up and extension of this competency-based training effort to include other interventions and locations.
Collapse
Affiliation(s)
- Katherine A Dondanville
- Department of Psychiatry and Behavioral Sciences, The University of Texas Health Science Center at San Antonio, 7703 Floyd Curl Drive, San Antonio, TX, 78229, USA.
| | - Brooke A Fina
- Department of Psychiatry and Behavioral Sciences, The University of Texas Health Science Center at San Antonio, 7703 Floyd Curl Drive, San Antonio, TX, 78229, USA
| | - Casey L Straud
- Department of Psychiatry and Behavioral Sciences, The University of Texas Health Science Center at San Antonio, 7703 Floyd Curl Drive, San Antonio, TX, 78229, USA.,Department of Psychology, The University of Texas at San Antonio, San Antonio, TX, 78249, USA
| | - Erin P Finley
- Department of Psychiatry and Behavioral Sciences, The University of Texas Health Science Center at San Antonio, 7703 Floyd Curl Drive, San Antonio, TX, 78229, USA.,South Texas Veterans Health Care System, 7400 Merton Minter Blvd, San Antonio, TX, USA
| | - Hannah Tyler
- Department of Psychiatry and Behavioral Sciences, The University of Texas Health Science Center at San Antonio, 7703 Floyd Curl Drive, San Antonio, TX, 78229, USA
| | - Vanessa Jacoby
- Department of Psychiatry and Behavioral Sciences, The University of Texas Health Science Center at San Antonio, 7703 Floyd Curl Drive, San Antonio, TX, 78229, USA
| | - Tabatha H Blount
- Department of Psychiatry and Behavioral Sciences, The University of Texas Health Science Center at San Antonio, 7703 Floyd Curl Drive, San Antonio, TX, 78229, USA
| | - John C Moring
- Department of Psychiatry and Behavioral Sciences, The University of Texas Health Science Center at San Antonio, 7703 Floyd Curl Drive, San Antonio, TX, 78229, USA
| | - Kristi E Pruiksma
- Department of Psychiatry and Behavioral Sciences, The University of Texas Health Science Center at San Antonio, 7703 Floyd Curl Drive, San Antonio, TX, 78229, USA
| | - Abby E Blankenship
- Department of Psychiatry and Behavioral Sciences, The University of Texas Health Science Center at San Antonio, 7703 Floyd Curl Drive, San Antonio, TX, 78229, USA
| | - Wyatt R Evans
- VA North Texas Health Care System, 4500 S Lancaster Rd, Dallas, TX, USA
| | - Mariya Zaturenskaya
- James A. Haley Veterans' Hospital, 13000 Bruce B. Downs Blvd, Tampa, FL, USA
| | | |
Collapse
|