1
|
Becker KD, Wu EG, Westman JG, Boyd MR, Guan K, Lakind D, Chu W, Knudsen KS, Bradley WJ, Park AL, Kenworthy LaMarca T, Lang E, Chorpita BF. The Interrater Reliability of a Coding System for Measuring Mental Health Professionals' Decisions and Actions. JOURNAL OF CLINICAL CHILD AND ADOLESCENT PSYCHOLOGY : THE OFFICIAL JOURNAL FOR THE SOCIETY OF CLINICAL CHILD AND ADOLESCENT PSYCHOLOGY, AMERICAN PSYCHOLOGICAL ASSOCIATION, DIVISION 53 2024:1-17. [PMID: 39137271 DOI: 10.1080/15374416.2024.2384027] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/15/2024]
Abstract
OBJECTIVE The clinical decisions and actions of evidence-based practice in psychology (EBPP) are largely underspecified and poorly understood, in part due to the lack of measurement methods. We tested the reliability of a behavioral coding system that characterizes a flow of interrelated activities that includes problem detection and prioritization, intervention selection and implementation, and review of intervention integrity and impact. METHOD The context included two publicly funded youth mental health service organizations located in geographically distinct and underresourced communities in the U.S. where service inequities are common. We sampled 84 digitally recorded and transcribed supervision events that included a sample of professionals who were mostly women (93.02%) and BIPOC (86.04%) whose self-reported race/ethnicity matched the youth populations they served. We coded these events for activities (e.g., considering) and their predicate content (i.e., problems or practices) and examined reliability of these codes applied to excerpts (i.e., small contiguous units of dialogue) as well as to complete events. RESULTS Interrater reliability estimates showed that, overall, coders reliably rated the occurrence and extensiveness of activities and content. Excerpt coding was generally more reliable than event coding. However, mathematical aggregation of excerpt coding offered a superior method for estimating event codes reliably, reducing individual subjectivity while providing event level synthesis of activities that are grounded in excerpt level details. CONCLUSIONS The assessment of clinical decisions and actions has the potential to unpack the black box of EBPP, with different methods best suited to different research questions and resource considerations.
Collapse
Affiliation(s)
| | - Eleanor G Wu
- Department of Psychology, University of South Carolina
| | | | | | - Karen Guan
- Department of Psychology, University of California
| | | | - Wendy Chu
- Department of Psychology, University of South Carolina
| | | | | | | | | | - Emily Lang
- Department of Psychology, University of California
| | | |
Collapse
|
2
|
Lau AS, Lind T, Cox J, Motamedi M, Lui JHL, Chlebowski C, Flores A, Diaz D, Roesch S, Brookman-Frazee L. Validating a Pragmatic Measure of Evidence-Based Practice (EBP) Delivery: Therapist Reports of EBP Strategy Delivery and Associations with Child Outcome Trajectories. ADMINISTRATION AND POLICY IN MENTAL HEALTH AND MENTAL HEALTH SERVICES RESEARCH 2024:10.1007/s10488-024-01395-x. [PMID: 39096408 DOI: 10.1007/s10488-024-01395-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 06/19/2024] [Indexed: 08/05/2024]
Abstract
Pragmatic measures of evidence-based practice (EBP) implementation can support and evaluate implementation efforts. We examined the predictive validity of therapist reports of EBP strategy delivery for children's mental health outcomes. Data were obtained from 1,380 sessions with 248 children delivered by 76 therapists in two county systems. Children (Mage=11.8 years, SD = 3.7) presented with internalizing (52%), externalizing (27%), trauma (16%), and other (5%) concerns. Therapists reported their delivery of EBP strategies on a revised version of the EBP Concordant Care Assessment (ECCA; Brookman-Frazee, et al., Administration and Policy in Mental Health and Mental Health Services Research, 48, 155-170, 2021) that included 25 content (e.g., parenting, cognitive behavioral) and 12 technique strategies (e.g., modeling, practice/role-play). On average, 5.6 ECCA session reports (SD = 2.3) were obtained for each client, and caregivers reported symptoms on the Brief Problem Checklist (Chorpita, et al., Journal of Consulting and Clinical Psychology, 78(4), 526-536, 2010) at baseline, weekly over two months, and again at four months. Multilevel models examined whether the mean extensiveness of each EBP strategy predicted trajectories of child outcomes. More individual technique (6 of 12) than content strategies (1 of 25) were associated with outcome trajectories. For techniques, more extensive use of Performance Feedback and Live Coaching and less extensive use of Addressing Barriers were associated with greater declines in total symptoms, and more extensive use of Establishing/Reviewing Goals, Tracking/Reviewing Progress, and Assigning/Reviewing Homework was associated with declines in externalizing symptoms. For content, more extensive use of Cognitive Restructuring was associated with declines in total symptoms. In addition, higher average extensiveness ratings of the top content strategy across sessions was associated with greater declines in total and externalizing symptoms. Therapist-reported delivery of some EBP strategies showed evidence of predictive validity and may hold utility in indexing quality of care.
Collapse
Affiliation(s)
- Anna S Lau
- Department of Psychology, University of California-Los Angeles, Los Angeles, CA, USA
| | - Teresa Lind
- Department of Child and Family Development, San Diego State University, San Diego, CA, USA.
- Child and Adolescent Services Research Center (CASRC), San Diego, CA, USA.
| | - Julia Cox
- Department of Psychology, University of California-Los Angeles, Los Angeles, CA, USA
| | - Mojdeh Motamedi
- Child and Adolescent Services Research Center (CASRC), San Diego, CA, USA
- Department of Psychiatry, University of California-San Diego, La Jolla, CA, USA
| | - Joyce H L Lui
- Department of Psychology, Concordia University, Montreal, QC, Canada
| | - Colby Chlebowski
- Child and Adolescent Services Research Center (CASRC), San Diego, CA, USA
- Department of Psychology, San Diego State University, San Diego, CA, USA
| | - Ashley Flores
- Department of Psychology, University of California-Los Angeles, Los Angeles, CA, USA
| | - Devynne Diaz
- Child and Adolescent Services Research Center (CASRC), San Diego, CA, USA
- Department of Psychiatry, University of California-San Diego, La Jolla, CA, USA
| | - Scott Roesch
- Child and Adolescent Services Research Center (CASRC), San Diego, CA, USA
- Department of Psychology, San Diego State University, San Diego, CA, USA
| | - Lauren Brookman-Frazee
- Child and Adolescent Services Research Center (CASRC), San Diego, CA, USA
- Department of Psychiatry, University of California-San Diego, La Jolla, CA, USA
| |
Collapse
|
3
|
Graziano PA, Sibley MH, Coxe SJ, Bickman L, Martin P, Scheres A, Hernandez ML. Community-Delivered Evidence-Based Practice and Usual Care for Adolescent Attention-Deficit/Hyperactivity Disorder: Examining Mechanistic Outcomes. Behav Ther 2024; 55:412-428. [PMID: 38418050 PMCID: PMC10902603 DOI: 10.1016/j.beth.2023.08.001] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/03/2022] [Revised: 08/01/2023] [Accepted: 08/04/2023] [Indexed: 03/01/2024]
Abstract
Previous research suggests that routine psychosocial care for adolescents with attention-deficit/hyperactivity disorder (ADHD) is an eclectic and individualized mix of diluted evidence-based practices (EBPs) and low-value approaches. This study evaluated the extent to which a community-delivered EBP and usual care (UC) for adolescents with ADHD produce differential changes in theorized behavioral, psychological, and cognitive mechanisms of ADHD. A randomized community-based trial was conducted with double randomization of adolescent and community therapists to EBP delivery supports (Supporting Teens' Autonomy Daily [STAND]) versus UC delivery. Participants were 278 culturally diverse adolescents (ages 11-17) with ADHD and caregivers. Mechanistic outcomes were measured at baseline, post-treatment, and follow-up using parent-rated, observational, and task-based measures. Results using linear mixed models indicated that UC demonstrated superior effects on parent-rated and task-based executive functioning relative to STAND. However, STAND demonstrated superior effects on adolescent motivation and reducing parental intrusiveness relative to UC when it was delivered by licensed therapists. Mechanisms of community-delivered STAND and UC appear to differ. UC potency may occur through improved executive functioning, whereas STAND potency may occur through improved teen motivation and reducing low-value parenting practices. However, when delivered by unlicensed, community-based therapists, STAND did not enact proposed mechanisms. Future adaptations of community-delivered EBPs for ADHD should increase supports for unlicensed therapists, who comprise the majority of the community mental health workforce.
Collapse
Affiliation(s)
| | - Margaret H Sibley
- University of Washington School of Medicine, Seattle Children's Research Institute, and Florida International University
| | | | | | | | | | | |
Collapse
|
4
|
Sibley MH, Graziano PA, Coxe SJ, Bickman L, Martin P, Flores S. A Randomized Community-Based Trial of Behavior Therapy vs. Usual Care for Adolescent ADHD: Secondary Outcomes and Effects on Comorbidity. Behav Ther 2023; 54:839-851. [PMID: 37597961 PMCID: PMC10440417 DOI: 10.1016/j.beth.2023.03.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/28/2022] [Revised: 02/27/2023] [Accepted: 03/03/2023] [Indexed: 08/21/2023]
Abstract
Though behavior therapy (BT) for ADHD in adolescence is evidence-based, almost no work examines its implementation and effectiveness in community settings. A recent randomized community-based trial of an evidence-based BT for adolescent ADHD (Supporting Teens' Autonomy Daily; STAND; N = 278) reported high clinician, parent, and youth acceptability but variable implementation fidelity. Primary outcome analyses suggested no significant differences between STAND and usual care (UC) unless the clinician delivering STAND was licensed. The present study reports secondary outcomes for this trial on indices of comorbidity (anxiety, depression, oppositional defiant disorder, conduct disorder) and ADHD outcomes not targeted by the active treatment (social skills, sluggish cognitive tempo). We also examine whether therapist licensure moderated treatment effects (as in primary outcome analyses). Using intent-to-treat and per protocol linear mixed models, patients randomized to STAND were compared to those randomized to UC over approximately 10 months of follow-up. Group × Time effects revealed that, overall, STAND did not outperform usual care when implemented by community clinicians. However, a Group × Time × Licensure interaction revealed a significant effect on conduct problems when STAND was delivered by licensed clinicians (d = .19-.47). When delivered in community settings, behavior therapy for adolescent ADHD can outperform UC with respect to conduct problems reduction. Community mental health clinics should consider: (1) assigning adolescent ADHD cases to licensed professionals to maximize impact and (2) choosing psychosocial approaches when ADHD presents with comorbid conduct problems. There is also a need to reduce implementation barriers for unlicensed clinicians in community settings.
Collapse
Affiliation(s)
- Margaret H Sibley
- University of Washington School of Medicine, Seattle Children's Research Institute.
| | | | | | - Leonard Bickman
- Center for Children & Families, Florida International University
| | | | - Sabrina Flores
- Center for Child Health, Behavior, and Development, Seattle Children's Research Institute
| |
Collapse
|
5
|
Developing an Implementation Model for ADHD Intervention in Community Clinics: Leveraging Artificial Intelligence and Digital Technology. COGNITIVE AND BEHAVIORAL PRACTICE 2023. [DOI: 10.1016/j.cbpra.2023.02.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/06/2023]
|
6
|
Are There Long-Term Effects of Behavior Therapy for Adolescent ADHD? A Qualitative Study. Child Psychiatry Hum Dev 2022:10.1007/s10578-021-01294-4. [PMID: 35039978 DOI: 10.1007/s10578-021-01294-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 11/28/2021] [Indexed: 11/03/2022]
Abstract
We utilized qualitative methodology to characterize potential long-term effects (therapeutic and iatrogenic) of behavior therapy for adolescents with ADHD. Forty-two in-depth interviews were conducted with adolescents with ADHD and parents, 4 years post-treatment. Grounded theory methods identified and reported prevalence of themes. All reported long-term effects were classified as benefits; no iatrogenic effects were noted. Long-term impact themes reported for a majority of participants included: development of organization skills (81.0%), enhanced motivation (57.1%), improved self-awareness (57.1%), improved parental knowledge of ADHD (76.2%), increased parent autonomy granting (61.9%), enhanced parental engagement with the youth (52.4%), and improved parent-teen relationships (52.4%). Fourteen themes were present for smaller subsamples, including reduced need for medication (3 of 9 medicated participants). Experimental studies of behavior therapy for adolescent ADHD should measure themes detected herein and directly test the possibility of long-term treatment effects.
Collapse
|
7
|
Sibley MH, Johansson M, Monroy JM, Hill D, LaCount P, Barney S, Molina N, Delgado A. Building a Theoretical Model for Supporting Teens' Autonomy Daily (STAND): A Network Analysis of Family-Perceived Changes. Behav Ther 2022; 53:49-63. [PMID: 35027158 PMCID: PMC8760466 DOI: 10.1016/j.beth.2021.05.009] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/15/2021] [Revised: 05/28/2021] [Accepted: 05/31/2021] [Indexed: 01/03/2023]
Abstract
Little is known about processes through which behavior therapy (BT) for adolescent ADHD improves outcomes. The purpose of this study was to build a theoretical model for the processes through which a BT for adolescent ADHD (Supporting Teens' Autonomy Daily; STAND) impacts functioning. Seventy-eight audio recordings from a standard therapeutic task in the final STAND session were analyzed as parents and adolescents (ages 11-16) reflected upon what changed during STAND and why. Qualitative coding sorted parent and teen statements into orthogonal categories of perceived changes. Network analysis examined inter-relations between categories. Results indicated twenty-one categories of perceived change areas. Parent use of behavioral strategies, adolescent motivation, and adolescent organization skills were central nodes in the network of perceived changes, with strong relations to academic and parent-teen relationship outcomes. A model is proposed in which skills training in STAND increases parent behavioral strategy use and teen organization skills, while Motivational Interviewing (MI) in STAND increase parent behavioral strategy use and initial adolescent motivation. In turn, parent behavioral strategy use is proposed to further reinforce teen motivation through contingency management, thereby increasing teen application of organization skills to daily life. As a result of improved teen motivation and organization skills, the model proposes that ADHD symptoms, academic problems, and parent-teen conflict abate. We discuss secondary mechanisms and outcomes in this model, the possibility of person-specific processes, implications for community-based adaptation of STAND, and plans to validate this conceptual model using sophisticated mediational models.
Collapse
Affiliation(s)
- Margaret H Sibley
- University of Washington School of Medicine, Seattle Children's Research Institute.
| | | | | | | | | | | | | | | |
Collapse
|
8
|
Sibley MH, Ortiz M, Rios-Davis A, Zulauf-McCurdy CA, Graziano PA, Bickman L. Stakeholder-Generated Implementation Strategies to Promote Evidence-Based ADHD Treatment in Community Mental Health. ADMINISTRATION AND POLICY IN MENTAL HEALTH AND MENTAL HEALTH SERVICES RESEARCH 2022; 49:44-58. [PMID: 33988847 DOI: 10.1007/s10488-021-01143-5] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/26/2021] [Indexed: 10/21/2022]
Abstract
Community implementation of evidence-based practices (EBPs) for Attention Deficit/Hyperactivity Disorder (ADHD) is greatly lacking. A recent randomized community-based trial of an EBP for ADHD (Supporting Teens' Autonomy Daily; STAND) demonstrated suboptimal implementation and effectiveness outcomes. In the present study, we conducted an Innovation Tournament (IT) with agency staff stakeholders (N = 26) to identify barriers to successful implementation of STAND and implementation strategies for a revised service delivery model. We conducted member-checking of agency staff-generated ideas with parents (N = 226) and subsequent querying of additional parent (N = 226) and youth-generated (N = 205) strategies to improve care. Go-Zone plots were utilized to identify strategies with the highest feasibility and importance. Practical barriers (i.e., transportation, scheduling difficulties) and parent/youth engagement were the most commonly cited obstacles to successful implementation of STAND in community contexts. Eighteen "winning" implementation strategies were identified that survived member checking. These were classified as train and educate stakeholders (n = 5; e.g., train agency supervisors to deliver supervision, digitize treatment materials and trainings), engage consumers (n = 9; e.g., begin treatment with rapport building sessions, increase psychoeducation), provide interactive assistance (n = 2; e.g., add group supervision, increase roleplay in supervision), and use of evaluative/iterative strategies (n = 2; e.g., perform fidelity checks, supervisor review of session recordings). Parents and youth desired longer duration of treatment and increased focus on maintenance. Strategies will be developed and tested as part of a pilot effectiveness trial designed to refine STAND's service delivery model.Trial Registration NCT02694939 www.clinicaltrials.gov.
Collapse
Affiliation(s)
- Margaret H Sibley
- Department of Psychiatry & Behavioral Sciences, University of Washington School of Medicine, Seattle, WA, USA.
- Seattle Children's Research Institute, Seattle, WA, USA.
| | | | | | - Courtney A Zulauf-McCurdy
- Department of Psychiatry & Behavioral Sciences, University of Washington School of Medicine, Seattle, WA, USA
| | | | | |
Collapse
|