Monitoring implementation in program evaluation with direct audio coding

This project explored the reliability and utility of transcription in coding qualitative data across two studies in a program evaluation context. The first study tested the method of direct audio coding, or coding audio files without transcripts, using qualitative data software. The presence and frequency of codes applied in direct audio coding and traditional transcription coding were compared and the two methods produced similar results. Direct audio coding was then employed in an evaluation study to monitor implementation and the method was found to be reliable. Implications are discussed with considerations for both researchers and practitioners.

Measuring Activation in Parents of Youth with Emotional and Behavioral Disorders

For parents of youth with emotional and behavioral disorders, activation, or having the knowledge, skills, and confidence to access and engage in appropriate services for their children, is important for managing their child's mental health care. The Parent Activation Measure (PAM) was modified to create the Parent Patient Activation Measure-Mental Health (P-PAM-MH) to measure activation as part of a randomized controlled trial of a peer parent support intervention for parents of youth with emotional and behavioral problems. Results from this study provide initial support for use of the P-PAM-MH as a measure of activation in this population and for the reliability and validity of the measure. Implications of the findings from this study for research and practice in behavioral health are discussed.

Parental report of outcomes from a randomized trial of in-home family services

This study conducted a randomized trial to examine the efficacy of the Boys Town In-Home Family Services (IHFS) program for families of high-risk youth. Participants were recruited from a state helpline for families struggling with poor family functioning and child emotional or behavioral issues. Consent was obtained for 300 of which 152 were randomly assigned to participate in IHFS for 3-4 months and 148 were assigned to the services as usual comparison group. For the families in the treatment group, 18% did not participant in the intervention, and 66% of families received 20 or more service hours. Parent report data were collected at intake, post, as well 6 and 12 months after post data collection. Data were collected on constructs such as caregiver strain, family functioning, parenting, family resources, and parent report of child behavior. Piecewise analyses of the intake to post data indicated significantly greater reductions in caregiver strain for the treatment condition. Given the conservative corrections for the use of multiple tests, no other measures demonstrated significant differences. For the piecewise model of the maintenance phase, there were no significant differences between groups aside from caregiver strain that showed a significant improvement for the comparison condition. Supplementary dose-response analyses indicated that for most families there was an ideal dosage of about 25-75 hr to bring about the largest improvements in caregiver strain, parenting skills, and child behavior. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

An Examination of the Psychometric Properties and Validation of the Family Resource Scale for Families Seeking Assistance With Their Child’s Behavioral Difficulties

Psychometric properties of the 30-item Family Resource Scale (FRS) were examined in a sample of families seeking assistance for their child’s behavioral difficulties ( N = 300). The FRS is a measure that assesses the adequacy of family resources across several contexts including intra-family support and personal resources. Our analyses supported a modified 29-item four-factor solution ( basic needs, extra money and time, time for family, and essential) compared with the seven-factor solution provided by the developers. Results showed that families’ perception of their resources as adequate across all domains was associated with better family functioning and less caregiver strain. Combined, our findings suggest modifications to the existent structure of the FRS.

Child maltreatment as a function of cumulative family risk: Findings from the intensive family preservation program

This study examined child maltreatment as a function of cumulative family risk in a sample of at-risk families (N=837) who were referred to an intensive family preservation program because of child behavior problems or suspected child abuse and neglect. The goal of this intensive family preservation program is to improve parenting skills and reduce immediate family stressors that may lead to an increased risk of child abuse and neglect. The findings indicate that the most prominent family risks comprising the cumulative risk scale in our sample were socio-economic disadvantage (e.g., income, unemployment, housing instability) and parental characteristics (e.g., mental/physical health, parental use of alcohol, domestic violence). Further, the results demonstrated a strong quadratic trend in the relationship between cumulative family risk and child maltreatment, and identified a risk threshold effect at three cumulative family risks after which the child risk for maltreatment increased exponentially. These findings are interpreted in the light of the current research on differentiative interventions, supporting differentiated services to the families with low vs. higher risk for child maltreatment.

Psychometrics of the Symptoms and Functioning Severity Scale for High-Risk Youth

Youth in residential care have significant mental health needs which require regular progress monitoring; however, very few emotional or behavioral assessments have been examined with this unique, high-risk population. This study examined the psychometrics of the Symptom Functioning and Severity Scale, a brief 24-item measure designed to assess the emotional and behavioral status of youth. This study examined the SFSS ratings from 143 youth with a disruptive behavior diagnosis living in a group-home facility in the Midwest and 52 of their service providers. Overall, the findings suggest that the psychometrics of the SFSS, when rated by staff or youth were similar to the original outpatient clinical samples. More specifically, the Rasch analyses indicate that the SFSS items and the overall scale is performing adequately, and the confirmatory factor analyses replicated the two-factor structure for staff. However, the fit of the two-factor model was less compelling for youth ratings. In all, the brief SFSS seems a promising measure for assessing problem severity for youth in residential care.

Examining Change in Therapeutic Alliance to Predict Youth Mental Health Outcomes

OBJECTIVE

To examine the link between therapeutic alliance and youth outcomes.

METHOD

The study was conducted at a group-home with 112 youth with a disruptive-behavior diagnosis. Therapeutic alliance was collected routinely via youth and staff report. Outcome data were collected using youth and staff reports of externalizing behavior as well as behavioral incidents occurring during care. Outcome data were collected following intake into services and at 6 and 12 months of care. Data were analyzed to examine (1) if youth behavior problems at intake were predictive of therapeutic alliance and (2) if changes in alliance were predictive of subsequent youth outcomes. These were conducted with a 6-month service-delivery model and replicated with a 12-month model.

RESULTS

There was some support for the first hypothesis, that initial levels of youth externalizing behavior would be related to alliance ratings; however, most of the effects were marginally significant. The second hypothesis, that changes in therapeutic alliance would be related to subsequent youth outcomes, was supported for the 6-month model, but not the 12-month model.

CONCLUSIONS

Changes in therapeutic alliance may be predictive of youth outcomes during care. Additional research into examining therapeutic alliance trajectories is warranted to improve mental health services for youth.

Psychometric Evaluation of the Symptoms and Functioning Severity Scale (SFSS) Short Forms with Out-of-Home Care Youth

BACKGROUND

There is a need for brief progress monitoring measures of behavioral and emotional symptoms for youth in out-of-home care. The Symptoms and Functioning Severity Scale (SFSS; Bickman et al., 2010) is one measure that has clinician and youth short forms (SFSS-SFs); however, the psychometric soundness of the SFSS-SFs with youth in out-of-home care has yet to be examined.

OBJECTIVE

The objective was to determine if the psychometric characteristics of the clinician and youth SFSS-SFs are viable for use in out-of-home care programs.

METHODS

The participants included 143 youth receiving residential treatment and 52 direct care residential staff. The current study assessed internal consistency and alternate forms reliability for SFSS-SFs for youth in a residential care setting. Further, a binary classification test was completed to determine if the SFSS-SFs similarly classified youth as the SFSS full version for low- and elevated-severity.

RESULTS

The internal consistency for the clinician and youth SFSS-SFs was adequate (α = .75 to .82) as was the parallel forms reliability (r = .85 to .97). The sensitivity (0.80 to 0.95), specificity (0.88 to 0.97), and overall accuracy (0.89 to 0.93) for differentiating low and elevated symptom severity was acceptable.

CONCLUSIONS

The clinician and youth SFSS-SFs have acceptable psychometrics and may be beneficial for progress monitoring and additional research should clarify their potential for progress monitoring of youth in out-of-home programs.

Convergent Validity of the Strength-Based Behavioral and Emotional Rating Scale with Youth in a Residential Setting

Strength-based assessment has been identified as an appropriate approach to use in planning treatment and evaluating outcomes of youth in residential settings. In previous research, the Behavioral and Emotional Rating Scale-2, a standardized and norm-referenced strength-based measure, has demonstrated adequate reliability and validity with youth served in community and educational settings. The purpose of the present study was to examine the internal reliability and convergent validity of the BERS-2 by comparing the test to the Child Behavior Checklist and the Symptoms and Functioning Severity Scale. The results indicate that the scores from the BERS-2 are internally consistent and converge with other behavioral and emotional measures which, taken together, suggest that the BERS-2 could be acceptable for assessing the emotional and behavioral strengths of youth in residential settings. Study limitations and future research directions are identified.

Therapeutic Alliance Between Youth and Staff in Residential Group Care: Psychometrics of the Therapeutic Alliance Quality Scale

Therapeutic alliance has been frequently studied in individual counseling sessions; however, research on therapeutic alliance in residential settings for youth with mental health diagnoses has been limited. This may be due, in part, to the presence of multiple service providers often in caregiving roles. The purpose of this study was to examine the psychometric quality of a widely utilized measure of therapeutic alliance used in psychotherapy with youth in residential care where the treatment is provided by a trained married couple. We also compared the relationship between youth ratings of their male and female service provider, as well as examined correlations in ratings between youth and staff on therapeutic alliance. Finally, we investigated the direction, magnitude, and trajectory of change in therapeutic alliance over a 12-month period following admission into residential care. The method was a longitudinal assessment of 135 youth and 124 staff regarding therapeutic alliance over the course of 12 months or discharge from services. Results indicated strong psychometric properties and high correlations for youth ratings of both their male and female service providers. However, the correlation was low between youth and service provider ratings of alliance. Longitudinal analyses indicated that rates of therapeutic alliance changed over time.

Prevalence of physical health problems among youth entering residential treatment

OBJECTIVE

To examine the prevalence of physical health problems among youth entering residential treatment.

PATIENTS AND METHODS

The sample included 1744 youth (mean age: 14.6 ± 1.8 years) entering a large residential treatment program between 2000 and 2010. Youth received an intake medical evaluation, including a review of available records, detailed medical history, and physical examination. Medical conditions present at the time of the evaluation were recorded by the examining physician and later coded by the research team. Only diagnoses recognized by the International Classification of Diseases, 10th Revision, were included in the analyses. To maintain the focus on physical health problems, behavioral and emotional disorders listed in the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision were excluded. Obesity, acne, and allergies were also excluded.

RESULTS

Approximately one-third (33.7%) of youth had a physical health diagnosis at the time of intake. Asthma was the most prevalent condition diagnosed (15.3% of the sample). Girls were significantly more likely to have a diagnosis than were boys (37.1% vs 31.5%). Age was not associated with diagnostic status. Rates of physical health conditions differed significantly by ethnicity: black (36.4%) and white (35.4%) youth had the highest rates, and Hispanic youth (23.2%) had the lowest.

CONCLUSIONS

Youth who enter residential treatment have high rates of physical health conditions. These problems could complicate mental health treatment and should be considered in multidisciplinary treatment planning.

Creating and Sustaining Effective Partnerships to Advance Research on Youth With Serious Emotional and Behavioral Disorders

A key barrier to conducting research involving children and families is the difficulty of creating partnerships among researchers, treatment agencies, and schools. This article describes several key factors that were essential to establishing an effective research collaboration between practitioners and university-based researchers, including a mutual respect for the unique needs of research and practice; a strategy for joint decision making; a partnership model of incremental growth; a plan for mentoring junior faculty and students; a format for regular contact between the partners; and a plan for long-term sustainability. This collaboration has conducted over a dozen evaluation studies, as well as examined best practice issues surrounding the needs of children and families with serious emotional and behavioral needs. Even more important has been the lines of research that have been developed from this partnership which keeps the collaboration focused. The lessons learned from this research partnership should inform other collaborations.