The 25-Item Ontario Child Health Study Emotional Behavioural Scales-Brief Version (OCHS-EBS-B): Test-Retest Reliability and Construct Validity When Used as Categorical Measures

OBJECTIVE

Child and youth mental health problems are often assessed by parent self-completed checklists that produce dimensional scale scores. When converted to binary ratings of disorder, little is known about their psychometric properties in relation to classifications based on lay-administered structured diagnostic interviews. In addition to estimating agreement, our objective is to test for statistical equivalence in the test-retest reliability and construct validity of two instruments used to classify child emotional, behavioural, and attentional disorders: the 25-item, parent completed Ontario Child Health Study Emotional Behavioural Scales-Brief Version (OCHS-EBS-B) and the Mini International Neuropsychiatric Interview for Children and Adolescents-parent version (MINI-KID-P).

METHODS

This study draws on independent samples (n = 452) and uses the confidence interval approach to test for statistical equivalence. Reliability is based on kappa (κ). Construct validity is based on standardized beta coefficients (β) estimated in structural equation models.

RESULTS

The average differences between the MINI-KID-P and OCHS-EBS-B in κ and β were -0.022 and -0.020, respectively. However, in both instances, criteria for statistical equivalence were met in only 5 of 12 comparisons. Based on κ, between-instrument agreement on the classifications of disorder went from 0.481 (attentional disorder) to 0.721 (emotional disorder) but were substantially higher (0.731 to 0.895, respectively) when corrected for attenuation due to measurement error.

CONCLUSIONS

Although falling short of equivalence, the results suggest on balance that the reliability and validity of the two instruments for classifying child psychiatric disorder assessed by parents are highly comparable. This conclusion is supported by the high levels of agreement between the instruments after correcting for attenuation due to measurement error.

Comparing the reliability and validity of youth-reported checklists and standardized interviews for categorical measurement of emotional and behavioral problems

INTRODUCTION

Self-completed checklists measuring youth mental health problems produce dimensional scale scores and can be converted to categorical classifications representing the presence/absence of psychopathology. We test whether categorical classifications from scale scores are equivalent psychometrically to categorical classifications of the same problems obtained by lay-administered standardized structured diagnostic interviews.

METHODS

The sample of n = 325 youth aged 12-18 (44% male) and their parent/caregivers come from combined test-retest reliability studies conducted in Ontario, Canada, from 2011 to 2015. Ontario Child Health Study Emotional Behavioural Scales-Brief Version (OCHS-EBS-B) scores converted to categorical classifications of emotional and behavioral problems were compared with interview classifications. We test hypotheses of statistical equivalence and inferiority, using a confidence interval approach to detect if differences lie within the smallest effect size of interest of ±0.18. We compare categorical classifications on: (1) test-retest reliability (ҡ), (2) content validity (between-instrument agreement), and (3) construct validity (strength of association with three mental health-related constructs).

RESULTS

Average test-retest reliabilities were 0.695 (checklists) and 0.670 (interviews). The reliability of checklist emotional problem classifications was not inferior to interview classifications and the difference in reliability between instruments for behavioral problems was small (-0.036). Average between-instrument agreement was ҡ = 0.586 (observed) and ҡ = 0.841 (corrected for attenuation due to measurement error) indicating high content overlap. Statistical equivalence criteria were met in 5 of 6 construct validity comparisons.

CONCLUSIONS

Categorical classifications of emotional and behavioral problems from youth-reported checklists are, on balance, equivalent to interview classifications. Checklists represent a simple, brief, inexpensive alternative to interviews.

Utilization of the Adolescent and Young Adult Psycho-Oncology Screening Tool in a Pediatric Hospital Adolescent/Young Adult Program

Purpose: The AYA Psycho-Oncology Screening Tool was developed to assess adolescent and young adult (AYA) patients' distress during cancer treatment. The on-treatment distress screening tool has been validated with AYAs and includes a 10-point distress thermometer (DT) and a 53-item problem checklist (PCL). However, previous studies have not solely examined AYA cancer distress within a children's hospital. Therefore, our project aimed to explore AYA distress in a pediatric cancer setting. Methods: AYA-aged participants (aged ≥15) were given the distress screener initially within 1 month of diagnosis and every 2, 4, or 6 months, depending on their previous distress score. Chi-square, independent t-tests, and binary logistic regressions were conducted for data analysis. Results: Between January 2021 and July 2022, we completed 123 screenings in 68 AYAs (age 15-30) on treatment. Average DT score was 2.96 with 30% of participants endorsing distress levels of 5 and above. There were statistically significant differences by sex as females endorsed higher levels of distress compared with males. Adolescents (<18) endorsed statistically significant higher frequency of emotional PCL items in comparison with young adults (≥18). There were no differences by race or diagnosis. Conclusions: Our team gained awareness of specific areas of concerns for AYAs, allowing for more targeted interventions for distressed participants. Certain demographic variables may put participants at risk for increased distress. As a result of the project, a protocol has been developed to follow up with participants if they report a certain distress score (5 or above) and/or endorse critical items.