The end of the primary outcome measure: A research agenda for constructing its replacement

Advancing equity in access to school mental health through multiple informant decision-making

There has been a substantial increase in the number of students with mental health needs, yet significant discrepancies exist in access to timely intervention. Traditional gatekeeping to intervention has been the provenance of single information sources. Multi-informant decision-making is a promising mechanism to improve equitable access. However, critical advancements are necessary to improve decision-making relating to (a) who is identified, (b) what type of need is determined, (c) the type of intervention necessary, and (d) where or under what circumstances to implement the intervention. We review critical components of effective mental health decision-making, contributors to inequities in school mental health services, and offer future directions for research and practice to increase equitable student outcomes.

Child-parent agreement on the SDQ: The role of child-parent attachment and parental feelings

OBJECTIVES

Children and their parents often provide divergent reports regarding their mental health on the Strengths and Difficulties Questionnaire (SDQ). These discrepancies may impede the diagnostic processes. The present study aimed to explore how a child's attachment to the parent and parental feelings may explain some of the variability between parent's and children's reports on the SDQ.

METHODS

Data were collected using self-report questionnaires from 277 children and their parents (n = 421) who were referred to a public mental health clinic. This information was classified into clinical categories (normal and abnormal) and analyzed using multinomial logistic regression.

RESULTS

The agreement rates between children and parents on the normality of children were high in general and across gender and age. Insecure attachment to parents positively and significantly predicted the agreement of child and parent reporting abnormality and disagreement when parents reported normality and children reported abnormality. Parental anger positively and significantly predicted disagreement in reports in which parents reported abnormal anger and children reported normal anger.

CONCLUSIONS

These findings highlight the importance of assessing informant variability in association with emotional and relationship variables as clinically meaningful information for a clinical diagnosis.

Cross-informant agreement between caregivers and teachers for prosocial behavior across child welfare settings

Healthy social functioning has been a growing focus for understanding children's positive development and well-being. Despite the flexibility and cost-effectiveness of behavior rating scales, there has been a rising issue of concern in both practice and research applications regarding rating scales and its low cross-informant agreement. The present study aimed to analyze the extent to which caregivers and teachers agree about children's prosocial behaviors, identify whether there were differences in the degree of agreement across child welfare settings, and ultimately offer recommendations for assessing behavior for children involved with child welfare services. To do so, the reports of 1,224 children, caregivers, and teachers from the National Survey of Child and Adolescent Well-Being (NSCAW) I who have been investigated by Child Protective Services (CPS) for abuse and/or neglect have been examined. Intra-class correlation coefficients (ICCs) of caregiver and teacher assessment of children's prosocial functioning were calculated to understand the degree of agreement in their perceptions of children's prosocial behaviors. ICC values presented show that agreement was low but significant across all samples and subscales (ICC range, .11-.22). It is suggested that future researchers pursue examination of the underlying mechanisms or factors that contribute to informant discrepancies.

The Operations Triad Model and Youth Mental Health Assessments: Catalyzing a Paradigm Shift in Measurement Validation

Researchers strategically assess youth mental health by soliciting reports from multiple informants. Typically, these informants (e.g., parents, teachers, youth themselves) vary in the social contexts where they observe youth. Decades of research reveal that the most common data conditions produced with this approach consist of discrepancies across informants' reports (i.e., informant discrepancies). Researchers should arguably treat these informant discrepancies as domain-relevant information: data relevant to understanding youth mental health domains (e.g., anxiety, depression, aggression). Yet, historically, in youth mental health research as in many other research areas, one set of paradigms has guided interpretations of informant discrepancies: Converging Operations and the Multi-Trait Multi-Method Matrix (MTMM). These paradigms (a) emphasize shared or common variance observed in multivariate data, and (b) inspire research practices that treat unique variance (i.e., informant discrepancies) as measurement confounds, namely random error and/or rater biases. Several yearsw ago, the Operations Triad Model emerged to address a conceptual problem that Converging Operations does not address: Some informant discrepancies might reflect measurement confounds, whereas others reflect domain-relevant information. However, addressing this problem requires more than a conceptual paradigm shift beyond Converging Operations. This problem necessitates a paradigm shift in measurement validation. We advance a paradigm (Classifying Observations Necessitates Theory, Epistemology, and Testing [CONTEXT]) that addresses problems with using the MTMM in youth mental health research. CONTEXT optimizes measurement validity by guiding researchers to leverage (a) informants that produce domain-relevant informant discrepancies, (b) analytic procedures that retain domain-relevant informant discrepancies, and (c) study designs that facilitate detecting domain-relevant informant discrepancies.

What too strict a method obscures about the validity of outcome measures

Objective: To assess the outcome of psychotherapeutic treatments, psychotherapy researchers often compare pre- and post-treatment scores on self-report outcome measures. In this paper, the common assumption is challenged that pre-to-post decreasing and increasing outcome scores are indicative of successful and failed therapies, respectively.Method: The outcome of 29 psychotherapeutic treatments was evaluated by means of quantitative analysis of pre- and post-treatment scores on commonly used outcome measures (such as the Symptom Checklist-90-R, the Inventory of Interpersonal Problems, and the General Health Questionnaire-12), as well as through consensual qualitative research.Results: Overall, a moderate to low convergence between qualitative and quantitative evaluations of outcome was observed. Detailed analyses of six cases are presented in which pre-to-post comparisons of outcome measures proved misleading.Conclusions: It is concluded that psychotherapy outcome research might benefit from assessment strategies that are sensitive to the singularities of individual treatments and to the complexity of the phenomenon of therapeutic outcome. Furthermore, classical psychometric evaluations of the validity of outcome measures might be supplemented with less-systematic evaluations that take any contingent source of information on outcome into account.

Time Spent Together in Intimate Relationships: Implications for Relationship Functioning

Current models of relationship functioning often emphasize conflict with a particular focus on the behaviors that occur in that context. Much less is known about the impact of time spent interacting in the absence of conflict. The primary aim of this study is to test associations between time spent in various forms of daily interaction (engaging in a shared activity, talking, and arguing) and multiple relationship outcomes while controlling for positive and negative communication during conflict. The present sample consists of 49 married couples (N = 98 individuals). Data were analyzed using multilevel models to account for non-independence of the data. Consistent with previous literature, communication behaviors were related to relationship outcomes. After controlling for communication, couples who spent more time arguing per day were less satisfied in their relationships, and perceived greater negative qualities in their relationships. Finally, couples who spend a larger proportion of their time together talking reported greater satisfaction, perceived more positive qualities in their relationships, and experienced greater closeness. These findings suggest that low salience interactions account for unique variance in relationship functioning above and beyond what is currently studied. Future research is needed to determine possible mechanisms by which low salience interactions are related to relationship outcomes.

A comprehensive mapping of outcomes following psychotherapy for adolescent depression: The perspectives of young people, their parents and therapists

As mental health systems move towards person-centred care, outcome measurement in clinical research and practice should track changes that matter to young people and their families. This study mapped the types of change described by three key stakeholder groups following psychotherapy for depression, and compared the salience of these outcomes with the frequency of their measurement in recent quantitative treatment effectiveness studies for adolescent depression.Using qualitative content analysis, this study identified and categorized outcomes across 102 semi-structured interviews that were conducted with depressed adolescents, their parents, and therapists, as part of a randomized superiority trial. Adolescents had been allocated to Cognitive-Behavioral Therapy, Short-Term Psychoanalytic Psychotherapy, or a Brief Psychosocial Intervention.The study mapped seven high-level outcome domains and 29 outcome categories. On average, participants discussed change in four domains and six outcome categories. The most frequently discussed outcome was an improvement in mood and affect (i.e., core depressive symptoms), but close to half of the participants also described changes in family functioning, coping and resilience, academic functioning, or social functioning. Coping had specific importance for adolescents, while parents and therapists showed particular interest in academic functioning. There was some variation in the outcomes discussed beyond these core themes, across stakeholder groups and treatment arms.Of the outcomes that were frequently discussed in stakeholder narratives, only symptomatic change has been commonly reported in recent treatment studies for adolescent depression. A shift towards considering multiple outcome domains and perspectives is needed to reflect stakeholder priorities and enable more nuanced insights into change processes.

A comparison of methods for analysing multiple outcome measures in randomised controlled trials using a simulation study

Multiple primary outcomes are sometimes collected and analysed in randomised controlled trials (RCTs), and are used in favour of a single outcome. By collecting multiple primary outcomes, it is possible to fully evaluate the effect that an intervention has for a given disease process. A simple approach to analysing multiple outcomes is to consider each outcome separately, however, this approach does not account for any pairwise correlations between the outcomes. Any cases with missing values must be ignored, unless an additional imputation step is performed. Alternatively, multivariate methods that explicitly model the pairwise correlations between the outcomes may be more efficient when some of the outcomes have missing values. In this paper, we present an overview of relevant methods that can be used to analyse multiple outcome measures in RCTs, including methods based on multivariate multilevel (MM) models. We perform simulation studies to evaluate the bias in the estimates of the intervention effects and the power of detecting true intervention effects observed when using selected methods. Different simulation scenarios were constructed by varying the number of outcomes, the type of outcomes, the degree of correlations between the outcomes and the proportions and mechanisms of missing data. We compare multivariate methods to univariate methods with and without multiple imputation. When there are strong correlations between the outcome measures (ρ > .4), our simulation studies suggest that there are small power gains when using the MM model when compared to analysing the outcome measures separately. In contrast, when there are weak correlations (ρ < .4), the power is reduced when using univariate methods with multiple imputation when compared to analysing the outcome measures separately.

Change processes underlying "good outcome": A qualitative study on recovered and improved patients' experiences in psychotherapy for major depression

Aim: Exploring change processes underlying "good outcome" in psychotherapy for major depression. We examined the perspectives of patients who "recovered" and "improved" (Jacobson & Truax) following time-limited CBT and PDT. Method: In the context of an RCT on the treatment of major depression, patients were selected based on their pre-post outcome scores on the BDI-II: we selected 28 patients who recovered and 19 who improved in terms of depressive symptoms. A grounded theory analysis was conducted on post-therapy client change interviews, resulting in an integrative conceptual model. Results: According to recovered and improved patients, change follows from an interaction between therapy, therapist, patient, and extra-therapeutic context. Both helping and hindering influences were mentioned within all four influencing factors. Differences between recovered and improved patients point at the role of patients' agency and patients' internal and external obstacles. However, patients marked as "improved" described heterogeneous experiences. CBT- and PDT-specific experiences were also observed, although our findings suggest the possible role of therapist-related influences. Conclusion: From patients' perspectives, various change processes underlie "good outcome" that do not necessarily imply an "all good process". This supports a holistic, multidimensional conceptualization of change processes in psychotherapy and calls for more fine-grained mixed-methods process-outcome research.

Changes in playfulness, creativity and honesty as possible outcomes of psychotherapy

Objectives: This study comprised an examination of whether clients' playfulness, creativity, honesty, humor, and happiness changed during psychotherapy. Methods: Sixty-two clients who underwent psychotherapy in a naturalistic setting completed questionnaires at five time points throughout treatment. An HLM analytic approach was applied to account for the hierarchical nature of the data. Results: Mental distress declined during treatment, while playfulness and creativity increased significantly. Honesty decreased significantly in the course of the treatment, while no significant change was found in the level of affiliative humor or the level of happiness. Conclusions: Changes in personality variables that can serve as positive constructs defining mental health, namely playfulness, creativity, and honesty, might be a possible outcome of psychotherapy.

Methods to adjust for multiple comparisons in the analysis and sample size calculation of randomised controlled trials with multiple primary outcomes

BACKGROUND

Multiple primary outcomes may be specified in randomised controlled trials (RCTs). When analysing multiple outcomes it's important to control the family wise error rate (FWER). A popular approach to do this is to adjust the p-values corresponding to each statistical test used to investigate the intervention effects by using the Bonferroni correction. It's also important to consider the power of the trial to detect true intervention effects. In the context of multiple outcomes, depending on the clinical objective, the power can be defined as: 'disjunctive power', the probability of detecting at least one true intervention effect across all the outcomes or 'marginal power' the probability of finding a true intervention effect on a nominated outcome. We provide practical recommendations on which method may be used to adjust for multiple comparisons in the sample size calculation and the analysis of RCTs with multiple primary outcomes. We also discuss the implications on the sample size for obtaining 90% disjunctive power and 90% marginal power.

METHODS

We use simulation studies to investigate the disjunctive power, marginal power and FWER obtained after applying Bonferroni, Holm, Hochberg, Dubey/Armitage-Parmar and Stepdown-minP adjustment methods. Different simulation scenarios were constructed by varying the number of outcomes, degree of correlation between the outcomes, intervention effect sizes and proportion of missing data.

RESULTS

The Bonferroni and Holm methods provide the same disjunctive power. The Hochberg and Hommel methods provide power gains for the analysis, albeit small, in comparison to the Bonferroni method. The Stepdown-minP procedure performs well for complete data. However, it removes participants with missing values prior to the analysis resulting in a loss of power when there are missing data. The sample size requirement to achieve the desired disjunctive power may be smaller than that required to achieve the desired marginal power. The choice between whether to specify a disjunctive or marginal power should depend on the clincial objective.

Informant discrepancies in assessments of psychosocial functioning in school-based services and research: Review and directions for future research

Psychosocial functioning plays a key role in students' wellbeing and performance inside and outside of school. As such, techniques designed to measure and improve psychosocial functioning factor prominently in school-based service delivery and research. Given that the different contexts (e.g., school, home, community) in which students exist vary in the degree to which they influence psychosocial functioning, educators and researchers often rely on multiple informants to characterize intervention targets, monitor intervention progress, and inform the selection of evidence-based services. These informants include teachers, students, and parents. Across research teams, domains, and measurement methodologies, researchers commonly observe discrepancies among informants' reports. We review theory and research-occurring largely outside of school-based service delivery and research-that demonstrates how patterns of informant discrepancies represent meaningful differences that can inform our understanding of psychosocial functioning. In turn, we advance a research agenda to improve use and interpretation of informant discrepancies in school-based services and research.

No Change? A Grounded Theory Analysis of Depressed Patients' Perspectives on Non-improvement in Psychotherapy

Aim: Understanding the effects of psychotherapy is a crucial concern for both research and clinical practice, especially when outcome tends to be negative. Yet, while outcome is predominantly evaluated by means of quantitative pre-post outcome questionnaires, it remains unclear what this actually means for patients in their daily lives. To explore this meaning, it is imperative to combine treatment evaluation with quantitative and qualitative outcome measures. This study investigates the phenomenon of non-improvement in psychotherapy, by complementing quantitative pre-post outcome scores that indicate no reliable change in depression symptoms with a qualitative inquiry of patients' perspectives. Methods: The study took place in the context of a Randomised Controlled Trial evaluating time-limited psychodynamic and cognitive behavioral therapy for major depression. A mixed methods study was conducted including patients' pre-post outcome scores on the BDI-II-NL and post treatment Client Change Interviews. Nineteen patients whose data showed no reliable change in depression symptoms were selected. A grounded theory analysis was conducted on the transcripts of patients' interviews. Findings: From the patients' perspective, non-improvement can be understood as being stuck between knowing versus doing, resulting in a stalemate. Positive changes (mental stability, personal strength, and insight) were stimulated by therapy offering moments of self-reflection and guidance, the benevolent therapist approach and the context as important motivations. Remaining issues (ambition to change but inability to do so) were attributed to the therapy hitting its limits, patients' resistance and impossibility and the context as a source of distress. "No change" in outcome scores therefore seems to involve a "partial change" when considering the patients' perspectives. Conclusion: The study shows the value of integrating qualitative first-person analyses into standard quantitative outcome evaluation and particularly for understanding the phenomenon of non-improvement. It argues for more multi-method and multi-perspective research to gain a better understanding of (negative) outcome and treatment effects. Implications for both research and practice are discussed.

Improving Interpretability of Subjective Assessments About Psychological Phenomena: A Review and Cross-Cultural Meta-Analysis

Attempts to understand subjectivity have historically involved distinguishing the strengths of subjective methods (e.g., survey ratings from informants) from those of alternative methods (e.g., observational/performance-based tasks). Yet a movement is underway in Psychology that considers the merits of intersubjectivity: Understanding the space between two or more informant’s subjective impressions of a common person or phenomenon. In mental health research, understanding differences between subjective impressions have less to do with informants’ characteristics and more to do with the social environments or contexts germane to the people or phenomena examined. Our article focuses on one relatively understudied social environment: the cultural context. We draw from seminal work on psychological universals, as well as emerging work on cultural norms (i.e., cultural tightness) to understand intersubjectivity effects through a cross-cultural lens. We report a meta-analysis of 314 studies of intersubjectivity effects in mental health, revealing that (a) this work involves independent research teams in more than 30 countries, (b) informants rating a target person’s mental health (e.g., parent and teacher ratings of a child’s behavior) commonly provide diverging estimates of that person’s mental health, and (c) greater convergence between subjective reports relates to a “tighter” or more norms-bound culture. Our article illustrates strategies for understanding divergence between subjective reports. In particular, we highlight theoretical and methodological frameworks for examining patterns of divergence between subjective reports in relation to data from nonsubjective methods. We also describe how research on intersubjectivity informs efforts to improve the interpretability of subjective assessments in multiple subdisciplines in Psychology.

Validity Beyond Measurement: Why Psychometric Validity Is Insufficient for Valid Psychotherapy Research

In psychotherapy research, "validity" is canonically understood as the capacity of a test to measure what is purported to measure. However, we argue that this psychometric understanding of validity prohibits working researchers from considering the validity of their research. Psychotherapy researchers often use measures with a different epistemic goal than test developers intended, for example when a depression symptom measure is used to indicate "treatment success" (cf. outcome measurement for evidence-based treatment). However, the validity of a measure does not cover the validity of its use as operationalization of another target concept within a research procedure, nor the validity of its function toward an epistemic goal. In this paper, we discuss the importance of considering validity of the epistemic process beyond the validity of measures per se, based on an empirical case example from our psychotherapy study ("SCS", Cornelis et al., 2017). We discuss why the psychometric understanding of validity is insufficient in covering epistemic validity, and we evaluate to what extent the available terminology regarding validity of research is sufficient for working researchers to accurately consider the validity of their overall epistemic process. As psychotherapy research is meant to offer a sound evidence-base for clinical practice, we argue that it is vital that psychotherapy researchers are able to discuss the validity of the epistemic choices made to serve the clinical goal.

Review: What Outcomes Count? A Review of Outcomes Measured for Adolescent Depression Between 2007 and 2017

OBJECTIVE

Measurement of treatment outcomes in childhood depression has traditionally focused on assessing symptoms from the clinician's perspective, without exploring other outcome domains or considering young people's perspectives. This systematic review explored the extent to which multidimensional and multi-informant outcome measurements have been used in clinical research for adolescent depression in the past decade and how patterns have evolved over time.

METHOD

Embase, Medline, and PsycINFO were searched, and studies that were published from 2007 through 2017 and assessed the effectiveness of treatments or service provision for adolescent depression were included. Of 7,483 studies screened, 95 met the inclusion criteria and were included for data extraction and analysis.

RESULTS

Ten outcomes domains were identified, 2 of which were assessed on average using 4 outcome measures. Most studies (94%) measured symptoms, followed by functioning (52%). Other domains such as personal growth, relationships, quality of life, and service satisfaction were each considered by less than 1 in 10 studies. Youth self-report was considered by 54% but tended to focus on secondary outcomes. Multidimensional and multi-informant measurements were more frequent in studies published since 2015.

CONCLUSION

Recent clinical research continues to prioritize symptoms measurement based on clinician report and has not yet fully embraced multidimensional and multi-informant approaches. In the context of significant policy shifts toward patient-centered and evidence-based care, measuring what matters most to patients has become a priority, but this is not yet widely reflected in clinical research.

Understanding Long-term Outcome from the Patients' Perspective: A Mixed Methods Naturalistic Study on Inpatient Psychotherapy

OBJECTIVE

The complex phenomenon of psychotherapy outcome requires further conceptual and methodological developments that facilitate clinically meaningful research findings. In this study, we rely on an idiosyncratic and process-oriented understanding of treatment effects in order to investigate long-term outcome. A conceptual model of long-term outcome is presented that comprises both a taxonomy of change and explanatory factors.

METHOD

A mixed methods naturalistic study was conducted in an inpatient psychotherapy setting. Long-term quantitative outcome data are complemented with a data-driven thematic analysis of interviews with 22 participants, five to six years after ending inpatient psychotherapy.

RESULTS

Long-term outcome findings show improved well-being for the majority of former patients and this until five to six years after treatment. From the patients' perspectives, long-term changes can be situated on different interrelated existential levels: reconnection to others and (the meaning of) life, a revelation, an altered self, life changes, and altered expectations and ideas about recovery and treatment. The complex interplay of the person, the therapy centre, the outside world and the evolution over time helped explain the experienced changes and individual differences.

CONCLUSION

The findings support the value of an idiosyncratic and process-oriented understanding of outcome and recovery as well as substantiate the importance of multiple methods and perspectives when studying the effects of psychotherapy.

The contribution of parent and youth information to identify mental health disorders or problems in adolescents

BACKGROUND

Discrepancies between multiple informants often create considerable uncertainties in delivering services to youth. The present study assessed the ability of the parent and youth scales of the Strength and Difficulties Questionnaire (SDQ) to predict mental health problems/disorders across several mental health domains as validated against two contrasting indices of validity for psychopathology derived from the Development and Well Being Assessment (DAWBA): (1) an empirically derived computer algorithm and (2) expert based ICD-10 diagnoses.

METHODS

Ordinal and logistic regressions were used to predict any problems/disorders, emotional problems/disorders and behavioural problems/disorders in a community sample (n = 252) and in a clinic sample (n = 95).

RESULTS

The findings were strikingly similar in both samples. Parent and youth SDQ scales were related to any problem/disorder. Youth SDQ symptom and impact had the strongest association with emotional problems/disorder and parent SDQ symptom score were most strongly related to behavioural problems/disorders. Both the SDQ total and the impact scores significantly predicted emotional problems/disorders in males whereas this was the case only for the total SDQ score in females.

CONCLUSION

The present study confirms and expands previous findings on parent and youth informant validity. Clinicians should include both parent and youth for identifying any mental health problems/disorders, youth information for detecting emotional problems/disorders, and parent information to detect behavioural problems/disorders. Not only symptom scores but also impact measures may be useful to detect emotional problems/disorders, particularly in male youth.

Subjective - Objective Sleep Comparisons and Discrepancies Among Clinically-Anxious and Healthy Children

We compared subjective and objective sleep patterns and problems, and examined cross-method correspondence across parent reports, child reports, and actigraphy-derived sleep variables in clinically-anxious children and healthy controls. In a multi-site, cross-sectional study, 75 pre-adolescent children (6 to 11 years; M = 8.7 years; SD = 1.4; n = 39/52 % female) were examined including 39 with a diagnosis of primary generalized anxiety disorder (GAD) and 36 controls recruited from university-based clinics in Houston, TX and Washington, DC. Structured interviews, validated sleep questionnaires, and 1 week of actigraphy data were utilized. Despite subjective reports of significantly greater sleep problems among anxious children, actigraphy data revealed no significant differences between the groups. All parents estimated earlier bedtimes and greater total sleep duration relative to actigraphy, and all children endorsed more sleep problems than parents. With few exceptions, subjective reports exhibited low and non-significant correspondence with actigraphy-based sleep patterns and problems. Our findings suggest that high rates of sleep complaints found among children with GAD (and their parents) are not corroborated by objective sleep abnormalities, with the exception of marginally prolonged sleep onset latency compared to controls. Objective-subjective sleep discrepancies were observed in both groups but more apparent overall in the GAD group. Frequent complaints of sleep problems and daytime tiredness among anxious youth might more accurately reflect difficulties prior to the actual sleep period, cognitive-affective biases associated with sleep, and/or poor sleep quality. Findings highlight the importance of considering sleep from multiple perspectives.

Are multiple primary outcomes analysed appropriately in randomised controlled trials? A review

OBJECTIVES

To review how multiple primary outcomes are currently considered in the analysis of randomised controlled trials. We briefly describe the methods available to safeguard the inferences and to raise awareness of the potential problems caused by multiple outcomes.

METHODS/DESIGN

We reviewed randomised controlled trials (RCTs) in neurology and psychiatry disease areas, as these frequently analyse multiple outcomes. We reviewed all published RCTs from July 2011 to June 2014 inclusive in the following high impact journals: The New England Journal of Medicine, The Lancet, The American Journal of Psychiatry, JAMA Psychiatry, The Lancet Neurology and Neurology. We examined the information presented in the abstract and the methods used for sample size calculation and statistical analysis. We recorded the number of primary outcomes, the methods used to account for multiple primary outcomes, the number of outcomes discussed in the abstract and the number of outcomes used in the sample size calculation.

RESULTS

Of the 209 RCTs that we identified, 60 (29%) analysed multiple primary outcomes. Of these, 45 (75%) did not adjust for multiplicity in their analyses. Had multiplicity been addressed, some of the trial conclusions would have changed. Of the 15 (25%) trials which accounted for multiplicity, Bonferroni's correction was the most commonly used method.

CONCLUSIONS

Our review shows that trials with multiple primary outcomes are common. However, appropriate steps are not usually taken in most of the analyses to safeguard the inferences against multiplicity. Authors should state their chosen primary outcomes clearly and justify their methods of analysis.

The validity of the multi-informant approach to assessing child and adolescent mental health

Child and adolescent patients may display mental health concerns within some contexts and not others (e.g., home vs. school). Thus, understanding the specific contexts in which patients display concerns may assist mental health professionals in tailoring treatments to patients' needs. Consequently, clinical assessments often include reports from multiple informants who vary in the contexts in which they observe patients' behavior (e.g., patients, parents, teachers). Previous meta-analyses indicate that informants' reports correlate at low-to-moderate magnitudes. However, is it valid to interpret low correspondence among reports as indicating that patients display concerns in some contexts and not others? We meta-analyzed 341 studies published between 1989 and 2014 that reported cross-informant correspondence estimates, and observed low-to-moderate correspondence (mean internalizing: r = .25; mean externalizing: r = .30; mean overall: r = .28). Informant pair, mental health domain, and measurement method moderated magnitudes of correspondence. These robust findings have informed the development of concepts for interpreting multi-informant assessments, allowing researchers to draw specific predictions about the incremental and construct validity of these assessments. In turn, we critically evaluated research on the incremental and construct validity of the multi-informant approach to clinical child and adolescent assessment. In so doing, we identify crucial gaps in knowledge for future research, and provide recommendations for "best practices" in using and interpreting multi-informant assessments in clinical work and research. This article has important implications for developing personalized approaches to clinical assessment, with the goal of informing techniques for tailoring treatments to target the specific contexts where patients display concerns. (PsycINFO Database Record

The validity of the multi-informant approach to assessing child and adolescent mental health

Child and adolescent patients may display mental health concerns within some contexts and not others (e.g., home vs. school). Thus, understanding the specific contexts in which patients display concerns may assist mental health professionals in tailoring treatments to patients' needs. Consequently, clinical assessments often include reports from multiple informants who vary in the contexts in which they observe patients' behavior (e.g., patients, parents, teachers). Previous meta-analyses indicate that informants' reports correlate at low-to-moderate magnitudes. However, is it valid to interpret low correspondence among reports as indicating that patients display concerns in some contexts and not others? We meta-analyzed 341 studies published between 1989 and 2014 that reported cross-informant correspondence estimates, and observed low-to-moderate correspondence (mean internalizing: r = .25; mean externalizing: r = .30; mean overall: r = .28). Informant pair, mental health domain, and measurement method moderated magnitudes of correspondence. These robust findings have informed the development of concepts for interpreting multi-informant assessments, allowing researchers to draw specific predictions about the incremental and construct validity of these assessments. In turn, we critically evaluated research on the incremental and construct validity of the multi-informant approach to clinical child and adolescent assessment. In so doing, we identify crucial gaps in knowledge for future research, and provide recommendations for "best practices" in using and interpreting multi-informant assessments in clinical work and research. This article has important implications for developing personalized approaches to clinical assessment, with the goal of informing techniques for tailoring treatments to target the specific contexts where patients display concerns. (PsycINFO Database Record

Introduction to the special issue: toward implementing physiological measures in clinical child and adolescent assessments

The National Institute of Mental Health recently launched the Research Domain Criteria (RDoC). The RDoC is an initiative to improve classification of mental health concerns by promoting research on the brain mechanisms underlying these concerns, with the ultimate goal of developing interventions that target these brain mechanisms. A key focus of RDoC involves opening new lines of research examining patients' responses on biological measures. The RDoC presents unique challenges to mental health professionals who work with children and adolescents. Indeed, mental health professionals rarely integrate biological measures into clinical assessments. Thus, RDoC's ability to improve patient care rests, in part, on the development of strategies for implementing biological measures within mental health assessments. Further, mental health professionals already carry out comprehensive assessments that frequently yield inconsistent findings. These inconsistencies have historically posed challenges to interpreting research findings as well as assessment outcomes in practice settings. In this introductory article, we review key issues that informed the development of a special issue of articles demonstrating methods for implementing low-cost measures of physiological functioning in clinical child and adolescent assessments. We also outline a conceptual framework, informed by theoretical work on using and interpreting multiple informants' clinical reports (De Los Reyes, Thomas, Goodman, & Kundey, 2013 ), to guide hypothesis testing when using physiological measures within clinical child and adolescent assessments. This special issue and the conceptual model described in this article may open up new lines of research testing paradigms for implementing clinically feasible physiological measures in clinical child and adolescent assessments.

Non-clinical health services – what do they contribute to patient care and how can we measure their contribution?

Background and aims

Assessing the contribution of ‘support services’ to improving healthcare is challenging. Despite involving a significant part of the NHS budget, there is very little published literature on methods and approaches. This article describes the development of an integrated approach to assessing and measuring the health impact of these services.

Methods and results

An empirical approach was developed by the main provider of national support services to NHS Scotland. This involved identifying meaningful ways to describe health impact of services several steps removed from patient care, applying this to the full range of national support services provided by NHS National Services Scotland and developing relevant measurement methodologies. Through this approach it was possible to assess the relative health impact of all 63 NSS services and to use this for planning and prioritisation.

Conclusion

Assessing the contribution of support services to improving patient outcomes is a neglected area, despite the significant dependency of direct patient care services upon these services and the resource involved. The methods described in this article have potential to be used across the range of support services within the NHS to improve quality and efficiency of healthcare services.

Direkte Veränderungsmessung in der Psychotherapie

Theoretischer Hintergrund: Verfahren zur direkten Veränderungsmessung bestimmen den Psychotherapieerfolg in ökonomischer Ein-Punkt-Messung. Hierzu wurde der Bochumer Veränderungsbogen-2000 (BVB-2000) als überarbeitete und gekürzte Variante des Veränderungsfragebogens des Erlebens und Verhaltens (VEV; Zielke & Kopf-Mehnert, 1978 ) entwickelt. Fragestellung: Der BVB-2000 soll einer testtheoretischen Reanalyse mit Fokus auf die konvergente Validität unterzogen werden. Kritische Veränderungswerte auf Basis der Werte einer klinischen Wartekontrollgruppe sollen die Möglichkeit bieten den Therapieerfolg auch in Einzelfällen zu bestimmen. Methode: Die Psychotherapie von N = 205 Patienten wird mit verschiedenen Instrumenten zur Therapieerfolgsmessung inklusive des BVB-2000 begleitet. Auf Basis einer Wartekontrollgruppe (N = 88) werden kritische Veränderungswerte für den BVB-2000 berechnet. Ergebnisse: Der BVB-2000 weist eine hohe interne Konsistenz auf (α = .96; 26 Items) und zeigt durchgängig statistisch bedeutsame Zusammenhänge mit anderen Therapieerfolgsmaßen vor allem der Zielerreichung. Kritische Veränderungswerte ermöglichen die Einschätzung des Therapieerfolgs im Einzelfall. Schlussfolgerungen: Der BVB-2000 ist ein verständliches, ökonomisches, reliables und valides Instrument zur Psychotherapieerfolgsmessung.

Discrepancies between parent and adolescent beliefs about daily life topics and performance on an emotion recognition task

Parents and children and adolescents commonly disagree in their perceptions of a variety of behaviors, including the family relationship and environment, and child and adolescent psychopathology. To this end, numerous studies have examined to what extent increased discrepant perceptions-particularly with regard to perceptions of the family relationship and environment-predict increased child and adolescent psychopathology. Parents' and children and adolescents' abilities to decode and identify others' emotions (i.e., emotion recognition) may play a role in the link between discrepant perceptions and child and adolescent psychopathology. We examined parents' and adolescents' emotion recognition abilities in relation to discrepancies between parent and adolescent perceptions of daily life topics. In a sample of 50 parents and adolescents ages 14-to-17 years (M = 15.4 years, 20 males, 54 % African-American), parents and adolescents were each administered a widely used performance-based measure of emotion recognition. Parents and adolescents were also administered a structured interview designed to directly assess each of their perceptions of the extent to which discrepancies existed in their beliefs about daily life topics (e.g., whether adolescents should complete their homework and carry out household chores). Interestingly, lower parent and adolescent emotion recognition performance significantly related to greater parent and adolescent perceived discrepant beliefs about daily life topics. We observed this relation whilst accounting for adolescent age and gender and levels of parent-adolescent conflict. These findings have important implications for understanding and using informant discrepancies in both basic developmental psychopathology research and applied research in clinic settings (e.g., discrepant views on therapeutic goals).

Newer modalities in the work-up of peripheral pulmonary nodules

Technological advances in recent years have translated into the availability of newer modalities to establish the cause of peripheral pulmonary nodules (PPN). Even though the verdict is still out on the ideal diagnostic modality, there is no doubt that the bronchoscope is becoming a popular tool in the armamentarium of physicians who deal with PPN. This article focuses on newer bronchoscopic modalities being studied for the work-up of PPN. The authors also summarize the value of established diagnostic modalities to provide a balanced perspective.

Strategic objectives for improving understanding of informant discrepancies in developmental psychopathology research

Developmental psychopathology researchers and practitioners commonly conduct behavioral assessments using multiple informants' reports (e.g., parents, teachers, practitioners, children, and laboratory observers). These assessments often yield inconsistent conclusions about important questions in developmental psychopathology research, depending on the informant (e.g., psychiatric diagnoses and risk factors of disorder). Researchers have theorized why informant discrepancies exist and advanced methodological models of informant discrepancies. However, over 50 years of empirical data has uncovered little knowledge about these discrepancies beyond that they exist, complicate interpretations of research findings and assessment outcomes in practice, and correlate with some characteristics of the informants providing reports (e.g., demographics and mood levels). Further, recent studies often yield take-home messages about the importance of taking a multi-informant approach to clinical and developmental assessments. Researchers draw these conclusions from their work, despite multi-informant approaches to assessment long being a part of best practices in clinical and developmental assessments. Consequently, developmental psychopathology researchers and practitioners are in dire need of a focused set of research priorities with the key goal of rapidly advancing knowledge about informant discrepancies. In this paper, I discuss these research priorities, review work indicating the feasibility of conducting research addressing these priorities, and specify what researchers and practitioners would gain from studies advancing knowledge about informant discrepancies in developmental psychopathology research.

Discrepancies between parent and adolescent beliefs about daily life topics and performance on an emotion recognition task

Parents and children and adolescents commonly disagree in their perceptions of a variety of behaviors, including the family relationship and environment, and child and adolescent psychopathology. To this end, numerous studies have examined to what extent increased discrepant perceptions-particularly with regard to perceptions of the family relationship and environment-predict increased child and adolescent psychopathology. Parents' and children and adolescents' abilities to decode and identify others' emotions (i.e., emotion recognition) may play a role in the link between discrepant perceptions and child and adolescent psychopathology. We examined parents' and adolescents' emotion recognition abilities in relation to discrepancies between parent and adolescent perceptions of daily life topics. In a sample of 50 parents and adolescents ages 14-to-17 years (M = 15.4 years, 20 males, 54 % African-American), parents and adolescents were each administered a widely used performance-based measure of emotion recognition. Parents and adolescents were also administered a structured interview designed to directly assess each of their perceptions of the extent to which discrepancies existed in their beliefs about daily life topics (e.g., whether adolescents should complete their homework and carry out household chores). Interestingly, lower parent and adolescent emotion recognition performance significantly related to greater parent and adolescent perceived discrepant beliefs about daily life topics. We observed this relation whilst accounting for adolescent age and gender and levels of parent-adolescent conflict. These findings have important implications for understanding and using informant discrepancies in both basic developmental psychopathology research and applied research in clinic settings (e.g., discrepant views on therapeutic goals).

Informant discrepancies in adult social anxiety disorder assessments: links with contextual variations in observed behavior

Multi-informant assessments of adult psychopathology often result in discrepancies among informants' reports. Among 157 adults meeting criteria for either the generalized (n = 106) or nongeneralized (n = 51) social anxiety disorder (SAD) subtype, we examined whether discrepancies between patients' and clinicians' reports of patients' symptoms related to variations in both SAD subtype and expressions of social skills deficits across multiple social interaction tasks. Latent class analyses revealed two behavioral patterns: (a) context-specific social skills deficits and (b) cross-context social skills deficits. Similarly, patients' symptom reports could be characterized by concordance or discordance with clinicians' reports. Patient-clinician concordance on relatively high levels of patients' symptoms related to an increased likelihood of the patient meeting criteria for the generalized relative to nongeneralized subtype. Further, patient-clinician concordance on relatively high levels of patients' symptoms related to an increased likelihood of consistently exhibiting social skills deficits across social interaction tasks (relative to context-specific social skills deficits). These relations were robust in accounting for patient age, clinical severity, and Axis I and II comorbidity. Further, clinical severity did not completely explain variability in patients' behavior on laboratory tasks or discrepancies between patient and clinician reports. Findings provide the first laboratory-based support for the ability of informant discrepancies to indicate cross-contextual variability in clinical adult assessments, and the first of any developmental period to indicate this for SAD assessments. These findings have important implications for clinical assessment and developmental psychopathology research.

Principles underlying the use of multiple informants' reports

Researchers use multiple informants' reports to assess and examine behavior. However, informants' reports commonly disagree. Informants' reports often disagree in their perceived levels of a behavior ("low" versus "elevated" mood), and examining multiple reports in a single study often results in inconsistent findings. Although researchers often espouse taking a multi-informant assessment approach, they frequently address informant discrepancies using techniques that treat discrepancies as measurement error. Yet, recent work indicates that researchers in a variety of fields often may be unable to justify treating informant discrepancies as measurement error. In this review, the authors advance a framework (Operations Triad Model) outlining general principles for using and interpreting informants' reports. Using the framework, researchers can test whether or not they can extract meaningful information about behavior from discrepancies among multiple informants' reports. The authors provide supportive evidence for this framework and discuss its implications for hypothesis testing, study design, and quantitative review.