When quantitative measures become a qualitative storybook: A phenomenological case analysis of validity and performativity of questionnaire administration in psychotherapy research

How do we know whether treatment has failed? Paradoxical outcomes in counseling with young people

Background

In both routine practice contexts and research studies, evidence from standardized self-report symptom measures, administered pre- and post-treatment, is predominantly used to determine whether psychotherapy has been successful. Understanding the nature of unsuccessful psychotherapy requires an ability to evaluate the credibility of outcome data generated by such techniques. An important body of research has identified discrepancies between outcomes assessed through symptom measures and those obtained from other sources. However, not enough is known about the extent to which such paradoxical outcomes exist.

Objective

This study analyzes the relationship between outcomes, as assessed by a standardized self-report measure, and as assessed by ratings of young people's descriptions of change at post-counseling interviews.

Methods

Participants were 50 young people (13-16 years old) who had taken part in a trial of up to 10 weeks of school-based humanistic counseling. Our primary standardized measure was the Young Person's CORE (YP-CORE). To assess young people's experiences of counseling change, three independent raters scrutinized transcripts of post-counseling interviews, and scored levels of helpfulness on a 1 (Not at all helpful) to 10 (Extremely helpful) scale. Inter-rater reliabilities were 0.94 (Cronbach's Alpha) and 0.96 (McDonald's Omega). Sensitivity analyses were conducted to explore relationships between helpfulness ratings and other outcome measures, i.e., satisfaction with counseling (ESQ) and the Goal-Based-Outcome Tool (GBO), and process measures, i.e., the Working Alliance Inventory (WAI-S) and the Barret Lennard Relationship Inventory (BLRI).

Results

Multilevel analysis indicated that helpfulness ratings were not significantly associated with changes in YP-CORE scores. Analyzed categorically, 38% of those showing reliable improvement on the standardized measure were below the median for self-described helpfulness, and 47% of those not showing reliable change were at or above the median for self-described helpfulness. Sensitivity analyses demonstrated closer correlations between helpfulness ratings and other outcome measures (ESQ and GBO), and between helpfulness ratings and process measures (WAI-S and BLRI).

Discussion

Our results raise questions about reliance on symptom change outcome measures for defining treatment success and failure, given their disparity with clients' own descriptions of the helpfulness of therapy. Implications for practice and research are discussed.

Rigorous idiography: Exploring subjective and idiographic data with rigorous methods-The method of derangements

Psychological research often seeks general rules applying across individuals, an aim that is in tension with examining that which is unique to any individual. There are general statistical regularities across individuals' subjective self-report which enable much psychology and psychotherapy research to combine data from self-report questionnaire responses with statistical and psychometric methods to create a fundamental part of Cronbach and Meehl's foundational nomological networks of validity. However, these methods only apply when most participants answer the same questions on measures creating nomothetic data and this has led to a neglect of idiographic data. This paper reviews a method of analysis of idiographic data, of "rigorous idiography": the method of derangements. This is a remarkably simple statistical test of whether purely idiographic data convey reliable information. We show how the method appeared to become stuck in a bibliometric backwater but we expand on its potential for research and practise and hope it will be taken up and used correctly and more widely.

Patient-reported outcomes of the nurse-patient relationship in psychiatric inpatient hospitals: A multicentred descriptive cross-sectional study

WHAT IS KNOWN ON THE SUBJECT?: Psychiatric and/or mental health nurses are struggling to measure the outcomes of the nurse-patient relationship. Collecting nurse-sensitive patient outcomes is a strategy to provide outcomes of a nurse-patient relationship from patients' perspectives. Because there was no validated scale, the Mental Health Nurse-Sensitive Patient Outcome-Scale (six-point Likert-scale) was recently developed and psychometrically evaluated. WHAT THE PAPER ADDS TO EXISTING KNOWLEDGE?: This is the first study using the Mental Health Nurse-Sensitive Patient Outcome-scale to measure nurse-sensitive patient outcomes of the nurse-patient relationship in psychiatric hospitals. Moderate to good average scores for the MH-NURSE-POS total (4.42) and domains scores (≥4.09). are observed. Especially outcomes related to 'motivation' to follow and stay committed to the treatment received high average scores (≥4.60). Our results are consistent with the patient-reported effect(s) of relation-based nursing in qualitative research. The scores generate evidence to support the outcomes of the nurse-patient relationship and implicates that further investment in (re)defining and elaborating nurse-patient relationships in mental healthcare is meaningful and justified. More comparative patient-reported data can determine how nurse-sensitive patient outcomes are affected by the patient, nurse, and context. WHAT ARE THE IMPLICATIONS FOR PRACTICE?: Demonstrating patient-reported outcomes of the nurse-patient relationship can be important to enhance the therapeutic alliance between nurses and patients, organize responsive nursing care, and create nursing visibility in mental healthcare. Further nursing staff training on interpersonal competencies, such as self-awareness and cultural sensitivity, can be pivotal to achieving the patient-reported outcomes for inpatients with mental health problems.

ABSTRACT

INTRODUCTION: Identifying patient-reported outcomes of the nurse-patient relationship is a priority in inpatient mental healthcare to guide clinical decision-making and quality improvement initiatives. Moreover, demonstrating nurse-sensitive patient outcomes can be a strategy to avoid further erosion of the specialism of psychiatric and/or mental health nursing.

AIM/QUESTION

To measure nurse-sensitive patient outcomes of the nurse-patient relationship.

METHOD

In a multicentred cross-sectional study, 296 inpatients admitted to five psychiatric hospitals completed the recently developed and validated Mental Health Nurse-Sensitive Patient Outcome-Scale (MH-NURSE-POS). The MH-NURSE-POS consists of 21 items (six-point Likert-scale) in four domains: 'growth', 'expression', 'control', and 'motivation'.

RESULTS

Participants displayed moderate to good average scores for the MH-NURSE-POS total (4.42) and domain scores (≥4.09). Especially outcomes related to 'motivation' to follow and stay committed to the treatment received high average scores (≥4.60).

DISCUSSION

The results demonstrate that patients perceive the nurse-patient relationship and the care given by psychiatric and/or mental health nurses as contributing to their treatment.

IMPLICATIONS FOR PRACTICES

Patient-reported outcomes can guide nurses and managers to provide and organize nursing care and to build a nurse-patient relationship that has a positive impact on these outcomes. Additionally, outcomes can create nursing visibility as a profession in- and outside mental healthcare.

Back to Basics: The Importance of Conceptual Clarification in Psychological Science

Although the lack of conceptual clarity has been observed to be a widespread and fundamental problem in psychology, conceptual clarification plays a mostly marginal role in psychological research. In this article, we argue that better conceptualization of psychological phenomena is needed to move psychology forward as a science. We first show how conceptual unclarity seeps through all aspects of psychological research, from everyday concepts to statistical measures. We then turn to recommendations on how to improve conceptual clarity in psychology, emphasizing the importance of seeing research as an iterative process in which it is necessary to revisit the phenomena that are the foundations of theories and models, as well as how they are conceptualized and measured.

"Infeliz" or "Triste": A Paradigm for Mixed Methods Exploration of Outcome Measures Adaptation Across Language Variants

The literature on measure translation tends to hold, overtly or covertly, a questionable assumption about the possibility of exact translation and almost completely ignores issues of within language variation. Equally, psychometric methods used to assess cross-cultural validity after translation focus on large sample tests of cross-sectional measurement invariance. Such invariance is often not found and is of dubious pertinence to change/outcome measures usually used in psychotherapy research. We present a sequential process of three substudies using quantitative and qualitative procedures to explore whether an outcome measure needs to be changed when used across language variation. Qualitative data confirmed that an item was not ideal in the new context. However, quantitative exploration showed that, although statistically significant and affected by gender and item order, the impact of changing the item in the overall score was small, allowing retention of the existing Spanish translation. We argue that the myth of perfect translation and over-reliance on large-sample psychometric testing pursuing measurement invariance limit exploration of language effects. We recommend that these be used in the companion of user-based, sequential, mixed-method exploration to support the development of a richer field of understanding of outcomes and change self-report measures across languages and cultures and both across and within languages.

Comparing helpful and hindering processes in good and poor outcome cases: A qualitative metasynthesis of eight Hermeneutic Single Case Efficacy Design studies

Objective: We tested qualitative metasynthesis of a series of Hermeneutic Single Case Efficacy Design (HSCED) studies as a method for comparing within-session processes that may explain good and poor therapeutic outcome. Method: We selected eight HSCED studies according to change in clients' scores on the Strathclyde Inventory (SI), a brief self-report instrument used to measure outcome in person-centered psychotherapy. Four of the case studies investigated the experience of clients whose pre-post change in SI scores showed improvement by the end of therapy, and the other four focused on clients whose change in SI scores indicated deterioration. We conducted a qualitative metasynthesis, adopting a generic descriptive-interpretive approach to analyze and compare the data generated by the HSCED studies. Results: In contrast to improvers, deteriorators appeared to be less ready to engage in therapeutic work at the beginning of therapy, and found the process more difficult; their therapists were less able to respond to these difficulties in a responsive, empathic manner; deteriorators were less able to cope successfully with changes of therapist and, eventually, gave up on therapy. Conclusion: We found that our qualitative metasynthesis of a series of HSCED studies produced a plausible explanation for the contrasting outcomes that occurred.

What too strict a method obscures about the validity of outcome measures

Objective: To assess the outcome of psychotherapeutic treatments, psychotherapy researchers often compare pre- and post-treatment scores on self-report outcome measures. In this paper, the common assumption is challenged that pre-to-post decreasing and increasing outcome scores are indicative of successful and failed therapies, respectively.Method: The outcome of 29 psychotherapeutic treatments was evaluated by means of quantitative analysis of pre- and post-treatment scores on commonly used outcome measures (such as the Symptom Checklist-90-R, the Inventory of Interpersonal Problems, and the General Health Questionnaire-12), as well as through consensual qualitative research.Results: Overall, a moderate to low convergence between qualitative and quantitative evaluations of outcome was observed. Detailed analyses of six cases are presented in which pre-to-post comparisons of outcome measures proved misleading.Conclusions: It is concluded that psychotherapy outcome research might benefit from assessment strategies that are sensitive to the singularities of individual treatments and to the complexity of the phenomenon of therapeutic outcome. Furthermore, classical psychometric evaluations of the validity of outcome measures might be supplemented with less-systematic evaluations that take any contingent source of information on outcome into account.

Change processes underlying "good outcome": A qualitative study on recovered and improved patients' experiences in psychotherapy for major depression

Aim: Exploring change processes underlying "good outcome" in psychotherapy for major depression. We examined the perspectives of patients who "recovered" and "improved" (Jacobson & Truax) following time-limited CBT and PDT. Method: In the context of an RCT on the treatment of major depression, patients were selected based on their pre-post outcome scores on the BDI-II: we selected 28 patients who recovered and 19 who improved in terms of depressive symptoms. A grounded theory analysis was conducted on post-therapy client change interviews, resulting in an integrative conceptual model. Results: According to recovered and improved patients, change follows from an interaction between therapy, therapist, patient, and extra-therapeutic context. Both helping and hindering influences were mentioned within all four influencing factors. Differences between recovered and improved patients point at the role of patients' agency and patients' internal and external obstacles. However, patients marked as "improved" described heterogeneous experiences. CBT- and PDT-specific experiences were also observed, although our findings suggest the possible role of therapist-related influences. Conclusion: From patients' perspectives, various change processes underlie "good outcome" that do not necessarily imply an "all good process". This supports a holistic, multidimensional conceptualization of change processes in psychotherapy and calls for more fine-grained mixed-methods process-outcome research.

Validity Beyond Measurement: Why Psychometric Validity Is Insufficient for Valid Psychotherapy Research

In psychotherapy research, "validity" is canonically understood as the capacity of a test to measure what is purported to measure. However, we argue that this psychometric understanding of validity prohibits working researchers from considering the validity of their research. Psychotherapy researchers often use measures with a different epistemic goal than test developers intended, for example when a depression symptom measure is used to indicate "treatment success" (cf. outcome measurement for evidence-based treatment). However, the validity of a measure does not cover the validity of its use as operationalization of another target concept within a research procedure, nor the validity of its function toward an epistemic goal. In this paper, we discuss the importance of considering validity of the epistemic process beyond the validity of measures per se, based on an empirical case example from our psychotherapy study ("SCS", Cornelis et al., 2017). We discuss why the psychometric understanding of validity is insufficient in covering epistemic validity, and we evaluate to what extent the available terminology regarding validity of research is sufficient for working researchers to accurately consider the validity of their overall epistemic process. As psychotherapy research is meant to offer a sound evidence-base for clinical practice, we argue that it is vital that psychotherapy researchers are able to discuss the validity of the epistemic choices made to serve the clinical goal.