What outcomes to measure in routine mental health services, and how to assess them: a systematic review

A Scoping Review of Mental Health and Wellbeing Outcome Measures for Children and Young People: Implications for Children in Out-of-home Care

Purpose

One of the challenges for mental health research is the lack of an agreed set of outcome measures that are used routinely and consistently between disciplines and across studies in order to build a more robust evidence base for how to better understand young people's mental health and effectively address diverse needs.

Methods

This study involved a scoping review of reviews on consensus of the use of mental health and wellbeing measures with children and young people. We were particularly interested to identify if there are differences in measures that are recommended for children and young people with care experience including those with developmental disabilities.

Findings

We identified 41 reviews, of which two had a focus on child welfare settings, three on childhood trauma and 14 focused on children and young people with developmental disabilities. Overall, our review highlights a lack of consensus and a diversity of measures within the field. We identified 60 recommended measures, of which only nine were recommended by more than one review.

Conclusions

Our review highlights the need for greater agreement in the use of mental health outcome measures. While our review highlights that there is value in identifying measures that can be used with any child or young person, researchers need to take into account additional considerations when working with children and young people with care experience and those with developmental disabilities, to ensure measures are accessible and sensitive to their life experiences.

Monitoring outcomes from intake to termination: A suggested new symptom and progress monitoring program for Bridgehaven Mental Health Services

In a doctoral clinical psychology program, students are frequently challenged to learn and implement new skills to improve the lives of their clients. Conducting a program evaluation, from beginning to end, is one such example. This article describes the experience of its authors in completing a program evaluation for a local agency in Louisville, KY as a class project. The project resulted in a proposed procedure to monitor therapeutic outcomes of the agency's clients (the agency refers to them as "members"). The authors discuss the class process, how they decided on a topic for the project, the use of monitoring outcomes in community mental health settings, why monitoring outcomes is so important, and the new proposed outcome assessment procedure, limitations, and future directions. Free measures were selected to meet the perceived needs of the agency and were presented to the staff in a final presentation. Finally, the authors examined their overall experience regarding participation in the project.

A scoping review on two-stage randomized preference trial in the field of mental health and addiction

BACKGROUND

Randomized Controlled Trial is the most rigorous study design to test the efficacy and effectiveness of an intervention. Patient preference may negatively affect patient performance and decrease the generalizability of a trial to clinical population. Patient preference trial have particular implications in the field of mental health and addiction since mental health interventions are generally complex, blinding of intervention is often difficult or impossible, patients may have strong preference, and outcome measures are often subjective patient self-report which may be greatly influenced if patient's preference did not match with the intervention received.

METHODS

In this review, we have surveyed the application of two-stage randomized preference trial with focus on studies in the field of mental health and addiction. The study selection followed the guideline provided by Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews.

RESULTS

Six two-stage randomized preference trials (ten publications) have been identified in the field of mental health field and addiction. In these trials, the pooled dropout rates were 18.3% for the preference arm, and 28.7% for the random arm, with a pooled RR of 0.70 (95% CI, 0.56-0.88; P = 0.010) indicating lower risk of dropout in the preference arm. The standardized preference effects varied widely from 0.07 to 0.57, and could be as large as the treatment effect in some of the trials.

CONCLUSION

This scoping review has shown that two-stage randomized preference trials are not as popular as expected in mental health research. The results indicated that two-stage randomized preference trials in mental health would be beneficial in retaining patients to expand the generalizability of the trial.

Representing the values of program participants: Endogenous evaluative criteria

Evaluative conclusions are grounded in implicit and explicit criteria that describe a successful or high-quality intervention. Most often, evaluative criteria are drawn from program objectives that reflect the values and priorities of program designers and funders. Yet, an exclusive focus on program goals risks overlooking the values of program participants, the extent to which their actual needs and priorities are addressed, and, in certain types of programs, the choices participants make and agency they exercise. This article presents concepts and methods to guide evaluators in drawing some of the criteria used in an evaluation from program participants. The article outlines a typology of evaluative criteria and seven methods for drawing outcomes-focused criteria from program participants. The article concludes with a discussion of implications and future directions for research and practice.

A systematic review of patient-reported outcome measurement (PROM) and provider assessment in mental health: goals, implementation, setting, measurement characteristics and barriers

PURPOSE

To review and integrate the literature on mental-health-related patient-reported outcome measures (PROMs) and routine outcome measures (ROMs), namely in the domains of goals, characteristics, implementation, settings, measurements and barriers. PROM/ROM aims mainly to ascertain treatment impact in routine clinical practice through systematic service users' health assessment using standardized self-report, caretaker and/or provider assessment.

DATA SOURCES

Psych INFO and PubMed including Medline, Biomed Central, EMBASE Psychiatry and Elsevier Science's Direct.

STUDY SELECTION

Systemized review of literature (2000-2018) on implementation and sustainability of PROMs/ROMs in adult mental health settings (MHS).

DATA EXTRACTION AND SYNTHESIS

Systemized review of literature (2000-2018) on numerous aspects of PROM/ROM implementation and sustainability in adult MHS worldwide.

RESULTS

Based on 103 articles, PROMs/ROMs were implemented mostly in outpatient settings for people with assorted mental health disorders receiving a diversity of services. Frequency of assessments and completion rates varied: one-third of projects had provider assessments; about half had both provider and self-assessments. Barriers to implementation: perceptions that PROM/ROM is intrusive to clinical practice, lack of infrastructure, fear that results may be used for cost containment and service eligibility instead of service quality improvement, difficulties with measures, ethical and confidentiality regulations and web security data management regulations.

CONCLUSION

Improving data input systems, sufficient training, regular feedback, measures to increase administrative and logistic support to improve implementation, acceptability, feasibility and sustainability, follow-up assessments and client attrition rate reduction efforts are only some measures needed to enhance PROM/ROM efficiency and efficacy.

Personally meaningful recovery in people with psychological trauma: Initial validity and reliability of the Individual Recovery Outcomes Counter (I.ROC)

Individual Recovery Outcomes Counter (I.ROC) is a brief tool measuring personal recovery designed for collaborative use within support. This study aimed to investigate the psychometric properties of a self-report version of the I.ROC within a trauma population. A total of 107 adults attending trauma interventions in an NHS service in Scotland completed I.ROC alongside measures of self-esteem, mental illness symptoms, and functional impairment. Scores on each measure were compared to evaluate the convergent validity of I.ROC. Internal consistency and factor analytical techniques were also used to assess the structural validity and reliability of the measure. Results of internal consistency, convergent validity, and factor analysis provide preliminary support for I.ROC's validity within a trauma population. Previously proposed models were a poor fit for the current sample; principal components analysis suggested a three-factor structure with acceptable internal consistency, comprising ten of the original twelve items (I.ROC-10). Correlations with all measures reached significance for the original and modified I.ROC and its subscales. I.ROC appears to be a valid and reliable tool for use in measuring recovery within a trauma population, but further research is needed to examine the structural validity of I.ROC.

Patient-reported outcome measurements (PROMs) and provider assessment in mental health: a systematic review of the context of implementation

Purpose

To review and integrate the vast amount of literature yielded by recent growing interest in patient-reported outcome measurement and routine outcome measures (PROMs/ROMs), in order to suggest options and improvements for implementation. PROMs are the systematic assessment of service users’ health using standardized self-report measures. Specifically, for ROMs, it includes routine provider or caretaker assessment measures. Both are administered to ascertain routinely, the impact of treatment in mental health settings and to improve care. A review is needed because of the large differences in setting, conceptualization, practice and implementation. Here, we examine the different major projects worldwide.

Data sources

Psych INFO and PubMed including Medline, Biomed Central, EMBASE Psychiatry and Elsevier Science’s Direct.

Study selection

We conducted a systematized review of the literature published from 2000 to 2018 on the implementation and sustainability of PROMs and ROMs in mental health services for adults.

Data extraction, synthesis and Results

We described and characterized the programs in different countries worldwide. We identified 103 articles that met the inclusion criteria, representing over 80 PROMs/ROMs initiatives in 15 countries. National policy and structure of mental health services were found to be major factors in implementation. We discuss the great variability in PROMs/ROMs models in different countries, making suggestions for their streamlining and improvement.

Conclusion

We extracted valuable information on the different characteristics of the numerous PROMs/ROMs initiatives worldwide. However, in the absence of a strong nationwide policy effort and support, implementation seems scattered and irregular. Thus, development of the implementation of PROMs/ROMs is left to groups of enthusiastic clinicians and researchers, making sustainability problematic.

Identifying social participation subgroups of individuals with severe mental illnesses: a latent class analysis

PURPOSE

To investigate factors that influence participation in and needs for work and other daytime activities among individuals with severe mental illnesses (SMI).

METHODS

A latent class analysis using routine outcome monitoring data from 1069 patients was conducted to investigate whether subgroups of individuals with SMI can be distinguished based on participation in work or other daytime activities, needs for care in these areas, and the differences between these subgroups.

RESULTS

Four subgroups could be distinguished: (1) an inactive group without daytime activities or paid employment and many needs for care in these areas; (2) a moderately active group with some daytime activities, no paid employment, and few needs for care; (3) an active group with more daytime activities, no paid employment, and mainly met needs for care; and (4) a group engaged in paid employment without needs for care in this area. Groups differed significantly from each other in age, duration in MHC, living situation, educational level, having a life partner or not, needs for care regarding social contacts, quality of life, psychosocial functioning, and psychiatric symptoms. Differences were not found for clinical diagnosis or gender.

CONCLUSIONS

Among individuals with SMI, different subgroups can be distinguished based on employment situation, daytime activities, and needs for care in these areas. Subgroups differ from each other on patient characteristics and each subgroup poses specific challenges, underlining the need for tailored rehabilitation interventions. Special attention is needed for individuals who are involuntarily inactive, with severe psychiatric symptoms and problems in psychosocial functioning.

Comparing outcomes: The Clinical Outcome in Routine Evaluation from an international point of view

OBJECTIVES

The Clinical Outcome in Routine Evaluation-Outcome Measure (CORE-OM) is a freely accessible self-assessment questionnaire with a total of 34 items measuring the progress of psychological or psychotherapeutic treatments according to four scales (well-being, problems, functioning, and risk). The CORE-OM originated in the United Kingdom and has been translated into 54 languages and dialects. The aim of this study is to systematically compare the translated versions.

METHOD

A total of 21 translations were compared using methods of systematic review and meta-analysis.

RESULTS

The results show a certain heterogeneity between the studies; however, the 21 translations can be declared as equivalent.

CONCLUSION

The factorial structure could not be replicated in any of translations. Therefore, further analysis of the CORE-OM domains is recommended. In addition, some supplementary restrictions on the translation process, data collection, and reporting of results are necessary to ensure comparability and quality of CORE-OM translations.

Development of a mental health recovery module for the WHOQOL

PURPOSE

The WHOQOL tools are widely used, multi-faceted, patient-rated, quality of life (QoL) measures, developed by the World Health Organization. The WHOQOL questionnaires are used to assess generic quality of life issues affected by all health problems. This study developed a module to use with the WHOQOL tools to improve their sensitivity to Health Related QoL issues relevant to mental health recovery.

METHODS

Using a sequential mixed-methods approach, two research stages occurred. A qualitative stage invited 88 participants with experience of mental health recovery, into focus groups and importance rating activities, to identify candidate items for the new module. Following this, a quantitative stage involved 667 participants with, and without, mental health/addiction issues completing online or paper-based questionnaires to analyze which candidate items differentiated between those with and without mental health/addiction issues. Classical test theory and iterative Partial Credit Rasch Analysis were used to identify the most suitable candidate items for a reliable and valid mental health recovery module to be used with the WHOQOL tools.

RESULTS

Seventeen candidate items captured important HRQoL facets relevant to mental health recovery. Rasch analysis removed 10 misfitting items. The final 7-item module, which demonstrated the best Rasch model fit, enquires about recovery beliefs, identifying strengths, self-awareness, acceptance, capacity to relate, feeling understood, and recovery progress. Ordinal-to-interval conversion tables have been developed to optimize measurement precision when using the module.

CONCLUSIONS

Important HRQoL issues central to mental health recovery can be reliably evaluated by using the recovery module with the WHOQOL tools.

Behavior Problems Inventory (BPI): Psychometric characteristics of an instrument for routine assessment of persons with psychoses and related disorders

This study explores the psychometric characteristics of the Behavior Problems Inventory (BPI), an instrument for routine clinical assessment of behavior problems in outpatients with psychosis based on information provided by key family informants. Six hundred and twenty-one patients diagnosed with psychosis and bipolar affective disorder (ICD-10 F20-F31) attended at Community Mental Health Units were evaluated in routine reviews using the BPI and the Social Functioning Scale (SFS). Twenty-five subjects were simultaneously administered the Social Behavior Schedule (SBS) and 28 were again administered the BPI eight weeks later. The instrument shows adequate psychometric characteristics with high internal consistency and robust temporal reliability, as well as satisfactory concurrent and construct validity. Factor analysis identified three factors: Underactivity/Social Withdrawal, Active Problems and Lack of Impulse Control, with adequate saturation of the items on each of the factors. The BPI is easy to apply, reliable and valid, takes up little of valuable clinical time, allowing routine assessment in public service contexts for persons diagnosed with psychosis and bipolar affective disorder for whom key family informants are available.

Development and Validation of the Health of the Nation Outcome Scales-Residential Facility (HoNOS-RF)

The Health of the Nation Outcome Scale (HoNOS) (Lora et al. Epidemiol Psichiatr Soc 10(3):198-212, 2001) is widely used. However, clinicians have expressed concerns about its ability to describe severe mentally ill patients, as it does not consider some relevant clinical aspects. This study aims to develop and validate the HoNOS-Residential Facility (HoNOS-RF) in order to pursue a thorough assessment of patients admitted to psychiatric residential facilities (RFs). The final version of the HoNOS-RF was administered to 409 patients admitted to four RFs. Exploratory factor analysis, Cronbach' alpha (α), Intraclass Correlation Coefficients (ICC) were used to assess construct validity, internal consistency and reliability, respectively. Concurrent criterion validity was assessed through correlations with the Brief Psychiatric Rating Scale-Expanded Version (BPRS-E) (Roncone et al. Acta Psychiatric Scand 100(3):229-36, 1999), Personal and Social Performance Scale (PSP) (Morosini et al. Acta Psychiatric Scand 101(4): 323-29, 2000), and comparisons across diagnostic groups. The final version of the HoNOS-RF consisted of 31 items, grouped into the following eight factors (overall explaining 55% of the total variability): personal and interpersonal functioning; environment; behavior and burden of care; cognitive function; somatic problems; anxiety-depression symptoms; psychotic symptoms; and other psychiatric symptoms. The scale showed high internal consistency (α = .807), and the correlations with PSP and BPRS-E ranged from adequate to moderate. The ICCs were in the excellent range for almost all items. These findings support the validity and the reliability of the HoNOS-RF, thus it may be a useful tool for the assessment of patients admitted to RFs, as it addresses clinical aspects that were mostly not included in the original version.

Implementing routine outcome monitoring in public mental health services in Israel: Shared and unique challenges

OBJECTIVES

This study aimed to assess the feasibility, benefits, and challenges of routine outcome monitoring (ROM) in a public mental health centre in Israel. This is the first initiation of ROM implementation in a setting of a large psychiatric hospital, affiliated with the largest health maintenance organization in Israel, Clalit Health Services.

METHODS

Four ambulatory units were invited to participate in the implementation of ROM. Prior to initiation of recruitment, staff meetings were held and broad descriptions of current knowledge regarding benefits and challenges were discussed. Upon initiation of recruitment, patients completed outcome and process questionnaires during each session, and reports of patients' progress were sent to therapists soon after sessions ended.

RESULTS

One hundred sixty-three patients were invited to participate in measurement and feedback of therapy outcomes. Overall, 101 patients and 32 therapists agreed and actively participated in the implementation phase of ROM, producing 535 evaluated measurements and reports. Clinical, methodological, legal, administrative, and ethical aspects of the implementation were encountered and documented throughout the implementation process.

CONCLUSIONS

Medical confidentiality issues, as well as the multidisciplinary nature of hospital staff work, serve as central and unique challenges for incorporating ROM in a public psychiatric hospital. Other challenges that were previously reported in other implementation studies were also encountered and included therapist overload and objections, attrition, and the need for organizational support. Recommendations for future pioneering efforts for ROM implementation in large psychiatric facilities are discussed.

An evidence-based approach to routine outcome assessment

Routine use of Health of the Nation Outcome Scales (HoNOS) has not produced the anticipated benefits for people using mental health services. Four HoNOS-specific reasons for this are: low relevance to clinical decision-making; not reflecting service user priorities; being staff-rated; and having a focus on deficits. More generally, the imposition of a centrally chosen measure on the mental health system leads to a clash of cultures, since frontline workers do not need a standardised measure to treat individuals. A better approach might be to use research from the emerging academic discipline of implementation science to inform the routine use of a standardised measure that is chosen by the people who will use it and hence is more concordant with existing clinical processes. This is illustrated using a case study of successful implementation of the Camberwell Assessment of Need (CAN) in community mental health services across Ontario, Canada.

Aristotle’s concept of mental imagery in sports

Throughout the past decade, there has been an accelerated expansion of heightening consciousness in athletic performance’s mental facets, which is reflected in the aggrandizement of research concern in cognitive sports psychology. Research on mental imagery has been considerably influenced by cognitive concepts whereby Aristotle, the Stageirite philosopher, delivers the primary systematic account of the significant role of mental imagery in cognition. In Aristotelian psychological theory, mental images perform much the same function that the rather broader concept of mental portrayal plays in contemporary cognitive science. With the assumption that cognition refers to any mental activity associated with acquiring, storing or using knowledge (including competent behaviour), the present paper, endeavors to forge an alliance between Aristotle’s notion of mental imagery, which draws attentions to some form of fundamental mental portrayal of athletic Knowledge, along with the contemporary notion of mental imagery in sports performers.

Development of the clinically adaptive multidimensional outcome survey

OBJECTIVE

Routine outcome monitoring (ROM) has been strongly endorsed by psychotherapy researchers, but has yet to achieve widespread implementation in clinical settings. This article describes the development of the Clinically Adaptive Multidimensional Outcome Survey (CAMOS), an innovative ROM system that allows for local adaptation while providing high quality data.

METHOD

Three-hundred and four clients at a university counseling center and 211 female patients at an eating disorder treatment facility were administered the CAMOS at intake, and 118 took the CAMOS at both intake and discharge. Two models were developed and compared. Both models were developed using exploratory and confirmatory factor analysis.

RESULTS

A five-factor model was found to have the best model fit, internal consistency, convergent validity, and discriminant validity.

CONCLUSIONS

The CAMOS has evidence to support its reliability and validity as a measure of various dimensions of distress. Distinctive tailoring features of the CAMOS compared to other ROM measures are described.

Improving the analysis of routine outcome measurement data: what a Bayesian approach can do for you

Since recent decades, clinicians offering interventions against mental problems must systematically collect data on how clients change over time. Since these data typically contain measurement error, statistical tests have been developed which should disentangle true changes from random error. These statistical tests can be subdivided into two types: classical tests and Bayesian tests. Over the past, there has been much confusion among analysts regarding the questions that are answered by each of these tests. In this paper we discuss each type of test in detail and explain which questions are, and which are not, answered by each of the types of tests. We then apply a test of each type on an empirical data set and compare the results. Copyright © 2015 John Wiley & Sons, Ltd.

The performance of the Health of the Nation Outcome Scales as measures of clinical severity

The aim of this study was to examine the performance of the Health of the Nation Outcome Scales (HoNOS) against other measures of functioning and mental health in a full three-year cohort of admissions to a psychiatric hospital. A sample of N=1719 patients (35.3% females, aged 17-78 years) was assessed using observer-rated measures and self-reports of psychopathology at admission. Self-reports were available from 51.7% of the sample (34.4% females, aged 17-76 years). Functioning and psychopathology were compared across five ICD-10 diagnostic groups: substance use disorders, schizophrenia and psychotic disorders, affective disorders, anxiety/somatoform disorders and personality disorders. Associations between the measures were examined, stratifying by diagnostic subgroup. The HoNOS were strongly linked to other measures primarily in psychotic disorders (except for the behavioral subscale), while those with substance use disorders showed rather poor links. Those with anxiety/somatoform disorders showed null or only small associations. This study raises questions about the overall validity of the HoNOS. It seems to entail different levels of validity when applied to different diagnostic groups. In clinical practice the HoNOS should not be used as a stand-alone instrument to assess outcome but rather as part of a more comprehensive battery including diagnosis-specific measures.

Development and evaluation of an Individualized Outcome Measure (IOM) for randomized controlled trials in mental health

Pre-defined, researcher-selected outcomes are routinely used as the clinical end-point in randomized controlled trials (RCTs); however, individualized approaches may be an effective way to assess outcome in mental health research. The present study describes the development and evaluation of the Individualized Outcome Measure (IOM), which is a patient-specific outcome measure to be used for RCTs of complex interventions. IOM was developed using a narrative review, expert consultation and piloting with mental health service users (n = 20). The final version of IOM comprises two components: Goal Attainment (GA) and Personalized Primary Outcome (PPO). For GA, patients identify one relevant goal at baseline and rate its attainment at follow-up. For PPO, patients choose an outcome domain related to their goal from a pre-defined list at baseline, and complete a standardized questionnaire assessing the chosen outcome domain at baseline and follow-up. A feasibility study indicated that IOM had adequate completion (89%) and acceptability (96%) rates in a clinical sample (n = 84). IOM was then evaluated in a RCT (ISRCTN02507940). GA and PPO components were associated with each other and with the trial primary outcome. The use of the PPO component of IOM as the primary outcome could be considered in future RCTs. Copyright © 2015 John Wiley & Sons, Ltd.

A systematic review of mental health outcome measures for young people aged 12 to 25 years

BACKGROUND

Mental health outcome measures are used to monitor the quality and effectiveness of mental health services. There is also a growing expectation for implementation of routine measurement and measures being used by clinicians as a feedback monitoring system to improve client outcomes. The recent focus in Australia and elsewhere targeting mental health services to young people aged 12-25 years has meant that outcome measures relevant to this age range are now needed. This is a shift from the traditional divide of child and adolescent services versus adult services with a transitioning age at 18 years. This systematic review is the first to examine mental health outcome measures that are appropriate for the 12 to 25 year age range.

METHODS

MEDLINE and PsychINFO databases were systematically searched to identify studies using mental health outcome measures with young people aged 12 to 25 years. The search strategy complied with the relevant sections of the PRISMA statement.

RESULTS

A total of 184 published articles were identified, covering 29 different outcome measures. The measures were organised into domains that consisted of eight measures of cognition and emotion, nine functioning measures, six quality of life measures, and six multidimensional mental health measures. No measures were designed specifically for young people aged 12 to 25 years and only two had been used by clinicians as a feedback monitoring system. Five measures had been used across the whole 12 to 25 year age range, in a range of mental health settings and were deemed most appropriate for this age group.

CONCLUSIONS

With changes to mental health service systems that increasingly focus on early intervention in adolescence and young adulthood, there is a need for outcome measures designed specifically for those aged 12 to 25 years. In particular, multidimensional measures that are clinically meaningful need to be developed to ensure quality and effectiveness in youth mental health. Additionally, outcome measures can be clinically useful when designed to be used within routine feedback monitoring systems.

Scoping review of research in Australia on the co-occurrence of physical and serious mental illness and integrated care

The physical health of people with serious mental illness (SMI) has become a focal area of research. The aim of the present study was to ascertain the attention and distribution of research from within Australia on physical illness and SMI co-occurrence, and to identify gaps. A scoping review of peer-reviewed research literature from Australia, published between January 2000 and March 2014, was undertaken through an electronic literature search and coding of papers to chart trends. Four trends are highlighted: (i) an almost threefold increase in publications per year from 2000-2006 to 2007-2013; (ii) a steady release of literature reviews, especially from 2010; (iii) health-related behaviours, smoking, integrated-care programmes, and antipsychotic side-effects as the most common topics presented; and (iv) paucity of randomized, controlled trials on integrated-care models. Despite a marked increase in research attention to poorer physical health, there remains a large gap between research and the scale of the problem previously identified. More papers were descriptive or reviews, rather than evaluations of interventions. To foster more research, 12 research gaps are outlined. Addressing these gaps will facilitate the reduction of inequalities in physical health for people with SMI. Mental health nurses are well placed to lead multidisciplinary, consumer-informed research in this area.

Health of the Nation Outcome Scales in a psychiatric inpatient setting: assessing clinical change

RATIONALE, AIMS AND OBJECTIVES

The aim of this study was to examine the validity of the Health of the Nation Outcome Scales (HoNOS) in terms of change in relation to sex, clinical characteristics and level of clinical change as assessed using other measures.

METHODS

The sample consisted of N = 690 admissions from one Swiss psychiatric hospital who were assessed at admission and discharge using the HoNOS and the clinical global impression (CGI) scale. Repeated measures analyses of variance were conducted to compare changes in HoNOS scores over time stratified by sex, diagnostic category and CGI level of change, controlled for age and previous hospitalizations. Two-way interactions between time and these factors were calculated and post hoc t-tests were conducted to compare changes in HoNOS scores between admission and discharge at factor levels.

RESULTS

HoNOS scores significantly decreased from admission to discharge in bivariate analyses although no main effect of time was found in multivariate models. Sex was found to moderate change in HoNOS behavioural subscale scores; primary diagnosis at admission moderated change in HoNOS total scores, the behavioural subscale and the social subscale; and CGI level of change moderated the change in all HoNOS scales.

CONCLUSIONS

Our findings confirmed the sensitivity to change of the HoNOS in psychiatric settings from admission to discharge. Furthermore, we found that this change reflected similar changes in the CGI, a well-established measure for the evaluation of clinical outcomes, which, in turn supports the validity of the HoNOS.

Evaluating pharmacy services satisfaction on an inpatient psychiatric unit

Introduction

This study analyzed patient satisfaction survey responses in which patients evaluated their experience with the pharmacy staff in an acute psychiatric unit.

Methods

Thirty-one patients rated pharmacy services upon discharge by completing a 13-item survey.

Results

Overall patient satisfaction was found to be high with a 77.4–100% satisfaction rating reported on every survey item.

Discussion

Results indicated that the pharmacy staff should focus their efforts on improving communication with patients by selecting a quiet meeting location and by making themselves more accessible to patients in order to better serve their needs.

Considerations and evidence for an ADHD outcome measure

OBJECTIVE

The 2011 American Academy of Pediatrics attention-deficit/hyperactivity disorder (ADHD) guideline emphasizes monitoring and measuring outcomes of children diagnosed with ADHD; however, recommendations for how to measure improvement are less clear. A long-term goal was to develop an outcome measure that assesses the quality of care for children with ADHD. As a first step in that process, we conducted a literature synthesis on the efficacy and effectiveness of guideline-recommended ADHD treatments on patient outcomes.

METHODS

A literature search was conducted in PubMed according to PRISMA protocol and using MeSH terms. US Preventive Services Task Force (USPSTF) criteria were used to assess the level of evidence. Studies of interest were published after 2002 and assessed prospective ADHD improvement using recommended ADHD treatments.

RESULTS

The systematic review resulted in 35 studies. According to USPSTF criteria, included studies were level I (n = 24), level II-1 (n = 1), and level II-2 (n = 10) and were rated as good (n = 20) or fair (n = 15). DSM-criteria-based rating scales were used most frequently to measure ADHD treatment outcomes. All included treatments resulted in ADHD improvement. Regardless of outcome measure, tool, or treatment type, symptom reduction and improvement were relatively large, with mean percentage reductions ranging from 20% to 86% on ADHD-Behavior Rating Scales scales, with only 1 study with <25% reduction. Effect sizes ranged from 0.15 to 4.57.

CONCLUSIONS

On the basis of this literature review, a consistent pattern of improvement in pediatric ADHD patients' core symptoms emerged across studies, study designs, and recommended treatment approaches. This evidence supports the notion that an improvement of core symptoms within 1 year could satisfy the requirements of an effective outcome measure, which should be further investigated.

New trends in assessing the outcomes of mental health interventions

Assessing the outcomes of interventions in mental health care is both important and challenging. The aim of this paper is to advance the field of outcomes research by proposing a taxonomy of the decisions that clinicians and researchers need to consider when evaluating outcomes. Our taxonomy has eight components, framed as decisions: Whose outcome will be considered? Which scientific stage is being investigated? What outcome domain(s) matter? What level of assessment will be used? Will clinical and/or recovery outcomes be assessed? Whose perspective will be considered? Will deficits and/or strengths be the focus? Will invariant or individualized measures be preferred? We propose a future focus on understanding what matters most to people using mental health services, and on the use of measures rated by service users as the primary approach to evaluating outcome.

Using Item and Test Information to Optimize Targeted Assessments of Psychological Distress

The need for efficient clinical assessment instruments has been growing during the past years. In the current application, the item information (item response theory) is used to evaluate and build fixed short versions. The method was applied to a questionnaire measuring psychological distress and data were collected from two mixed outpatient and general population samples. After fitting the partial credit model, two short versions were built: one to increase efficiency in screening applications; the other for the monitoring of high distress patients. A cross-validation bootstrap procedure is proposed to check whether the short versions are more efficient than alternative item selections. Using the partial credit model, the results from short and full versions can be compared on score level, which improves the flexibility of the assessment. The discussion focuses on the model selection and on how many items are realistically needed in routine assessments of psychological distress.

The validity of patient- and clinician-rated measures of needs and the therapeutic relationship in psychosis: a pooled analysis

Measuring outcomes of treatments for psychosis such as needs and the quality of the therapeutic relationship is important in research and routine care. However, evidence on the validity of existing outcome measures is limited. We aimed to test the convergent, discriminant, and predictive validity of two widely used patient- and clinician-rated measures of needs and the therapeutic relationship. Multitrait-multimethod (MTMM) analysis was conducted on the Camberwell Assessment of Need Short Appraisal Schedule (CANSAS) and the Helping Alliance Scale (HAS), both the clinician (CANSAS-C, HAS-C) and patient (CANSAS-P, HAS-P) versions, in a pooled sample of 605 psychotic patients and their clinicians. CANSAS-C and CANSAS-P items loaded substantially into one common unmet needs factor. By comparison, substantial factor loadings were found for HAS-C and HAS-P items on two separate clinician- and patient-rated therapeutic relationship factors. Common unmet needs and clinician-rated therapeutic relationship factors significantly predicted reduced psychiatric in-patient days. Our findings support the convergent validity of the CANSAS, discriminant validity of the HAS, and predictive validity of CANSAS and HAS-C. The findings may inform the use of CANSAS and HAS as psychosis outcome measures in research and routine care.

Conceptual framework for personal recovery in mental health: systematic review and narrative synthesis

BACKGROUND

No systematic review and narrative synthesis on personal recovery in mental illness has been undertaken.

AIMS

To synthesise published descriptions and models of personal recovery into an empirically based conceptual framework.

METHOD

Systematic review and modified narrative synthesis.

RESULTS

Out of 5208 papers that were identified and 366 that were reviewed, a total of 97 papers were included in this review. The emergent conceptual framework consists of: (a) 13 characteristics of the recovery journey; (b) five recovery processes comprising: connectedness; hope and optimism about the future; identity; meaning in life; and empowerment (giving the acronym CHIME); and (c) recovery stage descriptions which mapped onto the transtheoretical model of change. Studies that focused on recovery for individuals of Black and minority ethnic (BME) origin showed a greater emphasis on spirituality and stigma and also identified two additional themes: culturally specific facilitating factors and collectivist notions of recovery.

CONCLUSIONS

The conceptual framework is a theoretically defensible and robust synthesis of people's experiences of recovery in mental illness. This provides an empirical basis for future recovery-oriented research and practice.

REFOCUS Trial: protocol for a cluster randomised controlled trial of a pro-recovery intervention within community based mental health teams

BACKGROUND

There is a consensus about the importance of 'recovery' in mental health services, but the evidence base is limited.

METHODS/DESIGN

A two centre, cluster randomised controlled trial. Participants are community-based mental health teams, and service users aged 18-65 years with a primary clinical diagnosis of psychosis. In relation to the REFOCUS Manual researchintorecovery.com/refocus, which describes a 12-month, pro-recovery intervention based on the REFOCUS Model, the objectives are: (1) To establish the effectiveness of the intervention described in the REFOCUS Manual; (2) To validate the REFOCUS Model; (3) To establish and optimise trial parameters for the REFOCUS Manual; and (4) To understand the relationship between clinical outcomes and recovery outcomes. The hypothesis for the main study is that service users in the intervention arm will experience significantly greater increases in measures of personal recovery (as measured by the QPR) compared to service users receiving care from control teams. The hypothesis for the secondary study is that black service users in the intervention arm will experience significantly greater increases in measures of personal recovery (as measured by the QPR) and client satisfaction (as measured by the CSQ) compared to Black service users receiving care from control teams. The intervention comprises treatment as usual plus two components: recovery-promoting relationships and working practices. The control condition is treatment as usual. The primary outcme is the Process of Recovery Questionnaire (QPR). Secondary outcomes are satisfaction, Goal setting - Personal Primary Outcome, hope, well-being, empowerment, and quality of life. Primary outcomes for the secondary study will be QPR and satisfaction. Cost data will be estimated, and clinical outcomes will also be reported (symptomatology, need, social disability, functioning). 29 teams (15 intervention and 14 control) will be randomised. Within each team, 15 services users will be randomly chosen, giving a total sample of 435 service users (225 in intervention and 210 in control). Power for the main study: 336 service users will give power to detect a medium effect size of 0.4 (alpha 0.05, power = 0.8) on both QPR sub-scales. Power for the secondary study: 89 participants will give power to detect an effect size of 0.67 on both QPR sub-scales and on CSQ. A range of approaches are used to minimise bias, although service users and clinicians cannot be blinded.

DISCUSSION

This cluster-RCT will evaluate a pro-recovery intervention in community mental health teams.

TRIAL REGISTRATION

ISRCTN: ISRCTN02507940.

The relation between objective and subjective domains of recovery among persons with schizophrenia-related disorders

In recent years, growing emphasis has been placed on the vision of recovery, which is broadly organized into two types: clinical objective versus personal subjective. The purpose of the present study was to investigate the relation between objective clinical recovery as defined by symptom severity and level of functioning, and subjective personal recovery as defined by quality of life, domains of personal confidence and hope, willingness to ask for help, reliance on others and no domination by symptoms. One hundred and fifty-nine persons diagnosed with schizophrenia or schizoaffective disorder completed measures of recovery, quality of life, perceived social support and emotional loneliness. Clinicians used the Modified Brief Psychiatric Rating Scale and the Global Assessment Functioning Scale to assess the severity of symptoms and level of functioning. Results revealed no direct correlation between total score of observer ratings of symptoms and total score of subjective self-report of being in recovery. The relationship between total score of symptoms and total score of subjective self-report of recovery was moderated by the age of onset. Magnitude of the self-report of subjective recovery was related to higher levels of reported social support and lower levels of reported loneliness. Finally, analyses suggested that the impact of social support and loneliness upon self-reported recovery was mediated by quality of life. Taken together, results are consistent with literature suggesting that clinical objective recovery is not synonymous with personal subjective recovery yet can be conceptualized as complementary.

The incremental value of self-reported mental health measures in predicting functional outcomes of veterans

Research on patient-centered care supports use of patient/consumer self-report measures in monitoring health outcomes. This study examined the incremental value of self-report mental health measures relative to a clinician-rated measure in predicting functional outcomes among mental health service recipients. Participants (n = 446) completed the Behavior and Symptom Identification Scale, the Brief Symptom Inventory, and the Veterans/Rand Short Form-36 at enrollment in the study (T1) and 3 months later (T2). Global Assessment of Functioning (GAF) ratings, mental health service utilization, and psychiatric diagnoses were obtained from administrative data files. Controlling for demographic and clinical variables, results indicated that improvement based on the self-report measures significantly predicted one or more functional outcomes (i.e., decreased likelihood of post-enrollment psychiatric hospitalization and increased likelihood of paid employment), above and beyond the predictive value of the GAF. Inclusion of self-report measures may be a useful addition to performance measurement efforts.

The contribution of advisory committees and public involvement to large studies: case study

BACKGROUND

Many large studies have complex advisory committee structures, yet there is no empirical evidence regarding their optimal composition, scope and contribution. The aim of this study was to inform the committee and advice infrastructure for future research studies.

METHODS

In the context of a five-year study funded by the UK National Institute for Health Research, three advisory committees were formed. In addition, advice was obtained from individual experts. All recommendations received in the start-up phase (first seven months) of the study were recorded, along with the decision about implementation of the recommendation. A particular focus was on the impact of public involvement.

RESULTS

A total of 172 recommendations were made, including 70 from 20 individual experts. The recommendations were grouped into five emergent themes: Scientific, Pragmatic, Resources, Committee and Collaboration. Most recommendations related to strengthening existing components or adding new components to the study protocol. Very few recommendations either proposed removing study components or contradicted other recommendations. Three 'implementation criteria' were identified: scientific value, pragmatic feasibility, and paradigmatic consistency. 103 (60%) of recommendations were implemented and 25 (15%) were not implemented. The benefits identified by the research team were improved quality and confidence, and the costs were increased cognitive demands, protocol revision time, and slower progress.

CONCLUSIONS

The findings are discussed in the context of the wider literature on public involvement in research. Six recommendations are identified. First, have a clear rationale for each advisory committee expressed as terms of reference, and consider the best balance between committees and individual consultation with experts. Second, an early concern of committees is inter-committee communication, so consider cross-representation and copying minutes between committees. Third, match the scope of advisory committees to the study, with a less complex advisory structure for studies with more finalised designs. Fourth, public involvement has a mixed impact, and relies on relationships of trust, which take time to develop. Fifth, carefully consider the match between the scientific paradigm applied in the study and the contribution of different types of knowledge and expertise, and how this will impact on possibilities for taking on advice. Finally, responding to recommendations uses up research team resources, and the costs can be reduced by using the three implementation criteria.

Performance measurement in mental health care: present situation and future possibilities

SUMMARY

This paper describes performance measurement and its indicators for mental health care services. Performance measurement can serve several goals such as accountability, quality improvement and performance management. For all three purposes structure, process and outcome indicators should be measured. Literature was retrieved from Medline and PsychInfo in order to see which performance indicators were used for the three purposes of performance measurement in mental health care. The indicators were classified in structure, process and outcome indicators. The results show no big differences in the indicators used among studies. Performance management is the performance measurement purpose most referred to, followed by accountability, and quality improvement. Outcome and process indicators are used most, structure indicators are in the minority. Several levels of measurement, that is national or service level, came forward in the literature review. To overcome misinterpretation of data and to be able to improve quality and manage performances, performance indicator sets should refer to structure, process and outcome. Indicators should be chosen carefully with the aim of the measurement taken into mind. Based on this review, a conceptual framework is presented to support managers in their decisions about which indictors can best be used for performance measurement. Additionally, a model that provides an understanding of the use of information gained by performance measurement is given.

Outcome management in in-patient psychiatric care

OBJECTIVE

To delineate methods and to describe patient appraisal as well as effect of outcome management in in-patient psychiatric care.

METHOD

Two hundred and ninety-four adults with mental illness receiving in-patient treatment at a psychiatric hospital in rural Bavaria gave informed consent to participate in this cluster-randomised trial. Participants were asked to provide information on treatment outcome via weekly computerised standardised assessments. Patients and clinicians in the intervention group received continuous feedback of outcome.

RESULTS

Patients were willing and able to regularly provide outcome data and valued feedback. However, use of feedback in conversations between patient and clinician was rare. Outcome management failed to impact on patient-rated outcome during in-patient treatment.

CONCLUSION

Outcome management is feasible in people receiving in-patient psychiatric care, but failed to show an overall short-term effect. Strategies need to be developed to improve active use of routinely collected treatment outcome data in mental health care.

Modelling candidate effectiveness indicators for mental health services

OBJECTIVE

Although Australia and the UK have both made efforts to systematize outcome measurement in mental health services, surprisingly little attention has been paid to how best to analyse routine outcome data in order to determine how services are performing.

METHODS

Outcome data collected in acute inpatient and ambulatory mental health services across Australia during the 2006-2007 financial year were used. three approaches to measuring effectiveness were explored: effect size (ES); the reliable change index (RCI); and standard error of measurement (SEM).

RESULTS

The most conservative results were produced by the RCI and the least conservative by the medium ES statistic and the SEM. By way of example, only 38.0% of inpatient admission-discharge periods of care showed significant improvement for adults when the RCI was used, whereas 67.4% and 72.9% did so when the medium ES and the SEM statistics were used, respectively.

CONCLUSIONS

In any routine outcome measurement exercise, the degree of effectiveness demonstrated by services will depend on the specific statistical indicator used to judge effectiveness. Routine outcome measurement has the potential to answer a range of crucial performance-related questions, but only if the same metric is used. Discussion of the appropriate statistical approach to take to facilitate cross-service, cross-area and even cross-national comparisons warrants attention.

Patient health outcomes in psychiatric mental health nursing

This integrative literature review examined evidence concerning the relationship between psychiatric mental health nursing interventions and patient-focused outcomes. Empirical studies, published between 1997 and 2007, were identified and gathered by searching relevant databases and specific data sources. Although 156 articles were critically appraised, only 25 of them met the inclusion criteria. Findings from this review showed that the most frequently used outcome instruments assessed psychiatric symptom severity. Most of the instruments targeted two symptom categories: altered thoughts/perceptions and altered mood. Other outcome instruments were categorized in the following domains: self-care, functioning, quality of life and satisfaction. The most important finding of this review is the lack of consistently strong evidence to support decisions concerning which outcome instrument or combination of instruments to recommend for routine use in practice. Based on this review, additional research to conceptualize, measure and examine the feasibility of outcome instruments sensitive to psychiatric mental health nursing interventions is recommended.

Mental health nursing standards for Australia: a review of current literature

The expanding role of the mental health nurse brings new challenges and rewards. To support this, nationally adopted, formalized standards of practice are required. Currently, the Standards of Practice for Mental Health Nurses in Australia published by the Australian and New Zealand College of Mental Health Nurses in 1995 provide a guide for mental health nurses working in Australia. While these standards have played a role in supporting mental health nurses, they have not been widely adopted. This report reviews the current literature on standards for practice and describes an evidence-based rationale as to why a review and renewal of these is required and why it is important, not just for mental health nurses but to the field of mental health in general.

Outcomes of public concern in schizophrenia

BACKGROUND

Schizophrenia is known to be associated with a range of adverse outcomes, which have an impact atthe societal level and are therefore of public concern.

AIMS

To examine the epidemiology and methods for measuring six adverse outcomes in schizophrenia: violence, victimisation, suicide/self-harm, substance use, homelessness and unemployment.

METHOD

A review ofthe literature was carried out for each adverse outcome, with attention to critical appraisal of existing measurement tools.

RESULTS

Schizophrenia is associated strongly with all six outcomes, although research has mainly focused on violence. Each outcome acts as a risk factor for at least some of the other outcomes. There are few standardised or validated measures for these 'hard' outcomes. Each measure has inherent biases but a growing trend is for these to be minimised by using multiple measures.

CONCLUSIONS

A single instrument which systematically measures multiple societal outcomes of schizophrenia would be extremely useful for both clinical and research purposes.

Inpatient psychosocial rehabilitation in rural NSW: assessment of clinically significant change for people with severe mental illness

OBJECTIVE

The aim of the present study was to describe an inpatient psychosocial rehabilitation programme in rural New South Wales and to assess the effectiveness of the programme using measures of clinically significant change.

METHOD

The first 88 patients with a schizophrenia spectrum disorder to enter the Manara Clinic and Turon House, New South Wales, psychosocial rehabilitation programmes were assessed at admission and discharge using the Brief Psychiatric Rating Scale, the Health of the Nation Outcome Scales, and the Kessler-10 self-report measure.

RESULTS

Significant improvements in psychiatric symptomatology, psychosocial functioning, and psychological distress were found over the course of the inpatient stay. Clinical significance analyses using patients in the community as the reference group indicated a reliable and clinically significant improvement for 33% of inpatients on psychiatric symptomatology, 39% of inpatients on psychosocial functioning, and 21% of inpatients on psychological distress.

CONCLUSIONS

The psychosocial rehabilitation programme provides clinically significant initial benefits for patients with severe mental illnesses. More attention needs to be paid to evaluating which components of psychosocial rehabilitation contribute most to these benefits. Follow-up evaluation is required to determine whether the benefits of this programme are sustained in the community.

Evaluating mental health services in a Greek community. The factor of non-compliance to therapy

This study aims to evaluate rendered mental health services for children and adolescents through the investigation of those factors which are related to non-compliance with therapy and which affect treatment outcome. Data were collected from the files of all new cases who applied to a Community Mental Health Centre in Athens during 2000-2002 (N = 363). For each case, the following factors were examined: age, sex, family situation, parents' educational level, referral source, child's psychiatric and psychosocial diagnoses, type of proposed therapy, phase at which termination of therapy occurred and outcome. Around 45.7% of the sample did not complete therapy. The probability of treatment compliance increased when the patient was male, with a diagnosis of a specific developmental disorder, treated in a well-structured therapy programme, was from a healthy family environment and his mother was better educated. On the contrary, an adverse family situation (one-parent family, inadequate parental supervision) and the female sex had a negative association with treatment compliance. Most of the cases discontinued their treatment upon completion of the diagnostic procedure. Referral source did not influence treatment compliance. Evaluation of our service has shown that more attention should be paid to less-educated families and those in adverse situations, particularly when the patient is female.

Use of standardised outcome measures in adult mental health services: randomised controlled trial

BACKGROUND

Routine use of standardised outcome measures is not universal.

AIMS

To evaluate the effectiveness of standardised outcome assessment.

METHOD

A randomised controlled trial, involving 160 representative adult mental health patients and paired staff (ISRCTN16971059). The intervention group (n=101) (a) completed monthly postal questionnaires assessing needs, quality of life, mental health problem severity and therapeutic alliance, and (b) received 3-monthly feedback. The control group (n=59) received treatment as usual.

RESULTS

The intervention did not improve primary outcomes of patient-rated unmet need and of quality of life. Other subjective secondary outcome measures were also not improved. The intervention reduced psychiatric inpatient days (3.5 v.16.4 mean days, bootstrapped 95% CI1.6-25.7), and hence service use costs were 2586 UK pounds (95% CI 102-5391) less for intervention-group patients. Net benefit analysis indicated that the intervention was cost-effective.

CONCLUSIONS

Routine use of outcome measures as implemented in this study did not improve subjective outcomes, but was associated with reduced psychiatric inpatient admissions.

Patient-rated mental health needs and quality of life improvement

BACKGROUND

Patient-rated unmet need is cross-sectionally associated with quality of life. Its longitudinal relationship has not been established.

AIMS

To test the hypotheses that: (a) higher patient-rated unmet need is associated with lower individual quality of life assessments by a patient over time; and (b) reduction in patient-rated unmet need precedes improvement in quality of life.

METHOD

One hundred and one individuals using adult mental health services were asked to complete 6-monthly questionnaires, comprising quality of life (Manchester Short Assessment of Quality of Life, MANSA) and unmet need (Camberwell Assessment of Need Short Appraisal Schedule, CANSAS) assessments.

RESULTS

Seventy-three participants provided 240 separate pairs of consecutive assessments. Random effects regression models indicated an impact on current quality of life for both average level of unmet need (B= -0.23, 95% CI -0.29 to -0.17) and change in unmet need over the past month (B= -0.04, 95% CI -0.02).

CONCLUSIONS

Changes in patient-rated unmet needs may cause changes in quality of life.

Performance of routine outcome measures in adult mental health care

OBJECTIVE

To examine the performance of two widely used outcome measures in public adult mental health services.

METHOD

As part of a larger study, clinical staff in eight New Zealand District Health Board areas collected consumer outcomes data for a period of 6 months. Among other measures, the Health of the Nation Outcome Scales (HoNOS) and Life Skills Profile (LSP-16) were completed at the start and end of episodes of care, as well as at 90-day reviews.

RESULTS

The performance of these two measures was examined in relation to compliance with the data collection protocol, item completion rates, certain psychometric properties, the relationship between them and the relationship of each instrument to the age, sex, principal psychiatric diagnosis and legal status of the consumer. Compliance with the HoNOS was superior to the LSP-16, but the reverse was true for completion. The two instruments were strongly correlated, but not enough to suggest that one may substitute for the other. A five-subscale structure for the HoNOS was supported. Certain HoNOS subscales were associated with age, but the total score was not, and differences between the sexes were small. Some large diagnostic differences were observed on both instruments. All LSP-16 summary scores and all HoNOS summary scores apart from Depression were higher (worse) in involuntary episodes. Only one LSP-16 item, relating to work potential, was related to age and men generally scored higher (worse) than women.

CONCLUSION

From a technical perspective, the HoNOS and LSP-16 appear suitable for routine collection. New Zealand has adopted the HoNOS for routine use but not the LSP-16 because of local concerns about its appropriateness and acceptability. The clinician-rated measures reported here need to be complemented by both consumer-rated and family/carer measures.