A model for developing outcome measures from the perspectives of mental health service users

It's not evidence, it's insight: bringing patients' perspectives into health technology appraisal at NICE

PLAIN ENGLISH SUMMARY

Health technology appraisal involves reviewing the findings from clinical trials and economic data to produce guidance on how health technology should be used. This task is carried out by appraisal committees in NICE. One of the several ways in which patients can feed their views into these committees is via a written patient statement. We asked nine committee members about what difference the information from patients makes to their decision-making. The Committee members reported that written patient statements offer a different perspective when reviewing the clinical and economic data. This can have a profound impact when a committee draws conclusions based solely on the data, which may not reflect the reality of patients' lives. The patients' and carers' input provides meaning to the data, 'bringing the numbers to life'. It identifies if the technology has any wider impacts than what's been reported in the clinical trial, and also if the trial has measured what's important to patients. We conclude that the written patient statement adds value to the decision-making process by helping Committee members to make sense of the clinical and economic data-it makes them look at the evidence 'in a different light'. Patients' stories are very effective in this context, because they have the power to communicate and to challenge Committee members' assumptions. Understanding this difference between analysing research evidence and drawing on patients' insights is important in thinking about what's needed in a written patient statement and the best way to obtain it.

ABSTRACT

Background Health technology appraisal involves reviewing clinical and economic data to inform guidance on the use of technology. In England this task is carried out by appraisal committees within the National Institute for Health and Care Excellence (NICE). Patients are not committee members as they have a vested interest in the outcome, but one of the several ways they are involved is through submitting a written patient statement, which is considered by the committee during its deliberations. We aimed to find out how the written patient statement adds value to the decision-making process by exploring how it is used in practice. Methods Semi-structured interviews were conducted with nine members of NICE appraisal committees. The interviews were transcribed and analysed thematically. We drew on published evidence of the impact of patient involvement on clinical research and our experience of supporting organisations to produce written patient statements to analyse the findings. Results Committee members reported that written patient statements offer a different perspective when evaluating clinical and economic data. This can have a profound impact when a committee draws conclusions based on data that may not reflect the reality of the patient experience. Information from patients and carers also provides context and meaning to the data, by explaining its real-life implications. It identifies wider impacts of a technology that may not have been assessed in a clinical trial, as well as commenting on whether what has been measured in a trial is relevant to patients. The main barrier to using the written patient statement is the misperception that it is a form of research 'evidence', when in fact it takes the form of experiential knowledge - or insight. Conclusions The written patient statement adds value by aiding Committee members in their interpretation of existing evidence - it enables them to consider this evidence 'in a different light'. In this context, patients' experiential knowledge is effective because it is subjective, emotional and anecdotal. It then has the power to communicate and to challenge assumptions based on the data alone. Understanding this difference between using evidence and insights has implications for the content of a written patient statement and the approaches used to obtain it.

Defining continuity of care from the perspectives of mental health service users and professionals: an exploratory, comparative study

Background

Continuity of care (COC) is central to the organization and delivery of mental health services. Traditional definitions have excluded service users, and this lack of involvement has been linked to poor conceptual clarity surrounding the term. Consequently, very little is known about the differences and similarities in the conceptualization of COC by mental health service users and professionals.

Objective

To explore and compare mental health service users’ and professionals’ definitions of COC.

Methods

Using an exploratory, qualitative design, five focus groups with 32 service users each met twice. Data were analysed thematically to generate a service user‐defined model of COC. In a cross‐sectional survey, health and social care professionals (n = 184) defined COC; responses were analysed thematically. Service user and professional definitions were conceptually mapped and compared to identify similarities and differences.

Results

There was crossover between the service user and professional derived models of COC. Both contained temporal, quality, systemic, staff, hospital and needs‐related elements of COC. Service users prioritized access, information, peer support and avoiding services; health professionals most frequently referred to staff, cross‐sectional and temporal COC. Service users alone identified service avoidance, peer support and day centres as COC elements; professionals alone identified cross‐sectional working.

Conclusions

Important similarities and differences exist in service user and professional conceptualizations of COC. Further research is necessary to explore these differences, prior to integrating service user and professional perspectives in a validated COC framework which could enable the development and evaluation of interventions to improve COC, informing policy and practice.

Expectations of new treatment in rheumatoid arthritis: developing a patient-generated questionnaire

BACKGROUND

Service-user partnerships in research exist in mental health, but there have been few advances in other disciplines, apart from cancer.

OBJECTIVES

To develop a patient-generated expectancy measure for new treatments in rheumatoid arthritis (RA), using a participatory method.

METHOD

Stage 1: three repeated focus groups and two expert panels with patients with RA conducted by a patient researcher to generate items for the draft questionnaire. Stage 2: feasibility study of draft scale with consecutive outpatient attendees.

RESULTS

Patients identified 21 dimensions of new treatment expectations, grouped into (i) physical, (ii) psycho-social and (iii) expectations relating to the impact of treatment. This resulted in a draft instrument assessed in a feasibility study.

DISCUSSION AND CONCLUSION

The participatory research method was useful in involving patients actively in research and to produce collaboratively a feasible, valid and acceptable measure in RA. The scale will be included in a longitudinal observational study, with newly diagnosed patients, to assess (i) whether the new scale demonstrates sensitivity to change for expectations when receiving new treatment and (ii) participants' completion rate of the new scale compared with five instruments included in the future study.

Life in acute mental health settings: experiences and perceptions of service users and nurses

Background. Acute psychiatric provision in the UK today as well as globally has many critics including service users and nurses. Method. Four focus groups, each meeting twice, were held separately for service users and nurses. The analysis was not purely inductive but driven by concerns with the social position of marginalised groups - both patients and staff. Results. The main themes were nurse/patient interaction and coercion. Service users and nurses conceptualised these differently. Service users found nurses inaccessible and uncaring, whereas nurses also felt powerless because their working life was dominated by administration. Nurses saw coercive situations as a reasonable response to factors 'internal' to the patient whereas for service users they were driven to extreme behaviour by the environment of the ward and coercive interventions were unnecessary and heavy handed. Conclusion. This study sheds new light on living and working in acute mental health settings today by comparing the perceptions of service users and nurses and deploying service user and nurse researchers. The intention is to promote better practice by providing a window on the perceptions of both groups.

'Is it worth doing?' Measuring the impact of patient and public involvement in research

ABSTRACT

Much of the current debate around the impact of patient/public involvement on research focuses on the lack of empirical data. While a number of systematic literature reviews have reported the various ways in which involvement makes a difference to research and the people involved, this evidence has been criticised as being weak and anecdotal. It is argued that robust evidence is still required. This review reflects on the use of quantitative approaches to evaluating impact. It concludes that the statistical evidence is weakened by not paying sufficient attention to the context in which involvement takes place and the way it is carried out. However, if scientific (systematic, quantitative, empirical) approaches are designed in a way to take these factors into account, they might not generate knowledge that is useful beyond the original context. Such approaches might not therefore enhance our understanding of when, why and how involvement makes a difference. In the context of individual research projects where researchers collaborate with patients/the public, researchers often acquire 'new' knowledge about life with a health condition. This new understanding can be described as experiential knowledge-'knowledge in context'-that researchers gain through direct experience of working with patients/the public. On this basis, researchers' accounts of their experience potentially provide a source of insight and learning to influence others, in the same way that the patient experience helps to shape research. These accounts could be improved by increasing the detail provided about context and mechanism. One of the most important contextual factors that influence the outcome of involvement is the researchers themselves and the skills, assumptions, values and priorities they start with. At the beginning of any research project, the researchers 'don't know what they don't know' until they involve patients/the public. This means that the impact of involvement within any particular project is somewhat unpredictable. The answer to the question 'Is involvement worth doing?' will always be 'It depends'. Further exploration of the contextual and mechanistic factors which influence outcomes could give a stronger steer to researchers but may never accurately predict any specific impact.

PLAIN ENGLISH SUMMARY

In recent years, there has been considerable interest in finding out what difference patient and public involvement makes to research projects. The evidence published so far has been criticised for being weak and anecdotal. Some people argue we need robust evidence of impact from scientific studies of involvement. In this review, I consider examples of where impact has been measured using statistical methods. I conclude that the statistical evidence is weak, if the studies do not consider the context in which involvement takes place and the way that it is done. Studies designed to take this into account give us more confidence that the involvement did make a difference to that particular project. They do not tell us whether the same impact will occur in the same way in other projects and therefore have limited value. Researchers gain an understanding of involvement through their direct experience of working with patients and the public. This is 'knowledge in context' or 'insight' gained in the same way that patients gain expertise through their direct experience of a health condition. This means that detailed accounts of involvement from researchers already provide valuable learning to others, in the same way that patients' insights help shape research. However, the impact of involvement will always be somewhat unpredictable, because at the start of any project researchers 'don't know what they don't know'-they do not know precisely what problems they might anticipate, until the patients/public tell them.

User-generated quality standards for youth mental health in primary care: a participatory research design using mixed methods

OBJECTIVES

To develop user-generated quality standards for young people with mental health problems in primary care using a participatory research model.

METHODS

50 young people aged 16-25 from community settings and primary care participated in focus groups and interviews about their views and experiences of seeking help for mental health problems in primary care, cofacilitated by young service users and repeated to ensure respondent validation. A second group of young people also aged 16-25 who had sought help for any mental health problem from primary care or secondary care within the last 5 years were trained as focus groups cofacilitators (n=12) developed the quality standards from the qualitative data and participated in four nominal groups (n=28).

RESULTS

46 quality standards were developed and ranked by young service users. Agreement was defined as 100% of scores within a two-point region. Group consensus existed for 16 quality standards representing the following aspects of primary care: better advertising and information (three); improved competence through mental health training and skill mix within the practice (two); alternatives to medication (three); improved referral protocol (three); and specific questions and reassurances (five). Alternatives to medication and specific questions and reassurances are aspects of quality which have not been previously reported.

CONCLUSIONS

We have demonstrated the feasibility of using participatory research methods in order to develop user-generated quality standards. The development of patient-generated quality standards may offer a more formal method of incorporating the views of service users into quality improvement initiatives. This method can be adapted for generating quality standards applicable to other patient groups.

Service user- and carer-reported measures of involvement in mental health care planning: methodological quality and acceptability to users

BACKGROUND

Increasing service user and carer involvement in mental health care planning is a key healthcare priority but one that is difficult to achieve in practice. To better understand and measure user and carer involvement, it is crucial to have measurement questionnaires that are both psychometrically robust and acceptable to the end user.

METHODS

We conducted a systematic review using the terms "care plan$," "mental health," "user perspective$," and "user participation" and their linguistic variants as search terms. Databases were searched from inception to November 2012, with an update search at the end of September 2014. We included any articles that described the development, validation or use of a user and/or carer-reported outcome measures of involvement in mental health care planning. We assessed the psychometric quality of each instrument using the "Evaluating the Measurement of Patient-Reported Outcomes" (EMPRO) criteria. Acceptability of each instrument was assessed using novel criteria developed in consultation with a mental health service user and carer consultation group.

RESULTS

We identified eleven papers describing the use, development, and/or validation of nine user/carer-reported outcome measures. Psychometric properties were sparsely reported and the questionnaires met few service user/carer-nominated attributes for acceptability. Where reported, basic psychometric statistics were of good quality, indicating that some measures may perform well if subjected to more rigorous psychometric tests. The majority were deemed to be too long for use in practice.

DISCUSSION

Multiple instruments are available to measure user/carer involvement in mental health care planning but are either of poor quality or poorly described. Existing measures cannot be considered psychometrically robust by modern standards, and cannot currently be recommended for use. Our review has identified an important knowledge gap, and an urgent need to develop new user and carer measures of care-planning involvement.

Impact of patient involvement in mental health research: longitudinal study

BACKGROUND

It is deemed good practice to involve patients routinely in research but no study has investigated the practical benefits, particularly to successful recruitment.

AIMS

To identify whether patient involvement is associated with study success.

METHOD

All studies listed on the Mental Health Research Network (MHRN) portfolio database (n = 374) were interrogated using logistic regression, ANOVA and Pearson's correlation to identify associations with study characteristics, funding bodies and recruitment success.

RESULTS

Patient involvement increased over time although in some areas of research it was limited. Some funders, especially the National Institute for Health Research (NIHR), had more associated patient involvement than others. Studies that involved patients to a greater extent were more likely to have achieved recruitment targets (χ(2) = 4.58, P<0.05), defined as reaching at least 90% of the target.

CONCLUSIONS

This is the first time associations with study success have been identified for patient involvement. Researchers might now consider ways to involve patients more comprehensively as this is associated with study success. Further research is needed to explore this finding.

Patient and public involvement in health research: ethical imperative and/or radical challenge?

Patient and public involvement in health research, including mental health research, is promoted by research funders in England. However, it is poorly conceptualised. One argument is that patient and public involvement in research is an ethical imperative because those who research is for should have a stake in how it is done. This could be developed through concepts of citizenship and democratic science. More strongly, it can be argued that changing the knowledge producers will change knowledge itself. Starting with feminist standpoint epistemology, it is argued that a political conceptualisation best captures the new knowledge that marginalised health groups can produce.

Sur la contribution des personnes utilisatrices de services de santé mentale en tant que partenaires d’enseignement en psychiatrie

Le Collège royal des médecins et chirurgiens du Canada prévoit qu’au terme de leur formation, les résidents en psychiatrie pourront agir efficacement en tant que promoteurs de la santé. Ce commentaire présente une analyse d’entrevues réalisées dans le cadre d’un perfectionnement professionnel auprès d’experts mondialement reconnus du rétablissement en santé mentale. À la lumière de la Charte d’Ottawa pour la promotion de la santé, il en ressort que des personnes utilisatrices de services de santé mentale peuvent être de précieux partenaires d’enseignement à la fois en matière de promotion de la santé et de rétablissement. Le recours à des patients partenaires d’enseignement en psychiatrie peut également représenter une stratégie de lutte à la stigmatisation et aux préjugés en favorisant davantage de réciprocité entre les professionnels et les personnes traitées.

Developing an evaluation framework for consumer-centred collaborative care of depression using input from stakeholders

OBJECTIVE

To develop a framework for research and evaluation of collaborative mental health care for depression, which includes attributes or domains of care that are important to consumers.

METHODS

A literature review on collaborative mental health care for depression was completed and used to guide discussion at an interactive workshop with pan-Canadian participants comprising people treated for depression with collaborative mental health care, as well as their family members; primary care and mental health practitioners; decision makers; and researchers. Thematic analysis of qualitative data from the workshop identified key attributes of collaborative care that are important to consumers and family members, as well as factors that may contribute to improved consumer experiences.

RESULTS

The workshop identified an overarching theme of partnership between consumers and practitioners involved in collaborative care. Eight attributes of collaborative care were considered to be essential or very important to consumers and family members: respectfulness; involvement of consumers in treatment decisions; accessibility; provision of information; coordination; whole-person care; responsiveness to changing needs; and comprehensiveness. Three inter-related groups of factors may affect the consumer experience of collaborative care, namely, organizational aspects of care; consumer characteristics and personal resources; and community resources.

CONCLUSION

A preliminary evaluation framework was developed and is presented here to guide further evaluation and research on consumer-centred collaborative mental health care for depression.

Recovery of evidence-based practice

Consumer recovery is now enshrined in the national mental health policy of many countries. If this construct, which stems from the consumer/user/survivor movement, is truly to be the official and formal goal of mental health services, then it must be the yardstick against which evidence-based practice (EBP) is judged. From a consumer-recovery perspective, this paper re-examines aspects of services chosen for study, methodologies, outcomes measures, and standards of evidence associated with EBP, those previously having been identified as deficient and in need of expansion. One of the significant differences between previous investigations and the present study is that the work, writing, perspectives, and advocacy of the consumer movement has developed to such a degree that we now have a much more extensive body of material upon which to critique EBP and inform and support the expansion of EBP. Our examination reinforces previous findings and the ongoing need for expansion. The consumer recovery-focused direction, resources, frameworks, and approaches identified through the present paper should be used to expand the aspects of services chosen for study, methodologies, outcomes measures, and standards of evidence. This expansion will ultimately enable services to practice in a manner consistent with the key characteristics of supporting personal recovery.

Patient-reported outcome measures: are they patient-generated, patient-centred or patient-valued?

BACKGROUND

In the past two decades, there has been a growing interest in the development of a more patient-centred approach to assessing treatment outcomes. This interest has resulted in the increasing use of patient-reported outcome measures (PROMs) in both clinical trials and usual clinical practice.

AIMS

To briefly discuss the paucity of efficacy and effectiveness studies in the field of mental health (exemplified by schizophrenia, depression and opioid dependence) that significantly incorporate the patient's perspective. The limited concordance between the perspectives of patients and clinicians in outcome assessment is also addressed. Finally, we propose a new PROM classification system based on the degree to which these instruments incorporate the patient's perspective.

CONCLUSIONS

PROMs may differ little from traditional instruments unless they truly incorporate the patient's perspective and not just the perspectives of clinicians and researchers. Efforts to develop new PROMs that provide a more patient-centred outcome assessment should use qualitative and participatory methods to capture and incorporate patient perspectives and values.

Understanding service user-defined continuity of care and its relationship to health and social measures: a cross-sectional study

BACKGROUND

Despite the importance of continuity of care [COC] in contemporary mental health service provision, COC lacks a clearly agreed definition. Furthermore, whilst there is broad agreement that definitions should include service users' experiences, little is known about this. This paper aims to explore a new construct of service user-defined COC and its relationship to a range of health and social outcomes.

METHODS

In a cross sectional study design, 167 people who experience psychosis participated in structured interviews, including a service user-generated COC measure (CONTINU-UM) and health and social assessments. Constructs underlying CONTINU-UM were explored using factor analysis in order to understand service user-defined COC. The relationships between the total/factor CONTINU-UM scores and the health and social measures were then explored through linear regression and an examination of quartile results in order to assess whether service user-defined COC is related to outcome.

RESULTS

Service user-defined COC is underpinned by three sub-constructs: preconditions, staff-related continuity and care contacts, although internal consistency of some sub-scales was low. High COC as assessed via CONTINU-UM, including preconditions and staff-related COC, was related to having needs met and better therapeutic alliances. Preconditions for COC were additionally related to symptoms and quality of life. COC was unrelated to empowerment and care contacts unrelated to outcomes. Service users who had experienced a hospital admission experienced higher levels of COC. A minority of service users with the poorest continuity of care also had high BPRS scores and poor quality of life.

CONCLUSIONS

Service-user defined continuity of care is a measurable construct underpinned by three sub-constructs (preconditions, staff-related and care contacts). COC and its sub-constructs demonstrate a range of relationships with health and social measures. Clinicians have an important role to play in supporting service users to navigate the complexities of the mental health system. Having experienced a hospital admission does not necessarily disrupt the flow of care. Further research is needed to test whether increasing service user-defined COC can improve clinical outcomes. Using CONTINU-UM will allow researchers to assess service users' experiences of COC based on the elements that are important from their perspective.

VOICE: developing a new measure of service users' perceptions of inpatient care, using a participatory methodology

BACKGROUND

Service users express dissatisfaction with inpatient care and their concerns revolve around staff interactions, involvement in treatment decisions, the availability of activities and safety. Traditionally, satisfaction with acute care has been assessed using measures designed by clinicians or academics.

AIMS

To develop a patient-reported outcome measure of perceptions of acute care. An innovative participatory methodology was used to involve services users throughout the research process.

METHOD

A total of 397 participants were recruited for the study. Focus groups of service users were convened to discuss their experiences and views of acute care. Service user researchers constructed a measure from the qualitative data, which was validated by expert panels of service users and tested for its psychometric properties.

RESULTS

Views on Inpatient Care (VOICE) is easy to understand and complete and therefore is suitable for use by service users while in hospital. The 19-item measure has good validity and internal and test-retest reliability. Service users who have been compulsorily admitted have significantly worse perceptions of the inpatient environment.

CONCLUSIONS

A participatory methodology has been used to generate a self-report questionnaire measuring service users' perceptions of acute care. VOICE encompasses the issues that service users consider most important and has strong psychometric properties.

Rapid progress or lengthy process? Electronic personal health records in mental health

A major objective of many healthcare providers is to increase patients' participation in their own care. The introduction of electronic personal health records (ePHRs) may help to achieve this. An ePHR is an electronic database of an individual's health information, accessible to and maintained by the patient. ePHRs are very much in vogue, with an increasing number of studies reporting their potential utility as well as cost. However, the vast majority of these studies focus on general healthcare. Little attempt has been made to document the specific problems which might occur throughout the implementation of ePHRs in mental health. This review identifies such concerns through an electronic search of the literature. Several potential difficulties are highlighted and addressed, including access to information technology, identifying relevant populations and the handling of sensitive information. Special attention is paid to the concept of 'empowerment' and what this means in relation to ePHRs.