Keeping mental health reform on course: selecting indicators of mental health system performance

A systematic review of in-patient psychiatric care for people with intellectual disabilities and/or autism: effectiveness, patient safety and experience

BACKGROUND

An increasing number of children, adolescents and adults with intellectual disabilities and/or autism are being admitted to general psychiatric wards and cared for by general psychiatrists.

AIMS

The aim of this systematic review was to consider the likely effectiveness of in-patient treatment for this population, and compare and contrast differing models of in-patient care.

METHOD

A systematic search was completed to identify papers where authors had reported data about the effectiveness of in-patient admissions with reference to one of three domains: treatment effect (e.g. length of stay, clinical outcome, readmission), patient safety (e.g. restrictive practices) and patient experience (e.g. patient or family satisfaction). Where possible, outcomes associated with admission were considered further within the context of differing models of in-patient care (e.g. specialist in-patient services versus general mental health in-patient services).

RESULTS

A total of 106 studies were included and there was evidence that improvements in mental health, social functioning, behaviour and forensic risk were associated with in-patient admission. There were two main models of in-patient psychiatric care described within the literature: admission to a specialist intellectual disability or general mental health in-patient service. Patients admitted to specialist intellectual disability in-patient services had greater complexity, but there were additional benefits, including fewer out-of-area discharges and lower seclusion rates.

CONCLUSIONS

There was evidence that admission to in-patient services was associated with improvements in mental health for this population. There was some evidence indicating better outcomes for those admitted to specialist services.

What makes a good quality indicator set? A systematic review of criteria

BACKGROUND

While single indicators measure a specific aspect of quality (e.g. timely support during labour), users of these indicators, such as patients, providers and policy-makers, are typically interested in some broader construct (e.g. quality of maternity care) whose measurement requires a set of indicators. However, guidance on desirable properties of indicator sets is lacking.

OBJECTIVE

Based on the premise that a set of valid indicators does not guarantee a valid set of indicators, the aim of this review is 2-fold: First, we introduce content validity as a desirable property of indicator sets and review the extent to which studies in the peer-reviewed health care quality literature address this criterion. Second, to obtain a complete inventory of criteria, we examine what additional criteria of quality indicator sets were used so far.

METHODS

We searched the databases Web of Science, Medline, Cinahl and PsycInfo from inception to May 2021 and the reference lists of included studies. English- or German-language, peer-reviewed studies concerned with desirable characteristics of quality indicator sets were included. Applying qualitative content analysis, two authors independently coded the articles using a structured coding scheme and discussed conflicting codes until consensus was reached.

RESULTS

Of 366 studies screened, 62 were included in the review. Eighty-five per cent (53/62) of studies addressed at least one of the component criteria of content validity (content coverage, proportional representation and contamination) and 15% (9/62) addressed all component criteria. Studies used various content domains to structure the targeted construct (e.g. quality dimensions, elements of the care pathway and policy priorities), providing a framework to assess content validity. The review revealed four additional substantive criteria for indicator sets: cost of measurement (21% [13/62] of the included studies), prioritization of 'essential' indicators (21% [13/62]), avoidance of redundancy (13% [8/62]) and size of the set (15% [9/62]). Additionally, four procedural criteria were identified: stakeholder involvement (69% [43/62]), using a conceptual framework (44% [27/62]), defining the purpose of measurement (26% [16/62]) and transparency of the development process (8% [5/62]).

CONCLUSION

The concept of content validity and its component criteria help assessing whether conclusions based on a set of indicators are valid conclusions about the targeted construct. To develop a valid indicator set, careful definition of the targeted construct including its (sub-)domains is paramount. Developers of quality indicators should specify the purpose of measurement and consider trade-offs with other criteria for indicator sets whose application may reduce content validity (e.g. costs of measurement) in light thereof.

Development of mental health quality indicators (MHQIs) for inpatient psychiatry based on the interRAI mental health assessment

Background

Outcome quality indicators are rarely used to evaluate mental health services because most jurisdictions lack clinical data systems to construct indicators in a meaningful way across mental health providers. As a result, important information about the effectiveness of health services remains unknown. This study examined the feasibility of developing mental health quality indicators (MHQIs) using the Resident Assessment Instrument - Mental Health (RAI-MH), a clinical assessment system mandated for use in Ontario, Canada as well as many other jurisdictions internationally.

Methods

Retrospective analyses were performed on two datasets containing RAI-MH assessments for 1,056 patients from 7 facilities and 34,788 patients from 70 facilities in Ontario, Canada. The RAI-MH was completed by clinical staff of each facility at admission and follow-up, typically at discharge. The RAI-MH includes a breadth of information on symptoms, functioning, socio-demographics, and service utilization. Potential MHQIs were derived by examining the empirical patterns of improvement and incidence in depressive symptoms and cognitive performance across facilities in both sets of data. A prevalence indicator was also constructed to compare restraint use. Logistic regression was used to evaluate risk adjustment of MHQIs using patient case-mix index scores derived from the RAI-MH System for Classification of Inpatient Psychiatry.

Results

Subscales from the RAI-MH, the Depression Severity Index (DSI) and Cognitive Performance Scale (CPS), were found to have good reliability and strong convergent validity. Unadjusted rates of five MHQIs based on the DSI, CPS, and restraints showed substantial variation among facilities in both sets of data. For instance, there was a 29.3% difference between the first and third quartile facility rates of improvement in cognitive performance. The case-mix index score was significantly related to MHQIs for cognitive performance and restraints but had a relatively small impact on adjusted rates/prevalence.

Conclusions

The RAI-MH is a feasible assessment system for deriving MHQIs. Given the breadth of clinical content on the RAI-MH there is an opportunity to expand the number of MHQIs beyond indicators of depression, cognitive performance, and restraints. Further research is needed to improve risk adjustment of the MHQIs for their use in mental health services report card and benchmarking activities.

The mental health system in Lombardy, Italy: access to services and patterns of care

PURPOSE

The psychiatric reform in Italy devolved to the regions the responsibility of implementing community psychiatric care. The aim of this paper is to evaluate the mental health system in Lombardy by assessing changes in accessibility and patterns of care occurred between 1999 and 2009.

METHODS

Data on mental health services were collected through the regional mental health information system and analyzed in terms of treated prevalence, treated incidence, continuity of care and packages of care.

RESULTS

Both treated incidence and treated prevalence in Lombardy increased between 1999 and 2009. There was an increasing access to psychiatric services of people with a better social integration. Incidence of schizophrenic and personality disorders decreased and that of affective and neurotic disorders increased dramatically, while increase in prevalence concerned all diagnostic groups. The percentage of patients in continuous care remained stable and was generally low. The majority of cases, even those with schizophrenia, are cared for on outpatient basis. The percentage of patients receiving integrated multiprofessional care declined. Rates of admission to inpatient services remained low and within the inpatient sector a shift from hospital towards residential care emerged, with decreasing hospital utilization and an increase in size of patient population entering community residences. Treatment gap is still a problem in schizophrenic disorders.

CONCLUSIONS

The Lombardy mental health system is strongly based on community care. However, it is reaching a turning point and it needs to be improved in some key areas: the shifting balance towards the care of common mental disorders, in the absence of resource allocations targeted to severely mentally ill, may hinder the system ability to deal with more disabled people. A focus on early intervention and an improvement of continuity of care for people with severe mental disorder, by strengthening community teams, is a priority.

Performance measurement in mental health care: present situation and future possibilities

SUMMARY

This paper describes performance measurement and its indicators for mental health care services. Performance measurement can serve several goals such as accountability, quality improvement and performance management. For all three purposes structure, process and outcome indicators should be measured. Literature was retrieved from Medline and PsychInfo in order to see which performance indicators were used for the three purposes of performance measurement in mental health care. The indicators were classified in structure, process and outcome indicators. The results show no big differences in the indicators used among studies. Performance management is the performance measurement purpose most referred to, followed by accountability, and quality improvement. Outcome and process indicators are used most, structure indicators are in the minority. Several levels of measurement, that is national or service level, came forward in the literature review. To overcome misinterpretation of data and to be able to improve quality and manage performances, performance indicator sets should refer to structure, process and outcome. Indicators should be chosen carefully with the aim of the measurement taken into mind. Based on this review, a conceptual framework is presented to support managers in their decisions about which indictors can best be used for performance measurement. Additionally, a model that provides an understanding of the use of information gained by performance measurement is given.