Cancer Care Ontario and integrated cancer programs: portrait of a performance management system and lessons learned

What do end-users want to know about managing the performance of healthcare delivery systems? Co-designing a context-specific and practice-relevant research agenda

Background

Despite increasing interest in joint research priority-setting, few studies engage end-user groups in setting research priorities at the intersection of the healthcare and management disciplines. With health systems increasingly establishing performance management programmes to account for and incentivize performance, it is important to conduct research that is actionable by the end-users involved with or impacted by these programmes. The aim of this study was to co-design a research agenda on healthcare performance management with and for end-users in a specific jurisdictional and policy context.

Methods

We undertook a rapid review of the literature on healthcare performance management (n = 115) and conducted end-user interviews (n = 156) that included a quantitative ranking exercise to prioritize five directions for future research. The quantitative rankings were analysed using four methods: mean, median, frequency ranked first or second, and frequency ranked fifth. The interview transcripts were coded inductively and analysed thematically to identify common patterns across participant responses.

Results

Seventy-three individual and group interviews were conducted with 156 end-users representing diverse end-user groups, including administrators, clinicians and patients, among others. End-user groups prioritized different research directions based on their experiences and information needs. Despite this variation, the research direction on motivating performance improvement had the highest overall mean ranking and was most often ranked first or second and least often ranked fifth. The research direction was modified based on end-user feedback to include an explicit behaviour change lens and stronger consideration for the influence of context.

Conclusions

Joint research priority-setting resulted in a practice-driven research agenda capable of generating results to inform policy and management practice in healthcare as well as contribute to the literature. The results suggest that end-users are keen to open the “black box” of performance management to explore more nuanced questions beyond “does performance management work?” End-users want to know how, when and why performance management contributes to behaviour change (or fails to) among front-line care providers.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12961-021-00779-x.

Use of comparative performance indicators in rehabilitation

BACKGROUND

The development of performance indicators that enable benchmarking between organizations is an important mechanism for accountability, organizational learning, and performance improvement. In the province of Quebec (Canada), 21 rehabilitation organizations developed a common set of performance indicators through interorganizational collaboration.

PURPOSE

The aims of this study were to describe the rehabilitation organizations' use of a common set of performance indicators and to identify the factors influencing such use.

APPROACH

A qualitative survey was performed. Individual semistructured interviews were conducted with executives (n = 18) working at 16 rehabilitation organizations using a common set of performance indicators. A thematic analysis of the factors of use was performed according to the Consolidated Framework for Implementation Research. The use of performance indicators was categorized as purposeful, political, or passive.

FINDINGS

Our results showed that all organizations used the common set of performance indicators. Four factors were identified as important to all the rehabilitation organizations to explain their interest in comparative performance indicators: the need to develop their own performance indicators, the compatibility of performance information with organizational needs, complexity/simplicity of performance information, and the support offered by their common association. Sixty-three percent of rehabilitation organizations made purposeful or political use of performance indicators. Three main factors contributed to typify those organizations from the others: the perceived quality of the performance indicators, the leadership of decision makers, and the resources available.

PRACTICE IMPLICATIONS

Our results showed that use of performance indicators can support the initiation of projects for improving the quality of care. Key recommendations are proposed to decision makers that may enhance performance indicators' use.

Balanced performance measurement in research hospitals: the participative case study of a haematology department

Background

The paper aims to review, design and implement a multidimensional performance measurement system for a public research hospital in order to address the complexity of its multifaceted stakeholder requirements and its double institutional aim of care and research.

Method

The methodology relies on a participative case study performed by external researchers in close collaboration with the staff of an Italian research hospital.

Results

The paper develops and applies a customized version of balanced scorecard based on a new set of performance measures. Our findings suggest that it can be considered an effective framework for measuring the research hospital performance, thanks to a combination of generalizable and context-specific factors.

Conclusions

By showing how the balanced scorecard framework can be customized to research hospitals, the paper is especially of interest for complex healthcare organizations that are implementing management accounting practices. The paper contributes to the body of literature on the application of the balanced scorecard in healthcare through an examination of the challenges in designing and implementing this multidimensional performance tool. This is one of the first papers that show how the balanced scorecard model can be adapted to fit the specific requirements of public research hospitals.

An international comparison of costs of end-of-life care for advanced lung cancer patients using health administrative data

BACKGROUND

Patterns of end-of-life cancer care differ in Canada and the United States; yet little is known about differences in service-specific and overall costs.

AIM

The aim of this study was to compare end-of-life costs in Ontario, Canada, and the United States, using administrative health data.

DESIGN

Advanced-stage nonsmall cell lung cancer patients who died from cancer at age ⩾ 65.5 years in 2001-2005 were selected from the US Surveillance, Epidemiology, and End Results-Medicare database (N = 16,858) and the Ontario Cancer Registry (N = 8643). We estimated total and service-specific costs (2009 US dollars) in each of the last 6 months of life from the public payer perspectives for short-term and long-term survivors (lived < 180 and ⩾ 180 days post-diagnosis, respectively). Services were defined for comparisons between systems.

RESULTS

Mean monthly costs increased as death approached, were higher in short-term than long-term survivors, and were generally higher in the United States than in Ontario until the month before death, when they were similar (long-term survivors: US$10,464 and US$10,094 (p = 0.53), short-term survivors US$14,455 and US$12,836 (p = 0.11), in Surveillance, Epidemiology, and End Results-Medicare and Ontario, respectively). Costs for Medicare hospice and Ontario's palliative care components were similar and increased closer to death. Inpatient hospitalization was the main cost driver with similar costs in both cohorts, despite lower utilization in the United States. The compositions of many services and costs differed.

CONCLUSION

Costs for nonsmall cell lung cancer patients were slightly higher in the United States than Ontario until 1 month before death. Administrative data allow exploration and international comparisons of reimbursement policies, health-care delivery, and costs at the end of life.

Advancing the use of performance evaluation in health care

PURPOSE

The purpose of this paper is to develop a framework for health care performance evaluation that enables decision makers to identify areas indicative of corrective actions. The framework should provide information on strategic pro-/regress in an operational context that justifies the need for organizational adjustments.

DESIGN/METHODOLOGY/APPROACH

The study adopts qualitative methods for constructing the framework, subsequently implementing the framework in a Danish magnetic resonance imaging (MRI) unit. Workshops and interviews form the basis of the qualitative construction phase, and two internal and five external databases are used for a quantitative data collection.

FINDINGS

By aggregating performance outcomes, collective measures of performance are achieved. This enables easy and intuitive identification of areas not strategically aligned. In general, the framework has proven helpful in an MRI unit, where operational decision makers have been struggling with extensive amounts of performance information.

RESEARCH LIMITATIONS/IMPLICATIONS

The implementation of the framework in a single case in a public and highly political environment restricts the generalizing potential. The authors acknowledge that there may be more suitable approaches in organizations with different settings.

PRACTICAL IMPLICATIONS

The strength of the framework lies in the identification of performance problems prior to decision making. The quality of decisions is directly related to the individual decision maker. The only function of the framework is to support these decisions.

ORIGINALITY/VALUE

The study demonstrates a more refined and transparent use of performance reporting by combining strategic weight assignment and performance aggregation in hierarchies. In this way, the framework accentuates performance as a function of strategic progress or regress, thus assisting decision makers in exerting operational effort in pursuit of strategic alignment.

Dissemination of performance information and continuous improvement: A narrative systematic review

PURPOSE

Developing a performance measure and reporting the results to support decision making at an individual level has yielded poor results in many health systems. The purpose of this paper is to highlight the factors associated with the dissemination of performance information that generate and support continuous improvement in health organizations.

DESIGN/METHODOLOGY/APPROACH

A systematic data collection strategy that includes empirical and theoretical research published from 1980 to 2010, both qualitative and quantitative, was performed on Web of Science, Current Contents, EMBASE and MEDLINE. A narrative synthesis method was used to iteratively detail explicative processes that underlie the intervention. A classification and synthesis framework was developed, drawing on knowledge transfer and exchange (KTE) literature. The sample consisted of 114 articles, including seven systematic or exhaustive reviews.

FINDINGS

Results showed that dissemination in itself is not enough to produce improvement initiatives. Successful dissemination depends on various factors, which influence the way collective actors react to performance information such as the clarity of objectives, the relationships between stakeholders, the system's governance and the available incentives.

RESEARCH LIMITATIONS/IMPLICATIONS

This review was limited to the process of knowledge dissemination in health systems and its utilization by users at the health organization level. Issues related to improvement initiatives deserve more attention.

PRACTICAL IMPLICATIONS

Knowledge dissemination goes beyond better communication and should be considered as carefully as the measurement of performance. Choices pertaining to intervention should be continuously prompted by the concern to support organizational action.

ORIGINALITY/VALUE

While considerable attention was paid to the public reporting of performance information, this review sheds some light on a more promising avenue for changes and improvements, notably in public health systems.

Applying the balanced scorecard to local public health performance measurement: deliberations and decisions

Background

All aspects of the heath care sector are being asked to account for their performance. This poses unique challenges for local public health units with their traditional focus on population health and their emphasis on disease prevention, health promotion and protection. Reliance on measures of health status provides an imprecise and partial picture of the performance of a health unit. In 2004 the provincial Institute for Clinical Evaluative Sciences based in Ontario, Canada introduced a public-health specific balanced scorecard framework. We present the conceptual deliberations and decisions undertaken by a health unit while adopting the framework.

Discussion

Posing, pondering and answering key questions assisted in applying the framework and developing indicators. Questions such as: Who should be involved in developing performance indicators? What level of performance should be measured? Who is the primary intended audience? Where and how do we begin? What types of indicators should populate the health status and determinants quadrant? What types of indicators should populate the resources and services quadrant? What type of indicators should populate the community engagement quadrant? What types of indicators should populate the integration and responsiveness quadrants? Should we try to link the quadrants? What comparators do we use? How do we move from a baseline report card to a continuous quality improvement management tool?

Summary

An inclusive, participatory process was chosen for defining and creating indicators to populate the four quadrants. Examples of indicators that populate the four quadrants of the scorecard are presented and key decisions are highlighted that facilitated the process.

Quality improvement of paediatric care in the Netherlands

The development of the quality improvement programme of the Paediatric Association of the Netherlands is described within the setting of the national programme of the Dutch government. The programme is based on four pillars: site visits by peers (visitatie), continuous medical and professional education, development of clinical (evidence based) guidelines and patient safety with complication registration. The site visits by peers play a central role in assessing the quality improvement activities in hospital based paediatric care. The self assessment approach and the confidential character of the visits are well received by the surveyed specialists. Recent inclusion of quality criteria in the legally required 5 yearly medical specialist recertification process has boosted the care for quality, which could serve as example for other countries.