Evaluation of an international benchmarking initiative in nine eye hospitals

Impact of practicing internal benchmarking on continuous improvement of cataract surgery outcomes: a retrospective observational study at Aravind Eye Hospitals, India

OBJECTIVE

We aim to assess the effectiveness of a cataract surgery outcome monitoring tool used for continuous quality improvement. The objectives are to study: (1) the quality parameters, (2) the monitoring process followed and (3) the impact on outcomes.

DESIGN AND PROCEDURES

In this retrospective observational study we evaluated a quality improvement (QI) method which has been practiced at the focal institution since 2012: internal benchmarking of cataract surgery outcomes (CATQA). We evaluated quality parameters, procedures followed and clinical outcomes. We created tables and line charts to examine trends in key outcomes.

SETTING

Aravind Eye Care System, India.

PARTICIPANTS

Phacoemulsification surgeries performed on 718 120 eyes at 10 centres (five tertiary and five secondary eye centres) from 2012 to 2020 were included.

INTERVENTIONS

An internal benchmarking of surgery outcome parameters, to assess variations among the hospitals and compare with the best hospital.

OUTCOME MEASURES

Intraoperative complications, unaided visual acuity (VA) at postoperative follow-up visit and residual postoperative refractive error (within ±0.5D).

RESULTS

Over the study period the intraoperative complication rate decreased from 1.2% to 0.6%, surgeries with uncorrected VA of 6/12 or better increased from 80.8% to 89.8%, and surgeries with postoperative refractive error within ±0.5D increased from 76.3% to 87.3%. Variability in outcome measures across hospitals declined. Additionally, benchmarking was associated with improvements in facilities, protocols and processes.

CONCLUSION

Internal benchmarking was found to be an effective QI method that enabled the practice of evidence-based management and allowed for harnessing the available information. Continuous improvement in clinical outcomes requires systematic and regular review of results, identifying gaps between hospitals, comparisons with the best hospital and implementing lessons learnt from peers.

Development of a benchmark tool for cancer centers; results from a pilot exercise

BACKGROUND

Differences in cancer survival exist between countries in Europe. Benchmarking of good practices can assist cancer centers to improve their services aiming for reduced inequalities. The aim of the BENCH-CAN project was to develop a cancer care benchmark tool, identify performance differences and yield good practice examples, contributing to improving the quality of interdisciplinary care. This paper describes the development of this benchmark tool and its validation in cancer centers throughout Europe.

METHODS

A benchmark tool was developed and executed according to a 13 step benchmarking process. Indicator selection was based on literature, existing accreditation systems, and expert opinions. A final format was tested in eight cancer centers. Center visits by a team of minimally 3 persons, including a patient representative, were performed to verify information, grasp context and check on additional questions (through semi-structured interviews). Based on the visits, the benchmark methodology identified opportunities for improvement.

RESULTS

The final tool existed of 61 qualitative and 141 quantitative indicators, which were structured in an evaluative framework. Data from all eight participating centers showed inter-organization variability on many indicators, such as bed utilization and provision of survivorship care. Subsequently, improvement suggestions for centers were made; 85% of which were agreed upon.

CONCLUSION

A benchmarking tool for cancer centers was successfully developed and tested and is available in an open format. The tool allows comparison of inter-organizational performance. Improvement opportunities were successfully identified for every center involved and the tool was positively evaluated.

Clinical and Health System Determinants of Venous Thromboembolism Event Rates After Hip Arthroplasty: An International Comparison

BACKGROUND

Routinely collected hospital data provide increasing opportunities to assess the performance of health care systems. Several factors may, however, influence performance measures and their interpretation between countries.

OBJECTIVE

We compared the occurrence of in-hospital venous thromboembolism (VTE) in patients undergoing hip replacement across 5 countries and explored factors that could explain differences across these countries.

METHODS

We performed cross-sectional studies independently in 5 countries: Canada; France; New Zealand; the state of California; and Switzerland. We first calculated the proportion of hospital inpatients with at least one deep vein thrombosis (DVT) or pulmonary embolism by using numerator codes from the corresponding Patient Safety Indicator. We then compared estimates from each country against a reference value (benchmark) that displayed the baseline risk of VTE in such patients. Finally, we explored length of stay, number of secondary diagnoses coded, and systematic use of ultrasound to detect DVT as potential factors that could explain between-country differences.

RESULTS

The rates of VTE were 0.16% in Canada, 1.41% in France, 0.84% in New Zealand, 0.66% in California, and 0.37% in Switzerland, while the benchmark was 0.58% (95% confidence interval, 0.35-0.81). Factors that could partially explain differences in VTE rates between countries were hospital length of stay, number of secondary diagnoses coded, and proportion of patients who received lower limb ultrasound to screen for DVT systematically before hospital discharge. An exploration of the French data showed that the systematic use of ultrasound may be associated with over detection of DVT but not pulmonary embolism.

CONCLUSIONS

In-hospital VTE rates after arthroplasty vary widely across countries, and a combination of clinical, data-related, and health system factors explain some of the variations in VTE rates across countries.

Mapping standard ophthalmic outcome sets to metrics currently reported in eight eye hospitals

BACKGROUND

To determine alignment of proposed international standard outcomes sets for ophthalmic conditions to metrics currently reported by eye hospitals.

METHODS

Mixed methods comparative benchmark study, including eight eye hospitals in Australia, India, Singapore, Sweden, U.K., and U.S. All are major international tertiary care and training centers in ophthalmology. Main outcome measure is consistency of ophthalmic outcomes measures reported.

RESULTS

International agreed standard outcomes (ICHOM) sets are available for cataract surgery (10 metrics) and macular degeneration (7 metrics). The eight hospitals reported 22 different metrics for cataract surgery and 2 for macular degeneration, which showed only limited overlap with the proposed ICHOM metrics. None of the hospitals reported patient reported visual functioning or vision-related quality of life outcomes measures (PROMs). Three hospitals (38%) reported rates for uncomplicated cataract surgeries only. There was marked variation in how and at what point postoperatively visual outcomes following cataract, cornea, glaucoma, strabismus and oculoplastics procedures were reported. Seven (87.5%) measured post-operative infections and four (50%) measured 30 day unplanned reoperation rates.

CONCLUSIONS

Outcomes reporting for ophthalmic conditions currently widely varies across hospitals internationally and does not include patient-reported outcomes. Reaching consensus on measures and consistency in data collection will allow meaningful comparisons and provide an evidence base enabling improved sharing of "best practices" to improve eye care globally. Implementation of international standards is still a major challenge and practice-based knowledge on measures should be one of the inputs of the international standardization process.

Benchmarking specialty hospitals, a scoping review on theory and practice

BACKGROUND

Although benchmarking may improve hospital processes, research on this subject is limited. The aim of this study was to provide an overview of publications on benchmarking in specialty hospitals and a description of study characteristics.

METHODS

We searched PubMed and EMBASE for articles published in English in the last 10 years. Eligible articles described a project stating benchmarking as its objective and involving a specialty hospital or specific patient category; or those dealing with the methodology or evaluation of benchmarking.

RESULTS

Of 1,817 articles identified in total, 24 were included in the study. Articles were categorized into: pathway benchmarking, institutional benchmarking, articles on benchmark methodology or -evaluation and benchmarking using a patient registry. There was a large degree of variability:(1) study designs were mostly descriptive and retrospective; (2) not all studies generated and showed data in sufficient detail; and (3) there was variety in whether a benchmarking model was just described or if quality improvement as a consequence of the benchmark was reported upon. Most of the studies that described a benchmark model described the use of benchmarking partners from the same industry category, sometimes from all over the world.

CONCLUSIONS

Benchmarking seems to be more developed in eye hospitals, emergency departments and oncology specialty hospitals. Some studies showed promising improvement effects. However, the majority of the articles lacked a structured design, and did not report on benchmark outcomes. In order to evaluate the effectiveness of benchmarking to improve quality in specialty hospitals, robust and structured designs are needed including a follow up to check whether the benchmark study has led to improvements.

Benchmarking operating room departments in the Netherlands

Purpose

– Benchmarking is increasingly considered a useful management instrument to improve performance in healthcare. The purpose of this paper is to assess if a nationwide long-term benchmarking collaborative between operating room (OR) departments of university medical centres in the Netherlands leads to benefits in OR management and to evaluate if the initiative meets the requirements of the 4P-model.

Design/methodology/approach

– The evaluation was based on the 4P-model (purposes, performance indicators, participating organisations, performance management system), developed in former studies. A mixed-methods design was applied, consisting of document study, observations, interviews as well as analysing OR performance data using SPSS statistics.

Findings

– Collaborative benchmarking has benefits different from mainly performance improvement and identification of performance gaps. It is interesting that, since 2004, the OR benchmarking initiative still endures after already existing for ten years. A key benefit was pointed out by all respondents as “the purpose of networking”, on top of the purposes recognised in the 4P-model. The networking events were found to make it easier for participants to contact and also visit one another. Apparently, such informal contacts were helpful in spreading knowledge, sharing policy documents and initiating improvement. This benchmark largely met all key conditions of the 4P-model.

Research limitations/implications

– The current study has the limitations accompanied with any qualitative research and particularly related to interviewing. Qualitative research findings must be viewed within the context of the conducted case study. The experiences in this university hospital context in the Netherlands might not be transferable to other (general) hospital settings or other countries. The number of conducted interviews is restricted; nevertheless, all other data sources are extensive.

Originality/value

– A collaborative approach in benchmarking can be effective because participants use its knowledge-sharing infrastructure which enables operational, tactical and strategic learning. Organisational learning is to the advantage of overall OR management. Benchmarking seems a useful instrument in enabling hospitals to learn from each other, to initiate performance improvements and catalyse knowledge-sharing.

Benchmarking the Wilmer general eye services clinics: baseline metrics for surgical and outpatient clinic volume in an educational environment

BACKGROUND

The Wilmer General Eye Services (GES) at the Johns Hopkins Hospital is the clinic where residents provide supervised comprehensive medical and surgical care to ophthalmology patients. The clinic schedule and supervision structure allows for a progressive increase in trainee responsibility, with graduated autonomy and longitudinal continuity of care over the three years of ophthalmology residency training. This study sought to determine the number of cases the GES contributes to the resident surgical experiences. In addition, it was intended to create benchmarks for patient volumes, cataract surgery yield and room utilization as part of an educational initiative to introduce residents to metrics important for practice management.

METHODS

The electronic surgical posting system database was explored to determine the numbers of cases scheduled for patients seen by residents in the GES. In addition, aggregated residents' self-reported Accreditation Council for Graduate Medical Education (ACGME) surgical logs were collected for comparison. Finally transactional databases were queried to determine clinic volumes of new and established patients. The proportion of resident surgeries (1(st) surgeon and assistant) provided by GES patients, cataract surgery yield and new patient rates were calculated. Data was collected from July 1(st), 2014 until March 31(st), 2015 for all 16 residents (6 third year, 5 second year and 5 first year).

RESULTS

The percentage of cataract, oculoplastics, cornea and glaucoma surgeries in which a resident was 1(st) surgeon and the patient came from the GES was 91.3, 76.1, 65.6, and 93.9 respectively. The new patient rate was 28.1% and room utilization was 50.4%. Cataract surgery yield was 29.2 DISCUSSION: The GES provides a significant proportion of primary surgeon opportunities for the residents, and in some instances, the majority of cases. Compared to benchmarks available for private practices, the new patient rate is high while the cataract surgery yield is low. The room utilization is lower than the 85% preferred by the hospital system. These are the first benchmarks of this type for an academic resident ophthalmology practice in the United States.

CONCLUSIONS

Our study suggests that resident-hosted clinics can provide the majority of surgical opportunities for ophthalmology trainees, particulary with regard to cataract cases. However, because our study is the first academic resident practice to publish metrics of the type used in private practices, it is impossible to determine where our clinic stands compared to other training programs. Therefore, the authors strongly encourage ophthalmology training programs to explore and publish practice metrics. This will permit the creation of a benchmarking program that could be used to quantify efforts at enhancing ophthalmic resident education.

Diabetic retinopathy care--an international quality comparison

PURPOSE

The purpose of this paper is to study the quality indicator appropriateness and use it for international quality comparison on diabetic retinopathy (DR) patient care process in one American and one Dutch eye hospital.

DESIGN/METHODOLOGY/APPROACH

A 17-item DR quality indicator set was composed based on a literature review and systematically applied in two hospitals. Qualitative analysis entailed document study and 12 semi-structured face-to-face interviews with ophthalmologists, managers, and board members of the two hospitals.

FINDINGS

While the medical-clinical approach to DR treatment in both hospitals was similar, differences were found in quality of care perception and operationalization. Neither hospital systematically used outcome indicators for DR care. On the process level, the authors found larger differences. Similarities and differences were found in the structure of both hospitals. The hospitals' particular contexts influenced the interpretation and use of quality indicators.

PRACTICAL IMPLICATIONS

Although quality indicators and quality comparison between hospitals are increasingly used in international settings, important local differences influence their application. Context should be taken into account. Since that context is locally bound and directly linked to hospital setting, caution should be used interpreting the results of quality comparison studies.

ORIGINALITY/VALUE

International quality comparison is increasingly suggested as a useful way to improve healthcare. Little is known, however, about the appropriateness and use of quality indicators in local hospital care practices.

The PATH project in eight European countries: an evaluation

PURPOSE

This paper's aim is to evaluate the perceived impact and the enabling factors and barriers experienced by hospital staff participating in an international hospital performance measurement project focused on internal quality improvement.

DESIGN/METHODOLOGY/APPROACH

Semi-structured interviews involving international hospital performance measurement project coordinators, including 140 hospitals from eight European countries (Belgium, Estonia, France, Germany, Hungary, Poland, Slovakia and Slovenia). Inductively analyzing the interview transcripts was carried out using the grounded theory approach.

FINDINGS

Even when public reporting is absent, the project was perceived as having stimulated performance measurement and quality improvement initiatives in participating hospitals. Attention should be paid to leadership/ownership, context, content (project intrinsic features) and processes supporting elements.

RESEARCH LIMITATIONS/IMPLICATIONS

Generalizing the findings is limited by the study's small sample size. Possible implications for the WHO European Regional Office and for participating hospitals would be to assess hospital preparedness to participate in the PATH project, depending on context, process and structural elements; and enhance performance and practice benchmarking through suggested approaches.

ORIGINALITY/VALUE

This research gathered rich and unique material related to an international performance measurement project. It derived actionable findings.

Hospital process orientation from an operations management perspective: development of a measurement tool and practical testing in three ophthalmic practices

BACKGROUND

Although research interest in hospital process orientation (HPO) is growing, the development of a measurement tool to assess process orientation (PO) has not been very successful yet. To view a hospital as a series of processes organized around patients with a similar demand seems to be an attractive proposition, but it is hard to operationalize this idea in a measurement tool that can actually measure the level of PO. This research contributes to HPO from an operations management (OM) perspective by addressing the alignment, integration and coordination of activities within patient care processes. The objective of this study was to develop and practically test a new measurement tool for assessing the degree of PO within hospitals using existing tools.

METHODS

Through a literature search we identified a number of constructs to measure PO in hospital settings. These constructs were further operationalized, using an OM perspective. Based on five dimensions of an existing questionnaire a new HPO-measurement tool was developed to measure the degree of PO within hospitals on the basis of respondents' perception. The HPO-measurement tool was pre-tested in a non-participating hospital and discussed with experts in a focus group. The multicentre exploratory case study was conducted in the ophthalmic practices of three different types of Dutch hospitals. In total 26 employees from three disciplines participated. After filling in the questionnaire an interview was held with each participant to check the validity and the reliability of the measurement tool.

RESULTS

The application of the HPO-measurement tool, analysis of the scores and interviews with the participants resulted in the possibility to identify differences of PO performance and the areas of improvement--from a PO point of view--within each hospital. The result of refinement of the items of the measurement tool after practical testing is a set of 41 items to assess the degree of PO from an OM perspective within hospitals.

CONCLUSIONS

The development and practically testing of a new HPO-measurement tool improves the understanding and application of PO in hospitals and the reliability of the measurement tool. The study shows that PO is a complex concept and appears still hard to objectify.

A theoretical framework for holistic hospital management in the Japanese healthcare context

OBJECTIVE

This paper develops a conceptual framework for performance measurement as a pilot study on holistic hospital management in the Japanese healthcare context.

METHODS

We primarily used two data sources as well as expert statements obtained through interviews: a systematic review of literature and a questionnaire survey to healthcare experts. The systematic survey searched PubMed and PubMed Central, and 24 relevant papers were elicited. The expert questionnaire asked respondents to rate the degree of "usefulness" for each of 66 indicators on a three-point scale.

RESULTS

Applying the theoretical framework, a minimum set of performance indicators was selected for holistic hospital management, which well fit the healthcare context in Japan. This indicator set comprised 35 individual indicators and several factors measured through questionnaire surveys. The indicators were confirmed by expert judgments from viewpoints of face, content and construct validities as well as their usefulness.

CONCLUSION

A theoretical framework of performance measurement was established from primary healthcare stakeholders' perspectives. Performance indicators were largely divided into healthcare outcomes and performance shaping factors. Indicators in the former category may be applied for the detection of operational problems, while their latent causes can be effectively addressed by the latter category in terms of process, structure and culture/climate within the organization.

Hospital benchmarking: are U.S. eye hospitals ready?

BACKGROUND

Benchmarking is increasingly considered a useful management instrument to improve quality in health care, but little is known about its applicability in hospital settings.

PURPOSE

The aims of this study were to assess the applicability of a benchmarking project in U.S. eye hospitals and compare the results with an international initiative.

METHODOLOGY

We evaluated multiple cases by applying an evaluation frame abstracted from the literature to five U.S. eye hospitals that used a set of 10 indicators for efficiency benchmarking. Qualitative analysis entailed 46 semistructured face-to-face interviews with stakeholders, document analyses, and questionnaires.

FINDINGS

The case studies only partially met the conditions of the evaluation frame. Although learning and quality improvement were stated as overall purposes, the benchmarking initiative was at first focused on efficiency only. No ophthalmic outcomes were included, and clinicians were skeptical about their reporting relevance and disclosure. However, in contrast with earlier findings in international eye hospitals, all U.S. hospitals worked with internal indicators that were integrated in their performance management systems and supported benchmarking. Benchmarking can support performance management in individual hospitals. Having a certain number of comparable institutes provide similar services in a noncompetitive milieu seems to lay fertile ground for benchmarking. International benchmarking is useful only when these conditions are not met nationally.

PRACTICE IMPLICATIONS

Although the literature focuses on static conditions for effective benchmarking, our case studies show that it is a highly iterative and learning process. The journey of benchmarking seems to be more important than the destination. Improving patient value (health outcomes per unit of cost) requires, however, an integrative perspective where clinicians and administrators closely cooperate on both quality and efficiency issues. If these worlds do not share such a relationship, the added "public" value of benchmarking in health care is questionable.

Selecting indicators for international benchmarking of radiotherapy centres

Introduction: Benchmarking can be used to improve hospital performance. It is however not easy to develop a concise and meaningful set of indicators on aspects related to operations management. We developed an indicator set for managers and evaluated its use in an international benchmark of radiotherapy centres. The indicator set assessed the efficiency, patient-centeredness and timeliness of the services delivered.

Methods: We identified possible indicators from literature and professionals. Stakeholders’ feedback helped to produce a shortlist of indicators. For this indicator set, data were obtained in a pilot that included four European radiotherapy centres. With these data, the indicators were evaluated on definition clarity, data availability, reliability and discriminative value.

Results: Literature produced a gross list of 81 indicators. Based on stakeholder feedback, 33 indicators were selected and evaluated in the benchmark. Six negatively evaluated indicators were adapted, together with eight positively evaluated indicators 14 indicators seemed feasible. Examples of indicators concerned utilisation, waiting times, patient satisfaction and risk analysis.

Conclusions: This study provides a pragmatic indicator development process for international benchmarks on operations management. The presented indicators showed to be feasible for use in international benchmarking of radiotherapy centres. The pilot identified attainable performance levels and provided leads for improvements.

Overview of the quality assurance movement in health care

This chapter aims to describe the origin and current status of quality assurance (QA) in health care and to provide a background of similar developments in other industries, which have provided a major impetus for QA initiatives in health care. The interest in quality and safety in the health care sector has rapidly risen over the past decade. Without important lessons learnt from other industries, the interest and obtained improvements would have been far less fast. Knowledge on basic principles and challenges faced by other industries like the airline, car, and nuclear energy industry, that drove quality improvement projects, is of major relevance to understand the evolutions taking place in health care. To fully appreciate the QA movement, and design or implement quality improvement projects, its basic principles need to be understood. This chapter aims to give insights in basic principles underlying QA, and to discuss historical lessons that have been learnt from other industries. Furthermore, it discusses how to implement and assure a sustainable QA program.

International benchmarking of specialty hospitals. A series of case studies on comprehensive cancer centres

Background

Benchmarking is one of the methods used in business that is applied to hospitals to improve the management of their operations. International comparison between hospitals can explain performance differences. As there is a trend towards specialization of hospitals, this study examines the benchmarking process and the success factors of benchmarking in international specialized cancer centres.

Methods

Three independent international benchmarking studies on operations management in cancer centres were conducted. The first study included three comprehensive cancer centres (CCC), three chemotherapy day units (CDU) were involved in the second study and four radiotherapy departments were included in the final study. Per multiple case study a research protocol was used to structure the benchmarking process. After reviewing the multiple case studies, the resulting description was used to study the research objectives.

Results

We adapted and evaluated existing benchmarking processes through formalizing stakeholder involvement and verifying the comparability of the partners. We also devised a framework to structure the indicators to produce a coherent indicator set and better improvement suggestions. Evaluating the feasibility of benchmarking as a tool to improve hospital processes led to mixed results. Case study 1 resulted in general recommendations for the organizations involved. In case study 2, the combination of benchmarking and lean management led in one CDU to a 24% increase in bed utilization and a 12% increase in productivity. Three radiotherapy departments of case study 3, were considering implementing the recommendations.

Additionally, success factors, such as a well-defined and small project scope, partner selection based on clear criteria, stakeholder involvement, simple and well-structured indicators, analysis of both the process and its results and, adapt the identified better working methods to the own setting, were found.

Conclusions

The improved benchmarking process and the success factors can produce relevant input to improve the operations management of specialty hospitals.