Composite measures of healthcare quality: sensible in theory, problematic in practice

Developing quality indicators for cancer hospitals in China: a national modified Delphi process

OBJECTIVE

Although demand and supply of cancer care have been rapidly increasing in recent decades, there is a lack of systemic quality measurement for cancer hospitals in China. This study aimed to develop a set of core indicators for measuring quality of care for cancer hospitals in China.

DESIGN

The development of quality indicators was based on a literature review and a two-round modified Delphi survey. The theoretical framework and initial indicators were identified through the comprehensive literature review, and the selection of quality indicators relied on experts' consensus on the importance and feasibility of indicators by the modified Delphi process. In addition, indicator weight was identified using the analytical hierarchical process method and percentage weight method.

SETTING AND PARTICIPANTS

A panel of leading experts including oncologists, cancer care nurses, quality management experts from various regions of China were invited to participate in the two-round modified Delphi process from October to December 2020. A total of 25 experts completed the two-round modified Delphi process.

RESULTS

The experts reached consensus on a set of 47 indicators, comprising 17 structure indicators, 19 process indicators and 11 outcome indicators. Experts gave much higher weight to outcome indicators (accounting for 53.96% relative weight) than to structure (16.34%) and process (29.70%) indicators. In addition, experts also showed concerns and gave suggestions on data availability of specific outcome indicators.

CONCLUSIONS

Drawing on the comprehensive literature review and the modified Delphi process, this study developed a core set of quality indicators that can be used to evaluate quality performance of cancer hospitals. This is helpful in supporting quality cancer care in China and will provide new insights into the systemic measurement of cancer care internationally.

Biases in Electronic Health Records Data for Generating Real-World Evidence: An Overview

Electronic Health Records (EHR) are increasingly being perceived as a unique source of data for clinical research as they provide unprecedentedly large volumes of real-time data from real-world settings. In this review of the secondary uses of EHR, we identify the anticipated breadth of opportunities, pointing out the data deficiencies and potential biases that are likely to limit the search for true causal relationships. This paper provides a comprehensive overview of the types of biases that arise along the pathways that generate real-world evidence and the sources of these biases. We distinguish between two levels in the production of EHR data where biases are likely to arise: (i) at the healthcare system level, where the principal source of bias resides in access to, and provision of, medical care, and in the acquisition and documentation of medical and administrative data; and (ii) at the research level, where biases arise from the processes of extracting, analyzing, and interpreting these data. Due to the plethora of biases, mainly in the form of selection and information bias, we conclude with advising extreme caution about making causal inferences based on secondary uses of EHRs.

A composite measure for patient-reported outcomes in orthopedic care: design principles and validity checks

BACKGROUND

The complex, multidimensional nature of healthcare quality makes provider and treatment decisions based on quality difficult. Patient-reported outcome (PRO) measures can enhance patient centricity and involvement. The proliferation of PRO measures, however, requires a simplification to improve comprehensibility. Composite measures can simplify complex data without sacrificing the underlying information.

OBJECTIVE AND METHODS

We propose a five-step development approach to combine different PRO into one composite measure (PRO-CM): (i) theoretical framework and metric selection, (ii) initial data analysis, (iii) rescaling, (iv) weighting and aggregation, and (v) sensitivity and uncertainty analysis. We evaluate different rescaling, weighting, and aggregation methods by utilizing data of 3145 hip and 2605 knee replacement patients, to identify the most advantageous development approach for a PRO-CM that reflects quality variations from a patient perspective.

RESULTS

The comparison of different methods within steps (iii) and (iv) reveals the following methods as most advantageous: (iii) rescaling via z-score standardization and (iv) applying differential weights and additive aggregation. The resulting PRO-CM is most sensitive to variations in physical health. Changing weighting schemes impacts the PRO-CM most directly, while it proves more robust towards different rescaling and aggregation approaches.

CONCLUSION

Combining multiple PRO provides a holistic picture of patients' health improvement. The PRO-CM can enhance patient understanding and simplify reporting and monitoring of PRO. However, the development methodology of a PRO-CM needs to be justified and transparent to ensure that it is comprehensible and replicable. This is essential to address the well-known problems associated with composites, such as misinterpretation and lack of trust.

Methods to derive composite indicators used for quality and safety measurement and monitoring in healthcare: a scoping review protocol

INTRODUCTION

Composite indicators of quality and safety in healthcare summarise performance across multiple indicators into a single performance measure. Composite indicators can identify domains and drivers of quality, improve the ability to detect differences, aid prioritisation for quality improvement and facilitate decision making about future healthcare needs. However, the use of composite indicators can be controversial, particularly when used to rank healthcare providers. Many of the concerns around transparency, appropriateness and uncertainty may be addressed by a robust and transparent development and review process.The aim of this scoping review is to describe methodologies used at each of the stages of development of composite indicators of quality and safety in healthcare. This review will provide those tasked with developing or reviewing composite indicators with a valuable consolidated analysis of a substantial and wide-ranging literature.

METHODS AND ANALYSIS

The framework proposed by the Joanna Briggs Institute and enhancements proposed by Peters et al (2015, 2017, 2020) will be used in conducting this scoping review, and the Preferred Reporting Items for Systematic Reviews and Meta-Analysis extension for scoping reviews will guide the reporting. Grey literature and peer-reviewed documents will be in-scope. Electronic databases (PubMed, Embase, CINAHL, ABI/INFORM and SafetyLit) will be searched, and publications will be screened by two reviewers. Discussion, policy and guidance publications will be included if they discuss any aspect of the methods used in the development of a composite indicator of quality or safety in a healthcare setting. The search period ranges from 1 January 2000 to 31 December 2022. Data extraction will capture information on 11 stages of composite indicator development, augmenting a 10-stage framework developed by the European Commission Joint Research Centre.

ETHICS AND DISSEMINATION

Ethical approval is not required. Review findings will be published in a peer-reviewed journal and presented at scientific conferences.

It takes quality improvement to cross the chasm

Healthcare organisations in the USA rank significantly lower in quality of care compared with other developed nations. Research shows US performance emphasises expensive treatment over effective prevention programmes. This study demonstrates how a comprehensive quality improvement programme can improve health outcomes in a large county-based Medicaid health plan. The health plan serves a diverse community of members spanning racial and ethnic groups with varying levels of clinical risk and social determinants of health burdens. We used a regression discontinuity design to evaluate the impact of a comprehensive quality improvement programme vs using mainly pay-for-performance on Healthcare Effectiveness Data and Information Set (HEDIS) metrics over the course of 10 years. We found significant improvements in several HEDIS metrics that occurred after the quality improvement programme was implemented. These results demonstrate the importance of using a comprehensive quality improvement strategy along with pay-for-performance to improve health outcomes. It was determined that this research was exempt from institutional review board approval, as it used administrative healthcare data, and did not involve direct interventions with human subjects.

Defining a textbook outcome for the resection of colorectal liver metastases

INTRODUCTION

Textbook outcomes (TBO) are composite measures of care which may be superior in assessing quality compared to traditional methods. We aim to define TBO which are specific to surgical resection of colorectal liver metastases, and investigate their impact on survival.

METHODS

Single center analysis of all liver resections performed at our center from 2009 to 2020. A Cox model was used to identify perioperative outcomes which impacted on overall survival. These were retained with important postoperative outcomes to form a "TBO." The impact of a TBO on overall survival was investigated using Kaplan-Meier curve analysis.

RESULTS

TBO was achieved in 72.2% (197/273) of resections. Major morbidity (Clavien-Dindo ≥3) at 19.4% was the major limiting factor in not achieving a TBO. TBO was associated with improved 3-year (77% vs. 55%), 5-year (60.7% vs. 42.5%), and median (93 vs. 44 months) overall survival (log-rank test, p = 0.006). Multivariable analysis revealed age >65 years, American Society of Anaesthesiologists Grade III-IV, and resection of >2 segments as factors predictive of not achieving a TBO.

CONCLUSION

TBO is a useful composite measure in surgery for colorectal liver metastases. It can highlight areas which may be targeted for quality improvement and be useful as a tool to examine variation between centers. Achieving a TBO is associated with a significant improvement in survival.

The ultimate question? Evaluating the use of Net Promoter Score in healthcare: A systematic review

BACKGROUND

Patient experience is a complex phenomenon that presents challenges for appropriate and effective measurement. With the lack of a standardized measurement approach, efforts have been made to simplify the evaluation and reporting of patient experience by using single-item measures, such as the Net Promoter Score (NPS). Although NPS is widely used in many countries, there has been little research to validate its effectiveness and value in the healthcare setting. The aim of this study was to systematically evaluate the evidence that is available about the application of NPS in healthcare settings.

METHODS

Studies were identified using words and synonyms that relate to NPS, which was applied to five electronic databases: Medline, CINAHL, Proquest, Business Journal Premium, and Scopus. Titles and abstracts between January 2005 and September 2020 were screened for relevance, with the inclusion of quantitative and qualitative studies in the healthcare setting that evaluated the use of NPS to measure patient experience.

RESULTS

Twelve studies met the inclusion criteria. Four studies identified benefits associated with using NPS, such as ease of use, high completion rates and being well-understood by a range of patients. Three studies questioned the usefulness of the NPS recommendation question in healthcare settings, particularly when respondents are unable to select their service provider. The free-text comments section, which provides additional detail and contextual cues, was viewed positively by patients and staff in 4 of 12 studies. According to these studies, NPS can be influenced by a wide range of variables, such as age, condition/disease, intervention and cultural variation; therefore, caution should be taken when using NPS for comparisons. Four studies concluded that NPS adds minimal value to healthcare improvement.

CONCLUSION

The literature suggests that many of the proposed benefits of using NPS are not supported by research. NPS may not be sufficient as a stand-alone metric and may be better used in conjunction with a larger survey. NPS may be more suited for use in certain healthcare settings, for example, where patients have a choice of provider. Staff attitudes towards the use of NPS for patient surveying are mixed. More research is needed to validate the use of NPS as a primary metric of patient experience.

PATIENT OR PUBLIC CONTRIBUTION

Consumer representatives were provided with the research findings and their feedback was sought about the study. Consumers commented that they found the results to be useful and felt that this study highlighted important considerations when NPS data is used to evaluate patient experience.

Quality of acute myocardial infarction care in England and Wales during the COVID-19 pandemic: linked nationwide cohort study

BACKGROUND AND OBJECTIVE

The impact of the COVID-19 pandemic on the quality of care for patients with acute myocardial infarction (AMI) is uncertain. We aimed to compare quality of AMI care in England and Wales during and before the COVID-19 pandemic using the 2020 European Society of Cardiology Association for Acute Cardiovascular Care quality indicators (QIs) for AMI.

METHODS

Cohort study of linked data from the AMI and the percutaneous coronary intervention registries in England and Wales between 1 January 2017 and 27 May 2020 (representing 236 743 patients from 186 hospitals). At the patient level, the likelihood of attainment for each QI compared with pre COVID-19 was calculated using logistic regression. The date of the first national lockdown in England and Wales (23 March 2020) was chosen for time series comparisons.

RESULTS

There were 10 749 admissions with AMI after 23 March 2020. Compared with before the lockdown, patients admitted with AMI during the first wave had similar age (mean 68.0 vs 69.0 years), with no major differences in baseline characteristics (history of diabetes (25% vs 26%), renal failure (6.4% vs 6.9%), heart failure (5.8% vs 6.4%) and previous myocardial infarction (22.9% vs 23.7%)), and less frequently had high Global Registry of Acute Coronary Events risk scores (43.6% vs 48.6%). There was an improvement in attainment for 10 (62.5%) of the 16 measured QIs including a composite QI (43.8% to 45.2%, OR 1.06, 95% CI 1.02 to 1.10) during, compared with before, the lockdown.

CONCLUSION

During the first wave of the COVID-19 pandemic in England and Wales, quality of care for AMI as measured against international standards did not worsen, but improved modestly.

Evaluating the safety of mental health-related prescribing in UK primary care: a cross-sectional study using the Clinical Practice Research Datalink (CPRD)

Background

Most patients with mental illness are managed in primary care, yet there is a lack of data exploring potential prescribing safety issues in this setting for this population.

Objectives

Examine the prevalence of, between-practice variation in, and patient and practice-level risk factors for, 18 mental health-related potentially hazardous prescribing indicators and four inadequate medication monitoring indicators in UK primary care.

Method

Cross-sectional analyses of routinely collected electronic health records from 361 practices contributing to Clinical Practice Research Datalink GOLD database. The proportion of patients ‘at risk’ (based on an existing diagnosis, medication, age and/or sex) triggering each indicator and composite indicator was calculated. To examine between-practice variation, intraclass correlation coefficient (ICC) and median OR (MOR) were estimated using two-level logistic regression models. The relationship between patient and practice characteristics and risk of triggering composites including 16 of the 18 prescribing indicators and four monitoring indicators were assessed using multilevel logistic regression.

Results

9.4% of patients ‘at risk’ (151 469 of 1 611 129) triggered at least one potentially hazardous prescribing indicator; between practices this ranged from 3.2% to 24.1% (ICC 0.03, MOR 1.22). For inadequate monitoring, 90.2% of patients ‘at risk’ (38 671 of 42 879) triggered at least one indicator; between practices this ranged from 33.3% to 100% (ICC 0.26, MOR 2.86). Patients aged 35–44, females and those receiving more than 10 repeat prescriptions were at greatest risk of triggering a prescribing indicator. Patients aged less than 25, females and those with one or no repeat prescription were at greatest risk of triggering a monitoring indicator.

Conclusion

Potentially hazardous prescribing and inadequate medication monitoring commonly affect patients with mental illness in primary care, with marked between-practice variation for some indicators. These findings support health providers to identify improvement targets and inform development of improvement efforts to reduce medication-related harm.

Textbook outcome after rectal cancer surgery as a composite measure for quality of care: A population-based study

BACKGROUND

Textbook outcome is a composite measure of combined outcome indicators, which has been suggested to be of additional value over single outcome parameters in clinical auditing of surgical treatment. This study aimed to assess textbook outcome after rectal cancer surgery as short-term marker for quality of care.

MATERIALS AND METHODS

Patients who underwent elective rectal cancer surgery between 2012 and 2019 and registered in the Dutch ColoRectal Audit were included. Textbook outcome was achieved when the following criteria were met: 30-day and primary hospital admission survival, no reintervention, tumour-free margins, no postoperative complications, a hospital stay of less than 14 days and no readmission. Hospital variation was evaluated in case-mix corrected funnel-plots. A multilevel logistic regression analysis was performed to identify associated factors with textbook outcome.

RESULTS

The study population consisted of 20,521 patients who underwent primary rectal cancer surgery, of whom 56.3% achieved textbook outcome. Postoperative complications were the main contributor to not achieving textbook outcome. Case-mix corrected funnel plots demonstrated that underperforming hospitals in 2012-2015 were no underperformers in 2016-2019 anymore. Female sex, laparoscopic surgery, and rectal resection without defunctioning stoma creation were positively associated with textbook outcome.

CONCLUSION

Textbook outcome after rectal cancer resection is mainly driven by postoperative complications. Although textbook outcome showed some discriminating value for identifying underperforming hospitals, it does not fit the plan-do-check-act cycle of clinical auditing. In our opinion, textbook outcome has little added value to the current outcome indicators for rectal cancer surgery.

Development of a percutaneous coronary intervention patient level composite measure for a clinical quality registry

BACKGROUND

Composite measures combine data to provide a comprehensive view of patient outcomes. Despite composite measures being a valuable tool to assess post-intervention outcomes, the patient perspective is often missing. The purpose of this study was to develop a composite measure for an established cardiac outcome registry, by combining clinical outcomes following percutaneous coronary interventions (PCI) with a patient-reported outcome measure (PROM) developed specifically for this population (MC-PROM).

METHODS

Two studies were undertaken. Study 1: Patients who had undergone a PCI at one of the three participating registry hospital sites completed the 5-item MC-PROM. Clinical outcome data for the patients (e.g. death, myocardial infarction, repeat vascularisation, new bleeding event) were collected 30 days post-intervention as part of routine data collection for the cardiac registry. Exploratory factor analysis of clinical outcomes and MC-PROM data was conducted to determine the minimum number of constructs to be included in a composite measure. Study 2: Clinical experts participated in a Delphi technique, consisting of three rounds of online surveys, to determine the clinical outcomes to be included and the weighting of the clinical outcomes and MC-PROM score for the composite measure.

RESULTS

Study 1: Routine clinical outcomes and the MC-PROM data were collected from 266 patients 30 days post PCI. The MC-PROM score was not significantly correlated with any clinical outcomes. Study 2: There was a relatively consistent approach to the weighting of the clinical outcomes and MC-PROM items by the expert panel (n = 18) across the three surveys with the exception of the clinical outcome of 'deceased at 30 days'. The final composite measure included five clinical outcomes within 30 days weighted at 90% (new heart failure, new myocardial infarction, new stent thrombosis, major bleeding event, new stroke, unplanned cardiac rehospitalisation) and the MC-PROM score (comprising 10% of the total weighting).

CONCLUSIONS

A single patient level composite score, which incorporates weighted clinical outcomes and a PROM was developed. This composite score provides a more comprehensive reported measure of individual patient wellbeing at 30 days post their PCI-procedure, and may assist clinicians to further assess and address patient level factors that potentially impact on clinical recovery.

Adapting systems thinking and safety reporting in high-consequence industries to healthcare

This chapter uses a systems approach to represent the healthcare sector and positions safety reporting programs as feedback mechanisms to reactively, proactively, and predictively improve the overall reliability and safety of care practices. Drawing from the aviation sector, benefits and limitations of different safety reporting systems are explored and challenges to adapting such systems into healthcare are presented. Examples of successful adaptation and implementation in healthcare demonstrate that while adaptation is possible and could yield impressive outcomes, such programs remain susceptible to the natural tendency of the healthcare system to remain siloed and internally competitive, rather than holistic and team-oriented. Thus, one could conclude that in order for safety reporting programs to be self-sustaining, the systemic disincentives need to be examined carefully and intentionally removed, replacing them with meaningful incentives to collaboration and maximization of patient safety outcomes.

Inclusion of a care bundle for fever, hyperglycaemia and swallow management in a National Audit for acute stroke: evidence of upscale and spread

BACKGROUND

In the Quality in Acute Stroke Care (QASC) trial undertaken in stroke units (SUs) located in New South Wales (NSW), Australia (2005-2010), facilitated implementation of a nurse-led care bundle to manage fever, hyperglycaemia and swallowing (FeSS protocols) reduced death and disability for patients with stroke. We aimed to determine subsequent adherence to the bundled FeSS processes (reflective of the protocols) between 2013 and 2017 in Australian hospitals, and examine whether changes in adherence to these processes varied based on previous participation in the QASC trial or subsequent statewide scale-up (QASCIP-Quality in Acute Stroke Care Implementation Project) and presence of an SU.

METHODS

Cross-sectional, observational study using self-reported organisational survey and retrospective clinical audit data from the National Acute Services Stroke Audit (2013, 2015, 2017). Mixed-effects logistic regression was performed with dependent variables: (1) composite outcome measure reflecting compliance with the FeSS protocols and (2) individual FeSS processes, including the year of audit as an independent variable, adjusted for correlation of outcomes within hospital. Separate models including interaction terms between the year of audit and previous participation in QASC/QASCIP and year of audit and SU were also generated.

RESULTS

Hospital participation included the following: 2013-124 hospitals, 3741 cases; 2015-112 hospitals, 4087 cases; and 2017-117 hospitals, 4192 cases. An 80% increase in the odds of receiving the composite outcome in 2017 compared to 2013 was found (2013, 30%; 2017, 41%; OR 1.8; 95% CI 1.6, 2.0; p < 0.001). The odds of FeSS adherence from 2013 to 2017 was greater for hospitals that had participated in QASC/QASCIP relative to those that had not (participated OR 2.1; 95% CI 1.7, 2.7; not participated OR 1.6; 95% CI 1.4, 1.8; p = 0.03). Similar uptake in adherence was evident in hospitals with and without an SU between 2013 and 2017.

CONCLUSION

The use of the FeSS protocols within Australia increased from 2013 to 2017 with the inclusion of these care processes in the National Audit. Greater uptake in hospitals previously involved in QASC/QASCIP was evident. Our implementation methods may be useful for other national initiatives for improving access to evidence-based practice.

The implications of high bed occupancy rates on readmission rates in England: A longitudinal study

Hospital bed occupancy rates in the English National Health Service have risen to levels considered clinically unsafe. This study assesses the association of increased bed occupancy with changes in the percentage of overnight patients discharged from hospital on a given day, and their subsequent 30-day readmission rate. Longitudinal panel data methods are used to analyse secondary care records (n = 4,193,590) for 136 non-specialist Trusts between April 2014 and February 2016. The average bed occupancy rate across the study period was 90.4%. A 1% increase in bed occupancy was associated with a 0.49% rise in the discharge rate, and a 0.011% increase in the 30-day readmission rate for discharged patients. These associations became more pronounced once bed occupancy exceeded 95%. When bed occupancy rates were high, hospitals discharged a greater proportion of their patients. Those were mostly younger and less clinically complex, suggesting that hospitals are successfully prioritising early discharge amongst least vulnerable patients. However, while increased bed occupancy was not associated with a substantial increase in overall 30-day readmission rates, the relationship was more pronounced in older and sicker patients, indicating possible links with short-fallings in discharge processes.