Effect of a population-level performance dashboard intervention on maternal-newborn outcomes: an interrupted time series study

Quality Dashboards in Hospital Settings: A Systematic Review With Implications for Nurses

BACKGROUND

Dashboards visually display quality and safety data to aid nurses in making informed decisions.

PURPOSE

This systematic review evaluated quality improvement (QI) dashboard characteristics associated with interventions to improve patient outcomes and positive end-user evaluation.

METHODS

Literature was searched from 2012 to 2022 in PubMed, CINAHL, Scopus, MEDLINE, and Google Scholar.

RESULTS

Sixteen articles were included. Varied dashboard characteristics were noted, with mixed patient outcomes and end-user responses. Graphs and tabular presentations were associated with improved patient outcomes, whereas graphs were associated with end-user satisfaction. Benchmarks were noted with improved patient outcomes but not end-user satisfaction. Interactive dashboards were important for end users and improved patient outcomes.

CONCLUSION

Nurses can find dashboards helpful in guiding QI projects. Dashboards may include graphs and/or tables, benchmarks, and interactivity but should be useful, usable, and aligned to unit needs. Future research should focus on the use of quality dashboards in nursing practice.

Building a NICU quality & safety infrastructure

The American Academy of Pediatrics (AAP) Standards for Levels of Neonatal Care, published in 2023, highlights key components of a Neonatal Patient Safety and Quality Improvement Program (NPSQIP). A comprehensive Neonatal Intensive Care Unit (NICU) quality and safety infrastructure (QSI) is based on four foundational domains: quality improvement, quality assurance, safety culture, and clinical guidelines. This paper serves as an operational guide for NICU clinical leaders and quality champions to navigate these domains and develop their local QSI to include the AAP NPSQIP standards.

Effect of a Performance Feedback Dashboard on Hospitalist Laboratory Test Utilization

BACKGROUND

Healthcare spending continues to be an area of improvement across all forms of medicine. Overtreatment or low-value care, including overutilization of laboratory testing, has an estimated annual cost of waste of $75.7-$101.2 billion annually. Providing performance feedback to hospitalists has been shown to be an effective way to encourage the practice of quality-improvement-focused medicine. There remains limited data regarding the implementation of performance feedback and direct results on hospital laboratory testing spending in the short term.

OBJECTIVE

The objective of this project was to identify whether performance-based feedback on laboratory utilization between both hospitalists and resident teams results in more conservative utilization of laboratory testing.

DESIGN, SETTING, PARTICIPANTS

This quality improvement project was conducted at a tertiary academic medical center, including both direct-care and house-staff teams.

INTERVENTION OR EXPOSURE

A weekly performance feedback report was generated and distributed to providers detailing laboratory test utilization by all hospitalists in a ranked system, normalized by the census of patients, for 3 months.

MAIN OUTCOMES AND MEASURES

The outcome measure was cumulative laboratory utilization during the intervention period compared to baseline utilization during the corresponding 3 months in the year prior and the weekly trend in laboratory utilization over 52 weeks. The aggregate laboratory utilization rate during intervention and control time periods was defined as the total number of laboratory tests ordered divided by the total number of patient encounters. Additionally, the cost difference was averaged per quarter and reported. The week-by-week trend in laboratory utilization was evaluated using a statistical process control (SPC) chart.

RESULTS

We found that following intervention during January-March 2020, the cumulative complete blood count utilization rate decreased from 5.54 to 4.83 per patient encounter and the basic metabolic panels/CMP utilization rate decreased from 6.65 to 6.11 per patient encounter compared with January-March 2019. This equated to cost savings of ~$42,700 in total for the quarter. Nonrandom variation was seen on SPC charts in weekly laboratory utilization rates for common laboratory tests during the intervention period.

CONCLUSIONS

We found that our intervention did result in a decrease in laboratory test utilization rates across direct-care and house-staff teams. This study lays promising groundwork for one tool that can be used to eliminate a source of hospital waste and improve the quality and efficiency of patient care.

Planning and implementing practice changes in Ontario maternal-newborn hospital units: a secondary qualitative analysis

BACKGROUND

Moving evidence into practice is complex, and pregnant and birthing people and their infants do not always receive care that aligns with the best available evidence. Implementation science can inform how to effectively move evidence into practice. While there are a growing number of examples of implementation science being studied in maternal-newborn care settings, it remains unknown how real-world teams of healthcare providers and leaders approach the overall implementation process when making practice changes. The purpose of this study was to describe maternal-newborn hospital teams' approaches to implementing practice changes. We aimed to identify what implementation steps teams take (or not) and identify strengths and potential areas for improvement based on best practices in implementation science.

METHODS

We conducted a supplementary qualitative secondary analysis of 22 interviews completed in 2014-2015 with maternal-newborn nursing leaders in Ontario, Canada. We used directed content analysis to code the data to seven steps in an implementation framework (Implementation Roadmap): identify the problem and potential best practice; assemble local evidence; select and customize best practice; discover barriers and drivers; tailor implementation strategies; field-test, plan evaluation, prepare to launch; launch, evaluate, and sustain. Frequency counts are presented for each step.

RESULTS

Participants reported completing a median of 4.5 of 7 Implementation Roadmap steps (range = 3-7), with the most common being identifying a practice problem. Other steps were described less frequently (e.g., selecting and adapting evidence, field-testing, outcome evaluation) or discussed frequently but not optimally (e.g., barriers assessment). Participants provided examples of how they engaged point-of-care staff throughout the implementation process, but provided fewer examples of engaging pregnant and birthing people and their families. Some participants stated they used a formal framework or process to guide their implementation process, with the most common being quality improvement approaches and tools.

CONCLUSIONS

We identified variability across the 22 hospitals in the implementation steps taken. While we observed many strengths, we also identified areas where further support may be needed. Future work is needed to create opportunities and resources to support maternal-newborn healthcare providers and leaders to apply principles and tools from implementation science to their practice change initiatives.

Comparing Methods of Identifying Outlying Nurses in Audits of Low-Risk Cesarean Delivery Rates

BACKGROUND

The low-risk cesarean delivery (CD) rate is an established performance indicator for providers in maternity care for quality improvement purposes.

PURPOSE

Our objectives were to assess nurse performance using adjusted nurse-level CD rates and to compare methods of identifying nurse outliers.

METHODS

We conducted a retrospective, cohort study of 6970 births attended by 181 registered nurses in one hospital's maternity unit. Adjusted and unadjusted nurse-level CD rates were compared and agreement between 3 definitions (statistical, top decile, over a benchmark) of outliers calculated.

RESULTS

Adjusted nurse-level CD rates varied from 5.5% to 53.2%, and the unadjusted rates varied from 5% to 50%. Risk adjustment had little impact on the ranking of nurses, and outliers were consistently identified by 3 definitions.

CONCLUSIONS

Trade-offs between statistical certainty and feasibility need to be considered when classifying nurse outliers. Findings can help target interventions to improve nurse performance.

OUP accepted manuscript

Objectives

(1) Systematically review the literature on computerized audit and feedback (e-A&F) systems in healthcare. (2) Compare features of current systems against e-A&F best practices. (3) Generate hypotheses on how e-A&F systems may impact patient care and outcomes.

Methods

We searched MEDLINE (Ovid), EMBASE (Ovid), and CINAHL (Ebsco) databases to December 31, 2020. Two reviewers independently performed selection, extraction, and quality appraisal (Mixed Methods Appraisal Tool). System features were compared with 18 best practices derived from Clinical Performance Feedback Intervention Theory. We then used realist concepts to generate hypotheses on mechanisms of e-A&F impact. Results are reported in accordance with the PRISMA statement.

Results

Our search yielded 4301 unique articles. We included 88 studies evaluating 65 e-A&F systems, spanning a diverse range of clinical areas, including medical, surgical, general practice, etc. Systems adopted a median of 8 best practices (interquartile range 6–10), with 32 systems providing near real-time feedback data and 20 systems incorporating action planning. High-confidence hypotheses suggested that favorable e-A&F systems prompted specific actions, particularly enabled by timely and role-specific feedback (including patient lists and individual performance data) and embedded action plans, in order to improve system usage, care quality, and patient outcomes.

Conclusions

e-A&F systems continue to be developed for many clinical applications. Yet, several systems still lack basic features recommended by best practice, such as timely feedback and action planning. Systems should focus on actionability, by providing real-time data for feedback that is specific to user roles, with embedded action plans.

Protocol Registration

PROSPERO CRD42016048695.

An evaluation of an electronic audit and feedback system for patient safety in a tertiary hospital setting: A study protocol

An electronic audit and feedback (e-A&F) system was developed to support healthcare providers' awareness of their own performance, improve delivery of care and ultimately the safety of patients while in hospital. The point-of-care e-A&F system provides healthcare providers, from a 600-bed tertiary hospital in Western Australia, with near real-time feedback via web-based dashboards. The aim of this evaluation is to determine the implications of e-A&F across multiple dimensions and domains of care in a tertiary hospital setting. The study also aims to address the paucity in the literature by validating hypothesised design and implementation mechanisms on its effectiveness. Key datasets to be examined include those related to patient outcomes, staff behaviour and costs. Quantitative methods, such as interrupted time series analysis and multiple logistic regression analysis, amongst other methods, will be employed to achieve these aims.

Awareness of performance on outcomes after total hip and knee arthroplasty among Dutch orthopedic surgeons: how to improve feedback from arthroplasty registries

Background and purpose - The Netherlands Registry of Orthopedic Implants (LROI) uses audit and feedback (A&F) as the strategy to improve performance outcomes after total hip and knee arthroplasty (THA/TKA). Effectiveness of A&F depends on awareness of below-average performance to initiate improvement activities. We explored the awareness of Dutch orthopedic surgeons regarding their performance on outcomes after THA/TKA and factors associated with this awareness.Methods - An anonymous questionnaire was sent to all 445 eligible Dutch orthopedic surgeons performing THA/TKA. To assess awareness on own surgeon-group performance, they were asked whether their 1-year THA/TKA revision rates over the past 2 years were below average (negative outlier), average (non-outlier), above average (positive outlier) in the funnel plot on the LROI dashboard, or did not know. Associations were determined with (1) dashboard login at least once a year (yes/no); (2) correct funnel-plot interpretation (yes/no) and; (3) recall of their 1-year THA/TKA revision rate (yes/no).Results - 44% of respondents started the questionnaire, 158 THA and 156 TKA surgeons. 55% of THA surgeons and 55% of TKA surgeons were aware of their performance. Surgeons aware of their performance more often logged in on the LROI dashboard, more often interpreted funnel plots correctly, and more often recalled their revision rate. 38% of THA and 26% of TKA surgeons scored "good" on all 3 outcomes.Interpretation - Only half of the orthopedic surgeons were aware of their performance status regarding outcomes after THA/TKA. This suggests that to increase awareness, orthopedic surgeons need to be actively motivated to look at the dashboard more frequently and educated on interpretation of funnel plots for audit and feedback to be effective.

Obstetrical safety indicators for preventing hospital harms in low risk births: a scoping review protocol

INTRODUCTION

Optimising the safety of obstetric patient care is a primary concern for many hospitals. Performance indicators measuring aspects of patient care processes can lead to improvements in health systems and the prevention of harm to the patient. We present our protocol for a scoping review to identify indicators for obstetric safety in low risk births. We aim to identify indicators addressing preventable hospital harms, to summarise the data and synthesise results.

METHODS AND ANALYSIS

We will use methods described by Arksey and O'Malley and further expanded by Levac et al. We will search electronic databases such as Medline, Embase, CINAHL and the Cochrane Library, and websites from professional bodies and other organisations, using an iterative search strategy.Two reviewers will independently screen titles and abstracts of search results to determine eligibility for inclusion. If eligibility is not clear, the reviewers will screen the full text version. If reviewers' decisions regarding eligibility differ, a third reviewer will review the record. Two reviewers will independently extract data from records that meet our inclusion criteria using a standardised data collection form. We will narratively describe quantitative data, such as the frequency with which indicators are identified, and conduct a thematic analysis of the qualitative data. We will compile a comprehensive list of patient safety indicators and organise them according to concepts that best suit the data such as the Donabedian model or the Hospital Harm Framework. We will discuss the implications for future research, clinical practice and policy-making. We will report the conduct of the review using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for scoping reviews checklist.

ETHICS AND DISSEMINATION

The sources of information included in this scoping review will be available to the public. Therefore, ethics approval is not warranted. We will disseminate results in a peer-reviewed publication, conference/event presentation(s) and stakeholder communications.

Requirements for a quality dashboard: Lessons from National Clinical Audits

Healthcare organizations worldwide use quality dashboards to provide feedback to clinical teams and managers, in order to monitor care quality and stimulate quality improvement. However, there is limited evidence regarding the impact of quality dashboards and audit and feedback research focuses on feedback to individual clinicians, rather than to clinical and managerial teams. Consequently, we know little about what features a quality dashboard needs in order to provide benefit. We conducted 54 interviews across five healthcare organizations in the National Health Service in England, interviewing personnel at different levels of the organization, to understand how national (UK) clinical audit data are used for quality improvement and factors that support or constrain use of these data. The findings, organized around the themes of choosing performance indicators, assessing performance, identifying causes, communicating from ward to board, and data quality, have implications for the design of quality dashboards, which we have translated into a series of requirements.

How, in what contexts, and why do quality dashboards lead to improvements in care quality in acute hospitals? Protocol for a realist feasibility evaluation

INTRODUCTION

National audits are used to monitor care quality and safety and are anticipated to reduce unexplained variations in quality by stimulating quality improvement (QI). However, variation within and between providers in the extent of engagement with national audits means that the potential for national audit data to inform QI is not being realised. This study will undertake a feasibility evaluation of QualDash, a quality dashboard designed to support clinical teams and managers to explore data from two national audits, the Myocardial Ischaemia National Audit Project (MINAP) and the Paediatric Intensive Care Audit Network (PICANet).

METHODS AND ANALYSIS

Realist evaluation, which involves building, testing and refining theories of how an intervention works, provides an overall framework for this feasibility study. Realist hypotheses that describe how, in what contexts, and why QualDash is expected to provide benefit will be tested across five hospitals. A controlled interrupted time series analysis, using key MINAP and PICANet measures, will provide preliminary evidence of the impact of QualDash, while ethnographic observations and interviews over 12 months will provide initial insight into contexts and mechanisms that lead to those impacts. Feasibility outcomes include the extent to which MINAP and PICANet data are used, data completeness in the audits, and the extent to which participants perceive QualDash to be useful and express the intention to continue using it after the study period.

ETHICS AND DISSEMINATION

The study has been approved by the University of Leeds School of Healthcare Research Ethics Committee. Study results will provide an initial understanding of how, in what contexts, and why quality dashboards lead to improvements in care quality. These will be disseminated to academic audiences, study participants, hospital IT departments and national audits. If the results show a trial is feasible, we will disseminate the QualDash software through a stepped wedge cluster randomised trial.

Data accuracy in the Ontario birth Registry: a chart re-abstraction study

Background

Ontario’s birth Registry (BORN) was established in 2009 to collect, interpret, and share critical data about pregnancy, birth and the early childhood period to facilitate and improve the provision of healthcare. Since the use of routinely-collected health data has been prioritized internationally by governments and funding agencies to improve patient care, support health system planning, and facilitate epidemiological surveillance and research, high quality data is essential. The purpose of this study was to verify the accuracy of a selection of data elements that are entered in the Registry.

Methods

Data quality was assessed by comparing data re-abstracted from patient records to data entered into the Ontario birth Registry. A purposive sample of 10 hospitals representative of hospitals in Ontario based on level of care, birth volume and geography was selected and a random sample of 100 linked mother and newborn charts were audited for each site. Data for 29 data elements were compared to the corresponding data entered in the Ontario birth Registry using percent agreement, kappa statistics for categorical data elements and intra-class correlation coefficients (ICCs) for continuous data elements.

Results

Agreement ranged from 56.9 to 99.8%, but 76% of the data elements (22 of 29) had greater than 90% agreement. There was almost perfect (kappa 0.81–0.99) or substantial (kappa 0.61–0.80) agreement for 12 of the categorical elements. Six elements showed fair-to-moderate agreement (kappa <0.60). We found moderate-to-excellent agreement for four continuous data elements (ICC >0.50).

Conclusion

Overall, the data elements we evaluated in the birth Registry were found to have good agreement with data from the patients’ charts. Data elements that showed moderate kappa or low ICC require further investigation.

Use of a maternal newborn audit and feedback system in Ontario: a collective case study

Background

As part of a larger study examining the effectiveness of the Maternal Newborn Dashboard, an electronic audit and feedback system to improve maternal-newborn care practices and outcomes, the purpose of this study was to increase our understanding of factors explaining variability in performance after implementation of the Dashboard in Ontario, Canada.

Methods

A collective case study. A maximum variation sampling approach was used to invite hospitals reflecting different criteria to participate in a 1-day to 2-day site visit by the research team. The visits included: (1) semistructured interviews and focus groups with healthcare providers, leaders and personnel involved in clinical change processes; (2) observations and document review. Interviews and focus groups were audio-recorded and transcribed verbatim. Qualitative content analysis was used to code and categorise the data.

Results

Between June and November 2016, we visited 14 maternal-newborn hospitals. Hospitals were grouped into four quadrants based on their key indicator performance and level of engagement with the Dashboard. Findings revealed four overarching themes that contribute to the varying success of sites in achieving practice change on the Dashboard key performance indicators, namely, interdisciplinary collaboration and accountability, application of formal change strategies, team trust and use of evidence and data, as well as alignment with organisational priorities and support.

Conclusion

The diversity of facilitators and barriers across the 14 hospitals highlights the need to go beyond a ‘one size fits all’ approach when implementing audit and feedback systems. Future work to identify tools to assess barriers to practice change and to evaluate the effects of cointerventions to optimise audit and feedback systems for clinical practice change is needed.

Improving perioperative pain management: a preintervention and postintervention study in 7 developing countries

Supplemental Digital Content is Available in the Text.

Introduction:

The burden of untreated postoperative pain is high.

Objective:

This study assessed feasibility of using quality improvement (QI) tools to improve management of perioperative pain in hospitals in multiple developing countries.

Methods:

The International Pain Registry and Developing Countries working groups, from the International Association for the Study of Pain (IASP), sponsored the project and PAIN OUT, a QI and research network, coordinated it, and provided the research tools. The IASP published a call about the project on its website. Principal investigators (PIs) were responsible for implementing a preintervention and postintervention study in 1 to 2 surgical wards in their hospitals, and they were free to choose the QI intervention. Trained surveyors used standardized and validated web-based tools for collecting findings about perioperative pain management and patient reported outcomes (PROs). Four processes and PROs, independent of surgery type, assessed effectiveness of the interventions.

Results:

Forty-three providers responded to the call; 13 applications were selected; and PIs from 8 hospitals, in 14 wards, in 7 countries, completed the study. Interventions focused on teaching providers about pain management. Processes improved in 35% and PROs in 37.5% of wards.

Conclusions:

The project proved useful on multiple levels. It offered PIs a framework and tools to perform QI work and findings to present to colleagues and administration. Management practices and PROs improved on some wards. Interpretation of change proved complex, site-dependent, and related to multiple factors. PAIN OUT gained experience coordinating a multicentre, international QI project. The IASP promoted research, education, and QI work.

Segmented Linear Regression Models for Assessing Change in Retrospective Studies in Healthcare

Introduction

In retrospective studies, the effect of a given intervention is usually evaluated by using statistical tests to compare data from before and after the intervention. A problem with this approach is that the presence of underlying trends can lead to incorrect conclusions. This study aimed to develop a rigorous mathematical method to analyse temporal variation and overcome these limitations.

Methods

We evaluated hip fracture outcomes (time to surgery, length of stay, and mortality) from a total of 2777 patients between April 2011 and September 2016, before and after the introduction of a dedicated hip fracture unit (HFU). We developed a novel modelling method that fits progressively more complex linear sections to the time series using least squares regression. The method was used to model the periods before implementation, after implementation, and of the whole study period, comparing goodness of fit using F-tests.

Results

The proposed method offered reliable descriptions of the temporal evolution of the time series and augmented conclusions that were reached by mere group comparisons. Reductions in time to surgery, length of stay, and mortality rates that group comparisons would have credited to the hip fracture unit appeared to be due to unrelated underlying trends.

Conclusion

Temporal analysis using segmented linear regression models can reveal secular trends and is a valuable tool to evaluate interventions in retrospective studies.