Are audits wasting resources by measuring the wrong things? A survey of methods used to select audit review criteria

A realist synthesis of multicentre comparative audit implementation: exploring what works and in which healthcare contexts

BACKGROUND

Multicentre comparative clinical audits have the potential to improve patient care, allow benchmarking and inform resource allocation. However, implementing effective and sustainable large-scale audit can be difficult within busy and resource-constrained contemporary healthcare settings. There are little data on what facilitates the successful implementation of multicentre audits. As healthcare environments are complex sociocultural organisational environments, implementing multicentre audits within them is likely to be highly context dependent.

OBJECTIVE

We aimed to examine factors that were influential in the implementation process of multicentre comparative audits within healthcare contexts-what worked, why, how and for whom?

METHODS

A realist review was conducted in accordance with the Realist and Meta-narrative Evidence Syntheses: Evolving Standards reporting standards. A preliminary programme theory informed two systematic literature searches of peer-reviewed and grey literature. The main context-mechanism-outcome (CMO) configurations underlying the implementation processes of multicentre audits were identified and formed a final programme theory.

RESULTS

69 original articles were included in the realist synthesis. Four discrete CMO configurations were deduced from this synthesis, which together made up the final programme theory. These were: (1) generating trustworthy data; (2) encouraging audit participation; (3) ensuring audit sustainability; and (4) facilitating audit cycle completion.

CONCLUSIONS

This study elucidated contexts, mechanisms and outcomes influential to the implementation processes of multicentre or national comparative audits in healthcare. The relevance of these contextual factors and generative mechanisms were supported by established theories of behaviour and findings from previous empirical research. These findings highlight the importance of balancing reliability with pragmatism within complex adaptive systems, generating and protecting human capital, ensuring fair and credible leadership and prioritising change facilitation.

Improving the reporting of orthodontic clinical audits: an evaluation

Aims The aim of this study was to evaluate the reporting of orthodontic audits published between 2013-2019 following the introduction of a submission template in 2015.Methods An audit reporting checklist was developed, with each audit independently assessed by two assessors. Based on the previous quality checklist, an overall score of 4 or less represented poor reporting, 5-8 fair reporting and 9 or greater good reporting. All data variables were collected in a pre-piloted Excel data collection sheet.Results One hundred and fifty-nine audits were identified. A range of reporting scores were evident. The overall mean score was 10.1 (SD 1.5). Reporting scores showed improvement during the study timeframe, with a general increase in scores evident from 2015. Higher scores were achieved by multi-cycle audits (coefficient [coef]: 2.0, 95% CI: 1.38, 2.62, p <0.001). Lower scores were achieved by partial audits (coef: -1.8, 95% CI: -2.23, -1.36, p <0.001), but scores increased every year (coef: 0.2, 95% CI: 0.12, 0.27, p <0.001).Conclusions The reporting of orthodontic audits is rated as good, with yearly improvement in scores evident. The introduction of a submission template had a positive effect on the reporting of audits. Recommendations to further improve the quality of audits are outlined.

Obstetrical safety indicators for preventing hospital harms in low risk births: a scoping review protocol

INTRODUCTION

Optimising the safety of obstetric patient care is a primary concern for many hospitals. Performance indicators measuring aspects of patient care processes can lead to improvements in health systems and the prevention of harm to the patient. We present our protocol for a scoping review to identify indicators for obstetric safety in low risk births. We aim to identify indicators addressing preventable hospital harms, to summarise the data and synthesise results.

METHODS AND ANALYSIS

We will use methods described by Arksey and O'Malley and further expanded by Levac et al. We will search electronic databases such as Medline, Embase, CINAHL and the Cochrane Library, and websites from professional bodies and other organisations, using an iterative search strategy.Two reviewers will independently screen titles and abstracts of search results to determine eligibility for inclusion. If eligibility is not clear, the reviewers will screen the full text version. If reviewers' decisions regarding eligibility differ, a third reviewer will review the record. Two reviewers will independently extract data from records that meet our inclusion criteria using a standardised data collection form. We will narratively describe quantitative data, such as the frequency with which indicators are identified, and conduct a thematic analysis of the qualitative data. We will compile a comprehensive list of patient safety indicators and organise them according to concepts that best suit the data such as the Donabedian model or the Hospital Harm Framework. We will discuss the implications for future research, clinical practice and policy-making. We will report the conduct of the review using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for scoping reviews checklist.

ETHICS AND DISSEMINATION

The sources of information included in this scoping review will be available to the public. Therefore, ethics approval is not warranted. We will disseminate results in a peer-reviewed publication, conference/event presentation(s) and stakeholder communications.

A healthcare quality management system underpinning the 3-E model and its application in a new tertiary hospital in Australia

Objectives

Engaging, enhancing and embedding clinical audit improvement activities into everyday practice to develop capacity, capability and culture in continuous improvement.

Method

Through the implementation of an electronic quality management system called Governance, Evidence, Knowledge and Outcome (GEKO), the key aspects of governance, evidence knowledge and outcomes were able to be applied to quality initiatives. Implementation of the GEKO system incorporated the principles of total quality control and management to include strategic management control and marketing in parallel with leadership strategies.

The vision was to motivate staff to enable ownership of the quality cycle of continuous improvement of patient care to incorporate underlying systems and processes that impact on patient care.

Results

A continuous improvement pathway was successfully established 4 months post hospital commissioning. Over 890 (approximately 16% workforce) multidisciplinary and multi-professional staff received training and support for QIs in 12 months; over 535 quality proposals were received on GEKO. Submissions by profession: nursing and midwifery 46% (246), medical 33% (177), allied health 9% (48), pharmacy 5% (27), and non-clinical staff 7% (37). Average new submissions per month were 42. Reviews demonstrated the application of a rapid cycle approach to develop, test, modify and refine improvements and enhanced clinical care.

Conclusion

Appropriate governance structure, processes, extensive education and training together with collaborative relationships are the keys to embed clinical audit improvement into everyday practice. The availability of a quality management system like GEKO is very useful to make QI accessible to all staff.

Past, Present, and Future of Informed Consent in Pain and Genomics Research: Challenges Facing Global Medical Community

In recent decades, there has been a revision of the role of institutional review boards with the intention of protecting human subjects from harm and exploitation in research. Informed consent aims to protect the subject by explaining all of the benefits and risks associated with a specific research project. To date, there has not been a review published analyzing issues of informed consent in research in the field of genetic/Omics in subjects with chronic pain, and the current review aims to fill that gap in the ethical aspects of such investigation. Despite the extensive discussion on ethical challenges unique to the field of genetic/Omics, this is the first attempt at addressing ethical challenges regarding Informed Consent Forms for pain research as the primary focus. We see this contribution as an important one, for while ethical issues are too often ignored in pain research in general, the numerous arising ethical issues that are unique to pain genetic/Omics suggest that researchers in the field need to pay even greater attention to the rights of subjects/patients. This article presents the work of the Ethic Committee of the Pain-Omics Group (www.painomics.eu), a consortium of 11 centers that is running the Pain-Omics project funded by the European Community in the 7th Framework Program theme (HEALTH.2013.2.2.1-5-Understanding and controlling pain). The Ethic Committee is composed of 1 member of each group of the consortium as well as key opinion leaders in the field of ethics and pain more generally.

Prevention of in-hospital falls: development of criteria for the conduct of a multi-site audit

BACKGROUND

Patient falls are a significant issue for hospitals due to the high rates of morbidity and mortality associated with these events, as well as the financial costs for the healthcare system.

OBJECTIVES

To establish what constitutes best practice in terms of fall prevention in acute care facilities and use this to inform the development of best practice audit criteria.

METHODS

Criteria for clinical audit were developed from evidence derived from systematic reviews and guidelines. While these were drawn from the best available evidence, they were also developed in conjunction with clinicians undertaking a fall-prevention clinical audit and key stakeholders from the clinical settings to ensure their relevance and applicability to the acute care setting.

RESULTS

Current literature recommends a comprehensive and multifactorial approach to fall prevention. Eight audit criteria were derived from the best available evidence including the domains of physical environment, hospital culture and care processes, use of technology and targeted interventions.

CONCLUSIONS

Existing research evidence and consultation with stakeholders has allowed the development of applicable, evidence-based audit criteria for fall prevention in acute care settings. This model can promote engagement, impact clinical practice and lead to improved outcomes.

Prevention of falls in acute hospital settings: a multi-site audit and best practice implementation project

OBJECTIVE

To assess falls prevention practices in Australian hospitals and implement interventions to promote best practice.

DESIGN

A multi-site audit using eight evidence-based audit criteria. Following a baseline audit, barriers to compliance were identified and targeted. Two follow-up audit cycles assessed the sustainability of practice change.

SETTING

Nine acute care hospitals around Australia, including a mix of public and private. One medical ward and one surgical ward from each hospital were involved.

PARTICIPANTS

A clinical leader from each hospital, trained in evidence implementation, conducted the audits and implementation strategies in their setting.

INTERVENTIONS

Multi-component falls prevention interventions were utilized, designed to target specific barriers to compliance identified at each hospital. Common interventions involved staff and patient education.

MAIN OUTCOME MEASURE

Percentage compliance with falls prevention audit criteria and change in compliance between baseline and follow-up audits. Fall rate data were also analysed.

RESULTS

Mean overall compliance at baseline across all hospitals was 50.4% (range 30.8-76.6%). At the first follow-up, this had increased to 74.5% (range 59.4-87.4%), which was sustained at the second follow-up (74.1%, range 48.6-84.4%). There were no statistically significant differences between compliance rates in medical versus surgical wards or in private versus public hospitals. Despite sustained practice improvement, reported fall rates remained unchanged. The focus on staff education possibly led to improved reporting of falls, which may explain the apparent lack of effect on fall rates.

CONCLUSIONS

Clinical audit and feedback is an effective strategy to promote quality improvement in falls prevention practices in acute hospital settings.

Audit of audit: review of a clinical audit programme in a teaching hospital intensive care unit

OBJECTIVE

A comprehensive review of the clinical audit programme in a teaching hospital intensive care unit.

DESIGN

A retrospective analysis of the clinical audit projects undertaken within the intensive care unit over the preceding 2 years and compared with published national guidelines for clinical audit.

SETTING

A 27-bedded teaching hospital intensive care unit in the UK.

MEASUREMENTS

Each audit project was reviewed independently by two assessors. The following questions were assessed. 1. Were the projects true audits? 2. Were they prospective of retrospective? 3. Did the projects have input from appropriate members of the multidisciplinary team. 4. How many of the audit projects were re-audits? 5. Of the re-audits how many showed evidence of service improvement? each audit project was also scored against the Audit Project Assessment Tool produced by the UK Clinical Governance Support Team.

RESULTS

Of the twenty five audit projects reviewed twenty two were considered to be true audits. All of the projects used only retrospective data. Audit projects were contributed from all sections of the multidisciplinary critical care team but there were few truly multidisciplinary projects. Four of the audit projects were re-audits, of these three showed service improvement and one showed deterioration. Of the twenty two true audit projects reviewed, eleven were classified as good quality projects using the Audit Project Assessment Tool.

CONCLUSIONS

Despite the clinical audit programme being active and well supported, objective evidence of clinical governance benefit was lacking. The overall clinical audit programme has been revitalised by a series of improvements since undertaking this review and this approach is recommended to other organizations who are interested in improving their clinical audit performance.

Improved clinical outcomes combining house staff self-assessment with an audit-based quality improvement program

BACKGROUND

There is a focus on integrating quality improvement with medical education and advancement of the American College of Graduate Medical Education (ACGME) core competencies.

OBJECTIVE

To determine if audits of patients with unexpected admission to the medical intensive care unit using a self-assessment tool and a focused Morbidity and Mortality (M&M) conference improves patient care.

DESIGN

Charts from patients transferred from the general medical floor (GMF) to the medical intensive care unit (ICU) were reviewed by a multidisciplinary team. Physician and nursing self-assessment tools and a targeted monthly M&M conference were part of the educational component.

PARTICIPANTS

Physicians and nurses participated in root cause analysis.

MEASURES

Records of all patients transferred from a general medical floor (GMF) to the ICU were audited. One hundred ninety-four cases were reviewed over a 10-month period.

RESULTS

New policies regarding vital signs and house staff escalation of care were initiated. The percentage of calls for patients who met medical emergency response team/critical care consult criteria increased from 53% to 73%, nurse notification of a change in a patient's condition increased from 65% to 100%, nursing documentation of the change in the patients condition and follow-up actions increased from 65% percent to a high of 90%, the number of cardiac arrests on a GMF decreased from 3.1/1,000 discharges to 0.6/1,000 discharges (p = 0.002), and deaths on the Medicine Service decreased from 34/1,000 discharges to 24/1,000 discharges (p = 0.024).

CONCLUSION

We describe an audit-based program that involves nurses, house staff, a self-assessment tool and a focused M&M conference. The program resulted in significant policy changes, more rapid assessment of unstable patients and improved hospital outcomes.

What are the most effective strategies for improving quality and safety of health care?

There is now a plethora of different quality improvement strategies (QIS) for optimizing health care, some clinician/patient driven, others manager/policy-maker driven. Which of these are most effective remains unclear despite expressed concerns about potential for QIS-related patient harm and wasting of resources. The objective of this study was to review published literature assessing the relative effectiveness of different QIS. Data sources comprising PubMed Clinical Queries, Cochrane Library and its Effective Practice and Organization of Care database, and HealthStar were searched for studies of QIS between January 1985 and February 2008 using search terms based on an a priori QIS classification suggested by experts. Systematic reviews of controlled trials were selected in determining effect sizes for specific QIS, which were compared as a narrative meta-review. Clinician/patient driven QIS were associated with stronger evidence of efficacy and larger effect sizes than manager/policy-maker driven QIS. The most effective strategies (>10% absolute increase in appropriate care or equivalent measure) included clinician-directed audit and feedback cycles, clinical decision support systems, specialty outreach programmes, chronic disease management programmes, continuing professional education based on interactive small-group case discussions, and patient-mediated clinician reminders. Pay-for-performance schemes directed to clinician groups and organizational process redesign were modestly effective. Other manager/policy-maker driven QIS including continuous quality improvement programmes, risk and safety management systems, public scorecards and performance reports, external accreditation, and clinical governance arrangements have not been adequately evaluated with regard to effectiveness. QIS are heterogeneous and methodological flaws in much of the evaluative literature limit validity and generalizability of results. Based on current best available evidence, clinician/patient driven QIS appear to be more effective than manager/policy-maker driven QIS although the latter have, in many instances, attracted insufficient robust evaluations to accurately determine their comparative effectiveness.

Could clinical audit improve the diagnosis of pulmonary tuberculosis in Cuba, Peru and Bolivia?

OBJECTIVES

To assess the effectiveness of clinical audit in improving the quality of diagnostic care provided to patients suspected of tuberculosis; and to understand the contextual factors which impede or facilitate its success.

METHODS

Twenty-six health centres in Cuba, Peru and Bolivia were recruited. Clinical audit was introduced to improve the diagnostic care for patients attending with suspected TB. Standards were based on the WHO and TB programme guidelines relating to the appropriate use of microscopy, culture and radiological investigations. At least two audit cycles were completed over 2 years. Improvement was determined by comparing the performance between two six-month periods pre- and post-intervention. Qualitative methods were used to ascertain facilitating and limiting contextual factors influencing change among healthcare professionals' clinical behaviour after the introduction of clinical audit.

RESULTS

We found a significant improvement in 11 of 13 criteria in Cuba, in 2 of 6 criteria in Bolivia and in 2 of 5 criteria in Peru. Twelve out of 24 of the audit criteria in all three countries reached the agreed standards. Barriers to quality improvement included conflicting objectives for clinicians and TB programmes, poor coordination within the health system and patients' attitudes towards illness.

CONCLUSIONS

Clinical audit may drive improvements in the quality of clinical care in resource-poor settings. It is likely to be more effective if integrated within and supported by the local TB programmes. We recommend developing and evaluating an integrated model of quality improvement including clinical audit.

The structure of quality systems is important to the process and outcome, an empirical study of 386 hospital departments in Sweden

BACKGROUND

Clinicians, nurses, and managers in hospitals are continuously confronted by new technologies and methods that require changes to working practice. Quality systems can help to manage change while maintaining a high quality of care. A new model of quality systems inspired by the works of Donabedian has three factors: structure (resources and administration), process (culture and professional co-operation), and outcome (competence development and goal achievement). The objectives of this study were to analyse whether structure, process, and outcome can be used to describe quality systems, to analyse whether these components are related, and to discuss implications.

METHODS

A questionnaire was developed and sent to a random sample of 600 hospital departments in Sweden. The adjusted response rate was 75%. The data were analysed with confirmatory factor analysis and structural equation modeling in LISREL. This is to our knowledge the first large quantitative study that applies Donabedian's model to quality systems.

RESULTS

The model with relationships between structure, process, and outcome was found to be a reasonable representation of quality systems at hospital departments (p = 0.095, indicating no significant differences between the model and the data set). Structure correlated strongly with process (0.72) and outcome (0.60). Given structure, process also correlated with outcome (0.20).

CONCLUSION

The model could be used to describe and evaluate single quality systems or to compare different quality systems. It could also be an aid to implement a systematic and evidence-based system for working with quality improvements in hospital departments.

Clinical audit and reform of the UK research ethics review system

There is an international consensus that medical research involving humans should only be undertaken in accordance with ethical principles. Paradoxically though, there is no consensus over the kinds of activities that constitute research and should be subject to review. In the UK and elsewhere, research requiring review is distinguished from clinical audit. Unfortunately the two activities are not always easy to differentiate from one another. Moreover, as the volume of audit increases and becomes more formal in response to the demand for evidence-based practice in medicine, the overlap between research and audit grows more acute. Arguably, similar ethical standards and systems for ensuring that those standards are met should be applied regardless of whether or not a project is classified as research or audit. At a time when the research ethics review system in the UK is undergoing significant reform it is important that the opportunity is not missed to address the longstanding research-audit problem. We discuss suggestions for further reform that addresses this issue.

Discussion of quality and audit in health

Attaining quality health care has long been a social policy priority for countries internationally. This discussion considers issues important to understanding quality, and audit implementation in particular. The paper covers, first, the principles and practice of audit and, second, broader implementation issues, which together point to the further development of quality initiatives in health in the United Kingdom health care context. To close, the future of audit as a means of improving health care is elaborated.

Is audit research? The relationships between clinical audit and social-research

PURPOSE

Quality has an established history in health care. Audit, as a means of quality assessment, is well understood and the existing literature has identified links between audit and research processes. This paper reviews the relationships between audit and research processes, highlighting how audit can be improved through the principles and practice of social research.

DESIGN/METHODOLOGY/APPROACH

The review begins by defining the audit process. It goes on to explore salient relationships between clinical audit and research, grouped into the following broad themes: ethical considerations, highlighting responsibilities towards others and the need for ethical review for audit; asking questions and using appropriate methods, emphasising transparency in audit methods; conceptual issues, including identifying problematic concepts, such as "satisfaction", and the importance of reflexivity within audit; emphasising research in context, highlighting the benefits of vignettes and action research; complementary methods, demonstrating improvements for the quality of findings; and training and multidisciplinary working, suggesting the need for closer relationships between researchers and clinical practitioners.

FINDINGS

Audit processes cannot be considered research. Both audit and research processes serve distinct purposes.

ORIGINALITY/VALUE

Attention to the principles of research when conducting audit are necessary to improve the quality of audit and, in turn, the quality of health care.

Review of performance assessment and improvement in ambulatory medical care

Health care plans often consider quality of care as a means of containing rising health costs. The assessment of physician and group practice performance has become increasingly widespread in ambulatory care. This article reviews the three main methods used to improve and assess performance: practice audits, peer-review groups and practice visits. The focus is on Europe - which countries use which methods - and on the following aspects: which authorities or bodies are responsible for setting up and running the systems, are the systems mandatory or voluntary, who takes part in assessments and what is their motivation, are patients views taken into account. Many countries run parallel systems managed by authorities working at different hierarchical levels (national, regional or local). The reasons that underlie the choice of a particular system are discussed. They are mostly related to the national health care system and to cultural factors.