Deepening our Understanding of Quality in Australia (DUQuA): a study protocol for a nationwide, multilevel analysis of relationships between hospital quality management systems and patient factors

Safety climate, leadership and patient views associated with hip fracture care quality and clinician perceptions of hip fracture care performance

BACKGROUND

Hip fracture is a major public health concern for older adults, requiring surgical treatment for patients presenting at hospitals across Australia. Although guidelines have been developed to drive appropriate care of hip fracture patients in hospitals, data on health outcomes suggest these are not well-followed.

OBJECTIVE

This study aims to examine whether clinician measures of safety, teamwork and leadership, and patient perceptions of care are associated with key indicators of hip fracture care and the extent to which there is agreement between clinician perceptions of hip fracture care performance and actual hospital performance of hip fracture care.

METHODS

Retrospective analysis was performed on a series of questionnaires used to assess hospital department- and patient-level measures from the Deepening our Understanding of Quality in Australia study. Data were analysed from 32 public hospitals that encompassed 23 leading hip fracture clinicians, 716 patient medical records and 857 patients from orthopaedic public hospital wards.

RESULTS

Aggregated across all hospitals, only 5 of 12 of the key hip fracture indicators had ≥50% adherence. Adherence to indicators requiring actions to be performed within a recommended time period was poor (7.2-25.6%). No Patient Measure of Safety or clinician-based measures of teamwork, safety climate or leadership were associated with adherence to key indicators of hip fracture care. Simple proportionate agreement between clinician perceptions and actual hospital performance was generally strong, but few agreement coefficients were compelling.

CONCLUSION

The development of strong quality management processes requires ongoing effort. The findings of this study provide important insights into the relationship between hospital care and outcomes for hip fracture patients and could drive the design of targeted interventions for improved quality assurance of hip fracture care.

Using patient feedback to drive quality improvement in hospitals: a qualitative study

OBJECTIVES

Although different forms of patient feedback are available, their use in hospital management is still limited. The objective of this study is to explore how patient feedback is currently used in hospitals to improve quality.

DESIGN

This is a qualitative exploratory multiple case study. Data collection included nine interviews, of an average duration of 50 min, conducted between March and June 2019. Additionally, a document and secondary data analysis were performed.

SETTING

This study was conducted in three Brazilian hospitals selected for their solid patient feedback practises.

PARTICIPANTS

Managers from the customer service, quality, nursing, operations, projects and patient experience departments of the three hospitals.

RESULTS

Despite literature suggesting that organisational objectives regarding patient feedback are not clear, data show that there is managerial concern regarding the promotion of an environment capable of changing according to patient feedback. In these instances, organisational processes were structured to focus on patients' feedback and its receipt by the staff, including a non-punitive culture. Several patient feedback forms are available: voluntary events, patient surveys and informal feedback. Instruments to measure patient feedback focused on specific aspects of healthcare, to identify and clarify the problems for addressal by the management. The net promoter score was the main strategic indicator of patient feedback, used to assess the impact of improvement action.

CONCLUSIONS

The hospitals had established objectives that valued the patient's perspective. Involvement of the health team, availability of different channels for feedback and the use of quality tools are considered a good basis for using patient feedback to drive quality improvement.

Do quality management systems influence clinical safety culture and leadership? A study in 32 Australian hospitals

OBJECTIVE

This study aimed to explore the associations between the organization-level quality arrangements, improvement and implementation and department-level safety culture and leadership measures across 32 large Australian hospitals.

DESIGN

Quantitative observational study, using linear and multi-level modelling to identify relationships between quality management systems and clinician safety culture and leadership.

SETTING

Thirty-two large Australian public hospitals.

PARTICIPANTS

Quality audit at organization level, senior quality manager at each participating hospital, 1382 clinicians (doctors, nurses and allied health professionals).

MAIN OUTCOME MEASURES

Associations between organization-level quality measures and department-level clinician measures of teamwork climate, safety climate and leadership for acute myocardial infarction (AMI), hip fracture and stroke treatment conditions.

RESULTS

We received 1332 valid responses from participants. The quality management systems index (QMSI, a questionnaire-based measure of the hospitals' quality management structures) was 'positively' associated with all three department-level scales in the stroke department, with safety culture and leadership in the emergency department, but with none of the three scales in the AMI and hip fracture departments. The quality management compliance index (QMCI, an external audit-based measure of the quality improvement activities) was 'negatively' associated with teamwork climate and safety climate in AMI departments, after controlling for QMSI, but not in other departments. There was no association between QMCI and leadership in any department, after controlling for QMSI, and there was no association between the clinical quality implementation index (CQII, an external audit-based measure of the level of implementation of quality activities) and any of the three department-level scales in any of the four departments, after controlling for both QMSI and QMCI.

CONCLUSIONS

The influence of organization-level quality management systems on clinician safety culture and leadership varied depending on the hospital department, suggesting that whilst there was some consistency on patient safety attitudes and behaviours throughout the organizations, there were also other factors at play.

Using accreditation surveyors to conduct health services research: a qualitative, comparative study in Australia

OBJECTIVE

Healthcare accreditation surveyors are well positioned to gain access to hospitals and apply their existing data collection skills to research. Consequently, we contracted and trained a surveyor cohort to collect research data for the Deepening our Understanding of Quality in Australia (DUQuA) project. The aim of this study is to explore and compare surveyors' perceptions and experiences in collecting quality and safety data for accreditation and for health services research.

DESIGN

A qualitative, comparative study.

SETTING AND PARTICIPANTS

Ten surveyors participated in semi-structured interviews, which were audio recorded, transcribed and coded using Nvivo11. Interview transcripts of participants were analysed thematically and separately, providing an opportunity for comparison and for identifying common themes and subthemes.

INTERVENTION(S)

None.

MAIN OUTCOME MEASURE(S)

Topics addressed data collection for healthcare accreditation and research, including preparation and training, structure, organization, attitudes and behaviours of staff and perceptions of their role.

RESULTS

Five themes and ten subthemes emerged from the interviews: (1) overlapping facilitators for accreditation and research data collection, (2) accreditation-specific facilitators, (3) overlapping barriers for accreditation and research data collection, (4) research data collection-specific barriers and (5) needs and recommendations. Subthemes were (1.1) preparation and training availability, (1.2) prior knowledge and experiences; (2.1) ease of access, (2.2) high staff engagement, (3.1) time, (4.1) poor access and structure, (4.2) lack of staff engagement, (4.3) organizational changes; (5.1) short-notice accreditation and (5.2) preparation for future research.

CONCLUSIONS

Although hospital accreditation and research activities require different approaches to data collection, we found that suitably trained accreditation surveyors were able to perform both activities effectively. The barriers surveyors encountered when collecting data for research provide insight into the challenges that may be faced when visiting hospitals for short-notice accreditation.

Deepening our Understanding of Quality in Australia (DUQuA): An overview of a nation-wide, multi-level analysis of relationships between quality management systems and patient factors in 32 hospitals

OBJECTIVE

The Deepening our Understanding of Quality in Australia (DUQuA) project is a multisite, multi-level, cross-sectional study of 32 of the largest hospitals in Australia. This overview examines relationships between (i) organization-level quality management systems and department-level quality management strategies and (ii) patient-level measures (clinical treatment processes, patient-reported perceptions of care and clinical outcomes) within Australian hospitals.

DESIGN

We examined hospital quality improvement structures, processes and outcomes, collecting data at organization, department and patient levels for acute myocardial infarction (AMI), hip fracture and stroke. Data sources included surveys of quality managers, clinicians and patients, hospital visits, medical record reviews and national databases. Outcomes data and patient admissions data were analysed. Relationships between measures were evaluated using multi-level models. We based the methods on the Deepening our Understanding of Quality Improvement in Europe (DUQuE) framework, extending that work in parts and customizing the design to Australian circumstances.

SETTING, PARTICIPANTS AND OUTCOME MEASURES

The 32 hospitals, containing 119 participating departments, provided wide representation across metropolitan, inner and outer regional Australia. We obtained 31 quality management, 1334 clinician and 857 patient questionnaires, and conducted 2401 medical record reviews and 151 external assessments. External data via a secondary source comprised 14 460 index patient admissions across 14 031 individual patients. Associations between hospital, Emergency Department (ED) and department-level systems and strategies and five patient-level outcomes were assessed: 19 of 165 associations (11.5%) were statistically significant, 12 of 79 positive associations (15.2%) and 7 of 85 negative associations (8.2%).

RESULTS

We did not find clear relationships between hospital-level quality management systems, ED or department quality strategies and patient-level outcomes. ED-level clinical reviews were related to adherence to clinical practice guidelines for AMI, hip fracture and stroke, but in different directions. The results, when considered alongside the DUQuE results, are suggestive that front line interventions may be more influential than department-level interventions when shaping quality of care and that multi-pronged strategies are needed. Benchmark reports were sent to each participating hospital, stimulating targeted quality improvement activities.

CONCLUSIONS

We found no compelling relationships between the way care is organized and the quality of care across three targeted patient-level outcome conditions. The study was cross-sectional, and thus we recommend that the relationships studied should be assessed for changes across time. Tracking care longitudinally so that quality improvement activities are monitored and fed back to participants is an important initiative that should be given priority as health systems strive to develop their capacity for quality improvement over time.

Organization quality systems and department-level strategies: refinement of the Deepening our Understanding in Quality in Australia (DUQuA) organization and department-level scales

OBJECTIVE

The aim of this study was to develop and refine indices to measure organization and care pathway-level quality management systems in Australian hospitals.

DESIGN

A questionnaire survey and audit tools were derived from instruments validated as part of the Deepening Our Understanding of Quality improvement in Europe (DUQuE) study, adapted for Australian hospitals through expert opinion. Statistical processes were used to explore the factor structure, reliability and non-redundancy and descriptive statistics of the scales.

SETTING

Thirty-two large Australian public hospitals.

PARTICIPANTS

Audit of quality management processes at organization-level and care pathway processes at department level for three patient conditions (acute myocardial infarction (AMI), hip fracture and stroke) and senior quality manager, at each of the 32 participating hospitals.

MAIN OUTCOME MEASURE(S)

The degree of quality management evident at organization and care pathway levels.

RESULTS

Analysis yielded seven quality systems and strategies scales. The three hospital-level measures were: the Quality Management Systems Index (QMSI), the Quality Management Compliance Index (QMCI) and the Clinical Quality Implementation Index (CQII). The four department-level measures were: Specialised Expertise and Responsibility (SER), Evidence-Based Organisation of Pathways (EBOP), Patient Safety Strategies (PSS) and Clinical Review (CR). For QMCI, and for seven out of eight subscales in QMSI, adequate internal consistency (Cronbach's $\alpha$ >0.8) was achieved. For CQII, lack of variation and ceiling effects in the data resulted in very low internal consistency scores, but items were retained for theoretical reasons. Internal consistency was high for CR (Cronbach's $\alpha$ 0.74-0.88 across the three conditions), and this was supported by all item-total correlations exceeding the desired threshold. For EBOP, Cronbach's $\alpha$ was acceptable for hip fracture (0.80) and stroke (0.76), but only moderate for AMI (0.52). PSS and SER scales were retained for theoretical reasons, although internal consistencies were only moderate (SER) to poor (PSS).

CONCLUSIONS

The Deepening our Understanding of Quality in Australia (DUQuA) organization and department scales can be used by Australian hospital managers to assess and measure improvement in quality management at organization and department levels within their hospitals and are readily modifiable for other health systems depending on their needs.

The clinician safety culture and leadership questionnaire: refinement and validation in Australian public hospitals

OBJECTIVE

This study aimed to refine and validate a tool to measure safety culture and leadership in Australian hospitals.

DESIGN

The clinician safety culture and leadership questionnaire was constructed by combining and refining the following two previously validated scales: Safety Attitudes Questionnaire and the Leadership Effectiveness Survey. Statistical processes were used to explore the factor structure, reliability, validity and descriptive statistics of the new instrument.

SETTING

Thirty-two large Australian public hospitals.

PARTICIPANTS

1382 clinicians (doctors, nurses and allied health professionals).

MAIN OUTCOME MEASURE(S)

Descriptive statistics, structure and validity of clinician safety culture and leadership scale.

RESULTS

We received 1334 valid responses from participants. The distribution of ratings was left-skewed, with a small ceiling effect, meaning that scores were clustered toward the high end of the scale. Using confirmatory factor analysis, we confirmed the structure of the three scales as a combined measure of safety culture and leadership. The data were divided into equal calibration and validation datasets. For the calibration dataset, the Chi-square: df ratio was 4.4, the root mean square error of approximation RMSEA (a measure of spread of the data) was 0.071, the standardized root mean square residual SRMR (an absolute measure of the fit of the data) was 0.058 and the Confirmatory Fit Index (CFI) (another test confirming the fit of the data) was 0.82; while none of the indices suggested good fit, all but CFI fell within acceptable thresholds. All factors demonstrated adequate internal consistency and construct reliability, as desired. All three domains achieved discriminant validity through cross-loadings, meaning that the three domains were determined to be independent constructs. Results for the validation dataset were effectively identical to those found in the calibration dataset.

CONCLUSIONS

While the model may benefit from additional refinement, we have validated the tool for measuring clinician safety culture and leadership in our Australian sample. The DUQuA safety culture and leadership scale can be used by Australian hospitals to assess clinician safety culture and leadership, and is readily modifiable for other health systems depending on their needs.

Implementation and data-related challenges in the Deepening our Understanding of Quality in Australia (DUQuA) study: implications for large-scale cross-sectional research

Healthcare organisations vary in the degree to which they implement quality and safety systems and strategies. Large-scale cross-sectional studies have been implemented to explore whether this variation is associated with outcomes relevant at the patient level. The Deepening our Understanding of Quality in Australia (DUQuA) study draws from earlier research of this type, to examine these issues in 32 Australian hospitals. This paper outlines the key implementation and analysis challenges faced by DUQuA. Many of the logistical difficulties of implementing DUQuA derived from compliance with the administratively complex and time-consuming Australian ethics and governance system designed principally to protect patients involved in clinical trials, rather than for low-risk health services research. The complexity of these processes is compounded by a lack of organizational capacity for multi-site health services research; research is expected to be undertaken in addition to usual work, not as part of it. These issues likely contributed to a relatively low recruitment rate for hospitals (41% of eligible hospitals). Both sets of issues need to be addressed by health services researchers, policymakers and healthcare administrators, if health services research is to flourish. Large-scale research also inevitably involves multiple measurements. The timing for applying these measures needs to be coherent, to maximise the likelihood of finding real relationships between quality and safety systems and strategies, and patient outcomes; this timing was less than ideal in DUQuA, in part due to administrative delays. Other issues that affected our study include low response rates for measures requiring recruitment of clinicians and patients, missing data and a design that necessarily included multiple statistical comparisons. We discuss how these were addressed. Successful completion of these projects relies on mutual and ongoing commitment, and two-way communication between the research team and hospital staff at all levels. This will help to ensure that enthusiasm and engagement are established and maintained.

Bending the quality curve

With this paper, we initiate the Supplement on Deepening our Understanding of Quality in Australia (DUQuA). DUQuA is an at-scale, cross-sectional research programme examining the quality activities in 32 large hospitals across Australia. It is based on, with suitable modifications and extensions, the Deepening our Understanding of Quality improvement in Europe (DUQuE) research programme, also published as a Supplement in this Journal, in 2014. First, we briefly discuss key data about Australia, the health of its population and its health system. Then, to provide context for the work, we discuss previous activities on the quality of care and improvement leading up to the DUQuA studies. Next, we present a selection of key interventional studies and policy and institutional initiatives to date. Finally, we conclude by outlining, in brief, the aims and scope of the articles that follow in the Supplement. This first article acts as a framing vehicle for the DUQuA studies as a whole. Aggregated, the series of papers collectively attempts an answer to the questions: what is the relationship between quality strategies, both hospital-wide and at department level? and what are the relationships between the way care is organised, and the actual quality of care as delivered? Papers in the Supplement deal with a multiplicity of issues including: how the DUQuA investigators made progress over time, what the results mean in context, the scales designed or modified along the way for measuring the quality of care, methodological considerations and provision of lessons learnt for the benefit of future researchers.

The relationships between quality management systems, safety culture and leadership and patient outcomes in Australian Emergency Departments

Objective

We aimed to examine whether Emergency Department (ED) quality strategies, safety culture and leadership were associated with patient-level outcomes, after controlling for other organization-level factors, in 32 large Australian hospitals.

Design

Quantitative observational study, using linear and multi-level modelling to identify relationships between quality management systems at organization level; quality strategies at ED level for acute myocardial infarction (AMI), hip fracture and stroke; clinician safety culture and leadership and patient-level outcomes of waiting time and length of stay.

Setting

Thirty-two large Australian public hospitals.

Participants

Audit of quality management processes at organization and ED levels, senior quality manager at each of the 32 participating hospitals, 394 ED clinicians (doctors, nurses and allied health professionals).

Main Outcome Measure(s)

Within the multi-level model, associations were assessed between organization-level quality measures and ED quality strategies; organization-level quality measures and ED quality strategies and ward-level clinician measures of teamwork climate (TC), safety climate (SC) and leadership for AMI, hip fracture and stroke treatment conditions; and organization-level quality measures and ED quality strategies and ward-level clinician measures of TC, SC and leadership, and ED waiting time and length of stay (performance).

Results

We found seven statistically significant associations between organization-level quality systems and ED-level quality strategies; four statistically significant associations between quality systems and strategies and ED safety culture and leadership; and nine statistically significant associations between quality systems and strategies and ED safety culture and leadership, and ED waiting time and length of stay.

Conclusions

Organization-level quality structures influence ED-level quality strategies, clinician safety culture and leadership and, ultimately, waiting time and length of stay for patients. By focusing only on time-based measures of ED performance we risk punishing EDs that perform well on patient safety measures. We need to better understand the trade-offs between implementing safety culture and quality strategies and improving patient flow in the ED, and to place more emphasis on other ED performance measures in addition to time.

Do organization-level quality management systems influence department-level quality? A cross-sectional study across 32 large hospitals in Australia

Objective

Little is known about the influence that hospital quality systems have on quality at department level, in Australia and elsewhere. This study assessed the relationships between organizational-level quality management systems, and the extent to which hospital-level quality management systems and department-level quality management strategies are related.

Design

A multi-level, cross-sectional, mixed-method study.

Setting and participants

As part of the Deepening our Understanding of Quality in Australia (DUQuA) project, we invited all large hospitals in Australia (~200 or more beds) which provided acute myocardial infarction (AMI), hip fracture and stroke care. The quality managers of these hospitals were the respondents for one of seven measures of hospital quality management systems and strategies. Data across the six remaining measures were collected through site visits by external surveyors assessing the participating hospitals.

Main outcome measures

Relationships were assessed between three organization-level quality management system measures: a self-report measure assessing organization-level quality activities (quality management systems index, QMSI); externally assessed organization-level compliance to procedures used to plan, monitor and improve quality of care (quality management compliance index, QMCI); and externally assessed implementation of quality systems (clinical quality implementation index, CQII). Associations were also assessed between organization-level quality management systems and department-level quality management strategies: how clinical responsibilities are assigned for a particular condition; whether department organization processes are organized to facilitate evidence-based care recommendations; compliance with selected recommendations of international agencies; and whether clinical reviews are performed systematically.

Results

Of 78 invited hospitals, 32 participated in the study. QMSI was positively associated with QMCI and CQII, but after controlling for QMSI, no relationship was found between QMCI and CQII. There appears to be a cluster of relationships between QMSI and department-level measures, but this was not consistent across all departments.

Conclusion

This is the first national study undertaken in Australia to assess relationships within and between organization-level and department-level quality management systems. These quality management system tools align with many components of accreditation standards and may be useful for hospitals in continuously monitoring and driving improvement.

Descriptive Epidemiology of Safety Events at an Academic Medical Center

Background: Adverse safety events in healthcare are of great concern, and despite an increasing focus on the prevention of error and harm mitigation, the epidemiology of safety events remains incomplete. Methods: We performed an analysis of all reported safety events in an academic medical center using a voluntary incident reporting surveillance system for patient safety. Safety events were classified as: serious (reached the patient and resulted in moderate to severe harm or death); precursor (reached the patient and resulted in minimal or no detectable harm); and near miss (did not reach the patient). Results: During a three-year period, there were 31,817 events reported. Most of the safety events were precursor safety events (reached the patient and resulted in minimal harm or no detectable harm), corresponding to 77.3%. Near misses accounted for 10.8%, and unsafe conditions for 11.8%. The number of reported serious safety events was low, accounting for only 0.1% of all safety events. Conclusions: The reports analysis of these events should lead to a better understanding of risks in patient care and ways to mitigate it.

Validation of the patient measure of safety (PMOS) questionnaire in Australian public hospitals

Objective

Patients can provide a unique perspective on the safety of care in hospitals. Understanding that the extent to which the way hospitals are organized for quality and safety is associated with patient perceptions of care is becoming increasingly valued and necessary for the direction of targeted interventions across healthcare systems. The UK-developed patient measure of safety (PMOS) assesses eight domains of ward safety from the patient point of view and has recently been adapted and piloted in Australia. The aim of this study is to test the psychometric properties of PMOS-Australia (PMOS-A) amongst a large cohort of hospitalized patients.

Design

Cross-sectional questionnaire validation assessment.

Setting and participants

As part of the DUQuA project, the PMOS-A survey was distributed within acute myocardial infarction, hip fracture and stroke departments across 32 large public hospitals in Australia. Patients could complete the PMOS-A independently, or request the assistance of a family member/guardian, or staff on the wards—space was included to record mode of completion.

Main outcome measures

Confirmatory factor analysis (CFA) was undertaken on a calibration sample to generate the model, and a validation sample was used to cross-validate the model. A subset of only those participants who received assistance for PMOS-A completion was also tested using CFA on a calibration and validation sample. Model fit indices (chi-square to degrees of freedom ratio [Chi-square:DF], root mean square error of approximation [RMSEA], comparative fit indices [CFI], standardized root mean squared residual [SRMR]), Cronbach’s α, average inter-item correlations, construct reliability and cross-loadings were examined with reference to recommended thresholds to establish the extent of convergent validity and discriminant validity. A marker of criterion validity was assessed through testing associations between the PMOS-A and adherence to clinical guidelines.

Results

Across the calibration and validation samples of the full (N = 911) and assisted completers only subset (N = 490), three (Chi-square:DF, SRMR, RMSEA) of the four indices consistently or almost always met thresholds for acceptable model fit. CFI indices did not meet the recommended limits (0.72–0.78, against a target > 0.9). Positive relationships were found for all tests between PMOS-A and adherence to clinical guidelines, and these were significant when assessed in the calibration datasets for the full and assisted completion samples.

Conclusion

A sufficiently reliable and valid measure of patient perceptions of safety has been developed. These findings should provide adequate support to justify the use of this measure to assess patient perceptions of safety in Australian hospitals and can be modified for use elsewhere.

Older, vulnerable patient view: a pilot and feasibility study of the patient measure of safety (PMOS) with patients in Australia

OBJECTIVES

The UK-developed patient measure of safety (PMOS) is a validated tool which captures patient perceptions of safety in hospitals. We aimed (1) to investigate the extent to which the PMOS is appropriate for use with stroke, acute myocardial infarction (AMI) and hip fracture patients in Australian hospitals and (2) to pilot the PMOS for use in a large-scale, national study 'Deepening our Understanding of Quality in Australia' (DUQuA).

PARTICIPANTS

Stroke, AMI and hip fracture patients (n=34) receiving care in 3 wards in 1 large hospital.

METHODS

2 phases were conducted. First, a 'think aloud' study was used to determine the validity of PMOS with this population in an international setting, and to make amendments based on patient feedback. The second phase tested the revised measure to establish the internal consistency reliability of the revised subscales, and piloted the recruitment and administration processes to ensure feasibility of the PMOS for use in DUQuA.

RESULTS

Of the 43 questions in the PMOS, 13 (30%) were amended based on issues patients highlighted for improvement in phase 1. In phase 2, a total of 34 patients were approached and 29 included, with a mean age of 71.3 years (SD=16.39). Internal consistency reliability was established using interitem correlation and Cronbach's α for all but 1 subscale. The most and least favourably rated aspects of safety differed between the 3 wards. A study log was categorised into 10 key feasibility factors, including liaising with wards to understand operational procedures and identify patterns of patient discharge.

CONCLUSIONS

Capturing patient perceptions of care is crucial in improving patient safety. The revised PMOS is appropriate for use with vulnerable older adult groups. The findings from this study have informed key decisions made for the deployment of this measure as part of the DUQuA study.