Approach to improving quality: the role of quality measurement and a case study of the agency for healthcare research and quality pediatric quality indicators

Developing key performance indicators for a tertiary children's hospital network

Objective The aim of this study is to describe the experience of developing key performance indicators (KPIs) for Sydney Children's Hospital Network (SCHN), the largest paediatric healthcare entity in Australia. Methods Beginning with a published methodology, the process of developing KPIs involved five phases: (1) identification of potential KPIs referencing the organisational strategic plan and pre-existing internal and external documents; (2) consolidation into a pragmatic set; (3) analysis of potential KPIs against selection criteria; (4) mapping these back against the strategic plan and management structure; and (5) presentation to key stakeholders to ensure suitability and traction. Consistent with the strategic plan, a subset of indicators was selected to address quality of care for children from priority populations. Results A pragmatic list of 60 mandated and 50 potential KPIs was created from the 328 new and 397 existing potentially relevant KPIs generated by the executive team. Of these, 20 KPIs were selected as the most important; 65% were process measures. The majority of mandated KPIs were process measures. Of the KPIs selected to highlight inequities, there were proportionately more outcome measures (44% outcome, 27% process). Less than one-third could currently be measured by the organisation and were thus aspirational. Conclusion Developing a KPI suite requires substantial time, effort and organisational courage. A structured approach to performance measurement and improvement is needed to ensure a balanced suite of KPIs that can be expected to drive an organisation to improve child health outcomes. Future directions for SCHN include a systematic approach to implementation beyond the mandated KPIs, including KPIs that reflect equity and improved outcomes for priority populations, development of meaningful measures for the aspirational KPIs, adding structure KPIs and measurement of changes in child health outcomes related to the development of this KPI process. What is known about the topic? Health services are increasingly required to demonstrate accountability through KPIs. There is a body of literature on both theoretical frameworks for measuring performance and a long list of possible measures, however developing a meaningful suite of KPIs remains a significant challenge for individual organisations. What does this paper add? This paper describes lessons learned from the practical, pragmatic application of a published methodology to develop a suite of KPIs for the largest paediatric healthcare entity in Australia. It provides a select list of the highest-level KPIs selected by the organisation to stimulate further discussion among similar organisations in relation to KPI selection and implementation. What are the implications for practitioners? Developing and implementing a suite of meaningful KPIs for a large organisation requires courage, an understanding of health informatics, stakeholder engagement, stamina and pragmatism. The process we describe can be replicated and/or modified as needed, with discussion of key lessons learned to help practitioners plan ahead.

Pediatric Quality Metrics Related to Quality and Cost

The institution of pediatric quality in health care has grown in the past decade but continues to evolve. Children's health care emphasizes the importance of maintenance of health and prevention of illness, which can be measured based on immunization rates, routine or scheduled well care, and early intervention. Pediatric quality measures and indicators have become the basis for payment of services and a true goal to value. Designing processes such as pay-for-performance models, volume-based care, and coordination of care assist in assuring that children receive high-quality health care.

Development and Validation of the Agency for Healthcare Research and Quality Measures of Potentially Preventable Emergency Department (ED) Visits: The ED Prevention Quality Indicators for General Health Conditions

OBJECTIVE

To develop and validate rates of potentially preventable emergency department (ED) visits as indicators of community health.

DATA SOURCES

Agency for Healthcare Research and Quality, Healthcare Cost and Utilization Project 2008-2010 State Inpatient Databases and State Emergency Department Databases.

STUDY DESIGN

Empirical analyses and structured panel reviews.

METHODS

Panels of 14-17 clinicians and end users evaluated a set of ED Prevention Quality Indicators (PQIs) using a Modified Delphi process. Empirical analyses included assessing variation in ED PQI rates across counties and sensitivity of those rates to county-level poverty, uninsurance, and density of primary care physicians (PCPs).

PRINCIPAL FINDINGS

ED PQI rates varied widely across U.S. communities. Indicator rates were significantly associated with county-level poverty, median income, Medicaid insurance, and levels of uninsurance. A few indicators were significantly associated with PCP density, with higher rates in areas with greater density. A clinical and an end-user panel separately rated the indicators as having strong face validity for most uses evaluated.

CONCLUSIONS

The ED PQIs have undergone initial validation as indicators of community health with potential for use in public reporting, population health improvement, and research.

Costs of Venous Thromboembolism, Catheter-Associated Urinary Tract Infection, and Pressure Ulcer

OBJECTIVE

To estimate differences in the length of stay (LOS) and costs for comparable pediatric patients with and without venous thromboembolism (VTE), catheter-associated urinary tract infection (CAUTI), and pressure ulcer (PU).

METHODS

We identified at-risk children 1 to 17 years old with inpatient discharges in the Nationwide Inpatient Sample. We used a high dimensional propensity score matching method to adjust for case-mix at the patient level then estimated differences in the LOS and costs for comparable pediatric patients with and without VTE, CAUTI, and PU.

RESULTS

Incidence rates were 32 (VTE), 130 (CAUTI), and 3 (PU) per 10 000 at-risk patient discharges. Patients with VTE had an increased 8.1 inpatient days (95% confidence interval [CI]: 3.9 to 12.3) and excess average costs of $27 686 (95% CI: $11 137 to $44 235) compared with matched controls. Patients with CAUTI had an increased 2.4 inpatient days (95% CI: 1.2 to 3.6) and excess average costs of $7200 (95% CI: $2224 to $12 176). No statistical differences were found between patients with and without PU.

CONCLUSIONS

The significantly extended LOS highlights the substantial morbidity associated with these potentially preventable events. Hospitals seeking to develop programs targeting VTE and CAUTI should consider the improved turnover of beds made available by each event prevented.

Empirical examination of the indicator 'pediatric gastroenteritis hospitalization rate' based on administrative hospital data in Italy

Background

Awareness of the importance of strengthening investments in child health and monitoring the quality of services in the pediatric field is increasing. The Pediatric Quality Indicators developed by the US Agency for Healthcare Research and Quality (AHRQ), use hospital administrative data to identify admissions that could be avoided through high-quality outpatient care. Building on this approach, the purpose of this study is to perform an empirical examination of the ‘pediatric gastroenteritis admission rate’ indicator in Italy, under the assumption that lower admission rates are associated with better management at the primary care level and with overall better quality of care for children.

Methods

Following the AHRQ process for evaluating quality indicators, we examined age exclusion/inclusion criteria, selection of diagnostic codes, hospitalization type, and methodological issues for the ‘pediatric gastroenteritis admission rate’. The regional variability of hospitalizations was analyzed for Italian children aged 0–17 years discharged between January 1, 2009 and December 31, 2011. We considered hospitalizations for the following diagnoses: non-bacterial gastroenteritis, bacterial gastroenteritis and dehydration (along with a secondary diagnosis of gastroenteritis). The data source was the hospital discharge records database. All rates were stratified by age.

Results

In the study period, there were 61,130 pediatric hospitalizations for non-bacterial gastroenteritis, 5,940 for bacterial gastroenteritis, and 38,820 for dehydration. In <1-year group, the relative risk of hospitalization for non-bacterial gastroenteritis was 24 times higher than in adolescents, then it dropped to 14.5 in 1- to 4-year-olds and to 3.2 in 5- to 9-year-olds. At the national level, the percentage of admissions for bacterial gastroenteritis was small compared with non-bacterial, while including admissions for dehydration revealed a significant variability in diagnostic coding among regions that affected the regional performance of the indicator.

Conclusions

For broadest application, we propose a ‘pediatric gastroenteritis admission rate’ that consists of including bacterial gastroenteritis and dehydration diagnoses in the numerator, as well as infants aged <3 months. We also suggest adjusting for age and including day hospital admissions. Future evaluation by a clinical panel at the national level might be helpful to determine appropriate application for such measures, and make recommendations to policy makers.

Empirical validation of the "Pediatric Asthma Hospitalization Rate" indicator

Background

Quality assessment in pediatric care has recently gained momentum. Although many of the approaches to indicator development are similar regardless of the population of interest, few nationwide sets of indicators specifically designed for assessment of primary care of children exist. We performed an empirical analysis of the validity of “Pediatric Asthma Hospitalization Rate” indicator under the assumption that lower admission rates are associated with better performance of primary health care.

Methods

The validity of “Pediatric Asthma Hospitalization Rate” indicator proposed by the Agency for Healthcare Research and Quality in the Italian context was investigated with a focus on selection of diagnostic codes, hospitalization type, and risk adjustment. Seasonality and regional variability of hospitalization rates for asthma were analyzed for Italian children aged 2–17 years discharged between January 1, 2009, and December 31, 2011 using the hospital discharge records database. Specific rates were computed for age classes: 2–4, 5–9, 10–14, 15–17 years.

Results

In the years 2009–2011 the number of pediatric hospitalizations for asthma was 14,389 (average annual rate: 0.52 per 1,000) with a large variability across regions. In children aged 2–4 years, the risk of hospitalization for asthma was 14 times higher than in adolescents, then it dropped to 4 in 5- to 9-year-olds and to 1.1 in 10- to 14-year-olds. The inclusion of diagnoses of bronchitis revealed that asthma and bronchitis are equally represented as causes of hospital admissions and have a similar seasonality in preschool children, while older age groups experience hospital admissions mainly in spring and fall, this pattern being consistent with a diagnosis of atopic asthma. Rates of day hospital admissions for asthma were up to 5 times higher than the national average in Liguria and some Southern regions, and close to zero in some Northern regions.

Conclusions

The patterns of hospitalization for pediatric asthma in Italy showed that at least two different indicators are needed to measure accurately the quality of care provided to children. The candidate indicators should also include day hospital admissions to better assess accessibility. Future evaluation by a structured clinical panel review at the national level might be helpful to refine indicator definitions and risk groupings, to determine appropriate application for such measures, and to make recommendations to policy makers.

Understanding and benchmarking health service achievement of policy goals for chronic disease

BACKGROUND

Key challenges in benchmarking health service achievement of policy goals in areas such as chronic disease are: 1) developing indicators and understanding how policy goals might work as indicators of service performance; 2) developing methods for economically collecting and reporting stakeholder perceptions; 3) combining and sharing data about the performance of organizations; 4) interpreting outcome measures; 5) obtaining actionable benchmarking information. This study aimed to explore how a new Boolean-based small-N method from the social sciences-Qualitative Comparative Analysis or QCA-could contribute to meeting these internationally shared challenges.

METHODS

A 'multi-value QCA' (MVQCA) analysis was conducted of data from 24 senior staff at 17 randomly selected services for chronic disease, who provided perceptions of 1) whether government health services were improving their achievement of a set of statewide policy goals for chronic disease and 2) the efficacy of state health office actions in influencing this improvement. The analysis produced summaries of configurations of perceived service improvements.

RESULTS

Most respondents observed improvements in most areas but uniformly good improvements across services were not perceived as happening (regardless of whether respondents identified a state health office contribution to that improvement). The sentinel policy goal of using evidence to develop service practice was not achieved at all in four services and appears to be reliant on other kinds of service improvements happening.

CONCLUSIONS

The QCA method suggested theoretically plausible findings and an approach that with further development could help meet the five benchmarking challenges. In particular, it suggests that achievement of one policy goal may be reliant on achievement of another goal in complex ways that the literature has not yet fully accommodated but which could help prioritize policy goals. The weaknesses of QCA can be found wherever traditional big-N statistical methods are needed and possible, and in its more complex and therefore difficult to empirically validate findings. It should be considered a potentially valuable adjunct method for benchmarking complex health policy goals such as those for chronic disease.

Agency for Healthcare Research and Quality pediatric indicators as a quality metric for surgery in children: do they predict adverse outcomes?

BACKGROUND/PURPOSE

The pediatric quality indicators (PDIs) were developed by the Agency for Healthcare Research and Quality to compare patient safety and quality of pediatric care. These are being considered for mandatory reporting as well as pay-for-performance efforts. The present study evaluates the PDIs' predictive value for surgical outcomes in children.

METHODS

A cross-sectional study was performed using nationwide inpatient data from 1988 to 2007. Patients younger than 18 years with an inpatient surgical procedure were included and evaluated for 10 PDIs. Odds ratios for mortality, increase in length of stay, and total charges were calculated using multivariate regression adjusting for age, sex, race, region, hospital type, and comorbidities.

RESULTS

A total of 1,964,456 pediatric discharges were included. Mortality rates were 5.4% for patients with at least 1 PDI and 0.6% for those with none. Multivariate analysis showed that occurrence of any PDI was associated with a 20% increased risk of mortality. The PDIs were associated with an increased length of stay and total hospital charges.

CONCLUSION

The present study shows that PDIs are associated with increased mortality risk as well as increased hospital stay and total hospital charges. This provides positive evidence for the utility of these indicators as metrics for quality and patient safety.