Detection of postoperative respiratory failure: how predictive is the Agency for Healthcare Research and Quality's Patient Safety Indicator?

A scoping review of the methodological approaches used in retrospective chart reviews to validate adverse event rates in administrative data

Patient safety is a key quality issue for health systems. Healthcare acquired adverse events (AEs) compromise safety and quality; therefore, their reporting and monitoring is a patient safety priority. Although administrative datasets are potentially efficient tools for monitoring rates of AEs, concerns remain over the accuracy of their data. Chart review validation studies are required to explore the potential of administrative data to inform research and health policy. This review aims to present an overview of the methodological approaches and strategies used to validate rates of AEs in administrative data through chart review. This review was conducted in line with the Joanna Briggs Institute methodological framework for scoping reviews. Through database searches, 1054 sources were identified, imported into Covidence, and screened against the inclusion criteria. Articles that validated rates of AEs in administrative data through chart review were included. Data were extracted, exported to Microsoft Excel, arranged into a charting table, and presented in a tabular and descriptive format. Fifty-six studies were included. Most sources reported on surgical AEs; however, other medical specialties were also explored. Chart reviews were used in all studies; however, few agreed on terminology for the study design. Various methodological approaches and sampling strategies were used. Some studies used the Global Trigger Tool, a two-stage chart review method, whilst others used alternative single-, two-stage, or unclear approaches. The sources used samples of flagged charts (n = 24), flagged and random charts (n = 11), and random charts (n = 21). Most studies reported poor or moderate accuracy of AE rates. Some studies reported good accuracy of AE recording which highlights the potential of using administrative data for research purposes. This review highlights the potential for administrative data to provide information on AE rates and improve patient safety and healthcare quality. Nonetheless, further work is warranted to ensure that administrative data are accurate. The variation of methodological approaches taken, and sampling techniques used demonstrate a lack of consensus on best practice; therefore, further clarity and consensus are necessary to develop a more systematic approach to chart reviewing.

The Impact of Meaningful Use and Electronic Health Records on Hospital Patient Safety

The HITECH Act aimed to leverage Electronic Health Records (EHRs) to improve efficiency, quality, and patient safety. Patient safety and EHR use have been understudied, making it difficult to determine if EHRs improve patient safety. The objective of this study was to determine the impact of EHRs and attesting to Meaningful Use (MU) on Patient Safety Indicators (PSIs). A multivariate regression analysis was performed using a generalized linear model method to examine the impact of EHR use on PSIs. Fully implemented EHRs not attesting to MU had a positive impact on three PSIs, and hospitals that attested to MU had a positive impact on two. Attesting to MU or having a fully implemented EHR were not drivers of PSI-90 composite score, suggesting that hospitals may not see significant differences in patient safety with the use of EHR systems as hospitals move towards pay-for-performance models. Policy and practice may want to focus on defining metrics and PSIs that are highly preventable to avoid penalizing hospitals through reimbursement, and work toward adopting advanced analytics to better leverage EHR data. These findings will assist hospital leaders to find strategies to better leverage EHRs, rather than relying on achieving benchmarks of MU objectives.

Outcomes and risk factors for delayed-onset postoperative respiratory failure: a multi-center case-control study by the University of California Critical Care Research Collaborative (UC3RC)

BACKGROUND

Few interventions are known to reduce the incidence of respiratory failure that occurs following elective surgery (postoperative respiratory failure; PRF). We previously reported risk factors associated with PRF that occurs within the first 5 days after elective surgery (early PRF; E-PRF); however, PRF that occurs six or more days after elective surgery (late PRF; L-PRF) likely represents a different entity. We hypothesized that L-PRF would be associated with worse outcomes and different risk factors than E-PRF.

METHODS

This was a retrospective matched case-control study of 59,073 consecutive adult patients admitted for elective non-cardiac and non-pulmonary surgical procedures at one of five University of California academic medical centers between October 2012 and September 2015. We identified patients with L-PRF, confirmed by surgeon and intensivist subject matter expert review, and matched them 1:1 to patients who did not develop PRF (No-PRF) based on hospital, age, and surgical procedure. We then analyzed risk factors and outcomes associated with L-PRF compared to E-PRF and No-PRF.

RESULTS

Among 95 patients with L-PRF, 50.5% were female, 71.6% white, 27.4% Hispanic, and 53.7% Medicare recipients; the median age was 63 years (IQR 56, 70). Compared to 95 matched patients with No-PRF and 319 patients who developed E-PRF, L-PRF was associated with higher morbidity and mortality, longer hospital and intensive care unit length of stay, and increased costs. Compared to No-PRF, factors associated with L-PRF included: preexisiting neurologic disease (OR 4.36, 95% CI 1.81-10.46), anesthesia duration per hour (OR 1.22, 95% CI 1.04-1.44), and maximum intraoperative peak inspiratory pressure per cm H₂0 (OR 1.14, 95% CI 1.06-1.22).

CONCLUSIONS

We identified that pre-existing neurologic disease, longer duration of anesthesia, and greater maximum intraoperative peak inspiratory pressures were associated with respiratory failure that developed six or more days after elective surgery in adult patients (L-PRF). Interventions targeting these factors may be worthy of future evaluation.

Postoperative Respiratory Failure After Elective Abdominal Surgery: A Case-Control Study

INTRODUCTION

Postoperative respiratory failure (PRF) contributes significantly to morbidity and mortality. We sought to identify patient characteristics and perioperative risk factors associated with PRF in patients undergoing elective abdominal surgery to improve patient outcomes.

METHODS

We retrospectively reviewed patients undergoing elective abdominal surgery from 2011 to 2016 at our institution. An experimental group consisting of adult patients with the Patient Safety Indicator 11 diagnosis of PRF was compared with a time-matched control group.

RESULTS

Each group consisted of 233 patients. Comorbidities associated with PRF included ascites, coronary artery disease, chronic kidney disease, chronic obstructive pulmonary disease, diabetes mellitus type II, hypertension, and hypoalbuminemia (P < 0.05). American Society of Anesthesiologists score IV (20.2% versus 3.95%; P < 0.001), operative time (4.13 versus 2.55 h; P < 0.001), laparotomy with open operation (77.7% versus 45.5%; P < 0.001), and net intraoperative fluid balance (3635 versus 2410 mL; P < 0.001) were higher in patients with PRF. On multivariate analysis, age, American Society of Anesthesiologists score, chronic obstructive pulmonary disease, diabetes mellitus type II, laparotomy, and net intraoperative fluid balance maintained significance (P < 0.05).

CONCLUSIONS

We identified contributing pre- and intra-operative risk factors for PRF undergoing elective abdominal surgery. These findings may help identify those at increased risk for respiratory failure and mitigate complications.

Risk Factors Associated With Early Postoperative Respiratory Failure: A Matched Case-Control Study

BACKGROUND

Postoperative respiratory failure is the most common serious postoperative pulmonary complication, yet little is known about factors that can reduce its incidence. We sought to elucidate modifiable factors associated with respiratory failure that developed within the first 5 d after an elective operation.

MATERIALS AND METHODS

Matched case-control study of adults who had an operation at five academic medical centers between October 1, 2012 and September 30, 2015. Cases were identified using administrative data and confirmed via chart review by critical care clinicians. Controls were matched 1:1 to cases based on hospital, age, and surgical procedure.

RESULTS

Our total sample (n = 638) was 56.4% female, 71.3% white, and had a median age of 62 y (interquartile range 51, 70). Factors associated with early postoperative respiratory failure included male gender (odds ratio [OR] 1.72, 95% confidence interval [CI] 1.12-2.63), American Society of Anesthesiologists class III or greater (OR 2.85, 95% CI 1.74-4.66), greater number of preexisting comorbidities (OR 1.14, 95% CI 1.004-1.30), increased operative duration (OR 1.14, 95% CI 1.06-1.22), increased intraoperative positive end-expiratory pressure (OR 1.23, 95% CI 1.13-1.35) and tidal volume (OR 1.13, 95% CI 1.004-1.27), and greater net fluid balance at 24 h (OR 1.17, 95% CI 1.07-1.28).

CONCLUSIONS

We found greater intraoperative ventilator volume and pressure and 24-h fluid balance to be potentially modifiable factors associated with developing early postoperative respiratory failure. Further studies are warranted to independently verify these risk factors, explore their role in development of early postoperative respiratory failure, and potentially evaluate targeted interventions.

Postoperative respiratory failure: An update on the validity of the Agency for Healthcare Research and Quality Patient Safety Indicator 11 in an era of clinical documentation improvement programs

BACKGROUND

Administrative data can be used to identify cases of postoperative respiratory failure (PRF). We aimed to determine if recent changes to the Agency for Healthcare Research and Quality Patient Safety Indicator 11 (PSI 11) and adoption of clinical documentation improvement programs have improved the validity of PSI 11. We also analyzed reasons why PSI 11 was falsely triggered.

STUDY DESIGN

Cross-sectional study of all eligible discharges using health record data from five academic medical centers between October 1, 2012 and September 30, 2015.

RESULTS

Of 437 flagged records, 434 (99.3%) were accurately coded and 414 (94.7%) represented true clinical PRF. None of the false positive records involved respiratory failure present on admission. Most (78.3%) false positive records required airway protection but did not have respiratory failure.

CONCLUSION

The validity of PSI 11 has improved with recent changes to the code criterion and adoption of clinical documentation improvement programs.

Agency for Healthcare Research and Quality Patient Safety Indicators and Mortality in Surgical Patients

Patient safety indicators (PSI), developed by the Agency for Healthcare Research and Quality, use administrative billing data to measure and compare patient safety events at medical centers. We retrospectively examined whether PSIs accurately reflect patients’ risk of mortality, hospital length of stay, and intensive care unit (ICU) requirements at an academic medical center. Surgical patient records with PSIs were reviewed between October 2011 and September 2012 at our urban academic medical center. Primary outcomes studied included mortality, hospital length of stay, and ICU requirements. Subset analysis was performed for each PSI and its association with the outcome measures. PSIs were more common among surgical patients who died as compared with those alive at discharge (35.3 vs 2.7 PSIs/100 patients, P < 0.01). Although patients who died with PSIs had shorter hospital courses, they had a significantly greater ICU requirement than those without a PSI (96.0 vs 61.1%, P < 0.01) and patients who were alive at discharge (96.0 vs 48.0%, P < 0.01). The most frequently associated PSIs with mortality were postoperative metabolic derangements (41.7%), postoperative sepsis (38.5%), and pressure ulcers (33.3%). PSIs occur at a higher frequency in surgical patients who die and are associated with increased ICU requirements.

Opioid Use Disorders and the Risk of Postoperative Pulmonary Complications

BACKGROUND

As the rate of opioid use disorders continues to rise, perioperative physicians are increasingly faced with the challenge of providing analgesia to these patients after surgery. Due to the likelihood of opioid dose escalation in the perioperative period, we hypothesized that opioid-dependent patients would be at increased risk for postoperative pulmonary complications.

METHODS

A retrospective cross-sectional analysis of patients undergoing 6 representative elective surgical procedures was performed using the Nationwide Inpatient Sample from 2002 to 2011. The primary outcome was a composite including prolonged mechanical ventilation, reintubation, and acute respiratory failure. Secondary outcomes were length of stay, in-hospital mortality, and total hospital costs. Both multivariable logistic regression and propensity score matching were used to determine the impact of opioid use disorder on outcomes.

RESULTS

The total sample-weighted cohort consisted of 7,533,050 patients. Patients with opioid use disorders were more likely to suffer pulmonary complications, with a frequency of 4.2% compared to 1.6% in the nonopioid-dependent group (P < .001), and had a 1.62 times higher odds (95% confidence interval [CI], 1.16-2.27) in multivariable regression analysis. In a secondary subgroup analysis, only patients undergoing a colectomy had a greater odds of suffering pulmonary complications (odds ratio, 2.64; 95% CI, 1.42-4.91; P = .0021). Additionally, patients with an opioid use disorder had a longer length of stay (0.84 days [95% CI, 0.52-1.16; P < .001]) and greater costs ($1816 [95% CI, 935-2698; P < .001]).

CONCLUSIONS

This study demonstrates that patients with opioid use disorders are at increased risk for postoperative pulmonary complications, and have prolonged length of stay and resource utilization. Further research is needed regarding interventions to reduce the risk of complications in this subset of patients.

Key considerations when using health insurance claims data in advanced data analyses: an experience report

Health claims have become a popular source of data for healthcare analytics, with numerous applications ranging from disease burden estimation and policy evaluation to drug event detection and advanced predictive analytics. Independent of the application, a researcher utilising claims information will likely encounter challenges in using the data, which include dealing with several coding systems and coding irregularities. We highlight some of these challenges and approaches for successful analysis that may reduce implementation time and help in avoiding common pitfalls. We describe the experiences of a group of academic researchers in using an extensive seven-year repository of US medical and pharmaceutical claims data in a research study, and provide an overview of the challenges encountered with handling claims records for data analysis while sharing suggestions on how to address these challenges. To illustrate our experiences, we use the example of defining episodes of care for a bundled payment reimbursement system in the US context.

The Use of Claims-Based Data in Inpatient Public Reporting and Pay-for-Performance Programs: Is There Opportunity for Improvement?

PURPOSE

This study examined whether self-reported, facility-based data validation practices for claims submissions of cases flagged as Patient Safety Indicators (PSIs) match professional and regulatory standards.

METHODS

The National Association of Healthcare Quality members who work in an inpatient setting were invited to complete an anonymous survey to self-report their practices around facility-based data validation of PSI cases.

RESULTS

The authors found widespread variation in how PSI administrative data are internally validated; inconsistency in the education and training required of staff who participate in this process; and relatively poor compliance with physician query guidelines and documentation amendment standards.

CONCLUSIONS

The self-described wide variation and nonadherence to professional and regulatory standards within the facility-based validation process for PSIs raise concerns about the use of these data to make meaningful judgments about quality and safety. The authors recommend a standardized approach to reporting and validation be implemented for use of PSIs in public reporting and pay-for-performance programs.

Knowledge Gaps in the Perioperative Management of Adults with Obstructive Sleep Apnea and Obesity Hypoventilation Syndrome. An Official American Thoracic Society Workshop Report

The purpose of this workshop was to identify knowledge gaps in the perioperative management of obstructive sleep apnea (OSA) and obesity hypoventilation syndrome (OHS). A single-day meeting was held at the American Thoracic Society Conference in May, 2016, with representation from many specialties, including anesthesiology, perioperative medicine, sleep, and respiratory medicine. Further research is urgently needed as we look to improve health outcomes for these patients and reduce health care costs. There is currently insufficient evidence to guide screening and optimization of OSA and OHS in the perioperative setting to achieve these objectives. Patients who are at greatest risk of respiratory or cardiac complications related to OSA and OHS are not well defined, and the effectiveness of monitoring and other interventions remains to be determined. Centers involved in sleep research need to develop collaborative networks to allow multicenter studies to address the knowledge gaps identified below.

Validity of the Agency for Health Care Research and Quality Patient Safety Indicators and the Centers for Medicare and Medicaid Hospital-acquired Conditions: A Systematic Review and Meta-Analysis

BACKGROUND

The Agency for Health Care Research and Quality Patient Safety Indicators (PSIs) and Centers for Medicare and Medicaid Services Hospital-acquired Conditions (HACs) are increasingly being used for pay-for-performance and public reporting despite concerns over their validity. Given the potential for these measures to misinform patients, misclassify hospitals, and misapply financial and reputational harm to hospitals, these need to be rigorously evaluated. We performed a systematic review and meta-analysis to assess PSI and HAC measure validity.

METHODS

We searched MEDLINE and the gray literature from January 1, 1990 through January 14, 2015 for studies that addressed the validity of the HAC measures and PSIs. Secondary outcomes included the effects of present on admission (POA) modifiers, and the most common reasons for discrepancies. We developed pooled results for measures evaluated by ≥3 studies. We propose a threshold of 80% for positive predictive value or sensitivity for pay-for-performance and public reporting suitability.

RESULTS

Only 5 measures, Iatrogenic Pneumothorax (PSI 6/HAC 17), Central Line-associated Bloodstream Infections (PSI 7), Postoperative hemorrhage/hematoma (PSI 9), Postoperative deep vein thrombosis/pulmonary embolus (PSI 12), and Accidental Puncture/Laceration (PSI 15), had sufficient data for pooled meta-analysis. Only PSI 15 (Accidental Puncture and Laceration) met our proposed threshold for validity (positive predictive value only) but this result was weakened by considerable heterogeneity. Coding errors were the most common reasons for discrepancies between medical record review and administrative databases. POA modifiers may improve the validity of some measures.

CONCLUSION

This systematic review finds that there is limited validity for the PSI and HAC measures when measured against the reference standard of a medical chart review. Their use, as they currently exist, for public reporting and pay-for-performance, should be publicly reevaluated in light of these findings.

Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicator for Postoperative Respiratory Failure (PSI 11) does not identify accurately patients who received unsafe care

BACKGROUND

The Agency for Healthcare Research and Quality Patient Safety Indicator 11 is used to identify postoperative respiratory failure events and detect areas for quality improvement. This study examines the accuracy of Patient Safety Indicator 11 in identifying clinically valid patient safety events.

METHODS

All cases flagged for Patient Safety Indicator 11 from July 2013 to July 2015 by Agency for Healthcare Research and Quality QI Version 4.5 including International Classification of Diseases-9 codes were evaluated. Code-confirmed cases underwent independent review by 2 physicians. Inpatient electronic medical records were used to identify clinical factors for postoperative respiratory failure in each case to determine if postoperative respiratory failure was a result of unsafe care. The clinical true-positive rate and positive predictive value were calculated.

RESULTS

A total of 166 postoperative respiratory failure cases were reviewed; 51 were recoded and reversed due to coding or documentation errors; 115 cases met the Agency for Healthcare Research and Quality definition of postoperative respiratory failure. A total of 71 (61.7%) of the 115 cases were false positives and did not reflect unsafe care, while 44 cases were true positives with a positive predictive value of 38.3%. χ(2) analysis did not reveal an association between demographics, clinical characteristics, or operative procedure with true-positive cases.

CONCLUSION

Administrative coding data for Agency for Healthcare Research and Quality Patient Safety Indicator 11 do not identify accurately patients who received unsafe care when taking into account unpreventable clinical factors causing postoperative respiratory failure. The use of Agency for Healthcare Research and Quality Patient Safety Indicator 11 as a hospital performance measure should be reconsidered until inclusion and exclusion criteria are revised.

Intraoperative protective mechanical ventilation and risk of postoperative respiratory complications: hospital based registry study

OBJECTIVE

To evaluate the effects of intraoperative protective ventilation on major postoperative respiratory complications and to define safe intraoperative mechanical ventilator settings that do not translate into an increased risk of postoperative respiratory complications.

DESIGN

Hospital based registry study.

SETTING

Academic tertiary care hospital and two affiliated community hospitals in Massachusetts, United States.

PARTICIPANTS

69,265 consecutively enrolled patients over the age of 18 who underwent a non-cardiac surgical procedure between January 2007 and August 2014 and required general anesthesia with endotracheal intubation.

INTERVENTIONS

Protective ventilation, defined as a median positive end expiratory pressure (PEEP) of 5 cmH2O or more, a median tidal volume of less than 10 mL/kg of predicted body weight, and a median plateau pressure of less than 30 cmH2O.

MAIN OUTCOME MEASURE

Composite outcome of major respiratory complications, including pulmonary edema, respiratory failure, pneumonia, and re-intubation.

RESULTS

Of the 69,265 enrolled patients 34,800 (50.2%) received protective ventilation and 34,465 (49.8%) received non-protective ventilation intraoperatively. Protective ventilation was associated with a decreased risk of postoperative respiratory complications in multivariable regression (adjusted odds ratio 0.90, 95% confidence interval 0.82 to 0.98, P=0.013). The results were similar in the propensity score matched cohort (odds ratio 0.89, 95% confidence interval 0.83 to 0.97, P=0.004). A PEEP of 5 cmH2O and median plateau pressures of 16 cmH2O or less were associated with the lowest risk of postoperative respiratory complications.

CONCLUSIONS

Intraoperative protective ventilation was associated with a decreased risk of postoperative respiratory complications. A PEEP of 5 cmH2O and a plateau pressure of 16 cmH2O or less were identified as protective mechanical ventilator settings. These findings suggest that protective thresholds differ for intraoperative ventilation in patients with normal lungs compared with those used for patients with acute lung injury.

Using estimated true safety event rates versus flagged safety event rates: does it change hospital profiling and payment?

OBJECTIVE

To assess whether use of the AHRQ Patient Safety Indicator (PSI) composite measure versus modified composite measures leads to changes in hospital profiles and payments.

DATA SOURCES/STUDY SETTING

Retrospective analysis of 2010 Veterans Health Administration discharge data.

STUDY DESIGN

We used the AHRQ PSI software (v4.2) to obtain PSI-flagged events and composite scores for all 151 hospitals in the database (n = 517,814 hospitalizations). We compared the AHRQ PSI composite to two modified composites that estimated "true safety events" from previous chart abstraction findings: one with modified numerators based on the positive predictive value (PPV) of each PSI, and one with similarly modified numerators but whose denominators were based on the expected fraction of PSI-eligible cases that remained after removing those PSIs that were present-on-admission (POA).

PRINCIPAL FINDINGS

Although a small percentage (5-6 percent) of hospitals changed outlier status based on modified PSI composites, some of these changes were substantial; 30 and 19 percent of hospitals changed ≥20 ranks after adjustment for PPVs and POA flags, respectively. We estimate that 33 percent of hospitals would see a change of at least 10 percent in performance payments.

CONCLUSIONS

Changes in hospital profiles and payments would be substantial for some hospitals if the PSI composite score used weights reflecting the relative prevalence of true versus flagged events.

Detecting adverse events in surgery: comparing events detected by the Veterans Health Administration Surgical Quality Improvement Program and the Patient Safety Indicators

BACKGROUND

The Patient Safety Indicators (PSIs) use administrative data to screen for select adverse events (AEs). In this study, VA Surgical Quality Improvement Program (VASQIP) chart review data were used as the gold standard to measure the criterion validity of 5 surgical PSIs. Independent chart review was also used to determine reasons for PSI errors.

METHODS

The sensitivity, specificity, and positive predictive value of PSI software version 4.1a were calculated among Veterans Health Administration hospitalizations (2003-2007) reviewed by VASQIP (n = 268,771). Nurses re-reviewed a sample of hospitalizations for which PSI and VASQIP AE detection disagreed.

RESULTS

Sensitivities ranged from 31% to 68%, specificities from 99.1% to 99.8%, and positive predictive values from 31% to 72%. Reviewers found that coding errors accounted for some PSI-VASQIP disagreement; some disagreement was also the result of differences in AE definitions.

CONCLUSIONS

These results suggest that the PSIs have moderate criterion validity; however, some surgical PSIs detect different AEs than VASQIP. Future research should explore using both methods to evaluate surgical quality.

Detecting postoperative hemorrhage or hematoma from administrative data: the performance of the AHRQ Patient Safety Indicator

BACKGROUND

Patient Safety Indicator (PSI) 9, "postoperative hemorrhage or hematoma" (PHH), of the US Agency for Healthcare Research and Quality has been considered for public quality of care reporting. We sought to evaluate its performance in detecting true complications.

METHODS

We conducted a retrospective, cross-sectional study of hospitalizations that met PSI 9 eligibility criteria. We sampled records flagged positive and negative by PSI 9 from a diverse set of 31 hospitals between February 2006, and June 2009. Trained abstractors reviewed medical records using standard instruments. We determined the sensitivity, specificity, and positive (PPV) and negative (NPV) predictive values of the indicator.

RESULTS

Of 181 analyzable records flagged by PSI 9, 168 (93%; weighted PPV, 95% [95% confidence interval (CI), 90-98%]) involved an accurately coded event, but only 126 (70%; weighted PPV, 78% [95% CI, 58-90%]) represented true PHH. Thirty-two false positives involved only intraoperative hemorrhage. Among true positives, hypotension occurred in 28% and death attributed to the PHH in 4%. Thirty-two of 281 records flagged negative by PSI 9 (but enriched with questionably negative records) represented true PHH. The indicator's sensitivity was 42% (95% CI, 23-64%), specificity 99.9% (95% CI, 99.8-100%), and NPV 99.7% (95% CI, 99.0-99.9%). Modifying the indicator to include additional procedure codes improved both sensitivity (85% [95% CI, 67-94%]) and PPV (76% [95% CI, 60-88%]).

CONCLUSION

PSI 9 holds promise in detecting serious, possibly preventable complications. The indicator might be improved by specification of the 998.11 hemorrhage code to exclude purely intraoperative events and addition of procedure codes to the indicator's numerator criteria.

Comparison of the Agency for Healthcare Research and Quality Patient Safety Indicator Rates Among Veteran Dual Users

This study compares rates of 11 Agency for Healthcare Research and Quality Patient Safety Indicators (PSIs) among 266 203 veteran dual users (ie, those with hospitalizations in both the Veterans Health Administration [VA] and the private sector through Medicare fee-for-service coverage) during 2002 to 2007. PSI risk-adjusted rates were calculated using the PSI software (version 3.1a). Rates of pressure ulcer, central venous catheter-related bloodstream infections, and postoperative sepsis, areas in which the VA has focused quality improvement efforts, were found to be significantly lower in the VA than in the private sector. VA had significantly higher rates for 7 of the remaining 8 PSIs, although the rates of only 2 PSIs (postoperative hemorrhage/hematoma and accidental puncture or laceration) remained higher in the VA after sensitivity analyses were conducted. A better understanding of system-level differences in coding practices and patient severity, poorly documented in administrative data, is needed before conclusions about differences in quality can be drawn.

Validity of Agency for Healthcare Research and Quality Patient Safety Indicators at an Academic Medical Center

The Agency for Healthcare Research and Quality developed Patient Safety Indicators (PSI) to screen for in-hospital complications and patient safety events through International Classification of Diseases, 9th Revision, Clinical Modification coding. The purpose of this study was to validate 10 common surgically related PSIs at our academic medical center and investigate the causes for inaccuracies. We reviewed patient records between October 2011 and September 2012 at our urban academic medical center for 10 common surgically related PSIs. The records were reviewed for incorrectly identified PSIs and a subset was further reviewed for the contributing factors. There were 93,169 charts analyzed for PSIs and 358 PSIs were identified (3.84 per 1000 cases). The overall positive predictive value (PPV) was 83 per cent (95% confidence interval 79 to -86%). The lowest PPVs were associated with catheter-related bloodstream infections (67%), postoperative respiratory failure (71%), and pressure ulcers (79%). The most common contributing factors for incorrect PSIs were coding errors (30%), documentation errors (19%), and insufficient criteria for PSI in the chart (16%). We conclude that the validity of PSIs is low and could be improved by increased education for clinicians and coders. In their current form, PSIs remain suboptimal for widespread use in public reporting and pay-for-performance evaluation.

How valid is the AHRQ Patient Safety Indicator "postoperative physiologic and metabolic derangement"?

BACKGROUND

The Agency for Healthcare Research and Quality Patient Safety Indicator postoperative physiologic and metabolic derangement (PMD) uses ICD-9-CM codes to screen for potentially preventable acute kidney injury (AKI) requiring dialysis plus diabetes-related complications after elective surgery. Data on PMD's accuracy in identifying true events are limited. We examined the indicator's positive predictive value (PPV) in the Veterans Health Administration (VA).

STUDY DESIGN

Trained abstractors reviewed medical records of 119 PSI software-flagged PMD cases. We calculated PPVs overall and separately for renal- and diabetes-related complications. We also examined false positives to determine reasons for incorrect identification, and true positives to determine PMD-related outcomes and risk factors.

RESULTS

Overall 75 cases were true positives (PPV 63%, 95% CI 54% to 72%); 73 of 104 AKI cases were true positives (PPV 70%, 60% to 79%); only 2 of 15 diabetes cases were true positives (PPV 13%, 2% to 40%). Of all false positives, 70% represented nonelective admissions and 23% had the complication present on admission. Of AKI true positives, 37% died and 26% were discharged on dialysis; 55% had chronic kidney disease (≥ stage 3) present on admission. Cardiac surgery represented the largest category of AKI-associated index procedures (30%). AKI was most commonly attributed to perioperative renal hypoperfusion (84% of true positives), followed by nephrotoxins (33%) including contrast (11%).

CONCLUSIONS

Due to its low PPV, we recommend removing diabetes complications from the indicator and focusing on AKI. PMD's PPV could be significantly improved by using present-on-admission codes, and specific to the VA, by introduction of admission status codes. Many PMD-identified cases appeared to be at high risk based on patient- and procedure-related factors. The degree to which such cases are truly preventable events requires further assessment.

How valid is the AHRQ Patient Safety Indicator "postoperative hemorrhage or hematoma"?

BACKGROUND

Postoperative hemorrhage or hematoma (PHH), an Agency for Healthcare Research and Quality Patient Safety Indicator, uses administrative data to detect cases of potentially preventable postsurgical bleeding requiring a reparative procedure. How accurately it identifies true events is unknown. We therefore determined PHH's positive predictive value.

STUDY DESIGN

Using Patient Safety Indicator software (v.3.1a) and fiscal year 2003-2007 discharge data from 28 Veterans Health Administration hospitals, we identified 112 possible cases of PHH. Based on medical record abstraction, we characterized cases as true (TPs) or false positives (FPs), calculated positive predictive value, and analyzed FPs to ascertain reasons for incorrect identification and TPs to determine PHH-associated clinical consequences and risk factors.

RESULTS

Eighty-four cases were TPs (positive predictive value, 75%; 95% CI, 66-83%); 63% had a hematoma diagnosis, 30% had a hemorrhage diagnosis, 7% had both. Reasons for FPs included events present on admission (29%); hemorrhage/hematoma identified and controlled during the original procedure rather than postoperatively (21%); or postoperative hemorrhage/hematoma that did not require a procedure (18%). Most TPs (82%) returned to the operating room for hemorrhage/hematoma management; 64% required blood products and 7% died in-hospital. The most common index procedures resulting in postoperative hemorrhage/hematoma were vascular (38%); 56% were performed by a physician-in-training (under supervision). We found no substantial association between physician training status or perioperative anticoagulant use and bleeding risk.

CONCLUSIONS

PHH's accuracy could be improved by coding enhancements, such as adopting present on admission codes or associating a timing factor with codes dealing with bleeding control. The ability of PHH to identify events representing quality of care problems requires additional evaluation.

How valid is the AHRQ Patient Safety Indicator "postoperative respiratory failure"?

BACKGROUND

The Agency for Healthcare Research and Quality Patient Safety Indicator postoperative respiratory failure (PRF) uses administrative data to screen for potentially preventable respiratory failure after elective surgery based on a respiratory failure diagnosis or an intubation or ventilation procedure code. Data on PRF accuracy in identifying true events is scant; a recent study using University HealthSystem Consortium data found a positive predictive value (PPV) of 83%. We examined the indicator's PPV in the Veterans Health Administration.

STUDY DESIGN

We applied the Patient Safety Indicator software (v.3.1a) to fiscal year 2003-2007 VA discharge data. Trained abstractors reviewed medical records of 112 software-flagged PRF cases. We calculated the PPV and examined false positives to determine reasons for incorrect identification and true positives to determine clinical consequences and potential risk factors of PRF.

RESULTS

Seventy-five cases were true positive (PPV 67%; 95% CI, 57-76%); 13% were identified by a diagnosis code, 53% by a procedure code, 33% by both. Of false positives, 19% represented coding errors, 76% represented nonelective admissions. Of true positives, 28% of patients died, 56% had an American Society of Anesthesiologists level higher than II. Of associated index procedures, 53% were abdominal/pelvic, and 56% lasted >3 hours.

CONCLUSIONS

Based on our and University HealthSystem Consortium's findings, PRF should continue to be used as a screen for potential patient-safety events. Its PPV could be substantially improved in the Veterans Health Administration through introduction of an admission status code. Many PRF-identified cases appeared to be at high risk, based on patient and procedure-related factors. The degree to which such cases are truly preventable events requires additional assessment.

Positive predictive value of the AHRQ Patient Safety Indicator "Postoperative Sepsis": implications for practice and policy

BACKGROUND

Patient Safety Indicator (PSI) 13, or "Postoperative Sepsis," of the Agency for Healthcare Quality and Research (AHRQ), was recently adopted as part of a composite measure of patient safety by the Centers for Medicare and Medicaid Services (CMS). We sought to examine its positive predictive value (PPV) by determining how well it identifies true cases of postoperative sepsis.

STUDY DESIGN

Two retrospective cross-sectional studies of hospitalization records that met PSI 13 criteria were conducted, one within the Veterans Administration (VA) Hospitals from fiscal years (FY) 2003 to 2007, and one within community hospitals between October 1, 2005 and March 31, 2007. Trained abstractors reviewed medical records from each database using standardized abstraction instruments. We determined the PPV of the indicator and performed descriptive analyses of cases.

RESULTS

Of 112 cases flagged and reviewed within the VA system, 59 were true events of postoperative sepsis, yielding a PPV of 53% (95% CI 42% to 64%). Within the community hospital sector, of 164 flagged and reviewed cases, 67 were true cases of postoperative sepsis, yielding a PPV of 41% (95% CI 28% to 54%). False positives were due to infections that were present on admission, urgent or emergent cases, no clinical diagnosis of sepsis, or other coding limitations such as nonspecific shock in postoperative patients.

CONCLUSIONS

PSI 13 has relatively poor predictive ability to identify true cases of postoperative sepsis in both the VA and nonfederal sectors. The lack of information on diagnosis timing, confusion about the definition of elective admission, and coding limitations were the major reasons for false positives. As it currently stands, the use of PSI 13 as a stand-alone measure for hospital reporting appears premature.

Updating and validating the Charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries

With advances in the effectiveness of treatment and disease management, the contribution of chronic comorbid diseases (comorbidities) found within the Charlson comorbidity index to mortality is likely to have changed since development of the index in 1984. The authors reevaluated the Charlson index and reassigned weights to each condition by identifying and following patients to observe mortality within 1 year after hospital discharge. They applied the updated index and weights to hospital discharge data from 6 countries and tested for their ability to predict in-hospital mortality. Compared with the original Charlson weights, weights generated from the Calgary, Alberta, Canada, data (2004) were 0 for 5 comorbidities, decreased for 3 comorbidities, increased for 4 comorbidities, and did not change for 5 comorbidities. The C statistics for discriminating in-hospital mortality between the new score generated from the 12 comorbidities and the Charlson score were 0.825 (new) and 0.808 (old), respectively, in Australian data (2008), 0.828 and 0.825 in Canadian data (2008), 0.878 and 0.882 in French data (2004), 0.727 and 0.723 in Japanese data (2008), 0.831 and 0.836 in New Zealand data (2008), and 0.869 and 0.876 in Swiss data (2008). The updated index of 12 comorbidities showed good-to-excellent discrimination in predicting in-hospital mortality in data from 6 countries and may be more appropriate for use with more recent administrative data.

Designing an abstraction instrument: lessons from efforts to validate the AHRQ patient safety indicators

BACKGROUND

The U.S. Agency for Healthcare Research and Quality (AHRQ) and other organizations have developed quality indicators based on hospital administrative data. Characteristics of effective abstraction instruments were identified for determining both the positive predictive value (PPV) of Patient Safety Indicators (PSIs) and the extent to which hospitals and clinicians could have prevented adverse events.

METHODS

Through an iterative process involving nurse abstractors, physicians, and nurses with quality improvement experience, and health services researchers, 25 abstraction instruments were designed for 12 AHRQ provider-level morbidity PSIs. Data were analyzed from 13 of these instruments, and data are being collected using several more.

FINDINGS

Common problems in designing the instruments included avoiding uninformative questions and premature termination of the abstraction process, anticipating misinterpretation of questions, allowing an appropriate range of response options; using clear terminology, optimizing the flow of the abstraction process, balancing the utility of data against abstractor burden, and recognizing the needs of end users, such as hospitals and quality improvement professionals and researchers, for the abstracted information.

CONCLUSIONS

Designing medical record abstraction instruments for quality improvement research involves several potential pitfalls. Understanding how we addressed these challenges might help both investigators and users of outcome indicators to appreciate the strengths and limitations of outcome-based quality indicators and tools designed to validate or investigate such indicators within provider organizations.