Validating the Patient Safety Indicators in the Veterans Health Administration: Are They Ready for Prime Time?

Validity of 16 AHRQ Patient Safety Indicators to identify in-hospital complications: a medical record review across nine Swiss hospitals

The validity of the Agency for Healthcare Research and Quality's Patient Safety Indicators (PSIs) has been established in the USA and Canada. However, these indicators are also used for hospital benchmarking and cross-country comparisons in other nations with different health-care settings and coding systems as well as missing present on admission (POA) flags in the administrative data. This study sought to comprehensively assess and compare the validity of 16 PSIs in Switzerland, where they have not been previously applied. We performed a medical record review using administrative and electronic medical record data from nine Swiss hospitals. Seven independent reviewers evaluated 1245 cases at various hospitals using retrospective data from the years 2014-18. True positives, false positives, positive predictive values (PPVs), and reasons for misclassification were compared across all investigated PSIs, and the documentation quality of the PSIs was examined. PSIs 6 (iatrogenic pneumothorax), 10 (postoperative acute kidney injury), 11 (postoperative respiratory failure), 13 (postoperative sepsis), 14 (wound dehiscence), 17 (birth trauma), and 18 and 19 (obstetric trauma with or without instrument) showed high PPVs (range: 90-99%) and were not strongly influenced by missing POA information. In contrast, PSIs 3 (pressure ulcer), 5 (retained surgical item), 7 (central venous catheter-related bloodstream infection), 8 (fall with hip fracture), and 15 (accidental puncture/laceration) showed low PPVs (range: 18-49%). In the case of PSIs 3, 8, and 12 (perioperative embolism/thrombosis), the low PPVs were largely due to the lack of POA information. Additionally, it was found that the documentation of PSI 3 in discharge letters could be improved. We found large differences in validity across the 16 PSIs in Switzerland. These results can guide policymakers in Switzerland and comparable health-care systems in selecting and prioritizing suitable PSIs for quality initiatives. Furthermore, the national introduction of a POA flag would allow for the inclusion of additional PSIs in quality monitoring.

The deterrent effect of tort law: Evidence from medical malpractice reform

We examine whether caps on noneconomic damages in medical malpractice cases affect in-hospital patient safety. We use Patient Safety Indicators - measures of adverse events - as proxies for safety. In difference-in-differences ("DiD") analyses of five states that adopt caps during 2003-2005, we find that multiple measures of non-fatal patient safety events worsen after cap adoption relative to control states. DiD inference can be unreliable with a small number of treated units. We therefore develop a randomization inference-based test for inference with few treated units but multiple correlated outcomes and confirm the robustness of our results with this nonparametric approach. We also provide evidence that the decline in patient safety is unlikely to be driven by patient selection.

Should policy-makers and managers trust PSI? An empirical validation study of five patient safety indicators in a national health service

BACKGROUND

Patient Safety Indicators (PSI) are being modestly used in Spain, somewhat due to concerns on their empirical properties. This paper provides evidence by answering three questions: a) Are PSI differences across hospitals systematic -rather than random?; b) Do PSI measure differences among hospital-providers -as opposed to differences among patients?; and, c) Are measurements able to detect hospitals with a higher than "expected" number of cases?

METHODS

An empirical validation study on administrative data was carried out. All 2005 and 2006 publicly-funded hospital discharges were used to retrieve eligible cases of five PSI: Death in low-mortality DRGs (MLM); decubitus ulcer (DU); postoperative pulmonary embolism or deep-vein thrombosis (PE-DVT); catheter-related infections (CRI), and postoperative sepsis (PS). Empirical Bayes statistic (EB) was used to estimate whether the variation was systematic; logistic-multilevel modelling determined what proportion of the variation was explained by the hospital; and, shrunken residuals, as provided by multilevel modelling, were plotted to flag hospitals performing worse than expected.

RESULTS

Variation across hospitals was observed to be systematic in all indicators, with EB values ranging from 0.19 (CI95%:0.12 to 0.28) in PE-DVT to 0.34 (CI95%:0.25 to 0.45) in DU. A significant proportion of the variance was explained by the hospital, once patient case-mix was adjusted: from a 6% in MLM (CI95%:3% to 11%) to a 24% (CI95%:20% to 30%) in CRI. All PSI were able to flag hospitals with rates over the expected, although this capacity decreased when the largest hospitals were analysed.

CONCLUSION

Five PSI showed reasonable empirical properties to screen healthcare performance in Spanish hospitals, particularly in the largest ones.