Does the Complications Screening Program flag cases with process of care problems? Using explicit criteria to judge processes

Failure-to-rescue and interprovider comparisons after elective abdominal aortic aneurysm repair

BACKGROUND

The use of postoperative complication rates to derive metrics such as failure-to-rescue (FTR) is of increasing interest in assessing the quality of care. The aim of this study was to quantify FTR rates for elective abdominal aortic aneurysm (AAA) repair in England using administrative data, and to examine its validity against case-note review.

METHODS

A retrospective observational study using Hospital Episode Statistics (HES) data was combined with a multicentre audit of data quality. All elective AAA repairs done in England between 2005 and 2010 were identified. Postoperative complications were extracted, FTR rates quantified, and differences in FTR and in-hospital death rates established. A multicentre case-note review was performed to establish the accuracy of coding of complications, and the impact of inaccuracies on FTR rates derived from HES data.

RESULTS

A total of 19 638 elective AAA repairs were identified from HES; the overall mortality rate was 4·6 per cent. Patients with complications (19·2 per cent) were more likely to die than those without complications (odds ratio 12·22, 95 per cent c.i. 10·51 to 14·21; P < 0·001) and had longer hospital stays (P < 0·001). FTR rates correlated strongly with death rates, whereas complication rates did not. On case-note review (661 procedures), 41·5 per cent of patients had a complication recorded in the case notes. There was evidence of systematic under-reporting of complications in HES, leading to an overall misclassification rate of 36·3 (95 per cent c.i. 33·7 to 39·2) per cent (P < 0·001), which was less pronounced for surgical complications (12·6 (11·1 to 13·9) per cent; P <0·001). Despite this, the majority of FTR rates derived from HES were not significantly different from those derived from case-note data.

CONCLUSION

Postoperative complication and FTR rates after elective AAA repair can be derived from HES data. However, use of the metric for interprovider comparisons should be done cautiously, and only with concurrent case-note validation given the degree of miscoding identified.

Detecting postoperative hemorrhage or hematoma from administrative data: the performance of the AHRQ Patient Safety Indicator

BACKGROUND

Patient Safety Indicator (PSI) 9, "postoperative hemorrhage or hematoma" (PHH), of the US Agency for Healthcare Research and Quality has been considered for public quality of care reporting. We sought to evaluate its performance in detecting true complications.

METHODS

We conducted a retrospective, cross-sectional study of hospitalizations that met PSI 9 eligibility criteria. We sampled records flagged positive and negative by PSI 9 from a diverse set of 31 hospitals between February 2006, and June 2009. Trained abstractors reviewed medical records using standard instruments. We determined the sensitivity, specificity, and positive (PPV) and negative (NPV) predictive values of the indicator.

RESULTS

Of 181 analyzable records flagged by PSI 9, 168 (93%; weighted PPV, 95% [95% confidence interval (CI), 90-98%]) involved an accurately coded event, but only 126 (70%; weighted PPV, 78% [95% CI, 58-90%]) represented true PHH. Thirty-two false positives involved only intraoperative hemorrhage. Among true positives, hypotension occurred in 28% and death attributed to the PHH in 4%. Thirty-two of 281 records flagged negative by PSI 9 (but enriched with questionably negative records) represented true PHH. The indicator's sensitivity was 42% (95% CI, 23-64%), specificity 99.9% (95% CI, 99.8-100%), and NPV 99.7% (95% CI, 99.0-99.9%). Modifying the indicator to include additional procedure codes improved both sensitivity (85% [95% CI, 67-94%]) and PPV (76% [95% CI, 60-88%]).

CONCLUSION

PSI 9 holds promise in detecting serious, possibly preventable complications. The indicator might be improved by specification of the 998.11 hemorrhage code to exclude purely intraoperative events and addition of procedure codes to the indicator's numerator criteria.

Examining the relationship between processes of care and selected AHRQ patient safety indicators postoperative wound dehiscence and accidental puncture or laceration using the VA electronic medical record

This study examines whether Agency for Healthcare Research and Quality Patient Safety Indicators (PSIs) Postoperative Wound Dehiscence (PWD) and Accidental Puncture or Laceration (APL) events reflect problems with hospital processes of care (POC). The authors randomly selected 112 PSI-flagged PWD/APL discharges from 2002-2007 VA administrative data, identified true cases using chart review, and matched cases with controls. This yielded a total of 95 case-control pairs per PSI. Patient information and clinical processes on each case-control pair were abstracted from the electronic medical record (EMR). Although PWD cases and controls differed on incision and closure types, APL cases and controls were comparable in examined processes. Further exploration of the process differences between PWD cases and controls indicated that they were primarily caused by patients' underlying surgical problems rather than quality of care shortfalls. Documentation of POC was frequently missing in EMRs. Future studies should combine EMR review with alternative approaches, such as direct observation, to better assess POC.

Applying trigger tools to detect adverse events associated with outpatient surgery

OBJECTIVE

The objective of this study is to evaluate the performance of 5 triggers to detect adverse events (AEs) associated with outpatient surgery. Triggers use surveillance algorithms derived from clinical logic to flag cases where AEs have most likely occurred. Current efforts to detect AEs have focused primarily on the inpatient setting, despite the increase in outpatient surgery in all health care settings.

METHODS

Using trigger logic, we retrospectively evaluated data from 3 large health care systems' electronic medical records. Patients were eligible for inclusion if they had an outpatient (same-day) surgery in 2007 and at least 1 clinical note in the 6 months after the surgery. Two nurse abstractors reviewed a sample of trigger-flagged cases from each health care system. After reaching interrater reliability targets (κ > 0.60), we calculated the positive predictive value (PPV) of each trigger and the confidence interval of the estimate.

RESULTS

The surgical triggers flagged between 1% and 22% of the outpatient surgery cases, with a wide range in PPVs (6.0%-62.0%). The pulmonary embolism and deep vein thrombosis and emergency department triggers had the lowest proportion of flagged cases along with the highest PPVs, showing the most promise for screening cases with a high probability of AE occurrence.

CONCLUSIONS

Triggers may be useful in identifying a narrow set of surgeries for further review to determine if a surgical AE occurred, complementing existing tools and initiatives used to detect AEs. Improved detection of AEs in outpatient surgery should help target potential areas for quality improvement.

Measuring the cost of hospital adverse patient safety events

This paper estimates the excess cost of hospital inpatient care due to adverse safety events in the U.S. Department of Veterans Affairs (VA) hospitals during fiscal year 2007. We measured adverse events according to the Patient Safety Indicator (PSI) algorithms of the Agency for Healthcare Research and Quality. Patient level cost regression analyses were performed using generalized linear modeling techniques. Accounting for the heavily skewed distribution of costs among patients having adverse safety events, results suggested that the excess cost of nine different PSIs for VA patients are much higher than previously estimated. We tested sensitivity of results to whether costs were measured by VA's Decision Support System (DSS) that uses local costs of specific inputs, or by the average costing system developed by VA's Health Economics Resource Center. DSS costing appeared to better characterize the high cost patients.

Comparing patient outcomes across payer types: implications for using hospital discharge records to assess quality

OBJECTIVE

To explain observed differences in patient outcomes across payer types using hospital discharge records. Specifically, we address two mechanisms: hospital-payer matching versus unobserved patient heterogeneity.

DATA SOURCE

Florida's hospital discharge records (1996-2000) of major surgery patients with private health insurance between the ages of 18 and 65, Health Maintenance Organization (HMO) market penetration data, hospital systems data, and the Area Resource File.

STUDY DESIGN

The dependent variable is occurrence of one or more in-hospital complications as identified by the Complication Screening Program. The key independent variable is patients' primary-payer type (HMO, Preferred Provider Organization, and fee-for-service). We estimate five different logistic regression models, each representing a different assumption about the underlying factors that confound the causal relationship between the payer type and the likelihood of experiencing complications.

PRINCIPAL FINDING

We find that the observed differences in complication rates across payer types are largely driven by unobserved differences in patient health, even after adjusting for case mix using available data elements in the discharge records.

CONCLUSION

Because of the limitations inherent to hospital discharge records, making quality comparisons in terms of patient outcomes is challenging. As such, any efforts to assess quality in such a manner must be carried out cautiously.

How valid is the AHRQ Patient Safety Indicator "postoperative hemorrhage or hematoma"?

BACKGROUND

Postoperative hemorrhage or hematoma (PHH), an Agency for Healthcare Research and Quality Patient Safety Indicator, uses administrative data to detect cases of potentially preventable postsurgical bleeding requiring a reparative procedure. How accurately it identifies true events is unknown. We therefore determined PHH's positive predictive value.

STUDY DESIGN

Using Patient Safety Indicator software (v.3.1a) and fiscal year 2003-2007 discharge data from 28 Veterans Health Administration hospitals, we identified 112 possible cases of PHH. Based on medical record abstraction, we characterized cases as true (TPs) or false positives (FPs), calculated positive predictive value, and analyzed FPs to ascertain reasons for incorrect identification and TPs to determine PHH-associated clinical consequences and risk factors.

RESULTS

Eighty-four cases were TPs (positive predictive value, 75%; 95% CI, 66-83%); 63% had a hematoma diagnosis, 30% had a hemorrhage diagnosis, 7% had both. Reasons for FPs included events present on admission (29%); hemorrhage/hematoma identified and controlled during the original procedure rather than postoperatively (21%); or postoperative hemorrhage/hematoma that did not require a procedure (18%). Most TPs (82%) returned to the operating room for hemorrhage/hematoma management; 64% required blood products and 7% died in-hospital. The most common index procedures resulting in postoperative hemorrhage/hematoma were vascular (38%); 56% were performed by a physician-in-training (under supervision). We found no substantial association between physician training status or perioperative anticoagulant use and bleeding risk.

CONCLUSIONS

PHH's accuracy could be improved by coding enhancements, such as adopting present on admission codes or associating a timing factor with codes dealing with bleeding control. The ability of PHH to identify events representing quality of care problems requires additional evaluation.

Using ICD-10-AM Codes to Characterise Hospital-Acquired Complications

This paper describes the limitations of using the International Statistical Classification of Diseases and Related Health Problems, Tenth Revision, Australian Modification (ICD-10-AM) to characterise patient harm in hospitals. Limitations were identified during a project to use diagnoses flagged by Victorian coders as hospital-acquired to devise a classification of 144 categories of hospital acquired diagnoses (the Classification of Hospital Acquired Diagnoses or CHADx). CHADx is a comprehensive data monitoring system designed to allow hospitals to monitor their complication rates month-to-month using a standard method. Difficulties in identifying a single event from linear sequences of codes due to the absence of code linkage were the major obstacles to developing the classification. Obstetric and perinatal episodes also presented challenges in distinguishing condition onset, that is, whether conditions were present on admission or arose after formal admission to hospital. Used in the appropriate way, the CHADx allows hospitals to identify areas for future patient safety and quality initiatives. The value of timing information and code linkage should be recognised in the planning stages of any future electronic systems.

Clinical validation of the AHRQ postoperative venous thromboembolism patient safety indicator

BACKGROUND

The Agency for Healthcare Research and Quality (AHRQ) patient safety indicators (PSIs) screen for potentially preventable complications in hospitalized patients using hospital administrative data. The PSI for postoperative venous thromboembolism (VTE) relies on International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes for deep vein thrombosis (DVT) or pulmonary embolism (PE) in secondary diagnoses fields. In a clinical validation study of the PSI for postoperative VTE, natural language processing (NLP), supplemented by pharmacy and billing data, was used to identify VTE events missed by medical records coders.

METHODS

In a retrospective review of postsurgical discharges, charts were processed using the AHRQ PSI software. Cases were identified as possible false negatives by flagging charts for possible VTEs using pharmacy and billing data to identify all patients who were therapeutically anticoagulated or had placement of an inferior vena caval filter. All charts were reviewed by a physician blinded to screening results. Physician interpretation was considered the gold standard for VTE classification.

RESULTS

The AHRQ PSI had a positive predictive value (PPV) of .545 (95% confidence interval [CI], .453-.634) and a negative predictive value (NPV) of .997 (95% CI, .995-.999). Sensitivity was .87 and specificity was .98. Secondary coding review suggested that all 9 false-negative results were miscoded; if they had been properly coded, the sensitivity would increase to 1.00. Most false-positive cases resulted from superficial venous clots identified by the PSI due to coding ambiguity.

DISCUSSION

The VTE PSI performed well as a screening tool but generated a significant number of false-positive cases, a problem that could be substantially reduced with improved coding methods.

Comparison of mortality risk adjustment using a clinical data algorithm (American College of Surgeons National Surgical Quality Improvement Program) and an administrative data algorithm (Solucient) at the case level within a single institution

BACKGROUND

There is great interest in efficiently evaluating health care quality, but there is controversy over the use of administrative versus clinical data methods. We sought to compare actual mortality with risk-adjusted expected mortality in a sample population calculated by two different methods; one based on preexisting administrative records and one based on chart reviews.

STUDY DESIGN

We examined a sample of patients (n = 1,234) undergoing surgical procedures at an academic teaching hospital during 1 year. The first risk-adjustment method was that used by the National Surgical Quality Improvement Program, which is based on dedicated medical record review. The second method was that used by Solucient, LLC, which is based on preexisting administrative records.

RESULTS

The ratio of observed to expected mortality for this population set was higher using the National Surgical Quality Improvement Program algorithm (1.1; 95% CI, 0.8 to 1.5) than using the Solucient algorithm (0.9; 95% CI, 0.6 to 1.2) but neither estimate was notably different from 1.0. Similarly, when observed to expected mortality ratios were calculated separately for each quartile of mortality, there were no marked differences within quartiles, although minor differences with potential importance were noted. Fit was comparable by age categories, gender, and American Society of Anesthesiologists' categories. A number of actual deaths had higher predicted mortality scores using the Solucient algorithm.

CONCLUSIONS

Risk-adjusted mortality estimates were comparable using administrative or clinical data. Minor performance differences might still have implications. Because of the potential lower cost of using administrative data, this type of algorithm can be an efficient alternative and should continue to be investigated.

Failure to rescue: validation of an algorithm using administrative data

BACKGROUND

Failure to rescue (FTR), the rate of death in patients suffering 1 of 6 in-hospital complications, is an Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicator calculated from administrative data.

OBJECTIVE

: We sought to assess the accuracy of the AHRQ FTR algorithm.

METHODS

We undertook a retrospective chart review of 60 denominator cases of FTR identified by the algorithm at each of 40 University HealthSystem Consortium institutions. The primary outcome was the overall accuracy of the algorithm compared with chart review. We also assessed accuracy by complication type, patient characteristics, institution, service assignment, and mortality.

RESULTS

Of 2354 cases, 1193 (50.7%) were accurately identified by the algorithm as having had at least one of the FTR-qualifying complications during hospitalization. Of the 3073 complications identified in these patients, 1497 (48.7%) were correctly flagged by the algorithm, 907 (29.5%) were present on admission, 419 (13.6%) were not confirmed by chart review, and 250 (8.1%) met a predefined complication-specific criterion for exclusion. The case accuracy rate varied significantly by institution (mean, 50.7%; range, 18.3-100%; P < 0.001), service assignment (surgical service, 62.9% vs. nonsurgical service, 42.9%; P < 0.001), and mortality (alive, 43.9% vs. dead, 67.5%; P < 0.001) but was not affected by patients' age, gender, race, or insurance status.

CONCLUSIONS

As currently calculated from administrative data, the FTR algorithm misidentifies half of the cases on average, is least accurate for nonsurgical cases, and is widely variable across institutions. This indicator may be useful internally to flag possible cases of quality failure but has limitations for external institutional comparisons. Improvements in coding quality and consistency across institutions are needed.

The Validity of ICD-9-CM Codes in Identifying Postoperative Deep Vein Thrombosis and Pulmonary Embolism

BACKGROUND

Deep vein thrombosis and pulmonary embolism (DVT/PE) are common complications after surgery and are associated with substantial excess mortality and length of stay. International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes recorded in hospital claims have been used to identify and study DVT/PE, but the validity of this method is not well studied.

METHODS

Identification of postoperative DVT/PE events were compared using ICD-9-CM codes and medical record abstraction in random samples of hospital discharges of Medicare beneficiaries in 2002-2004.

RESULTS

Among 20,868 eligible surgical hospitalizations, 232 DVT cases and 95 PE cases were identified by ICD-9-CM codes; 108 DVT cases and 31 PE cases by medical record abstraction; 72 DVT cases and 23 PE cases by both methods. The resulting estimates of PPV of ICD9-CM coding were 31% (72/232 cases) for DVT, 24% (23/95) for PE, and 29% (90/308) for DVT/PE combined. The resulting sensitivity estimates were 67% (72/108 cases) for DVT, 74% (23/31) for PE, and 68% (90/133) for DVT/PE combined.

DISCUSSION

ICD-9-CM codes in Medicare claims are sensitive but have limited predictive validity in identifying postoperative DVT/PE. Improvements in the validity are needed before the indicator can be used for safety performance assessment.

Tracking rates of Patient Safety Indicators over time: lessons from the Veterans Administration

BACKGROUND

The Patient Safety Indicators (PSIs), developed by the Agency for Healthcare Research and Quality, are useful screening tools for highlighting areas in which quality should be further investigated and providing useful benchmarks for tracking progress.

OBJECTIVES

Our objectives were to: 1) provide a descriptive analysis of the incidence of PSI events from 2001 to 2004 in the Veterans Health Administration (VA); 2) examine trends in national PSI rates at the hospital discharge level over time; and 3) assess whether hospital characteristics (eg, teaching status, number of beds, and degree of quality improvement implementation) and baseline safety-related hospital performance predict future hospital safety-related performance.

METHODS

We examined changes in risk-adjusted PSI rates at the discharge level, calculated the correlation between hospitals' risk-adjusted PSI rates in 2001 with subsequent years, and developed generalized linear models to examine predictors of hospitals' 2004 risk-adjusted PSI rates.

RESULTS

Risk-adjusted rates of 2 of the 15 PSIs demonstrated significant trends over time. Rates of iatrogenic pneumothorax increased over time, whereas rates of failure to rescue decreased. Most PSIs demonstrated consistent rates over time. After accounting for patient and hospital characteristics, hospitals' baseline risk-adjusted PSI rates were the most important predictors of their 2004 risk-adjusted rates for 8 PSIs.

CONCLUSIONS

The PSIs are useful tools for tracking and monitoring patient safety events in the VA. Future research should investigate whether trends reflect better or worse care or increased attention to documenting patient safety events.

Impact of nursing overtime on nurse-sensitive patient outcomes in New York hospitals, 1995-2000

During the past several years, nurses and their advocates have expressed concern about heavy use of overtime in hospitals and claimed that it undermines the quality of nursing care. Using staffing and discharge data covering 1995 to 2000 from 161 acute general hospitals in New York State, this study uses multi variate regression to analyze the relationship between overtime and the rates of six nurse-sensitive patient outcomes and mortality. We find an association of overtime with lower rates of mortality in medical and surgical patients but do not consider these findings definitive. Because overtime use is episodic and unit specific, further study of these issues using data that examines the occurrence of adverse events by unit during periods of heavy nurse overtime is recommended.

Evaluating the Patient Safety Indicators

BACKGROUND

The Patient Safety Indicators (PSIs), an administrative data-based tool developed by the Agency for Healthcare Research and Quality, are increasingly being used to screen for potential in-hospital patient safety problems. Although the Veterans Health Administration (VA) is a national leader in patient safety, accurate information on the epidemiology of patient safety events in the VA is still unavailable.

OBJECTIVES

Our objectives were to: (1) apply the AHRQ PSI software to VA administrative data to identify potential instances of compromised patient safety; (2) determine occurrence rates of PSI events in the VA; and (3) examine the construct validity of the PSIs.

METHODS

We examined differences between observed and risk-adjusted PSI rates in the VA, compared VA and non-VA PSI rates, and investigated the construct validity of the PSIs by examining correlations of the PSIs with other outcomes of VA hospitalizations.

RESULTS

We identified 11,411 PSI events in the VA nationwide in FY'01. Observed PSI rates per 1000 discharges ranged from 0.007 for "transfusion reaction" to 155.5 for "failure to rescue." There were significant, although small, differences between VA and non-VA risk-adjusted PSI rates. Hospitalizations with PSI events had longer lengths of stay, higher mortality, and higher costs than those without PSI events.

CONCLUSIONS

Our results suggest that the PSIs may be useful as a patient safety screening tool in the VA. Our PSI rates were consistent with the national incidence of low rates; however, differences between VA and non-VA rates suggest that inadequate case-mix adjustment may be contributing to these findings.

Readmissions for venous thromboembolism: expanding the definition of patient safety indicators

BACKGROUND

The Agency for Healthcare Research and Quality (AHRQ) defines its venous thromboembolism (VTE) patient safety indicator (PSI) as surgical cases with a secondary diagnosis of pulmonary embolism or deep vein thrombosis. Short-term readmissions for VTE are excluded because most state administrative databases are unable to track readmissions.

METHODS

Patients meeting the AHRQ VTE PSI definition and those readmitted with a VTE principal diagnosis within 30 days of a prior surgical hospitalization were identified on the basis of inpatient discharge data.

RESULTS

A total of 4,906 surgical discharges in New York met the AHRQ VTE PSI definition in 2001. An additional 1,059 cases of VTE were found when surgical patients with a short-term readmission for VTE were identified. Patients readmitted with VTE were less likely to die but were more likely to have a pulmonary embolism and were more likely to be white and non-Hispanic compared to those who met the AHRQ VTE PSI definition.

DISCUSSION

Short-term readmissions for VTE represent potentially important cases to capture when monitoring adverse events. Prophylaxis, monitoring, and patient education may be required after hospital discharge to prevent or treat VTE as early as possible. Data systems that can track patients across multiple admissions to identify complications resulting in short-term readmissions are needed.

Using routine comparative data to assess the quality of health care: understanding and avoiding common pitfalls

Measuring the quality of health care has become a major concern for funders and providers of health services in recent decades. One of the ways in which quality of care is currently assessed is by taking routinely collected data and analysing them quantitatively. The use of routine data has many advantages but there are also some important pitfalls. Collating numerical data in this way means that comparisons can be made--whether over time, with benchmarks, or with other healthcare providers (at individual or institutional levels of aggregation). Inevitably, such comparisons reveal variations. The natural inclination is then to assume that such variations imply rankings: that the measures reflect quality and that variations in the measures reflect variations in quality. This paper identifies reasons why these assumptions need to be applied with care, and illustrates the pitfalls with examples from recent empirical work. It is intended to guide not only those who wish to interpret comparative quality data, but also those who wish to develop systems for such analyses themselves.

A national profile of patient safety in U.S. hospitals

Measures based on routinely collected data would be useful to examine the epidemiology of patient safety. Extending previous work, we established the face and consensual validity of twenty Patient Safety Indicators (PSIs). We generated a national profile of patient safety by applying these PSIs to the HCUP Nationwide Inpatient Sample. The incidence of most nonobstetric PSIs increased with age and was higher among African Americans than among whites. The adjusted incidence of most PSIs was highest at urban teaching hospitals. The PSIs may be used in AHRQ's National Quality Report, while providers may use them to screen for preventable complications, target opportunities for improvement, and benchmark performance.

Measuring errors and adverse events in health care

In this paper, we identify 8 methods used to measure errors and adverse events in health care and discuss their strengths and weaknesses. We focus on the reliability and validity of each, as well as the ability to detect latent errors (or system errors) versus active errors and adverse events. We propose a general framework to help health care providers, researchers, and administrators choose the most appropriate methods to meet their patient safety measurement goals.

A complication conference for internal quality control at the Neurosurgical Department of the University of Heidelberg

The registration of adverse events is an important issue in the field of medicine. Even today adverse event screening and registration is not part of the routine in most medical areas. In 1994, the Department of Neurosurgery at the University of Heidelberg implemented a conference for screening and registering adverse events. The aim was to record all complications occurring for an internal quality control. High priority was given to improving the process of data screening and registering. The conference is held every 2 weeks and all medical staff and residents of the department are obligated to be present. Screening of the adverse events encompasses all operations performed during a bi-weekly period. Every single operation is revised for an adverse event during or following the hospital stay. Adverse events are registered on a standardized data sheet and later transferred to a database for use in further investigations. After 6 years, the conference has been fully accepted and become an integral part of the workflow of the department. During this period, 8160 operations were screened and 1335 adverse events registered. The next step will be to integrate the data-collection process into the daily ward rounds using a personal digital assistant (PDA). This process is less time consuming and may perhaps augment the number of registered cases.

Discrepancies between explicit and implicit review: physician and nurse assessments of complications and quality

OBJECTIVE

To identify and characterize discrepancies between explicit and implicit medical record review of complications and quality of care.

SETTING

Forty-two acute-care hospitals in California and Connecticut in 1994.

STUDY DESIGN

In a retrospective chart review of 1,025 Medicare beneficiaries age >65, we compared explicit (nurse) and implicit (physician) reviews of complications and quality in individual cases. To understand discrepancies, we calculated the kappa statistic and examined physicians' comments.

DATA COLLECTION

With Medicare discharge abstracts, we used the Complications Screening Program to identify and then select a stratified random sample of cases flagged for 1 of 15 surgical complications, 5 medical complications, and unflagged controls. Peer Review Organization nurses and physicians performed chart reviews.

PRINCIPAL FINDINGS

Agreement about complications was fair (kappa = 0.36) among surgical and was moderate (kappa = 0.59) among medical cases. In discordant cases, physicians said that complications were insignificant, attributable to a related diagnosis, or present on admission. Agreement about quality was poor among surgical and medical cases (kappa = 0.00 and 0.13, respectively). In discordant cases, physicians said that quality problems were unavoidable, small lapses in otherwise satisfactory care, present on admission, or resulted in no adverse outcome.

CONCLUSIONS

We identified many discrepancies between explicit and implicit review of complications and quality. Physician reviewers may not consider process problems that are ubiquitous in hospitals to represent substandard quality.

Three decades of research on computer applications in health care: medical informatics support at the Agency for Healthcare Research and Quality

The Agency for Healthcare Research and Quality and its predecessor organizations-collectively referred to here as AHRQ-have a productive history of funding research and development in the field of medical informatics, with grant investments since 1968 totaling $107 million. Many computerized interventions that are commonplace today, such as drug interaction alerts, had their genesis in early AHRQ initiatives. This review provides a historical perspective on AHRQ investment in medical informatics research. It shows that grants provided by AHRQ resulted in achievements that include advancing automation in the clinical laboratory and radiology, assisting in technology development (computer languages, software, and hardware), evaluating the effectiveness of computer-based medical information systems, facilitating the evolution of computer-aided decision making, promoting computer-initiated quality assurance programs, backing the formation and application of comprehensive data banks, enhancing the management of specific conditions such as HIV infection, and supporting health data coding and standards initiatives. Other federal agencies and private organizations have also supported research in medical informatics, some earlier and to a greater degree than AHRQ. The results and relative roles of these related efforts are beyond the scope of this review.

Does clinical evidence support ICD-9-CM diagnosis coding of complications?

BACKGROUND

Hospital discharge diagnoses, coded by use of the International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM), increasingly determine reimbursement and support quality monitoring. Prior studies of coding validity have investigated whether coding guidelines were met, not whether the clinical condition was actually present.

OBJECTIVE

To determine whether clinical evidence in medical records confirms selected ICD-9-CM discharge diagnoses coded by hospitals.

RESEARCH DESIGN AND SUBJECTS

Retrospective record review of 485 randomly sampled 1994 hospitalizations of elderly Medicare beneficiaries in Califomia and Connecticut.

MAIN OUTCOME MEASURE

Proportion of patients with specified ICD-9-CM codes representing potential complications who had clinical evidence confirming the coded condition.

RESULTS

Clinical evidence supported most postoperative acute myocardial infarction diagnoses, but fewer than 60% of other diagnoses had confirmatory clinical evidence by explicit clinical criteria; 30% of medical and 19% of surgical patients lacked objective confirmatory evidence in the medical record. Across 11 surgical and 2 medical complications, objective clinical criteria or physicians' notes supported the coded diagnosis in >90% of patients for 2 complications, 80% to 90% of patients for 4 complications, 70% to <80% of patients for 5 complications, and <70% for 2 complications. For some complications (postoperative pneumonia, aspiration pneumonia, and hemorrhage or hematoma), a large fraction of patients had only a physician's note reporting the complication.

CONCLUSIONS

Our findings raise questions about whether the clinical conditions represented by ICD-9-CM codes used by the Complications Screening Program were in fact always present. These findings highlight concerns about the clinical validity of using ICD-9-CM codes for quality monitoring.

Use of administrative data to find substandard care: validation of the complications screening program

OBJECTIVE

The use of administrative data to identify inpatient complications is technically feasible and inexpensive but unproven as a quality measure. Our objective was to validate whether a screening method that uses data from standard hospital discharge abstracts identifies complications of care and potential quality problems.

DESIGN

This was a case-control study with structured implicit physician reviews.

SETTING

Acute-care hospitals in California and Connecticut in 1994.

PATIENTS

The study included 1,025 Medicare beneficiaries greater than 265 years of age.

METHODS

Using administrative data, we stratified acute-care hospitals by observed-to-expected complication rates and randomly selected hospitals within each state. We randomly selected cases flagged with 1 of 17 surgical complications and 6 medical complications. We randomly selected controls from unflagged cases.

MAIN OUTCOME MEASURE

Peer-review organization physicians' judgments about the presence of the flagged complication and potential quality-of-care problems.

RESULTS

Physicians confirmed flagged complications in 68.4% of surgical and 27.2% of medical cases. They identified potential quality problems in 29.5% of flagged surgical and 15.7% of medical cases but in only 2.1% of surgical and medical controls. The rate of physician-identified potential quality problems among flagged cases exceeded 25% in 9 surgical screens and 1 medical screen. Reviewers noted several potentially mitigating circumstances that affected their judgments about quality, including factors related to the patients' illness, the complexity of the case, and technical difficulties that clinicians encountered.

CONCLUSIONS

For some types of complications, screening administrative data may offer an efficient approach for identifying potentially problematic cases for physician review. Understanding the basis for physicians' judgments about quality requires more investigation.

Identification of in-hospital complications from claims data. Is it valid?

OBJECTIVES

This study examined the validity of the Complications Screening Program (CSP) by testing whether (1) ICD-9-CM codes used to identify a complication are coded completely and accurately and (2) the CSP algorithm successfully separates conditions present on admission from those occurring in the hospital.

METHODS

We compared diagnosis and procedure codes contained in the Medicare claim with codes abstracted from an independent re-review of more than 1,200 medical records from Connecticut and California.

RESULTS

Eighty-nine percent of the surgical cases and 84% of the medical cases had their CSP trigger codes corroborated by re-review of the medical record. For 13% of the surgical cases and 58% of the medical cases, the condition represented by the code was judged to be present on admission rather than occurring in-hospital. The positive predictive value of the claim was greater than 80% for the surgical risk pool, suggesting the value of the CSP as a screening tool.

CONCLUSIONS

The CSP has validity as a screen for most surgical complications but only for 1 medical complication. The CSP does not have validity as a "stand-alone" tool to identify more than a few in-hospital surgery-related events. The addition of an indicator to the Medicare claim to capture the timing of secondary diagnoses would improve the validity of the CSP for identifying both surgical and medical in-hospital events.