An empirical assessment of the validity of explicit and implicit process-of-care criteria for quality assessment

OBJECTIVE

To evaluate the validity of three criteria-based methods of quality assessment: unit weighted explicit process-of-care criteria; differentially weighted explicit process-of-care criteria; and structured implicit process-of-care criteria.

METHODS

The three methods were applied to records of index hospitalizations in a study of unplanned readmission involving roughly 2,500 patients with one of three diagnoses treated at 12 Veterans Affairs hospitals. Convergent validity among the three methods was estimated using Spearman rank correlation. Predictive validity was evaluated by comparing process-of-care scores between patients who were or were not subsequently readmitted within 14 days.

RESULTS

The three methods displayed high convergent validity and substantial predictive validity. Index-stay mean scores, using explicit criteria, were generally lower in patients subsequently readmitted, and differences between readmitted and nonreadmitted patients achieved statistical significance as follows: mean readiness-for-discharge scores were significantly lower in patients with heart failure or with diabetes who were readmitted; and mean admission work-up scores were significantly lower in patients with lung disease who were readmitted. Scores derived from the structured implicit review were lower in patients eventually readmitted but significantly so only in diabetics.

CONCLUSIONS

These three criteria-based methods of assessing process of care appear to be measuring the same construct, presumably "quality of care." Both the explicit and implicit methods had substantial validity, but the explicit method is preferable. In this study, as in others, it had greater inter-rater reliability.

The association of quality of care and occurrence of in-hospital, treatment-related complications

BACKGROUND

Little data exist supporting the association of quality of care and nonfatal adverse outcomes in hospitalized patients, yet those outcomes are routinely scrutinized in quality assessment efforts.

OBJECTIVE

To determine whether measurable differences in quality of care are associated with the occurrence of non-fatal, in-hospital, and treatment-related complications.

DESIGN

Retrospective cohort study.

SUBJECTS

A total of 2,268 patients who were discharged alive from 9 Southwestern Veterans Affairs Medical Centers with congestive heart failure (CHF), chronic obstructive pulmonary disease (COPD) or diabetes mellitus.

MEASURES

Retrospective chart review was performed to collect information on patient severity of illness, in-hospital complication occurrence, and process quality of care. Process quality was assessed as the adherence scores for admission work-up and for treatment during the hospital stay. Process quality represents the proportion of applicable admission or treatment criteria that were met by that patient's care providers. Once severity of illness was taken into account Cox proportional hazards regression was used to assess the independent contribution of process quality of care to complication occurrence.

RESULTS

Higher admission work-up adherence scores for COPD patients and higher treatment adherence scores for COPD and diabetes patients were associated with a lower risk of complication occurrence. The adjusted risk ratios of complications for higher versus lower adherence scores (with 95% CI) were 0.64 (0.43, 0.97) and 0.52 (0.33, 0.80) for admission and treatment, respectively, in COPD patients, and 0.51 (0.31, 0.83) for treatment in diabetics. No significant association was found in CHF patients.

CONCLUSION

Better admission work-up and treatment quality in COPD patients, as well as treatment quality in diabetic patients, are associated with lower risk of nonfatal treatment-related complications in the study population.

Dyspepsia: how should we measure it?

This study developed and validated a multidimensional measure of dyspepsia. A questionnaire was administered to 126 patients with dyspepsia who presented for care at a VA outpatient clinic and a family physician's private office. Dyspepsia-specific health was measured by self-report using: (1) an existing dyspepsia scale that produces an aggregate score by summing ratings across pain and non-pain symptoms; (2) adaptations of two scales originally designed to measure back pain; and (3) a new scale measuring satisfaction with dyspepsia-related health. Generic health was measured using the SF-36. Results from factor analysis revealed four dimensions of dyspepsia-related health: pain intensity, pain disability, non-pain symptoms, and satisfaction with dyspepsia-related health. After refinements, scales representing the four dimensions conformed to psychometric standards for reliability, and convergent and discriminant validity. The importance of measuring dyspepsia using a multidimensional approach was confirmed by demonstrating that classification of dyspepsia severity depended on the dimension that was assessed. We conclude that dyspepsia is best measured using a multidimensional approach.

International Classification of Diseases, 9th Revision, Clinical Modification codes in discharge abstracts are poor measures of complication occurrence in medical inpatients

OBJECTIVES

The authors tested the ability of International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) codes in discharge abstracts to identify medical inpatients who experienced an in-hospital complication, using complications identified through chart review as the gold standard.

METHODS

Two sets of ICD-9-CM codes were used: an inclusive set including many medical diagnoses that may also be coexistent complicating conditions on admission rather than complications and an exclusive set consisting primarily of ICD-9-CM-specified complication and adverse drug event codes.

RESULTS

Neither set performed well as a diagnostic test for complication occurrence according to receiver operating characteristic analysis (ROC areas were 0.61 for the inclusive set and 0.55 for the exclusive set). Sensitivities of the ICD-9-CM codes for complications were 0.34 for the inclusive set and 0.14 for the exclusive set. Corresponding positive predictive values were 0.32 and 0.37, respectively. Sensitivities of code definitions for individual complications were generally poor, less than 0.5 in most cases.

CONCLUSIONS

The authors conclude that ICD-9-CM codes in discharge abstracts are poor measures of complication occurrence.

Identifying complications and low provider adherence to normative practices using administrative data

OBJECTIVE

This study investigated whether unexpected length of stay (LOS) could be used as an indicator to identify hospital patients who experienced complications or whose care exhibited low adherence to normative practices.

DATA SOURCES AND STUDY SETTING

We analyzed 1,477 cases admitted for one of three medical conditions. All cases were discharged from one of nine participating Department of Veterans Affairs (VA) hospitals from October 1987 through September 1989. Analyses used administrative data and information abstracted through chart reviews that included severity of illness indicators, complications, and explicit process of care criteria reflecting adherence to normative practices.

STUDY DESIGN

We developed separate multiple linear regression models for each disease using LOS as the dependent measure and variables that could be assumed present at the time of admission as explanatory variables. Unexpectedly long LOS (i.e., discharges with high residuals) was used to target complications and unexpectedly short LOS was used to target cases whose care might have exhibited low adherence to normative practices. Information gleaned from chart reviews served as the gold standard for determining actual complications and low adherence.

PRINCIPAL FINDINGS

Analyses of administrative data showed that unexpectedly long LOS identified complications with sensitivities ranging from 40 through 62 percent across the three conditions. Positive predictive values all were at greater than chance levels (p < .05). This represented substantial improvement over identification of complications using ICD-9-CM codes contained in the administrative database where sensitivities were from 26 through 39 percent. Unexpectedly short LOS identified low provider adherence with sensitivities ranging from 33 through 45 percent with positive predictive values all above chance levels (p < .05). The addition to the LOS models of chart-based severity of illness information helped explain LOS, but failed to facilitate identification of complications or low adherence beyond what was accomplished using administrative data.

CONCLUSIONS

Administrative data can be used to target cases when seeking to identify complications or low provider adherence to normative practices. Targeting can be accomplished through the creation of indirect measures based on unexpected LOS. Future efforts should be devoted to validating unexpected LOS as a hospital-level quality indicator.

RELEVANCE/IMPACT

Scrutiny of unexpected LOS holds promise for enhancing the usefulness of administrative data as a resource for quality initiatives.

Expected source of payment and use of hospital services for coronary atherosclerosis

To study associations between payer and provision of services for patients hospitalized for coronary atherosclerosis, the authors analyzed abstracts of 24,424 discharges from California acute care hospitals during 1989. Services examined included receipt of coronary artery bypass surgery, percutaneous transluminal coronary angioplasty (PTCA), long length of stay (LOS) without revascularization, and overall LOS. Regression techniques controlled demographic factors and comorbidities. The privately insured were 96% more likely to undergo revascularization (either bypass or PTCA) than Medicaid discharges and 117% more likely than the uninsured. Odds of revascularization for Medicare and health maintenance organization discharges resembled those for the privately insured. Analyzed separately, PTCA was far more likely among the privately insured than Medicaid beneficiaries and the uninsured. In addition, the adjusted odds for PTCA were 52% greater for the privately insured than for health maintenance organization discharges. The greatest likelihood of long LOS without revascularization and the greatest overall LOS was observed for Medicaid discharges. Strong associations, consistent with financial incentives to provide care, exist between payer and provision of services. Future studies need to address whether variations in process result from differences in thresholds for procedure performance, differences in admission practices, or both.

In-hospital complications among survivors of admission for congestive heart failure, chronic obstructive pulmonary disease, or diabetes mellitus

OBJECTIVE

To determine the frequency of hospital complications among survivors of inpatient treatment for congestive heart failure (CHF), chronic obstructive pulmonary disease (COPD), or diabetes mellitus (DM).

DESIGN

Retrospective cohort study.

SETTING

Nine Veterans Affairs hospitals in the southern United States.

PATIENTS

1,837 men veterans discharged alive following hospitalization for CHF, COPD, or DM between January 1987 and December 1989. This patient population represents a subset of cases gathered to study the process of care in the hospital and subsequent early readmission; thus, veterans who died in the hospital were not included.

MEASUREMENTS

Medical record review to record the occurrence of any of 30 in-hospital complications such as cardiac arrest, nosocomial infections, or delirium (overall agreement between two reviewers = 84%, kappa = 0.37).

RESULTS

Complications occurred in 15.7% of the CHF cases, 13.1% of the COPD cases, and 14.8% of the DM cases. Hypoglycemic reactions were the most frequent individual adverse events in the CHF and DM cases (3.6% and 11.4% of the cases, respectively), and theophylline toxicity was most frequent among the COPD cases (4.9%). Patient age, the presence of comorbid diseases, and the Acute Physiology Score (APS) of APACHE II were associated with complication occurrence. For each disease, the patients who had a complication had significantly longer mean hospital stays than did the patients who did not have complications (14.6 to 14.9 days vs 7.2 to 8.2 days, p < 0.01).

CONCLUSIONS

Complications are frequent among patients discharged alive with CHF, COPD, or DM. The patients who experienced complications were more ill on admission and had longer hospital stays.

Using administrative databases to evaluate the quality of medical care: a conceptual framework

Health care is consuming an ever larger share of national resources in the United States. Measures to contain costs and evidence of unexplained variation in patient outcomes have led to concern about inadequacy in the quality of health care. As a measure of quality, the evaluation of hospitals through analysis of their discharge databases has been suggested because of the scope and economy offered by this methodology. However, the value of the information obtained through these analyses has been questioned because of the inadequacy of the clinical data contained in administrative databases and the resultant inability to control accurately for patient variation. We suggest, however, that the major shortcoming of prior attempts to use large databases to perform across-facility evaluation has resulted from the lack of a conceptual framework to guide the analysis. We propose a framework which identifies component areas and clarifies the underlying assumptions of the analytic process. For each area, criteria are identified which will maximize the validity of the results. Hospitals identified as having unexpectedly high unfavorable outcomes when our framework is applied will be those where poor quality will most likely be found by primary review of the process of care.(ABSTRACT TRUNCATED AT 250 WORDS)

The association between the quality of inpatient care and early readmission

OBJECTIVE

To determine whether the quality of care during a hospital stay is associated with unplanned readmission within 14 days.

DESIGN

Case-control study.

SETTING

12 Veterans Affairs hospitals.

PATIENTS

Men discharged after a hospitalization for diabetes (n = 593), chronic obstructive lung disease (n = 1172), or heart failure (n = 748). The ratio of controls (men without an unplanned readmission within 14 days to any Veterans Affairs hospital) to cases (men with such a readmission) was 3:1.

MAIN OUTCOME MEASURES

Unplanned readmission to any of the 159 Veterans Affairs hospitals within 14 days of discharge. Quality of care during the index stay was assessed by chart review using disease-specific explicit criteria for the process of inpatient care, which were developed by panels composed of expert physicians. Adherence scores (the percentage of applicable criteria that were met) were calculated for the admission workup, evaluation and treatment, and readiness for discharge.

RESULTS

After adjustment was made for demographic characteristics, severity of illness, and need for care, adherence scores correlated with early unplanned readmission (P < 0.05). For patients with diabetes and heart failure, decreased readiness-for-discharge adherence scores correlated with increased risk for readmission (P = 0.001 and P = 0.016, respectively). In patients with obstructive lung disease, decreased admission-workup scores correlated with increased risk for readmission (P = 0.013). One of 7 readmissions in patients with diabetes, 1 of 5 readmissions in patients with heart failure, and 1 of 12 readmissions in patients with obstructive lung disease were attributable to substandard care.

CONCLUSIONS

Lower quality of inpatient care increases the risk for unplanned early readmission in patients with heart failure, diabetes, or obstructive lung disease. Under certain circumstances, readmission is associated with remediable deficiencies in the process of inpatient care.

Administrative databases, case-mix adjustments and hospital resource use: the appropriateness of controlling patient characteristics

Hospital administrative databases are used for studying resource use and medical outcomes. The ability to use administrative data to make comparisons among providers requires accurate adjustment of rates based on case mix. Adjustment for case mix often incorporates pre-existing patient conditions such as comorbidities. We tested the hypothesis that some circulatory comorbidities can appear positively or negatively associated with percutaneous transluminal coronary angioplasty (PTCA), not for clinical reasons, but because of the population used for modeling. When statistical models included all discharges with a principal or primary diagnosis of coronary atherosclerosis, or angina with coronary atherosclerosis, multivariate analysis revealed that discharges with dysrhythmias and the more severely ill were less likely to receive PTCA. However, when analysis excluded discharges treated with options (e.g. bypass) reserved for patients with more severe conditions, the presence of dysrhythmias and more severe illness increased the odds of receiving PTCA. Variability involving the direction of association between patient characteristics and a specific intervention illustrates that rates adjusted for patient characteristics cannot be properly interpreted without a clear understanding of the rationale underlying strategies for case-mix adjustment.

Selecting disease-outcome pairs for monitoring the quality of hospital care

Health care payors and providers are increasingly monitoring hospital discharge data bases for adverse events as markers for quality of care. The principal criticisms of these analyses have focused on the impediments to risk adjustment posed by the incompleteness and inaccuracy of the data bases. However, efforts to address the inadequacies of the data bases will not correct deficiencies of the analytic process. These deficiencies arise from the application of one adverse outcome to all disease states. Instead, analysis should be restricted to comparisons of subgroups of patients in which a close fit exists between the quality of care for the disease state and the expected outcome. Furthermore, these disease-outcome pairs should be minimally subject to measurement error. The authors present a conceptual framework for developing such meaningful disease-outcome pairs, and using the hospital discharge data base of the Department of Veterans Affairs, show how the framework can be used to devise a monitoring strategy for re-admission.

A method of developing and weighting explicit process of care criteria for quality assessment

The use of explicit criteria to evaluate how well processes of care conform to accepted standards is a key method of quality assessment. Synthesizing four decades of literature, we devised an inexpensive, 6-step method of developing reliable, content-valid, explicit process criteria. This paper describes the method using a set of congestive heart failure criteria. In step 1 of the Criteria Development Method, criteria are derived from state-of-the-art clinical literature. In step 2, criteria are refined by expert panels. In this study, panelists refined the items by mail in a three-round Delphi process. In step 3, decisions about unit-or differential item weighting are made; we derived differential item weights from the panelists' third-round ratings. Step 4 consists of flagging items which may yield little information, i.e., consensus items of low import, and nonconsensus items. Numeric flags were computed using third-round median ratings and their interquartile ranges. Selection of a scoring method to summarize scores and communicate results is done in step 5. In step 6, chart reviewers are trained, inter-rater reliability is measured, and items with poor reliability are culled. This straightforward developmental method can be used to devise explicit process criteria for use in ambulatory or hospital settings and to evaluate care delivered by different types of providers. The method yields reliable criteria representing accepted standards of current clinical practice. This high content validity is a sine qua non for convergent and predictive validity, both of which must be demonstrated in empirical studies in which the criteria are compared against external yardsticks.

The impact of status asthmaticus practice guidelines on patient outcome and physician behavior

This study assesses the effects of a status asthmaticus guideline on patient outcome and pediatrician behavior in a staff model health maintenance organization (HMO). The guidelines were drafted by an asthma specialist in the HMO and then discussed with key clinical personnel. A preprinted protocol order form was developed to help implement the guideline into clinical practice. The medical records of pediatric patients admitted to the hospital with status asthmaticus before (N = 67) and after (N = 59) guideline development and implementation were reviewed. This study demonstrates that locally developed, treatment-specific guidelines based on scientific evidence and combined with a staff consensus process and a user-friendly protocol form can influence physician behavior and patient outcome positively.

Loss, widowhood, and psychological distress among the elderly

Psychological response to recent nonspousal familial loss was examined in a sample of elderly men and women (N = 825). Loss was related to a higher level of depressive symptomatology in men, but not women. Both the presence of a spouse and membership in a church or temple moderated the impact of loss on depression among men, such that widowed men who experienced a loss, and men who experienced a loss and did not belong to a church/temple, showed elevated depression scores. Widowed men who experienced recent nonspousal familial loss and did not belong to a church/temple were most depressed of all, with fully 100% of the respondents with these characteristics scoring above the cutpoint for depression established in community studies. Discussion centers on the role of social ties in buffering distress and gender differences in coping with stress. Outreach by churches/temples and other community organizations is suggested as appropriate intervention.

Loss, widowhood, and psychological distress among the elderly

Psychological response to recent nonspousal familial loss was examined in a sample of elderly men and women (N = 825). Loss was related to a higher level of depressive symptomatology in men, but not women. Both the presence of a spouse and membership in a church or temple moderated the impact of loss on depression among men, such that widowed men who experienced a loss, and men who experienced a loss and did not belong to a church/temple, showed elevated depression scores. Widowed men who experienced recent nonspousal familial loss and did not belong to a church/temple were most depressed of all, with fully 100% of the respondents with these characteristics scoring above the cutpoint for depression established in community studies. Discussion centers on the role of social ties in buffering distress and gender differences in coping with stress. Outreach by churches/temples and other community organizations is suggested as appropriate intervention.