Improving the statistical approach to health care provider profiling

Assessing mammographers' accuracy. A comparison of clinical and test performance

Direct estimation of mammographers' clinical accuracy requires the ability to capture screening assessments and correctly identify which screened women have breast cancer. This clinical information is often unavailable and when it is available its observational nature can cause analytic problems. Problems with clinical data have led some researchers to evaluate mammographers using a single set of films. Research based on these test film sets implicitly assumes a correspondence between mammographers' accuracy in the test setting and their accuracy in a clinical setting. However, there is no evidence supporting this basic assumption. In this article we use hierarchical models and data from 27 mammographers to directly compare accuracy estimated from clinical practice data to accuracy estimated from a test film set. We found moderate positive correlation [ rho; = 0.206 with 95% credible interval (-0.142-0. 488)] between mammographers' overall preponderance to call a mammogram positive. However, we found no evidence of correlation between clinical and test accuracy [ rho; = -0.025 with 95% credible interval (-0.484-0.447)]. This study is limited by the relatively small number of mammographers evaluated, by the somewhat restricted range of observed sensitivities and specificities, and by differences in the types of films evaluated in test and clinical datasets. Nonetheless, these findings raise important questions about how mammographer accuracy should be measured.

Medical profiling: improving standards and risk adjustments using hierarchical models

The conclusions from a profile analysis to identify performance extremes can be affected substantially by the standards and statistical methods used and by the adequacy of risk adjustment. Medically meaningful standards are proposed to replace common statistical standards. Hierarchical regression methods can handle several levels of random variation, make risk adjustments for the providers' case-mix differences, and address the proposed standards. These methods determine probabilities needed to make meaningful profiles of medical units based on standards set by all appropriate parties.

Influence of hospital and clinician workload on survival from colorectal cancer: cohort study

OBJECTIVE

To determine whether clinician or hospital caseload affects mortality from colorectal cancer.

DESIGN

Cohort study of cases ascertained between 1990 and 1994 by a region-wide colorectal cancer register.

OUTCOME MEASURES

Mortality within a median follow up period of 54 months after diagnosis.

RESULTS

Of the 3217 new patients registered over the period, 1512 (48%) died before 31 December 1996. Strong predictors of survival both in a logistic regression (fixed follow up) and in a Cox's proportional hazards model (variable follow up) were Duke's stage, the degree of tumour differentiation, whether the liver was deemed clear of cancer by the surgeon at operation, and the type of intervention (elective or emergency and curative or palliative intent). In a multilevel model, surgeon's caseload had no significant effect on mortality at 2 years. Hospital workload, however, had a significant impact on survival. The odds ratio for death within 2 years for cases managed in a hospital with a caseload of between 33 and 46 cases per year, 47 and 54 cases per year, and >/=55 cases per year (compared to one with </=23 cases per year) were respectively 1.48 (95% confidence interval 1.03 to 2.13), 1.52 (1.08 to 2.13), and 1.18 (0.83 to 1.68).

CONCLUSIONS

There was no detectable caseload effect for surgeons managing colorectal cancer, but survival of patients treated in hospitals with caseloads above 33 cases per year was slightly worse than for those treated in hospitals with fewer caseloads. Imprecise measurement of clinician specific "events rates" and the lack of routinely collected case mix data present major challenges for clinical audit and governance in the years ahead.

A method for estimating solid organ donor potential by organ procurement region

OBJECTIVES

This study sought to develop a methodology for estimating potential solid organ donors and measuring donation performance in a geographic region based on readily available data on the hospitals in that region.

METHODS

Medical records were reviewed in a stratified random sample of 89 hospitals from 3 regions to attain a baseline of donor potential. Data on a range of hospital characteristics were collected and tested as predictors of donor potential through the use of hierarchical Poisson regression modeling.

RESULTS

Five hospital characteristics predicted donor potential: hospital deaths, hospital Medicare case-mix index, total hospital staffed beds, medical school affiliation, and trauma center certification. Regional estimates were attained by aggregating individual hospital estimates. Confidence intervals for these regional estimates indicated that actual donations represented from 28% to 44% of the potential in the regions studied.

CONCLUSIONS

This methodology accurately estimates organ donor potential within 3 geographic regions and lays the foundation for evaluating organ donation effectiveness nationwide. Additional research is needed to test the validity of the model in other geographic regions and to further explore organ donor potential in hospitals with fewer than 50 beds.

Using outcomes to make inferences about nursing home quality

Recent concerns about containing the growth of public expenditures on nursing home care and the development of prospective and casemix reimbursement systems with incentives for cost containment have increased the importance of monitoring quality in nursing homes. The current view is that quality assurance systems should include more outcome measures to improve quality. This article discusses why it is difficult to develop facility-level outcome measures that can be used to evaluate and compare the quality of care of nursing homes. The article places the current interest in outcomes measures in its historical policy context and reviews important conceptual and methodological issues associated with outcome-based quality assessment. The authors discuss the difficulty in isolating the facility effect when studying nursing home outcomes and implications of using different estimation approaches. In conclusion, they discuss the need to integrate research with outcome-based quality assurance systems to allow ongoing evaluation and quality improvement.

Variations in standardized hospital mortality rates for six common medical diagnoses: implications for profiling hospital quality

OBJECTIVES

The authors determined whether standardized hospital mortality rates varied for six common medical diagnoses.

METHODS

The retrospective cohort study included 89,851 patients aged 18 years and older discharged from 30 hospitals in a large metropolitan area in 1991 to 1993 with a principal diagnosis of acute myocardial infarction, congestive heart failure, pneumonia, stroke, obstructive lung disease, or gastrointestinal hemorrhage. For each hospital, standardized mortality ratios (observed/predicted mortality) were determined using validated risk-adjustment models that were based on clinical data elements abstracted from patients' hospital records. Hospitals also were categorized into quintiles on the basis of standardized mortality ratios. Correlations between standardized mortality ratios and agreement between quintile rankings were determined for each pair of diagnoses.

RESULTS

Correlations between hospital-standardized mortality ratios for individual diagnoses were generally weak. For the 15 possible pairs of diagnoses, Pearson coefficients ranged from -0.10 to 0.43; only six were 0.30 or greater. Agreement between hospital quintile rankings was also generally low, with weighted kappa values ranging from -0.12 to 0.42. Three of 15 kappa values were less than 0 (ie, agreement lower than chance), and only four exceeded 0.20, the threshold for "fair" agreement. Although simulated analyses found that random variation and relatively low hospital volumes accounted for some of the difference in standardized mortality ratios for diagnoses, a large proportion of the difference remained unexplained.

CONCLUSIONS

Standardized hospital mortality rates varied for six diagnoses that likely are managed by similar practitioners. Although variability may be decreased by restricting analyses to hospitals with large volumes, the findings indicate that for many hospitals, diagnosis-specific mortality rates may be an inconsistent measure of hospital quality, even when data are aggregated for multiple years.