Assessing the accuracy of profiling methods for identifying top providers: performance of mental health care providers

Reformulating provider profiling by grouping providers treating similar patients prior to evaluating performance

Standard approaches to comparing health providers' performance rely on hierarchical logistic regression models that adjust for patient characteristics at admission. Estimates from these models may be misleading when providers treat different patient populations and the models are misspecified. To address this limitation, we propose a novel profiling approach that identifies groups of providers treating similar populations of patients and then evaluates providers' performance within each group. The groups of providers are identified using a Bayesian multilevel finite mixture of general location models. To compare the performance of our proposed profiling approach to standard methods, we use patient-level data from 119 skilled nursing facilities in Massachusetts. We use simulated and observed outcome data to explore the performance of these profiling methods in different settings. In simulations, our proposed method classifies providers to groups with similar patients' admission characteristics. In addition, in the presence of limited overlap in patient characteristics across providers and misspecifications of the outcome model, the provider-level estimates obtained using our approach identified providers that under- and overperformed compared to the standard regression-based approaches more accurately.

Assessing the Impacts of Misclassified Case-Mix Factors on Health Care Provider Profiling: Performance of Dialysis Facilities

Quantitative metrics are used to develop profiles of health care institutions, including hospitals, nursing homes, and dialysis clinics. These profiles serve as measures of quality of care, which are used to compare institutions and determine reimbursement, as a part of a national effort led by the Center for Medicare and Medicaid Services in the United States. However, there is some concern about how misclassification in case-mix factors, which are typically accounted for in profiling, impacts results. We evaluated the potential effect of misclassification on profiling results, using 20 744 patients from 2740 dialysis facilities in the US Renal Data System. In this case study, we compared 30-day readmission as the profiling outcome measure, using comorbidity data from either the Center for Medicare and Medicaid Services Medical Evidence Report (error-prone) or Medicare claims (more accurate). Although the regression coefficient of the error-prone covariate demonstrated notable bias in simulation, the outcome measure—standardized readmission ratio—and profiling results were quite robust; for example, correlation coefficient of 0.99 in standardized readmission ratio estimates. Thus, we conclude that misclassification on case-mix did not meaningfully impact overall profiling results. We also identified both extreme degree of case-mix factor misclassification and magnitude of between-provider variability as 2 factors that can potentially exert enough influence on profile status to move a clinic from one performance category to another (eg, normal to worse performer).

A Strategy for Risk-adjusted Ranking of Surgeons and Practices Based on Patient-reported Outcomes After Elective Lumbar Surgery

STUDY DESIGN

This study retrospectively analyzes prospectively collected data.

OBJECTIVE

The primary aim of this study is to present a scheme for patient-reported outcome (PRO)-based, risk-adjusted rankings of spine surgeons and sites that perform elective lumbar surgery, using the Quality and Outcomes Database (QOD).

SUMMARY OF BACKGROUND DATA

There is currently no means of determining which spine surgeons or centers are positive or negative outliers with respect to PROs for elective lumbar surgery. This is a critical gap as we move toward a value-based model of health care in which providers assume more accountability for the effectiveness of their treatments.

METHODS

Random effects regression models were fit for the following outcomes, with QOD site as a fixed effect but surgeon ID as a random effect: Oswestry Disability Index, EQ-5D, back pain and leg pain, and satisfaction. Hierarchical Bayesian models were also fit for each outcome, with QOD site as a random effect and surgeon as a nested random effect.

RESULTS

Our study cohort consists of 8834 patients who underwent surgery by 124 QOD surgeons, for the degenerative lumbar diseases. Nonoverlapping Bayesian credible intervals demonstrate that the variance attributed to QOD site was greater than the nested variance attributed to surgeon ID for the included PROs.

CONCLUSION

This study presents a novel strategy for the risk-adjusted, PRO-based ranking of spine surgeons and practices. This can help identify positive and negative outliers, thereby forming the basis for large-scale quality improvement. Assuming adequate coverage of baseline risk adjustment, the choice of surgeon matters when considering PROs after lumbar surgery; however, the choice of site appears to matter more.

LEVEL OF EVIDENCE

3.

Albuminuria measurement in diabetic care: a multilevel analysis measuring the influence of accreditation on institutional performance

Background

Studies assessing institutional performance regarding quality of care are frequently performed using single-level statistical analyses investigating differences between provider averages of various quality indicators. However, such analyses are insufficient as they do not consider patients’ heterogeneity around those averages. Hence, we apply a multilevel analysis of individual-patient heterogeneity that distinguishes between ‘general’ (‘latent quality’ or measures of variance) and ‘specific’ (measures of association) contextual effects. We assess general contextual effects of the hospital departments and the specific contextual effect of a national accreditation programme on adherence to the standard benchmark for albuminuria measurement in Danish patients with diabetes.

Methods

From the Danish Adult Diabetes Database, we extracted data on 137 893 patient cases admitted to hospitals between 2010 and 2013. Applying multilevel logistic and probit regression models for every year, we quantified general contextual effects of hospital department by the intraclass correlation coefficient (ICC) and the area under the receiver operating characteristic curve (AUC) values. We evaluated the specific effect of hospital accreditation using the ORs and the change in the department variance.

Results

In 2010, the department context had considerable influence on adherence with albuminuria measurement (ICC=21.8%, AUC=0.770), but the general effect attenuated along with the implementation of the national accreditation programme. The ICC value was 16.5% in 2013 and the rate of compliance with albuminuria measurement increased from 91.6% in 2010 to 96% in 2013.

Conclusions

Parallel to implementation of the national accreditation programme, departments’ compliance with the standard benchmark for albuminuria measurement increased and the ICC values decreased, but remained high. While those results indicate an overall quality improvement, further intervention focusing on departments with the lowest compliance could be considered.

The role of the clinical departments for understanding patient heterogeneity in one-year mortality after a diagnosis of heart failure: A multilevel analysis of individual heterogeneity for profiling provider outcomes

Purpose

To evaluate the general contextual effect (GCE) of the hospital department on one-year mortality in Swedish and Danish patients with heart failure (HF) by applying a multilevel analysis of individual heterogeneity.

Methods

Using the Swedish patient register, we obtained data on 36,943 patients who were 45–80 years old and admitted for HF to the hospital between 2007 and 2009. From the Danish Heart Failure Database (DHFD), we obtained data on 12,001 patients with incident HF who were 18 years or older and treated at hospitals between June 2010 and June2013. For each year, we applied two-step single and multilevel logistic regression models. We evaluated the general effects of the department by quantifying the intra-class correlation coefficient (ICC) and the increment in the area under the receiver operating characteristic curve (AUC) obtained by adding the random effects of the department in a multilevel logistic regression analysis.

Results

One-year mortality for Danish incident HF patients was low in the three audit years (around 11.1% -13.1%) and departments performed homogeneously (ICC ≈1.5% - 3.5%). The discriminatory accuracy of a model including age and gender was rather high (AUC≈ 0.71–0.73) but the increment in AUC after adding the department random effects into these models was only about 0.011–0.022 units in the three years.

One-year mortality in Swedish patients with first hospitalization for heart failure, was relatively higher for 2007–2009 (≈21.3% - 22%) and departments performed homogeneously (ICC ≈ 1.5% - 3%). The discriminatory accuracy of a model including age, gender and patient risk score was rather high (AUC≈ 0.726–0.728) but the increment in AUC after adding the department random effects was only about 0.010–0.017 units in the three years.

Conclusion

Using the DHFD standard benchmark for one-year mortality, Danish departments had a good, homogeneous performance. In reference to literature, Swedish departments had a homogeneous performance and the mortality rates for patients with first hospitalization for heart failure were similar to those reported since 2000. Considering this, if health authorities decide to further reduce mortality rates, a comprehensive quality strategy should focus on all Swedish hospitals. Yet, a complementary assessment for the period after the study period is required to confirm whether department performance is still homogeneous or not to determine the most appropriate action.

Reliability of Physician-Level Measures of Patient Experience in Primary Care

BACKGROUND

Patient experience measures are widely used to compare performance at the individual physician level.

OBJECTIVE

To assess the impact of unmeasured patient characteristics on visit-level patient experience measures and the sample sizes required to reliably measure patient experience at the primary care physician (PCP) level.

DESIGN

Repeated cross-sectional design.

SETTING

Academic family medicine practice in California.

PARTICIPANTS

One thousand one hundred forty-one adult patients attending 1319 visits with 56 PCPs (including 45 resident and 11 faculty physicians).

MEASUREMENTS

Post-visit patient experience surveys including patient measures used for standard adjustment as recommend by the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Consortium and additional patient characteristics used for expanded adjustment (including attitudes toward healthcare, global life satisfaction, patient personality, current symptom bother, and marital status).

RESULTS

The amount of variance in patient experience explained doubled with expanded adjustment for patient characteristics compared with standard adjustment (R² = 20.0% vs. 9.6%, respectively). With expanded adjustment, the amount of variance attributable to the PCP dropped from 6.1% to 3.4% and the required sample size to achieve a reliability of 0.90 in the physician-level patient experience measure increased from 138 to 255 patients per physician. After ranking of the 56 PCPs by average patient experience, 8 were reclassified into or out of the top or bottom quartiles of average experience with expanded as compared to standard adjustment [14.3% (95% CI: 7.0-25.2%)].

CONCLUSIONS

Widely used methods for measuring PCP-level patient experience may not account sufficiently for influential patient characteristics. If methods were adapted to account for these characteristics, patient sample sizes for reliable between-physician comparisons may be too large for most practices to obtain.

Provider Differences in Use of Implanted Ports in Older Adults With Cancer

BACKGROUND

Identifying unwarranted variation in health care can highlight opportunities to reduce harm. One often discretionary process in oncology is use of implanted ports to administer intravenous chemotherapy. While there are benefits, ports carry risks. This study's objective was to assess provider-driven variation in port use among cancer patients receiving chemotherapy.

RESEARCH DESIGN

Retrospective assessment using population-based SEER-Medicare data to assess differences in port use across health care providers of older adults with cancer. Participants included over 18,000 patients ages 66 and older diagnosed with breast, colorectal, lung, or pancreatic cancer in 2005-2007, treated by approximately 2900 providers. We identified port use for patients receiving treatment from hospital outpatient facilities versus physicians' offices. Our main analysis assessed the likelihood of a patient receiving a port given port use by the provider's last patient. For a subset of high-use providers, we examined individual provider-level variation by estimating the risk-adjusted likelihood of insertion.

RESULTS

Patients receiving chemotherapy in hospital outpatient facilities were significantly less likely to receive a port than those treated in physicians' offices, with adjusted odds ratios (AOR) varying from 0.50 to 0.75 across cancer sites. Implanting a port was associated with increased likelihood of port insertion in the provider's next patient (AOR varied from 1.71 to 2.25). Significant between-provider variation was found among providers with at least 10 patients.

CONCLUSIONS

Our findings support the idea that there is provider-driven variation in the use of ports for chemotherapy administration. This variation highlights an opportunity to standardize practice and reduce unnecessary use.