Investigating Risk Adjustment Methods for Health Care Provider Profiling When Observations are Scarce or Events Rare

Dealing with confounding in observational studies: A scoping review of methods evaluated in simulation studies with single-point exposure

The aim of this article was to perform a scoping review of methods available for dealing with confounding when analyzing the effect of health care treatments with single-point exposure in observational data. We aim to provide an overview of methods and their performance assessed by simulation studies indexed in PubMed. We searched PubMed for simulation studies published until January 2021. Our search was restricted to studies evaluating binary treatments and binary and/or continuous outcomes. Information was extracted on the methods' assumptions, performance, and technical properties. Of 28,548 identified references, 127 studies were eligible for inclusion. Of them, 84 assessed 14 different methods (ie, groups of estimators that share assumptions and implementation) for dealing with measured confounding, and 43 assessed 10 different methods for dealing with unmeasured confounding. Results suggest that there are large differences in performance between methods and that the performance of a specific method is highly dependent on the estimator. Furthermore, the methods' assumptions regarding the specific data features also substantially influence the methods' performance. Finally, the methods result in different estimands (ie, target of inference), which can even vary within methods. In conclusion, when choosing a method to adjust for measured or unmeasured confounding it is important to choose the most appropriate estimand, while considering the population of interest, data structure, and whether the plausibility of the methods' required assumptions hold.

Constructing inverse probability weights for institutional comparisons in healthcare

In comparing quality of care between hospitals, disease-specific quality indicators measure structural, process, or outcome elements related to the care of a particular condition. Such comparisons can be framed in terms of causal contrasts, answering the question of whether a patient (or a population of patients on average) would receive different care if treated at the care level of a different hospital. Fair comparisons have to be adjusted for patient case-mix, which is equivalent to controlling for confounding by the patient-level factors, including demographic factors, comorbidities, and disease progression. The methodological choice for such comparisons is usually between direct and indirect standardization methods. In this article, we discuss the alternative of inverse probability weighting as a tool for standardization in hospital comparisons. This involves fitting multinomial logistic hospital assignment models and using these to construct the inverse probability weights. The challenge in the present context is the presence of large number of hospitals being compared, many of which have a small patient volume. We propose methods to include small categories in the weighted analysis, as well as metrics and visualizations for checking the positivity/overlap and covariate balance in constructing such weights. The methods are illustrated in a running example using linked administrative data on surgical treatment of kidney cancer patients in Ontario.

Outlier classification performance of risk adjustment methods when profiling multiple providers

BACKGROUND

When profiling multiple health care providers, adjustment for case-mix is essential to accurately classify the quality of providers. Unfortunately, misclassification of provider performance is not uncommon and can have grave implications. Propensity score (PS) methods have been proposed as viable alternatives to conventional multivariable regression. The objective was to assess the outlier classification performance of risk adjustment methods when profiling multiple providers.

METHODS

In a simulation study based on empirical data, the classification performance of logistic regression (fixed and random effects), PS adjustment, and three PS weighting methods was evaluated when varying parameters such as the number of providers, the average incidence of the outcome, and the percentage of outliers. Traditional classification accuracy measures were considered, including sensitivity and specificity.

RESULTS

Fixed effects logistic regression consistently had the highest sensitivity and negative predictive value, yet a low specificity and positive predictive value. Of the random effects methods, PS adjustment and random effects logistic regression performed equally well or better than all the remaining PS methods for all classification accuracy measures across the studied scenarios.

CONCLUSIONS

Of the evaluated PS methods, only PS adjustment can be considered a viable alternative to random effects logistic regression when profiling multiple providers in different scenarios.