Classification accuracy of claims-based methods for identifying providers failing to meet performance targets

The Importance of Outcomes Ascertainment for Accurate Assessment of the Mammography Screening Cancer Detection Rate: A Simulation Study

PURPOSE

Cancer detection rate (CDR), an important metric in the mammography screening audit, is designed to ensure adequate sensitivity. Most practices use biopsy results as the reference standard; however, commonly ascertainment of biopsy results is incomplete. We used simulation to determine the relationship between the cancer ascertainment rate of biopsy (AR-biopsy), CDR estimation, and associated error rates in classifying whether practices and radiologists meet the established ACR benchmark of 2.5 per 1,000.

MATERIALS AND METHODS

We simulated screening mammography volume, number of cancers detected, and CDR, using negative binomial and beta-binomial distributions, respectively. Simulations were performed at both the practice and radiologist level. Average CDR was based on linearly rescaling a published CDR by the AR-biopsy. CDR distributions were simulated for AR-biopsy between 5% and 100% in steps of five percentage points and were summarized with boxplots and smoothed histograms over the range of AR-biopsy, to quantify the proportion of practices and radiologists meeting the ACR benchmark at each level of AR-biopsy.

RESULTS

Decreasing AR-biopsy led to an increasing probability of categorizing CDR performance as being below the ACR benchmark. Our simulation predicts that at the practice level, an AR-biopsy of 65% categorizes 17.6% below the benchmark (compared to 1.6% at an AR-biopsy of 100%), and at the radiologist level, an AR-biopsy of 65% categorizes 34.7% as being below the benchmark (compared to 11.6% at an AR-biopsy of 100%).

CONCLUSIONS

Our simulation demonstrates that decreasing the AR-biopsy (in currently clinically relevant ranges) has the potential to artifactually lower the assessed CDR on both the practice and radiologist levels and may, in turn, increase the chance of erroneous categorization of underperformance per the ACR benchmark.

Statistical inference for association studies using electronic health records: handling both selection bias and outcome misclassification

Health research using electronic health records (EHR) has gained popularity, but misclassification of EHR-derived disease status and lack of representativeness of the study sample can result in substantial bias in effect estimates and can impact power and type I error. In this paper, we develop new strategies for handling disease status misclassification and selection bias in EHR-based association studies. We first focus on each type of bias separately. For misclassification, we propose three novel likelihood-based bias correction strategies. A distinguishing feature of the EHR setting is that misclassification may be related to patient-varying factors, and the proposed methods leverage data in the EHR to estimate misclassification rates without gold standard labels. For addressing selection bias, we describe how calibration and inverse probability weighting methods from the survey sampling literature can be extended and applied to the EHR setting. Addressing misclassification and selection biases simultaneously is a more challenging problem than dealing with each on its own, and we propose several new strategies. For all methods proposed, we derive valid standard error estimators and provide software for implementation. We provide a new suite of statistical estimation and inference strategies for addressing misclassification and selection bias simultaneously that is tailored to problems arising in EHR data analysis. We apply these methods to data from The Michigan Genomics Initiative, a longitudinal EHR-linked biorepository.

An analytic framework for exploring sampling and observation process biases in genome and phenome-wide association studies using electronic health records

Large-scale association analyses based on observational health care databases such as electronic health records have been a topic of increasing interest in the scientific community. However, challenges due to nonprobability sampling and phenotype misclassification associated with the use of these data sources are often ignored in standard analyses. The extent of the bias introduced by ignoring these factors is not well-characterized. In this paper, we develop an analytic framework for characterizing the bias expected in disease-gene association studies based on electronic health records when disease status misclassification and the sampling mechanism are ignored. Through a sensitivity analysis approach, this framework can be used to obtain plausible values for parameters of interest given summary results from standard analysis. We develop an online tool for performing this sensitivity analysis. Simulations demonstrate promising properties of the proposed method. We apply our approach to study bias in disease-gene association studies using electronic health record data from the Michigan Genomics Initiative, a longitudinal biorepository effort within The University Michigan health system.

The emerging landscape of health research based on biobanks linked to electronic health records: Existing resources, statistical challenges, and potential opportunities

Biobanks linked to electronic health records provide rich resources for health-related research. With improvements in administrative and informatics infrastructure, the availability and utility of data from biobanks have dramatically increased. In this paper, we first aim to characterize the current landscape of available biobanks and to describe specific biobanks, including their place of origin, size, and data types. The development and accessibility of large-scale biorepositories provide the opportunity to accelerate agnostic searches, expedite discoveries, and conduct hypothesis-generating studies of disease-treatment, disease-exposure, and disease-gene associations. Rather than designing and implementing a single study focused on a few targeted hypotheses, researchers can potentially use biobanks' existing resources to answer an expanded selection of exploratory questions as quickly as they can analyze them. However, there are many obvious and subtle challenges with the design and analysis of biobank-based studies. Our second aim is to discuss statistical issues related to biobank research such as study design, sampling strategy, phenotype identification, and missing data. We focus our discussion on biobanks that are linked to electronic health records. Some of the analytic issues are illustrated using data from the Michigan Genomics Initiative and UK Biobank, two biobanks with two different recruitment mechanisms. We summarize the current body of literature for addressing these challenges and discuss some standing open problems. This work complements and extends recent reviews about biobank-based research and serves as a resource catalog with analytical and practical guidance for statisticians, epidemiologists, and other medical researchers pursuing research using biobanks.

Estimation of data adaptive minimal clinically important difference with a nonconvex optimization procedure

Understanding the limitation of solely relying on statistical significance, researchers have proposed methods to draw biomedical conclusions based on clinical significance. The minimal clinically important significance is one of the most fundamental concepts to study clinical significance. Based on an anchor question usually available in the patients' reported outcome, Hedayat et al. presented a method to estimate minimal clinically important significance using the classification technique. However, their method implicitly requires that the binary outcome of the anchor question is equally likely, i.e. the balanced outcome assumption. This assumption cannot be guaranteed a priori when one designs the study; hence, it cannot be satisfied in general. In this paper, we propose a data adaptive method, which can overcome this limitation. Compared to Hedayat et al., our method uses a faster gradient based algorithm and adopts a more flexible structure of the minimal clinically important significance at the individual level. We conduct comprehensive simulation studies and apply our method to the chondral lesions and meniscus procedure study to demonstrate its usefulness and also its outperformance.

Outlier classification performance of risk adjustment methods when profiling multiple providers

BACKGROUND

When profiling multiple health care providers, adjustment for case-mix is essential to accurately classify the quality of providers. Unfortunately, misclassification of provider performance is not uncommon and can have grave implications. Propensity score (PS) methods have been proposed as viable alternatives to conventional multivariable regression. The objective was to assess the outlier classification performance of risk adjustment methods when profiling multiple providers.

METHODS

In a simulation study based on empirical data, the classification performance of logistic regression (fixed and random effects), PS adjustment, and three PS weighting methods was evaluated when varying parameters such as the number of providers, the average incidence of the outcome, and the percentage of outliers. Traditional classification accuracy measures were considered, including sensitivity and specificity.

RESULTS

Fixed effects logistic regression consistently had the highest sensitivity and negative predictive value, yet a low specificity and positive predictive value. Of the random effects methods, PS adjustment and random effects logistic regression performed equally well or better than all the remaining PS methods for all classification accuracy measures across the studied scenarios.

CONCLUSIONS

Of the evaluated PS methods, only PS adjustment can be considered a viable alternative to random effects logistic regression when profiling multiple providers in different scenarios.

The Importance of Integrating Clinical Relevance and Statistical Significance in the Assessment of Quality of Care--Illustrated Using the Swedish Stroke Register

Background

When profiling hospital performance, quality inicators are commonly evaluated through hospital-specific adjusted means with confidence intervals. When identifying deviations from a norm, large hospitals can have statistically significant results even for clinically irrelevant deviations while important deviations in small hospitals can remain undiscovered. We have used data from the Swedish Stroke Register (Riksstroke) to illustrate the properties of a benchmarking method that integrates considerations of both clinical relevance and level of statistical significance.

Methods

The performance measure used was case-mix adjusted risk of death or dependency in activities of daily living within 3 months after stroke. A hospital was labeled as having outlying performance if its case-mix adjusted risk exceeded a benchmark value with a specified statistical confidence level. The benchmark was expressed relative to the population risk and should reflect the clinically relevant deviation that is to be detected. A simulation study based on Riksstroke patient data from 2008–2009 was performed to investigate the effect of the choice of the statistical confidence level and benchmark value on the diagnostic properties of the method.

Results

Simulations were based on 18,309 patients in 76 hospitals. The widely used setting, comparing 95% confidence intervals to the national average, resulted in low sensitivity (0.252) and high specificity (0.991). There were large variations in sensitivity and specificity for different requirements of statistical confidence. Lowering statistical confidence improved sensitivity with a relatively smaller loss of specificity. Variations due to different benchmark values were smaller, especially for sensitivity. This allows the choice of a clinically relevant benchmark to be driven by clinical factors without major concerns about sufficiently reliable evidence.

Conclusions

The study emphasizes the importance of combining clinical relevance and level of statistical confidence when profiling hospital performance. To guide the decision process a web-based tool that gives ROC-curves for different scenarios is provided.