Detecting and visualizing outliers in provider profiling via funnel plots and mixed effect models

Contribution of medico-administrative databases to health vigilance: example of post-oocyte retrieval infections

STUDY QUESTION

Can we monitor post-oocyte retrieval infections in the French national health data system to complement the French ART vigilance system?

SUMMARY ANSWER

Medico-administrative databases provide a more comprehensive view of post-oocyte retrieval infections and can be used to detect abnormal increases in frequency and outlier ART centers as a complementary tool to the ART vigilance system.

WHAT IS KNOWN ALREADY

The various studies of ART complications are reassuring, showing relatively low overall complication rates. Nonetheless, the European Union has set up a vigilance system to monitor these complications. However, this system is not an exhaustive source of information and does not provide a complete overview of post-ART complications.

STUDY DESIGN, SIZE, DURATION

The study population was identified from the comprehensive French national hospital discharge database. It included women under 46 years of age undergoing an oocyte retrieval in 2019, classified into three population subgroups according to the indication of oocyte retrieval: infertility (IF) , fertility preservation (FP), and oocyte donation (OD) . The study population included 52 098 women who had undergone 65 948 oocyte retrievals in 2019.

PARTICIPANTS/MATERIALS, SETTING, METHODS

Hospital stays and delivery of antibiotics within 31 days after oocyte retrieval were analyzed. Women and infections were characterized according to various characteristics (age, comorbidities, indication of oocyte retrieval, type of hospital stay, length of hospital stay, type of antibiotherapy, etc.). Multivariate analysis was performed to determine the relation between the occurrence of infection and women's characteristics, and results are expressed as odds ratios (ORs) and 95% CI. A funnel plot and a box plot were used to compare the infection rate per center with the national average and to detect outliers.

MAIN RESULTS AND THE ROLE OF CHANCE

Infections in the month following the oocyte retrieval represented 6.9% of the procedures in 2019 (n = 4522). Of these infections, 112 were hospitalized (0.2% of oocyte retrievals), and 4410 were non-hospitalized (6.7% of oocyte retrievals). The hospitalized infections were essentially gynecological infections (40.9%) and urinary tract infections (23.5%). In 87.9% of non-hospitalized infections, a single antibiotic therapy was prescribed. Mixed-effect model analysis showed that the risk of infection was significantly higher in women under 30 years of age, in the FP population, in supplementary universal health coverage (CMU-C) beneficiaries, and women with endometriosis. Funnel plot and box plot analysis showed that three ART centers have an infection rate significantly higher than the national average. In the three centers that stand out from all the others, the objective is to return to these centers to understand the possible reasons for this observed rate and to implement corrective measures.

LIMITATIONS, REASONS FOR CAUTION

Despite all its advantages, the French national health data system presents some limitations, such as the risk of inappropriate coding. Another limitation of this study is that we cannot confirm an attributable relation between the infection and the ART procedure, even if the delay of 31 days after oocyte retrieval is consistent with the occurrence of a post-retrieval complication. In addition, antibiotics may be prescribed as a 'precautionary' measure in certain situations (women with a susceptibility to infection, complicated procedures), or as antibiotic prophylaxis for embryo transfer.

WIDER IMPLICATIONS OF THE FINDINGS

Despite the limits in identifying post-ART infections in medico-administrative databases, this approach is a promising way to complement the ART vigilance reporting system. This concept developed for infections will also be generalized to other complications with regular feedback to professionals.

STUDY FUNDING/COMPETING INTEREST(S)

No specific funding was sought for the study. The study was supported by the Agence de la biomédecine, France. The authors declare that they have no conflict of interest.

TRIAL REGISTRATION NUMBER

N/A.

Reformulating provider profiling by grouping providers treating similar patients prior to evaluating performance

Standard approaches to comparing health providers' performance rely on hierarchical logistic regression models that adjust for patient characteristics at admission. Estimates from these models may be misleading when providers treat different patient populations and the models are misspecified. To address this limitation, we propose a novel profiling approach that identifies groups of providers treating similar populations of patients and then evaluates providers' performance within each group. The groups of providers are identified using a Bayesian multilevel finite mixture of general location models. To compare the performance of our proposed profiling approach to standard methods, we use patient-level data from 119 skilled nursing facilities in Massachusetts. We use simulated and observed outcome data to explore the performance of these profiling methods in different settings. In simulations, our proposed method classifies providers to groups with similar patients' admission characteristics. In addition, in the presence of limited overlap in patient characteristics across providers and misspecifications of the outcome model, the provider-level estimates obtained using our approach identified providers that under- and overperformed compared to the standard regression-based approaches more accurately.

Evaluations of statistical methods for outlier detection when benchmarking in clinical registries: a systematic review

OBJECTIVES

Benchmarking is common in clinical registries to support the improvement of health outcomes by identifying underperforming clinician or health service providers. Despite the rise in clinical registries and interest in publicly reporting benchmarking results, appropriate methods for benchmarking and outlier detection within clinical registries are not well established, and the current application of methods is inconsistent. The aim of this review was to determine the current statistical methods of outlier detection that have been evaluated in the context of clinical registry benchmarking.

DESIGN

A systematic search for studies evaluating the performance of methods to detect outliers when benchmarking in clinical registries was conducted in five databases: EMBASE, ProQuest, Scopus, Web of Science and Google Scholar. A modified healthcare modelling evaluation tool was used to assess quality; data extracted from each study were summarised and presented in a narrative synthesis.

RESULTS

Nineteen studies evaluating a variety of statistical methods in 20 clinical registries were included. The majority of studies conducted application studies comparing outliers without statistical performance assessment (79%), while only few studies used simulations to conduct more rigorous evaluations (21%). A common comparison was between random effects and fixed effects regression, which provided mixed results. Registry population coverage, provider case volume minimum and missing data handling were all poorly reported.

CONCLUSIONS

The optimal methods for detecting outliers when benchmarking clinical registry data remains unclear, and the use of different models may provide vastly different results. Further research is needed to address the unresolved methodological considerations and evaluate methods across a range of registry conditions.

PROSPERO REGISTRATION NUMBER

CRD42022296520.

Test-specific funnel plots for healthcare provider profiling leveraging individual- and summary-level information

In addition to applications in meta-analysis, funnel plots have emerged as an effective graphical tool for visualizing the detection of health care providers with unusual performance. Although there already exist a variety of approaches to producing funnel plots in the literature of provider profiling, limited attention has been paid to elucidating the critical relationship between funnel plots and hypothesis testing. Within the framework of generalized linear models, here we establish methodological guidelines for creating funnel plots specific to the statistical tests of interest. Moreover, we show that the test-specific funnel plots can be created merely leveraging summary statistics instead of individual-level information. This appealing feature inhibits the leak of protected health information and reduces the cost of inter-institutional data transmission. Two data examples, one for surgical patients from Michigan hospitals and the other for Medicare-certified dialysis facilities, demonstrate the applicability to different types of providers and outcomes with either individual- or summary-level information.

Measuring severe neonatal morbidity using hospital discharge data in France

BACKGROUND

Measuring infant health at birth is key for surveillance and research in obstetrics and neonatology, but there is no international consensus on morbidity indicators. The Neonatal Adverse Outcome Indicator (NAOI) is a composite indicator, developed in Australia, which measures the burden of severe neonatal morbidity using hospital discharge data.

OBJECTIVE

To evaluate the applicability of the NAOI in France for surveillance and research.

METHODS

We constituted a cohort of live births ≥24 weeks' gestational age in Metropolitan France from 2014 to 2015 using hospital discharge, insurance claims and cause of death data. Outlier hospitals were identified using funnel plots of standardised morbidity ratios (SMR), and their coding patterns were assessed. We compared the NAOI and its component codes with published Australian and English data and estimated unadjusted and adjusted risk ratios for known risk factors for neonatal morbidity.

RESULTS

We included 1,459,123 births (511 hospitals). Twenty-eight hospitals had SMR above funnel plot control limits. Newborns with NAOI morbidities in these hospitals had lower mortality and shorter stays than in other hospitals. Amongst within-limit hospitals, NAOI prevalence was 4.8%, comparable to Australia (4.6%) and England (5.4%). Most individual components had a similar prevalence, with the exception of respiratory support, intravenous fluid procedures and infection. NAOI was lowest at 39 weeks (2.2%) with higher risks for maternal age ≥40 (relative risk [RR] 1.47, 95% confidence interval [CI] 1.42, 1.51), state medical insurance (RR 1.60, 95% CI 1.52, 1.68), male sex (RR 1.21, 95% CI 1.19, 1.23) and birthweight <3^rd percentile (RR 4.60, 95% CI 4.51, 4.69).

CONCLUSIONS

The NAOI provides valuable information on population prevalence of severe neonatal morbidity and its risk factors. Whilst the prevalence was similar in high-income countries with comparable neonatal mortality levels, ensuring valid comparisons between countries and hospitals will require further work to harmonize coding procedures, especially for infection and respiratory morbidity.

Qualitative Data Clustering to Detect Outliers

Detecting outliers is a widely studied problem in many disciplines, including statistics, data mining, and machine learning. All anomaly detection activities are aimed at identifying cases of unusual behavior compared to most observations. There are many methods to deal with this issue, which are applicable depending on the size of the data set, the way it is stored, and the type of attributes and their values. Most of them focus on traditional datasets with a large number of quantitative attributes. The multitude of solutions related to detecting outliers in quantitative sets, a large and still has a small number of research solutions is a problem detecting outliers in data containing only qualitative variables. This article was designed to compare three different categorical data clustering algorithms: K-modes algorithm taken from MacQueen’s K-means algorithm and the STIRR and ROCK algorithms. The comparison concerned the method of dividing the set into clusters and, in particular, the outliers detected by algorithms. During the research, the authors analyzed the clusters detected by the indicated algorithms, using several datasets that differ in terms of the number of objects and variables. They have conducted experiments on the parameters of the algorithms. The presented study made it possible to check whether the algorithms similarly detect outliers in the data and how much they depend on individual parameters and parameters of the set, such as the number of variables, tuples, and categories of a qualitative variable.

Development and validation of the predictive risk of death model for adult patients admitted to intensive care units in Japan: an approach to improve the accuracy of healthcare quality measures

BACKGROUND

The Acute Physiology and Chronic Health Evaluation (APACHE) III-j model is widely used to predict mortality in Japanese intensive care units (ICUs). Although the model's discrimination is excellent, its calibration is poor. APACHE III-j overestimates the risk of death, making its evaluation of healthcare quality inaccurate. This study aimed to improve the calibration of the model and develop a Japan Risk of Death (JROD) model for benchmarking purposes.

METHODS

A retrospective analysis was conducted using a national clinical registry of ICU patients in Japan. Adult patients admitted to an ICU between April 1, 2018, and March 31, 2019, were included. The APACHE III-j model was recalibrated with the following models: Model 1, predicting mortality with an offset variable for the linear predictor of the APACHE III-j model using a generalized linear model; model 2, predicting mortality with the linear predictor of the APACHE III-j model using a generalized linear model; and model 3, predicting mortality with the linear predictor of the APACHE III-j model using a hierarchical generalized additive model. Model performance was assessed with the area under the receiver operating characteristic curve (AUROC), the Brier score, and the modified Hosmer-Lemeshow test. To confirm model applicability to evaluating quality of care, funnel plots of the standardized mortality ratio and exponentially weighted moving average (EWMA) charts for mortality were drawn.

RESULTS

In total, 33,557 patients from 44 ICUs were included in the study population. ICU mortality was 3.8%, and hospital mortality was 8.1%. The AUROC, Brier score, and modified Hosmer-Lemeshow p value of the original model and models 1, 2, and 3 were 0.915, 0.062, and < .001; 0.915, 0.047, and < .001; 0.915, 0.047, and .002; and 0.917, 0.047, and .84, respectively. Except for model 3, the funnel plots showed overdispersion. The validity of the EWMA charts for the recalibrated models was determined by visual inspection.

CONCLUSIONS

Model 3 showed good performance and can be adopted as the JROD model for monitoring quality of care in an ICU, although further investigation of the clinical validity of outlier detection is required. This update method may also be useful in other settings.

Multi-level models for heart failure patients' 30-day mortality and readmission rates: the relation between patient and hospital factors in administrative data

Background

This study aims at gathering evidence about the relation between 30-day mortality and 30-day unplanned readmission and patient and hospital factors. By definition, we refer to 30-day mortality and 30-day unplanned readmission as the number of deaths and non-programmed hospitalizations for any cause within 30 days after the incident heart failure (HF). In particular, the focus is on the role played by hospital-level factors.

Methods

A multi-level logistic model that combines patient- and hospital-level covariates has been developed to better disentangle the role played by the two groups of covariates. Later on, hospital outliers in term of better-than-expected/worst-than-expected performers have been identified by comparing expected cases vs. observed cases. Hospitals performance in terms of 30-day mortality and 30-day unplanned readmission rates have been visualized through the creation of funnel plots. Covariates have been selected coherently to past literature. Data comes from the hospital discharge forms for Heart Failure patients in the Lombardy Region (Northern Italy). Considering incident cases for HF in the timespan 2010–2012, 78,907 records for adult patients from 117 hospitals have been collected after quality checks.

Results

Our results show that 30-day mortality and 30-day unplanned readmissions are explained by hospital-level covariates, paving the way for the design and implementation of evidence-based improvement strategies. While the percentage of surgical DRG (OR = 1.001; CI (1.000–1.002)) and the hospital type of structure (Research hospitals vs. non-research public hospitals (OR = 0.62; CI (0.48–0.80)) and Non-research private hospitals vs. non-research hospitals OR = 0.75; CI (0.63–0.90)) are significant for mortality, the mean length of stay (OR = 0.96; CI (0.95–0.98)) is significant for unplanned readmission, showing that mortality and readmission rates might be improved through different strategies.

Conclusion

Our results confirm that hospital-level covariates do affect quality of care, and that 30-day mortality and 30-day unplanned readmission are affected by different managerial choices. This confirms that hospitals should be accountable for their “added value” to quality of care.

Detecting and Visualizing Outliers in Provider Profiling Using Funnel Plots and Mixed Effects Models-An Example from Prescription Claims Data

When prescribing a drug for a patient, a physician also has to consider economic aspects. We were interested in the feasibility and validity of profiling based on funnel plots and mixed effect models for the surveillance of German ambulatory care physicians’ prescribing. We analyzed prescriptions issued to patients with a health insurance card attending neurologists’ and psychiatrists’ ambulatory practices in the German federal state of Saarland. The German National Association of Statutory Health Insurance Physicians developed a prescribing assessment scheme (PAS) which contains a systematic appraisal of the benefit of drugs for so far 12 different indications. The drugs have been classified on the basis of their clinical evidence as “standard”, “reserve” or “third level” medication. We had 152.583 prescriptions in 56 practices available for analysis. A total of 38.796 patients received these prescriptions. The funnel plot approach with additive correction for overdispersion was almost equivalent to a mixed effects model which directly took the multilevel structure of the data into account. In the first case three practices were labeled as outliers, the mixed effects model resulted in two outliers. We suggest that both techniques should be routinely applied within a surveillance system of prescription claims data.

Primary percutaneous coronary intervention at centers with and without on-site surgical support: Insights from the Blue Cross Blue Shield of Michigan Cardiovascular Consortium (BMC2)

BACKGROUND

Primary percutaneous coronary intervention (PPCI) is being increasingly performed nationally at sites without on-site cardiac surgery; however, recent guidelines only provide a Class IIa recommendation for this practice. The state of Michigan has permitted PPCI without on-site surgery under a closely monitored system that mandates auditing of all procedures and quarterly feedback on quality and outcomes. This study sought to compare outcomes of patients undergoing PPCI at centers with and without on-site surgery in the state of Michigan.

METHODS

Consecutive patients who underwent PPCI at 47 hospitals in Michigan from January 2010 to December 2015 were included. From this cohort, 4,091 patients from sites with and without on-site cardiac surgery were propensity matched in a 1:1 fashion to compare baseline characteristics, procedural details, and in-hospital outcomes.

RESULTS

Of the 25,886 PPCIs performed at 47 hospitals in Michigan from 2010 to 2015, 21,610 (83.5%) were performed at sites with on-site surgery and 4,276 (16.5%) at sites without on-site surgery. Using propensity score matched cohorts (4,091 patients for each site type), we found no significant differences in baseline characteristics. Overall mortality (5.4% vs 5.8%; P=.442); composite outcome of in-hospital mortality, contrast-induced nephropathy, bleeding, and stroke (13.8% vs 12.8%; P=.152); and individual outcomes within the composite group showed no significant differences. Additionally, there were no clinically meaningful differences in rates of urgent/emergent coronary artery bypass graft or length of stay. Significant differences, however, were found in procedural access site, antiplatelet therapy, contrast volume, and anticoagulant strategy.

CONCLUSIONS

Primary PCI performed at centers with and without cardiac surgery have comparable outcomes and complication rates when performed with close monitoring of quality and outcomes.

Investigating consistent patterns of variation in short-notice cancellations of elective operations: The potential for learning and improvement through multi-site evaluations

As part of efforts to increase productivity in healthcare, there is considerable interest in the extent and causes of variation in the performance of provider organisations. In this study, we use publically available data from the English NHS to examine the characteristics of variation in the rates of short-notice cancellations of elective operations due to hospital reasons (e.g. lack of the required resources such as operating theatres and beds). We find that the variation between hospital trusts is very non-random. There is a fourfold difference in the cancellation rates between the top and bottom deciles of performance. Little is known about the causes of this. There is a large and striking consistency in the relative performance of hospital trusts on cancellation rates over the last five years. Thus, the best and worst performers are consistently relatively very good or very poor, so a multi-site comparison of practices, and accounting for confounds like patient demographics, could be very valuable to inform both this knowledge gap and practice in healthcare. Of particular interest is that the cancellation rates could be a symptom of deeper issues with the efficiency of patient flows within hospitals.

Appropriateness and Outcomes of Percutaneous Coronary Intervention at Top-Ranked and Nonranked Hospitals in the United States

OBJECTIVES

This study sought to compare the appropriate use and outcomes of percutaneous coronary intervention (PCI) between top-ranked and nonranked hospitals.

BACKGROUND

The U.S. News & World Report "Best Hospitals" rankings are an influential consumer-directed publication of hospital quality, and are commonly used in promotional campaigns by hospital systems.

METHODS

Hospitals in the National Cardiovascular Data Registry CathPCI registry between July 1, 2014, and June 30, 2015, were classified as top-ranked if they were included in the 2015 U.S. News & World Report 50 best "Cardiology and Heart Surgery" hospitals. The remaining were classified as nonranked. We compared in-hospital mortality, post-procedural bleeding, post-procedural acute kidney injury, and the proportion of appropriate PCI procedures between top-ranked and nonranked hospitals.

RESULTS

A total of 509,153 PCIs at 654 hospitals were included, of which 55,550 (10.9%) were performed at 44 top-ranked hospitals. After adjusting for patient case mix, PCIs performed at top-ranked hospitals had similar odds of in-hospital mortality (adjusted odds ratio [aOR]: 0.96; 95% confidence interval [CI]: 0.83 to 1.12; p = 0.64), acute kidney injury (aOR: 1.10; 95% CI: 0.98 to 1.22; p = 0.099), and bleeding (aOR: 1.15; 95% CI: 0.99 to 1.31; p = 0.052). Top-ranked hospitals had a slightly lower proportion of appropriate PCI compared with nonranked hospitals (89.2% vs. 92.8%; OR: 0.56; 95% CI: 0.45 to 0.69; p < 0.001).

CONCLUSIONS

PCI performed at top-ranked hospitals was not associated with superior outcomes compared with PCI at nonranked hospitals. The inclusion of metrics based on clinical data may be important for hospital quality rankings.

Selecting long-term care facilities with high use of acute hospitalisations: issues and options

BACKGROUND

This paper considers approaches to the question "Which long-term care facilities have residents with high use of acute hospitalisations?" It compares four methods of identifying long-term care facilities with high use of acute hospitalisations by demonstrating four selection methods, identifies key factors to be resolved when deciding which methods to employ, and discusses their appropriateness for different research questions.

METHODS

OPAL was a census-type survey of aged care facilities and residents in Auckland, New Zealand, in 2008. It collected information about facility management and resident demographics, needs and care. Survey records (149 aged care facilities, 6271 residents) were linked to hospital and mortality records routinely assembled by health authorities. The main ranking endpoint was acute hospitalisations for diagnoses that were classified as potentially avoidable. Facilities were ranked using 1) simple event counts per person, 2) event rates per year of resident follow-up, 3) statistical model of rates using four predictors, and 4) change in ranks between methods 2) and 3). A generalized mixed model was used for Method 3 to handle the clustered nature of the data.

RESULTS

3048 potentially avoidable hospitalisations were observed during 22 months' follow-up. The same "top ten" facilities were selected by Methods 1 and 2. The statistical model (Method 3), predicting rates from resident and facility characteristics, ranked facilities differently than these two simple methods. The change-in-ranks method identified a very different set of "top ten" facilities. All methods showed a continuum of use, with no clear distinction between facilities with higher use.

CONCLUSION

Choice of selection method should depend upon the purpose of selection. To monitor performance during a period of change, a recent simple rate, count per resident, or even count per bed, may suffice. To find high-use facilities regardless of resident needs, recent history of admissions is highly predictive. To target a few high-use facilities that have high rates after considering facility and resident characteristics, model residuals or a large increase in rank may be preferable.

Ensemble of trees approaches to risk adjustment for evaluating a hospital's performance

A commonly used method for evaluating a hospital's performance on an outcome is to compare the hospital's observed outcome rate to the hospital's expected outcome rate given its patient (case) mix and service. The process of calculating the hospital's expected outcome rate given its patient mix and service is called risk adjustment (Iezzoni 1997). Risk adjustment is critical for accurately evaluating and comparing hospitals' performances since we would not want to unfairly penalize a hospital just because it treats sicker patients. The key to risk adjustment is accurately estimating the probability of an Outcome given patient characteristics. For cases with binary outcomes, the method that is commonly used in risk adjustment is logistic regression. In this paper, we consider ensemble of trees methods as alternatives for risk adjustment, including random forests and Bayesian additive regression trees (BART). Both random forests and BART are modern machine learning methods that have been shown recently to have excellent performance for prediction of outcomes in many settings. We apply these methods to carry out risk adjustment for the performance of neonatal intensive care units (NICU). We show that these ensemble of trees methods outperform logistic regression in predicting mortality among babies treated in NICU, and provide a superior method of risk adjustment compared to logistic regression.