1
|
Lemardeley G, Porcu-Buisson G, Pirrello O, Gane J, Dieterlé S, Astrugue C, Charbonnier T, Lucas-Samuel S, Couchoud C. Contribution of medico-administrative databases to health vigilance: example of post-oocyte retrieval infections. Hum Reprod 2024; 39:190-200. [PMID: 37953342 DOI: 10.1093/humrep/dead232] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2023] [Revised: 10/19/2023] [Indexed: 11/14/2023] Open
Abstract
STUDY QUESTION Can we monitor post-oocyte retrieval infections in the French national health data system to complement the French ART vigilance system? SUMMARY ANSWER Medico-administrative databases provide a more comprehensive view of post-oocyte retrieval infections and can be used to detect abnormal increases in frequency and outlier ART centers as a complementary tool to the ART vigilance system. WHAT IS KNOWN ALREADY The various studies of ART complications are reassuring, showing relatively low overall complication rates. Nonetheless, the European Union has set up a vigilance system to monitor these complications. However, this system is not an exhaustive source of information and does not provide a complete overview of post-ART complications. STUDY DESIGN, SIZE, DURATION The study population was identified from the comprehensive French national hospital discharge database. It included women under 46 years of age undergoing an oocyte retrieval in 2019, classified into three population subgroups according to the indication of oocyte retrieval: infertility (IF) , fertility preservation (FP), and oocyte donation (OD) . The study population included 52 098 women who had undergone 65 948 oocyte retrievals in 2019. PARTICIPANTS/MATERIALS, SETTING, METHODS Hospital stays and delivery of antibiotics within 31 days after oocyte retrieval were analyzed. Women and infections were characterized according to various characteristics (age, comorbidities, indication of oocyte retrieval, type of hospital stay, length of hospital stay, type of antibiotherapy, etc.). Multivariate analysis was performed to determine the relation between the occurrence of infection and women's characteristics, and results are expressed as odds ratios (ORs) and 95% CI. A funnel plot and a box plot were used to compare the infection rate per center with the national average and to detect outliers. MAIN RESULTS AND THE ROLE OF CHANCE Infections in the month following the oocyte retrieval represented 6.9% of the procedures in 2019 (n = 4522). Of these infections, 112 were hospitalized (0.2% of oocyte retrievals), and 4410 were non-hospitalized (6.7% of oocyte retrievals). The hospitalized infections were essentially gynecological infections (40.9%) and urinary tract infections (23.5%). In 87.9% of non-hospitalized infections, a single antibiotic therapy was prescribed. Mixed-effect model analysis showed that the risk of infection was significantly higher in women under 30 years of age, in the FP population, in supplementary universal health coverage (CMU-C) beneficiaries, and women with endometriosis. Funnel plot and box plot analysis showed that three ART centers have an infection rate significantly higher than the national average. In the three centers that stand out from all the others, the objective is to return to these centers to understand the possible reasons for this observed rate and to implement corrective measures. LIMITATIONS, REASONS FOR CAUTION Despite all its advantages, the French national health data system presents some limitations, such as the risk of inappropriate coding. Another limitation of this study is that we cannot confirm an attributable relation between the infection and the ART procedure, even if the delay of 31 days after oocyte retrieval is consistent with the occurrence of a post-retrieval complication. In addition, antibiotics may be prescribed as a 'precautionary' measure in certain situations (women with a susceptibility to infection, complicated procedures), or as antibiotic prophylaxis for embryo transfer. WIDER IMPLICATIONS OF THE FINDINGS Despite the limits in identifying post-ART infections in medico-administrative databases, this approach is a promising way to complement the ART vigilance reporting system. This concept developed for infections will also be generalized to other complications with regular feedback to professionals. STUDY FUNDING/COMPETING INTEREST(S) No specific funding was sought for the study. The study was supported by the Agence de la biomédecine, France. The authors declare that they have no conflict of interest. TRIAL REGISTRATION NUMBER N/A.
Collapse
Affiliation(s)
- G Lemardeley
- Medical and Scientific Department, Agence de la biomédecine, Saint Denis La Plaine, France
| | - G Porcu-Buisson
- Reproductive Medicine, Institut de Médecine de la Reproduction (IMR), Marseille, France
| | - O Pirrello
- Gynecology Department, CMCO, Schiltigheim, France
| | - J Gane
- Medical and Scientific Department, Agence de la biomédecine, Saint Denis La Plaine, France
| | - S Dieterlé
- Medical and Scientific Department, Agence de la biomédecine, Saint Denis La Plaine, France
| | - C Astrugue
- Medical and Scientific Department, Agence de la biomédecine, Saint Denis La Plaine, France
| | - T Charbonnier
- Medical and Scientific Department, Agence de la biomédecine, Saint Denis La Plaine, France
| | - S Lucas-Samuel
- Medical and Scientific Department, Agence de la biomédecine, Saint Denis La Plaine, France
| | - C Couchoud
- Medical and Scientific Department, Agence de la biomédecine, Saint Denis La Plaine, France
| |
Collapse
|
2
|
Silva GC, Gutman R. Reformulating provider profiling by grouping providers treating similar patients prior to evaluating performance. Biostatistics 2023; 24:962-984. [PMID: 35661195 DOI: 10.1093/biostatistics/kxac019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2021] [Revised: 05/03/2022] [Accepted: 05/05/2022] [Indexed: 10/19/2023] Open
Abstract
Standard approaches to comparing health providers' performance rely on hierarchical logistic regression models that adjust for patient characteristics at admission. Estimates from these models may be misleading when providers treat different patient populations and the models are misspecified. To address this limitation, we propose a novel profiling approach that identifies groups of providers treating similar populations of patients and then evaluates providers' performance within each group. The groups of providers are identified using a Bayesian multilevel finite mixture of general location models. To compare the performance of our proposed profiling approach to standard methods, we use patient-level data from 119 skilled nursing facilities in Massachusetts. We use simulated and observed outcome data to explore the performance of these profiling methods in different settings. In simulations, our proposed method classifies providers to groups with similar patients' admission characteristics. In addition, in the presence of limited overlap in patient characteristics across providers and misspecifications of the outcome model, the provider-level estimates obtained using our approach identified providers that under- and overperformed compared to the standard regression-based approaches more accurately.
Collapse
Affiliation(s)
- Gabriella C Silva
- Department of Biostatistics, School of Public Health, Brown University, 121 South Main Street, Providence, RI 02906 USA
| | - Roee Gutman
- Department of Biostatistics, School of Public Health, Brown University, 121 South Main Street, Providence, RI 02906 USA
| |
Collapse
|
3
|
Hansen J, Ahern S, Earnest A. Evaluations of statistical methods for outlier detection when benchmarking in clinical registries: a systematic review. BMJ Open 2023; 13:e069130. [PMID: 37451708 PMCID: PMC10351235 DOI: 10.1136/bmjopen-2022-069130] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/13/2022] [Accepted: 06/26/2023] [Indexed: 07/18/2023] Open
Abstract
OBJECTIVES Benchmarking is common in clinical registries to support the improvement of health outcomes by identifying underperforming clinician or health service providers. Despite the rise in clinical registries and interest in publicly reporting benchmarking results, appropriate methods for benchmarking and outlier detection within clinical registries are not well established, and the current application of methods is inconsistent. The aim of this review was to determine the current statistical methods of outlier detection that have been evaluated in the context of clinical registry benchmarking. DESIGN A systematic search for studies evaluating the performance of methods to detect outliers when benchmarking in clinical registries was conducted in five databases: EMBASE, ProQuest, Scopus, Web of Science and Google Scholar. A modified healthcare modelling evaluation tool was used to assess quality; data extracted from each study were summarised and presented in a narrative synthesis. RESULTS Nineteen studies evaluating a variety of statistical methods in 20 clinical registries were included. The majority of studies conducted application studies comparing outliers without statistical performance assessment (79%), while only few studies used simulations to conduct more rigorous evaluations (21%). A common comparison was between random effects and fixed effects regression, which provided mixed results. Registry population coverage, provider case volume minimum and missing data handling were all poorly reported. CONCLUSIONS The optimal methods for detecting outliers when benchmarking clinical registry data remains unclear, and the use of different models may provide vastly different results. Further research is needed to address the unresolved methodological considerations and evaluate methods across a range of registry conditions. PROSPERO REGISTRATION NUMBER CRD42022296520.
Collapse
Affiliation(s)
- Jessy Hansen
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
| | - Susannah Ahern
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
| | - Arul Earnest
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
| |
Collapse
|
4
|
Wu W, Kuriakose JP, Weng W, Burney RE, He K. Test-specific funnel plots for healthcare provider profiling leveraging individual- and summary-level information. HEALTH SERVICES AND OUTCOMES RESEARCH METHODOLOGY 2023; 23:45-58. [PMID: 37621728 PMCID: PMC10449097 DOI: 10.1007/s10742-022-00285-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/01/2022] [Revised: 06/05/2022] [Accepted: 06/15/2022] [Indexed: 10/17/2022]
Abstract
In addition to applications in meta-analysis, funnel plots have emerged as an effective graphical tool for visualizing the detection of health care providers with unusual performance. Although there already exist a variety of approaches to producing funnel plots in the literature of provider profiling, limited attention has been paid to elucidating the critical relationship between funnel plots and hypothesis testing. Within the framework of generalized linear models, here we establish methodological guidelines for creating funnel plots specific to the statistical tests of interest. Moreover, we show that the test-specific funnel plots can be created merely leveraging summary statistics instead of individual-level information. This appealing feature inhibits the leak of protected health information and reduces the cost of inter-institutional data transmission. Two data examples, one for surgical patients from Michigan hospitals and the other for Medicare-certified dialysis facilities, demonstrate the applicability to different types of providers and outcomes with either individual- or summary-level information.
Collapse
Affiliation(s)
- Wenbo Wu
- Department of Biostatistics, University of Michigan, 1415 Washington Heights, Ann Arbor, MI 48109, USA
| | - Jonathan P. Kuriakose
- Robert Wood Johnson Medical School, Rutgers University, 125 Paterson Street, New Brunswick, NJ 08901, USA
| | - Wenjing Weng
- Michigan Surgical Quality Collaborative, University of Michigan, 2800 Plymouth Road, Ann Arbor, MI 48109, USA
| | - Richard E. Burney
- Michigan Surgical Quality Collaborative, University of Michigan, 2800 Plymouth Road, Ann Arbor, MI 48109, USA
- Department of Surgery, University of Michigan, 1500 E. Medical Center Drive, Ann Arbor, MI 48109, USA
| | - Kevin He
- Department of Biostatistics, University of Michigan, 1415 Washington Heights, Ann Arbor, MI 48109, USA
| |
Collapse
|
5
|
Lebreton E, Menguy C, Fresson J, Egorova NN, Crenn Hebert C, Zeitlin J. Measuring severe neonatal morbidity using hospital discharge data in France. Paediatr Perinat Epidemiol 2022; 36:190-201. [PMID: 34797588 DOI: 10.1111/ppe.12816] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/25/2021] [Revised: 08/11/2021] [Accepted: 08/22/2021] [Indexed: 11/28/2022]
Abstract
BACKGROUND Measuring infant health at birth is key for surveillance and research in obstetrics and neonatology, but there is no international consensus on morbidity indicators. The Neonatal Adverse Outcome Indicator (NAOI) is a composite indicator, developed in Australia, which measures the burden of severe neonatal morbidity using hospital discharge data. OBJECTIVE To evaluate the applicability of the NAOI in France for surveillance and research. METHODS We constituted a cohort of live births ≥24 weeks' gestational age in Metropolitan France from 2014 to 2015 using hospital discharge, insurance claims and cause of death data. Outlier hospitals were identified using funnel plots of standardised morbidity ratios (SMR), and their coding patterns were assessed. We compared the NAOI and its component codes with published Australian and English data and estimated unadjusted and adjusted risk ratios for known risk factors for neonatal morbidity. RESULTS We included 1,459,123 births (511 hospitals). Twenty-eight hospitals had SMR above funnel plot control limits. Newborns with NAOI morbidities in these hospitals had lower mortality and shorter stays than in other hospitals. Amongst within-limit hospitals, NAOI prevalence was 4.8%, comparable to Australia (4.6%) and England (5.4%). Most individual components had a similar prevalence, with the exception of respiratory support, intravenous fluid procedures and infection. NAOI was lowest at 39 weeks (2.2%) with higher risks for maternal age ≥40 (relative risk [RR] 1.47, 95% confidence interval [CI] 1.42, 1.51), state medical insurance (RR 1.60, 95% CI 1.52, 1.68), male sex (RR 1.21, 95% CI 1.19, 1.23) and birthweight <3rd percentile (RR 4.60, 95% CI 4.51, 4.69). CONCLUSIONS The NAOI provides valuable information on population prevalence of severe neonatal morbidity and its risk factors. Whilst the prevalence was similar in high-income countries with comparable neonatal mortality levels, ensuring valid comparisons between countries and hospitals will require further work to harmonize coding procedures, especially for infection and respiratory morbidity.
Collapse
Affiliation(s)
- Elodie Lebreton
- INSERM UMR 1153, Obstetrical, Perinatal and Pediatric Epidemiology Research Team (Epopé), Center for Epidemiology and Statistics Sorbonne Paris Cité, DHU Risks in Pregnancy, Paris Descartes University, Paris, France.,Non Communicable Diseases and Trauma Division, Santé publique France, The National Public Health Agency, Saint-Maurice, France.,Perinat-ARS-IDF, Regional Health Agency of Ile-de-France (ARS-IDF), Saint-Denis, France
| | - Claudie Menguy
- Non Communicable Diseases and Trauma Division, Santé publique France, The National Public Health Agency, Saint-Maurice, France.,Perinat-ARS-IDF, Regional Health Agency of Ile-de-France (ARS-IDF), Saint-Denis, France
| | - Jeanne Fresson
- INSERM UMR 1153, Obstetrical, Perinatal and Pediatric Epidemiology Research Team (Epopé), Center for Epidemiology and Statistics Sorbonne Paris Cité, DHU Risks in Pregnancy, Paris Descartes University, Paris, France.,Department of Medical Information, Maternity of University Hospital - CHRU Nancy, Nancy, France
| | - Natalia N Egorova
- Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai, New York, USA
| | - Catherine Crenn Hebert
- Perinat-ARS-IDF, Regional Health Agency of Ile-de-France (ARS-IDF), Saint-Denis, France.,Maternity Unit, University Hospital (APHP), Hôpital Louis Mourier, Colombes, France
| | - Jennifer Zeitlin
- INSERM UMR 1153, Obstetrical, Perinatal and Pediatric Epidemiology Research Team (Epopé), Center for Epidemiology and Statistics Sorbonne Paris Cité, DHU Risks in Pregnancy, Paris Descartes University, Paris, France
| |
Collapse
|
6
|
Qualitative Data Clustering to Detect Outliers. ENTROPY 2021; 23:e23070869. [PMID: 34356410 PMCID: PMC8307081 DOI: 10.3390/e23070869] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/17/2021] [Revised: 06/26/2021] [Accepted: 07/01/2021] [Indexed: 11/17/2022]
Abstract
Detecting outliers is a widely studied problem in many disciplines, including statistics, data mining, and machine learning. All anomaly detection activities are aimed at identifying cases of unusual behavior compared to most observations. There are many methods to deal with this issue, which are applicable depending on the size of the data set, the way it is stored, and the type of attributes and their values. Most of them focus on traditional datasets with a large number of quantitative attributes. The multitude of solutions related to detecting outliers in quantitative sets, a large and still has a small number of research solutions is a problem detecting outliers in data containing only qualitative variables. This article was designed to compare three different categorical data clustering algorithms: K-modes algorithm taken from MacQueen’s K-means algorithm and the STIRR and ROCK algorithms. The comparison concerned the method of dividing the set into clusters and, in particular, the outliers detected by algorithms. During the research, the authors analyzed the clusters detected by the indicated algorithms, using several datasets that differ in terms of the number of objects and variables. They have conducted experiments on the parameters of the algorithms. The presented study made it possible to check whether the algorithms similarly detect outliers in the data and how much they depend on individual parameters and parameters of the set, such as the number of variables, tuples, and categories of a qualitative variable.
Collapse
|
7
|
Endo H, Uchino S, Hashimoto S, Aoki Y, Hashiba E, Hatakeyama J, Hayakawa K, Ichihara N, Irie H, Kawasaki T, Kumasawa J, Kurosawa H, Nakamura T, Ohbe H, Okamoto H, Shigemitsu H, Tagami T, Takaki S, Takimoto K, Uchida M, Miyata H. Development and validation of the predictive risk of death model for adult patients admitted to intensive care units in Japan: an approach to improve the accuracy of healthcare quality measures. J Intensive Care 2021; 9:18. [PMID: 33588956 PMCID: PMC7885245 DOI: 10.1186/s40560-021-00533-z] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/23/2020] [Accepted: 01/26/2021] [Indexed: 12/25/2022] Open
Abstract
BACKGROUND The Acute Physiology and Chronic Health Evaluation (APACHE) III-j model is widely used to predict mortality in Japanese intensive care units (ICUs). Although the model's discrimination is excellent, its calibration is poor. APACHE III-j overestimates the risk of death, making its evaluation of healthcare quality inaccurate. This study aimed to improve the calibration of the model and develop a Japan Risk of Death (JROD) model for benchmarking purposes. METHODS A retrospective analysis was conducted using a national clinical registry of ICU patients in Japan. Adult patients admitted to an ICU between April 1, 2018, and March 31, 2019, were included. The APACHE III-j model was recalibrated with the following models: Model 1, predicting mortality with an offset variable for the linear predictor of the APACHE III-j model using a generalized linear model; model 2, predicting mortality with the linear predictor of the APACHE III-j model using a generalized linear model; and model 3, predicting mortality with the linear predictor of the APACHE III-j model using a hierarchical generalized additive model. Model performance was assessed with the area under the receiver operating characteristic curve (AUROC), the Brier score, and the modified Hosmer-Lemeshow test. To confirm model applicability to evaluating quality of care, funnel plots of the standardized mortality ratio and exponentially weighted moving average (EWMA) charts for mortality were drawn. RESULTS In total, 33,557 patients from 44 ICUs were included in the study population. ICU mortality was 3.8%, and hospital mortality was 8.1%. The AUROC, Brier score, and modified Hosmer-Lemeshow p value of the original model and models 1, 2, and 3 were 0.915, 0.062, and < .001; 0.915, 0.047, and < .001; 0.915, 0.047, and .002; and 0.917, 0.047, and .84, respectively. Except for model 3, the funnel plots showed overdispersion. The validity of the EWMA charts for the recalibrated models was determined by visual inspection. CONCLUSIONS Model 3 showed good performance and can be adopted as the JROD model for monitoring quality of care in an ICU, although further investigation of the clinical validity of outlier detection is required. This update method may also be useful in other settings.
Collapse
Affiliation(s)
- Hideki Endo
- Department of Healthcare Quality Assessment, Graduate School of Medicine, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan. .,Department of Health Policy and Management, School of Medicine, Keio University, 35 Shinanomachi, Shinjuku-ku, Tokyo, 160-8582, Japan.
| | - Shigehiko Uchino
- Intensive Care Unit, The Jikei University School of Medicine, 3-19-18 Nishi-Shinbashi, Minato-ku, Tokyo, 105-8471, Japan
| | - Satoru Hashimoto
- Department of Anesthesiology and Intensive Care Medicine, Kyoto Prefectural University of Medicine, 465 Kajii-cho, Kawaramachi-Hirokoji, Kamigyo-ku, Kyoto, 602-8566, Japan
| | - Yoshitaka Aoki
- Department of Anesthesiology and Intensive Care Medicine, Hamamatsu University School of Medicine, 1-20-1 Handayama, Higashi-ku, Hamamatsu, Shizuoka, 431-3192, Japan
| | - Eiji Hashiba
- Division of Intensive Care, Hirosaki University Hospital, 53 Honcho, Hirosaki, Aomori, 036-8203, Japan
| | - Junji Hatakeyama
- Department of Emergency and Critical Care Medicine, National Hospital Organization Tokyo Medical Center, 2-5-1, Higashigaoka, Meguro-ku, Tokyo, 152-8902, Japan
| | - Katsura Hayakawa
- Department of Emergency and Critical Care Medicine, Saitama Red Cross Hospital, 1-5 Shintoshin, Chuo-ku, Saitama, 330-8553, Japan
| | - Nao Ichihara
- Department of Healthcare Quality Assessment, Graduate School of Medicine, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan
| | - Hiromasa Irie
- Department of Anesthesiology, Kurashiki Central Hospital, 1-1-1 Miwa, Kurashiki, Okayama, 710-8602, Japan
| | - Tatsuya Kawasaki
- Department of Pediatric Critical Care, Shizuoka Children's Hospital, 860 Urushiyama, Aoi-ku, Shizuoka, Shizuoka, 420-8660, Japan
| | - Junji Kumasawa
- Department of Critical Care Medicine, Sakai City Medical Center, 1-1-1 Ebaraji-cho, Nishi-ku, Sakai, Osaka, 593-8304, Japan
| | - Hiroshi Kurosawa
- Department of Pediatric Critical Care Medicine, Hyogo Prefectural Kobe Children's Hospital, 1-6-7 Minatojima Minamimachi, Chuo-ku, Kobe, Hyogo, 650-0047, Japan
| | - Tomoyuki Nakamura
- Department of Anesthesiology and Critical Care Medicine, Fujita Health University School of Medicine, 1-98 Dengakugakubo, Kutsukake-cho, Toyoake, Aichi, 470-1192, Japan
| | - Hiroyuki Ohbe
- Department of Clinical Epidemiology and Health Economics, School of Public Health, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-0033, Japan
| | - Hiroshi Okamoto
- Department of Critical Care Medicine, St. Luke's International Hospital, 9-1 Akashi-cho, Chuo-ku, Tokyo, 104-8560, Japan
| | - Hidenobu Shigemitsu
- Department of Intensive Care Medicine, Graduate School of Medicine, Tokyo Medical and Dental University, 1-5-45 Yushima, Bunkyo-ku, Tokyo, 113-8519, Japan
| | - Takashi Tagami
- Department of Emergency and Critical Care Medicine, Nippon Medical School Musashikosugi Hospital, 1-396 Kosugimachi, Nakahara-ku, Kawasaki, Kanagawa, 211-8533, Japan
| | - Shunsuke Takaki
- Department of Anesthesiology and Critical Care Medicine, Yokohama City University, 3-9 Fukuura, Kanazawa-ku, Yokohama, Kanagawa, 236-0004, Japan
| | - Kohei Takimoto
- Department of Intensive Care Medicine, Kameda Medical Center, 929 Higashi-cho, Kamogawa, Chiba, 296-8602, Japan
| | - Masatoshi Uchida
- Department of Emergency and Critical Care Medicine, Dokkyo Medical University, 880 Kitakobayashi, Mibu-machi, Shimotsuga-gun, Tochigi, 321-0293, Japan
| | - Hiroaki Miyata
- Department of Healthcare Quality Assessment, Graduate School of Medicine, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan.,Department of Health Policy and Management, School of Medicine, Keio University, 35 Shinanomachi, Shinjuku-ku, Tokyo, 160-8582, Japan
| |
Collapse
|
8
|
Multi-level models for heart failure patients' 30-day mortality and readmission rates: the relation between patient and hospital factors in administrative data. BMC Health Serv Res 2019; 19:1012. [PMID: 31888610 PMCID: PMC6936032 DOI: 10.1186/s12913-019-4818-2] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2018] [Accepted: 12/09/2019] [Indexed: 01/16/2023] Open
Abstract
Background This study aims at gathering evidence about the relation between 30-day mortality and 30-day unplanned readmission and patient and hospital factors. By definition, we refer to 30-day mortality and 30-day unplanned readmission as the number of deaths and non-programmed hospitalizations for any cause within 30 days after the incident heart failure (HF). In particular, the focus is on the role played by hospital-level factors. Methods A multi-level logistic model that combines patient- and hospital-level covariates has been developed to better disentangle the role played by the two groups of covariates. Later on, hospital outliers in term of better-than-expected/worst-than-expected performers have been identified by comparing expected cases vs. observed cases. Hospitals performance in terms of 30-day mortality and 30-day unplanned readmission rates have been visualized through the creation of funnel plots. Covariates have been selected coherently to past literature. Data comes from the hospital discharge forms for Heart Failure patients in the Lombardy Region (Northern Italy). Considering incident cases for HF in the timespan 2010–2012, 78,907 records for adult patients from 117 hospitals have been collected after quality checks. Results Our results show that 30-day mortality and 30-day unplanned readmissions are explained by hospital-level covariates, paving the way for the design and implementation of evidence-based improvement strategies. While the percentage of surgical DRG (OR = 1.001; CI (1.000–1.002)) and the hospital type of structure (Research hospitals vs. non-research public hospitals (OR = 0.62; CI (0.48–0.80)) and Non-research private hospitals vs. non-research hospitals OR = 0.75; CI (0.63–0.90)) are significant for mortality, the mean length of stay (OR = 0.96; CI (0.95–0.98)) is significant for unplanned readmission, showing that mortality and readmission rates might be improved through different strategies. Conclusion Our results confirm that hospital-level covariates do affect quality of care, and that 30-day mortality and 30-day unplanned readmission are affected by different managerial choices. This confirms that hospitals should be accountable for their “added value” to quality of care.
Collapse
|
9
|
Roshanghalb A, Mazzali C, Lettieri E, Paganoni AM. Chapter 10 Performance Measurement in Health Care: The Case of Best/Worst Performers Through Administrative Data. PERFORMANCE MEASUREMENT AND MANAGEMENT CONTROL: THE RELEVANCE OF PERFORMANCE MEASUREMENT AND MANAGEMENT CONTROL RESEARCH 2018. [DOI: 10.1108/s1479-351220180000033010] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/12/2023]
|
10
|
Hirsch O, Donner-Banzhoff N, Schulz M, Erhart M. Detecting and Visualizing Outliers in Provider Profiling Using Funnel Plots and Mixed Effects Models-An Example from Prescription Claims Data. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2018; 15:ijerph15092015. [PMID: 30223551 PMCID: PMC6163340 DOI: 10.3390/ijerph15092015] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/14/2018] [Revised: 09/11/2018] [Accepted: 09/13/2018] [Indexed: 12/04/2022]
Abstract
When prescribing a drug for a patient, a physician also has to consider economic aspects. We were interested in the feasibility and validity of profiling based on funnel plots and mixed effect models for the surveillance of German ambulatory care physicians’ prescribing. We analyzed prescriptions issued to patients with a health insurance card attending neurologists’ and psychiatrists’ ambulatory practices in the German federal state of Saarland. The German National Association of Statutory Health Insurance Physicians developed a prescribing assessment scheme (PAS) which contains a systematic appraisal of the benefit of drugs for so far 12 different indications. The drugs have been classified on the basis of their clinical evidence as “standard”, “reserve” or “third level” medication. We had 152.583 prescriptions in 56 practices available for analysis. A total of 38.796 patients received these prescriptions. The funnel plot approach with additive correction for overdispersion was almost equivalent to a mixed effects model which directly took the multilevel structure of the data into account. In the first case three practices were labeled as outliers, the mixed effects model resulted in two outliers. We suggest that both techniques should be routinely applied within a surveillance system of prescription claims data.
Collapse
Affiliation(s)
- Oliver Hirsch
- Department of General Practice/Family Medicine, Philipps University Marburg, Karl-von-Frisch-Str.4, 35043 Marburg, Germany.
| | - Norbert Donner-Banzhoff
- Department of General Practice/Family Medicine, Philipps University Marburg, Karl-von-Frisch-Str.4, 35043 Marburg, Germany.
| | - Maike Schulz
- Central Research Institute of Ambulatory Health Care in Germany (ZI), Salzufer 8, 10587 Berlin, Germany.
| | - Michael Erhart
- Central Research Institute of Ambulatory Health Care in Germany (ZI), Salzufer 8, 10587 Berlin, Germany.
| |
Collapse
|
11
|
Afana M, Gurm HS, Seth M, Frazier KM, Fielding S, Koenig GC. Primary percutaneous coronary intervention at centers with and without on-site surgical support: Insights from the Blue Cross Blue Shield of Michigan Cardiovascular Consortium (BMC2). Am Heart J 2018; 195:99-107. [PMID: 29224652 DOI: 10.1016/j.ahj.2017.10.004] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/23/2017] [Accepted: 10/07/2017] [Indexed: 10/18/2022]
Abstract
BACKGROUND Primary percutaneous coronary intervention (PPCI) is being increasingly performed nationally at sites without on-site cardiac surgery; however, recent guidelines only provide a Class IIa recommendation for this practice. The state of Michigan has permitted PPCI without on-site surgery under a closely monitored system that mandates auditing of all procedures and quarterly feedback on quality and outcomes. This study sought to compare outcomes of patients undergoing PPCI at centers with and without on-site surgery in the state of Michigan. METHODS Consecutive patients who underwent PPCI at 47 hospitals in Michigan from January 2010 to December 2015 were included. From this cohort, 4,091 patients from sites with and without on-site cardiac surgery were propensity matched in a 1:1 fashion to compare baseline characteristics, procedural details, and in-hospital outcomes. RESULTS Of the 25,886 PPCIs performed at 47 hospitals in Michigan from 2010 to 2015, 21,610 (83.5%) were performed at sites with on-site surgery and 4,276 (16.5%) at sites without on-site surgery. Using propensity score matched cohorts (4,091 patients for each site type), we found no significant differences in baseline characteristics. Overall mortality (5.4% vs 5.8%; P=.442); composite outcome of in-hospital mortality, contrast-induced nephropathy, bleeding, and stroke (13.8% vs 12.8%; P=.152); and individual outcomes within the composite group showed no significant differences. Additionally, there were no clinically meaningful differences in rates of urgent/emergent coronary artery bypass graft or length of stay. Significant differences, however, were found in procedural access site, antiplatelet therapy, contrast volume, and anticoagulant strategy. CONCLUSIONS Primary PCI performed at centers with and without cardiac surgery have comparable outcomes and complication rates when performed with close monitoring of quality and outcomes.
Collapse
|
12
|
Proudlove NC, Samarasinghe BS, Walshe K. Investigating consistent patterns of variation in short-notice cancellations of elective operations: The potential for learning and improvement through multi-site evaluations. Health Serv Manage Res 2017; 31:111-119. [PMID: 29256264 DOI: 10.1177/0951484817745064] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/17/2023]
Abstract
As part of efforts to increase productivity in healthcare, there is considerable interest in the extent and causes of variation in the performance of provider organisations. In this study, we use publically available data from the English NHS to examine the characteristics of variation in the rates of short-notice cancellations of elective operations due to hospital reasons (e.g. lack of the required resources such as operating theatres and beds). We find that the variation between hospital trusts is very non-random. There is a fourfold difference in the cancellation rates between the top and bottom deciles of performance. Little is known about the causes of this. There is a large and striking consistency in the relative performance of hospital trusts on cancellation rates over the last five years. Thus, the best and worst performers are consistently relatively very good or very poor, so a multi-site comparison of practices, and accounting for confounds like patient demographics, could be very valuable to inform both this knowledge gap and practice in healthcare. Of particular interest is that the cancellation rates could be a symptom of deeper issues with the efficiency of patient flows within hospitals.
Collapse
Affiliation(s)
- Nathan C Proudlove
- 1 Alliance Manchester Business School, University of Manchester, Manchester, UK
| | | | - Kieran Walshe
- 1 Alliance Manchester Business School, University of Manchester, Manchester, UK
| |
Collapse
|
13
|
Sukul D, Bhatt DL, Seth M, Zakroysky P, Wojdyla D, Rumsfeld JS, Wang T, Rao SV, Gurm HS. Appropriateness and Outcomes of Percutaneous Coronary Intervention at Top-Ranked and Nonranked Hospitals in the United States. JACC Cardiovasc Interv 2017; 11:342-350. [PMID: 29471947 DOI: 10.1016/j.jcin.2017.10.042] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/22/2017] [Revised: 10/24/2017] [Accepted: 10/26/2017] [Indexed: 11/28/2022]
Abstract
OBJECTIVES This study sought to compare the appropriate use and outcomes of percutaneous coronary intervention (PCI) between top-ranked and nonranked hospitals. BACKGROUND The U.S. News & World Report "Best Hospitals" rankings are an influential consumer-directed publication of hospital quality, and are commonly used in promotional campaigns by hospital systems. METHODS Hospitals in the National Cardiovascular Data Registry CathPCI registry between July 1, 2014, and June 30, 2015, were classified as top-ranked if they were included in the 2015 U.S. News & World Report 50 best "Cardiology and Heart Surgery" hospitals. The remaining were classified as nonranked. We compared in-hospital mortality, post-procedural bleeding, post-procedural acute kidney injury, and the proportion of appropriate PCI procedures between top-ranked and nonranked hospitals. RESULTS A total of 509,153 PCIs at 654 hospitals were included, of which 55,550 (10.9%) were performed at 44 top-ranked hospitals. After adjusting for patient case mix, PCIs performed at top-ranked hospitals had similar odds of in-hospital mortality (adjusted odds ratio [aOR]: 0.96; 95% confidence interval [CI]: 0.83 to 1.12; p = 0.64), acute kidney injury (aOR: 1.10; 95% CI: 0.98 to 1.22; p = 0.099), and bleeding (aOR: 1.15; 95% CI: 0.99 to 1.31; p = 0.052). Top-ranked hospitals had a slightly lower proportion of appropriate PCI compared with nonranked hospitals (89.2% vs. 92.8%; OR: 0.56; 95% CI: 0.45 to 0.69; p < 0.001). CONCLUSIONS PCI performed at top-ranked hospitals was not associated with superior outcomes compared with PCI at nonranked hospitals. The inclusion of metrics based on clinical data may be important for hospital quality rankings.
Collapse
Affiliation(s)
- Devraj Sukul
- Department of Internal Medicine, Division of Cardiovascular Medicine, University of Michigan, Ann Arbor, Michigan
| | - Deepak L Bhatt
- Brigham and Women's Hospital Heart & Vascular Center, Harvard Medical School, Boston, Massachusetts
| | - Milan Seth
- Department of Internal Medicine, Division of Cardiovascular Medicine, University of Michigan, Ann Arbor, Michigan
| | | | - Daniel Wojdyla
- Duke Clinical Research Institute, Durham, North Carolina
| | - John S Rumsfeld
- Department of Internal Medicine, Division of Cardiology, University of Colorado School of Medicine, Denver, Colorado
| | - Tracy Wang
- Duke Clinical Research Institute, Durham, North Carolina
| | - Sunil V Rao
- Duke Clinical Research Institute, Durham, North Carolina
| | - Hitinder S Gurm
- Department of Internal Medicine, Division of Cardiovascular Medicine, University of Michigan, Ann Arbor, Michigan; Division of Cardiovascular Medicine, VA Ann Arbor Healthcare System, Ann Arbor, Michigan.
| |
Collapse
|
14
|
Abramo G, D’Angelo AC, Grilli L. From rankings to funnel plots: The question of accounting for uncertainty when assessing university research performance. J Informetr 2016. [DOI: 10.1016/j.joi.2016.07.005] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|
15
|
Heterogeneity, school-effects and the North/South achievement gap in Italian secondary education: evidence from a three-level mixed model. STAT METHOD APPL-GER 2016. [DOI: 10.1007/s10260-016-0363-x] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
16
|
Broad JB, Ashton T, Lumley T, Boyd M, Kerse N, Connolly MJ. Selecting long-term care facilities with high use of acute hospitalisations: issues and options. BMC Med Res Methodol 2014; 14:93. [PMID: 25052433 PMCID: PMC4118262 DOI: 10.1186/1471-2288-14-93] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2013] [Accepted: 06/27/2014] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND This paper considers approaches to the question "Which long-term care facilities have residents with high use of acute hospitalisations?" It compares four methods of identifying long-term care facilities with high use of acute hospitalisations by demonstrating four selection methods, identifies key factors to be resolved when deciding which methods to employ, and discusses their appropriateness for different research questions. METHODS OPAL was a census-type survey of aged care facilities and residents in Auckland, New Zealand, in 2008. It collected information about facility management and resident demographics, needs and care. Survey records (149 aged care facilities, 6271 residents) were linked to hospital and mortality records routinely assembled by health authorities. The main ranking endpoint was acute hospitalisations for diagnoses that were classified as potentially avoidable. Facilities were ranked using 1) simple event counts per person, 2) event rates per year of resident follow-up, 3) statistical model of rates using four predictors, and 4) change in ranks between methods 2) and 3). A generalized mixed model was used for Method 3 to handle the clustered nature of the data. RESULTS 3048 potentially avoidable hospitalisations were observed during 22 months' follow-up. The same "top ten" facilities were selected by Methods 1 and 2. The statistical model (Method 3), predicting rates from resident and facility characteristics, ranked facilities differently than these two simple methods. The change-in-ranks method identified a very different set of "top ten" facilities. All methods showed a continuum of use, with no clear distinction between facilities with higher use. CONCLUSION Choice of selection method should depend upon the purpose of selection. To monitor performance during a period of change, a recent simple rate, count per resident, or even count per bed, may suffice. To find high-use facilities regardless of resident needs, recent history of admissions is highly predictive. To target a few high-use facilities that have high rates after considering facility and resident characteristics, model residuals or a large increase in rank may be preferable.
Collapse
Affiliation(s)
- Joanna B Broad
- Freemasons' Department of Geriatric Medicine, University of Auckland, C/- WDHB, Box 93503, Takapuna, Auckland 0740, New Zealand.
| | | | | | | | | | | |
Collapse
|
17
|
Liu Y, Traskin M, Lorch SA, George EI, Small D. Ensemble of trees approaches to risk adjustment for evaluating a hospital's performance. Health Care Manag Sci 2014; 18:58-66. [PMID: 24777832 DOI: 10.1007/s10729-014-9272-4] [Citation(s) in RCA: 17] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2013] [Accepted: 02/10/2014] [Indexed: 01/07/2023]
Abstract
A commonly used method for evaluating a hospital's performance on an outcome is to compare the hospital's observed outcome rate to the hospital's expected outcome rate given its patient (case) mix and service. The process of calculating the hospital's expected outcome rate given its patient mix and service is called risk adjustment (Iezzoni 1997). Risk adjustment is critical for accurately evaluating and comparing hospitals' performances since we would not want to unfairly penalize a hospital just because it treats sicker patients. The key to risk adjustment is accurately estimating the probability of an Outcome given patient characteristics. For cases with binary outcomes, the method that is commonly used in risk adjustment is logistic regression. In this paper, we consider ensemble of trees methods as alternatives for risk adjustment, including random forests and Bayesian additive regression trees (BART). Both random forests and BART are modern machine learning methods that have been shown recently to have excellent performance for prediction of outcomes in many settings. We apply these methods to carry out risk adjustment for the performance of neonatal intensive care units (NICU). We show that these ensemble of trees methods outperform logistic regression in predicting mortality among babies treated in NICU, and provide a superior method of risk adjustment compared to logistic regression.
Collapse
Affiliation(s)
- Yang Liu
- Department of Mathematics, University of Pennsylvania, Philadelphia, PA, USA,
| | | | | | | | | |
Collapse
|