1
|
Lee Y, Reese PP, Tran AH, Schaubel DE. Prognostic score-based methods for estimating center effects based on survival probability: Application to post-kidney transplant survival. Stat Med 2024; 43:3036-3050. [PMID: 38780593 DOI: 10.1002/sim.10092] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2022] [Revised: 03/25/2024] [Accepted: 04/16/2024] [Indexed: 05/25/2024]
Abstract
In evaluating the performance of different facilities or centers on survival outcomes, the standardized mortality ratio (SMR), which compares the observed to expected mortality has been widely used, particularly in the evaluation of kidney transplant centers. Despite its utility, the SMR may exaggerate center effects in settings where survival probability is relatively high. An example is one-year graft survival among U.S. kidney transplant recipients. We propose a novel approach to estimate center effects in terms of differences in survival probability (ie, each center versus a reference population). An essential component of the method is a prognostic score weighting technique, which permits accurately evaluating centers without necessarily specifying a correct survival model. Advantages of our approach over existing facility-profiling methods include a metric based on survival probability (greater clinical relevance than ratios of counts/rates); direct standardization (valid to compare between centers, unlike indirect standardization based methods, such as the SMR); and less reliance on correct model specification (since the assumed model is used to generate risk classes as opposed to fitted-value based 'expected' counts). We establish the asymptotic properties of the proposed weighted estimator and evaluate its finite-sample performance under a diverse set of simulation settings. The method is then applied to evaluate U.S. kidney transplant centers with respect to graft survival probability.
Collapse
Affiliation(s)
- Youjin Lee
- Department of Biostatistics, Brown University, Providence, Rhode Island
| | - Peter P Reese
- Department of Biostatistics, Epidemiology & Informatics, University of Pennsylvania, Philadelphia, Pennsylvania
- Department of Medicine, Renal-Electrolyte and Hypertension Division, University of Pennsylvania, Philadelphia, Pennsylvania
| | - Amelia H Tran
- Department of Biostatistics, Epidemiology & Informatics, University of Pennsylvania, Philadelphia, Pennsylvania
| | - Douglas E Schaubel
- Department of Biostatistics, Epidemiology & Informatics, University of Pennsylvania, Philadelphia, Pennsylvania
| |
Collapse
|
2
|
Dexter F, Epstein RH, Dillman D, Hindman BJ, Mueller RN. Predictive Validity of Anesthesiologists' Quality of Clinical Supervision and Nurse Anesthetists' Work Habits Assessed by Their Associations With Operating Room Times. Anesth Analg 2024:00000539-990000000-00860. [PMID: 38990773 DOI: 10.1213/ane.0000000000007076] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/13/2024]
Abstract
BACKGROUND At all Joint Commission-accredited hospitals, the anesthesia department chair must report quantitative assessments of anesthesiologists' and nurse anesthetists' (CRNAs') clinical performance at least annually. Most metrics lack evidence of usefulness, cost-effectiveness, reliability, or validity. Earlier studies showed that anesthesiologists' clinical supervision quality and CRNAs' work habits have content, convergent, discriminant, and construct validity. We evaluated predictive validity by testing for (expected) small but statistically significant associations between higher quality of supervision (work habits) and reduced probabilities of cases taking longer than estimated. METHODS Supervision quality of each anesthesiologist was evaluated daily by assigned trainees using the 9-item de Oliveira Filho scale. The work habits of each CRNA were evaluated daily by assigned anesthesiologists using a 6-item scale. Both are scored binary, 1 if all items are rated the maximum, 0 otherwise. From 40,718 supervision evaluations and 53,722 work habit evaluations over 8 fiscal years, 16 mixed-effects logistic regression models were estimated, with raters as fixed effects and ratees (anesthesiologists or CRNAs) as random effects. Empirical Bayes means in the logit scale were obtained for 561 anesthesiologist-years and 605 CRNA-years. The binary-dependent variable was whether the case took longer than estimated from the historical mean time for combinations of scheduled procedures and surgeons. From 264,060 cases, 8 mixed-effects logistic regression models were fitted, 1 per fiscal year, using ratees as random effects. Predictive validity was tested by pairing the 8 one-year analyses of clinical supervision, and the 8 one-year analyses of work habits, by ratee, with the 8 one-year analyses of whether OR time was longer than estimated. Bivariate errors in variable linear least squares linear regressions minimized total variances. RESULTS Among anesthesiologists, 8.2% (46/561) had below-average supervision quality, and 17.7% (99/561), above-average. Among CRNAs, 6.3% (38/605) had below-average work habits, and 10.9% (66/605) above-average. Increases in the logits of the quality of clinical supervision were associated with decreases in the logits of the probabilities of cases taking longer than estimated, unitless slope = -0.0361 (SE, 0.0053), P < .00001. Increases in the logits of CRNAs' work habits were associated with decreases in the logits of probabilities of cases taking longer than estimated, slope = -0.0238 (SE, 0.0054), P < .00001. CONCLUSIONS Predictive validity was confirmed, providing further evidence for using supervision and work habits scales for ongoing professional practice evaluations. Specifically, OR times were briefer when anesthesiologists supervised residents more closely, and when CRNAs had better work habits.
Collapse
Affiliation(s)
- Franklin Dexter
- From the Department of Anesthesia, University of Iowa, Iowa City, Iowa
| | - Richard H Epstein
- Department of Anesthesiology, Perioperative Medicine and Pain Management, University of Miami, Miami, Florida
| | - Dawn Dillman
- From the Department of Anesthesia, University of Iowa, Iowa City, Iowa
| | - Bradley J Hindman
- From the Department of Anesthesia, University of Iowa, Iowa City, Iowa
| | - Rashmi N Mueller
- From the Department of Anesthesia, University of Iowa, Iowa City, Iowa
| |
Collapse
|
3
|
Hansen J, Ahern S, Earnest A. Evaluations of statistical methods for outlier detection when benchmarking in clinical registries: a systematic review. BMJ Open 2023; 13:e069130. [PMID: 37451708 PMCID: PMC10351235 DOI: 10.1136/bmjopen-2022-069130] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/13/2022] [Accepted: 06/26/2023] [Indexed: 07/18/2023] Open
Abstract
OBJECTIVES Benchmarking is common in clinical registries to support the improvement of health outcomes by identifying underperforming clinician or health service providers. Despite the rise in clinical registries and interest in publicly reporting benchmarking results, appropriate methods for benchmarking and outlier detection within clinical registries are not well established, and the current application of methods is inconsistent. The aim of this review was to determine the current statistical methods of outlier detection that have been evaluated in the context of clinical registry benchmarking. DESIGN A systematic search for studies evaluating the performance of methods to detect outliers when benchmarking in clinical registries was conducted in five databases: EMBASE, ProQuest, Scopus, Web of Science and Google Scholar. A modified healthcare modelling evaluation tool was used to assess quality; data extracted from each study were summarised and presented in a narrative synthesis. RESULTS Nineteen studies evaluating a variety of statistical methods in 20 clinical registries were included. The majority of studies conducted application studies comparing outliers without statistical performance assessment (79%), while only few studies used simulations to conduct more rigorous evaluations (21%). A common comparison was between random effects and fixed effects regression, which provided mixed results. Registry population coverage, provider case volume minimum and missing data handling were all poorly reported. CONCLUSIONS The optimal methods for detecting outliers when benchmarking clinical registry data remains unclear, and the use of different models may provide vastly different results. Further research is needed to address the unresolved methodological considerations and evaluate methods across a range of registry conditions. PROSPERO REGISTRATION NUMBER CRD42022296520.
Collapse
Affiliation(s)
- Jessy Hansen
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
| | - Susannah Ahern
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
| | - Arul Earnest
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
| |
Collapse
|
4
|
Pavlou M, Ambler G, Omar RZ, Goodwin AT, Trivedi U, Ludman P, de Belder M. Outlier identification and monitoring of institutional or clinician performance: an overview of statistical methods and application to national audit data. BMC Health Serv Res 2023; 23:23. [PMID: 36627627 PMCID: PMC9832645 DOI: 10.1186/s12913-022-08995-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2022] [Accepted: 12/20/2022] [Indexed: 01/12/2023] Open
Abstract
BACKGROUND Institutions or clinicians (units) are often compared according to a performance indicator such as in-hospital mortality. Several approaches have been proposed for the detection of outlying units, whose performance deviates from the overall performance. METHODS We provide an overview of three approaches commonly used to monitor institutional performances for outlier detection. These are the common-mean model, the 'Normal-Poisson' random effects model and the 'Logistic' random effects model. For the latter we also propose a visualisation technique. The common-mean model assumes that the underlying true performance of all units is equal and that any observed variation between units is due to chance. Even after applying case-mix adjustment, this assumption is often violated due to overdispersion and a post-hoc correction may need to be applied. The random effects models relax this assumption and explicitly allow the true performance to differ between units, thus offering a more flexible approach. We discuss the strengths and weaknesses of each approach and illustrate their application using audit data from England and Wales on Adult Cardiac Surgery (ACS) and Percutaneous Coronary Intervention (PCI). RESULTS In general, the overdispersion-corrected common-mean model and the random effects approaches produced similar p-values for the detection of outliers. For the ACS dataset (41 hospitals) three outliers were identified in total but only one was identified by all methods above. For the PCI dataset (88 hospitals), seven outliers were identified in total but only two were identified by all methods. The common-mean model uncorrected for overdispersion produced several more outliers. The reason for observing similar p-values for all three approaches could be attributed to the fact that the between-hospital variance was relatively small in both datasets, resulting only in a mild violation of the common-mean assumption; in this situation, the overdispersion correction worked well. CONCLUSION If the common-mean assumption is likely to hold, all three methods are appropriate to use for outlier detection and their results should be similar. Random effect methods may be the preferred approach when the common-mean assumption is likely to be violated.
Collapse
Affiliation(s)
| | | | | | - Andrew T. Goodwin
- grid.440194.c0000 0004 4647 6776Department of Cardiothoracic Surgery, South Tees Hospitals NHS Foundation Trust, Middlesbrough, UK ,grid.139534.90000 0001 0372 5777National Institute for Cardiovascular Outcomes Research (NICOR), Barts Health NHS Trust, London, UK
| | - Uday Trivedi
- Department of Cardiac Surgery, University Hospital Sussex NHS Foundation Trust, Brighton, UK
| | - Peter Ludman
- grid.139534.90000 0001 0372 5777National Institute for Cardiovascular Outcomes Research (NICOR), Barts Health NHS Trust, London, UK ,grid.6572.60000 0004 1936 7486Institute of Cardiovascular Sciences, University of Birmingham, Birmingham, UK
| | - Mark de Belder
- grid.139534.90000 0001 0372 5777National Institute for Cardiovascular Outcomes Research (NICOR), Barts Health NHS Trust, London, UK
| |
Collapse
|
5
|
Abdel-Latif ME, Adegboye O, Nowak G, Elfaki F, Bajuk B, Glass K, Harley D. Variation in hospital morbidities in an Australian neonatal intensive care unit network. Arch Dis Child Fetal Neonatal Ed 2023:archdischild-2022-324940. [PMID: 36593112 DOI: 10.1136/archdischild-2022-324940] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/24/2022] [Accepted: 12/20/2022] [Indexed: 01/03/2023]
Abstract
OBJECTIVE There is an expectation among the public and within the profession that the performance and outcome of neonatal intensive care units (NICUs) should be comparable between centres with a similar setting. This study aims to benchmark and audit performance variation in a regional Australian network of eight NICUs. DESIGN Cohort study using prospectively collected data. SETTING All eight perinatal centres in New South Wales and the Australian Capital Territory, Australia. PATIENTS All live-born infants born between 23+0 and 31+6 weeks gestation admitted to one of the tertiary perinatal centres from 2007 to 2020 (n=12 608). MAIN OUTCOME MEASURES Early and late confirmed sepsis, intraventricular haemorrhage, medically and surgically treated patent ductus arteriosus, chronic lung disease (CLD), postnatal steroid for CLD, necrotising enterocolitis, retinopathy of prematurity (ROP), surgery for ROP, hospital mortality and home oxygen. RESULTS NICUs showed variations in maternal and neonatal characteristics and resources. The unadjusted funnel plots for neonatal outcomes showed apparent variation with multiple centres outside the 99.8% control limits of the network values. The hierarchical model-based risk-adjustment accounting for differences in patient characteristics showed that discharged home with oxygen is the only outcome above the 99.8% control limits. CONCLUSIONS Hierarchical model-based risk-adjusted estimates of morbidity rates plotted on funnel plots provide a robust and straightforward visual graphical tool for presenting variations in outcome performance to detect aberrations in healthcare delivery and guide timely intervention. We propose using hierarchical model-based risk adjustment and funnel plots in real or near real-time to detect aberrations and start timely intervention.
Collapse
Affiliation(s)
- Mohamed E Abdel-Latif
- Department of Neonatology, Centenary Hospital for Women and Children, Canberra Hospital, Garran, Canberra, ACT, Australia .,Department of Public Health, College of Science Health and Engineering, La Trobe University, Bundoora, Melbourne, VIC, Australia.,Discipline of Neonatology, School of Medicine and Psychology, College of Health and Medicine, Australian National University, Acton, Canberra, ACT, Australia
| | - Oyelola Adegboye
- Public Health and Tropical Medicine, College of Public Health, Medical and Veterinary Sciences, James Cook University, Townsville, Queensland, Australia.,Australian Institute of Tropical Health and Medicine, James Cook University, Townsville, QLD, Australia
| | - Gen Nowak
- Research School of Finance, Actuarial Studies, and Statistics, College of Business and Economics, Australian National University, Acton, Canberra, ACT, Australia
| | - Faiz Elfaki
- Department of Mathematics, Physics, and Statistics, Qatar University, Doha, Qatar
| | - Barbara Bajuk
- Critical Care Program, Sydney Children's Hospitals Network, Westmead, Sydney, NSW, Australia
| | - Kathryn Glass
- National Centre for Epidemiology and Population Health, Australian National University, Acton, Canberra, ACT, Australia
| | - David Harley
- National Centre for Epidemiology and Population Health, Australian National University, Acton, Canberra, ACT, Australia.,University of Queensland Centre for Clinical Research (UQCCR), University of Queensland, Brisbane, QLD, Australia
| |
Collapse
|
6
|
Ferreira G, Lobo M, Richards B, Dinh M, Maher C. Hospital variation in admissions for low back pain following an emergency department presentation: a retrospective study. BMC Health Serv Res 2022; 22:835. [PMID: 35818074 PMCID: PMC9275239 DOI: 10.1186/s12913-022-08134-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2022] [Accepted: 05/23/2022] [Indexed: 11/10/2022] Open
Abstract
Background One in 6 patients with low back pain (LBP) presenting to emergency departments (EDs) are subsequently admitted to hospital each year, making LBP the ninth most common reason for hospital admission in Australia. No studies have investigated and quantified the extent of clinical variation in hospital admission following an ED presentation for LBP. Methods We used routinely collected ED data from public hospitals within the state of New South Wales, Australia, to identify presentations of patients aged between 18 and 111 with a discharge diagnosis of LBP. We fitted a series of random effects multilevel logistic regression models adjusted by case-mix and hospital variables. The main outcome was the hospital-adjusted admission rate (HAAR). Data were presented as funnel plots with 95% and 99.8% confidence limits. Hospitals with a HAAR outside the 95% confidence limit were considered to have a HAAR significantly different to the state average. Results We identified 176,729 LBP presentations across 177 public hospital EDs and 44,549 hospital admissions (25.2%). The mean (SD) age was 51.8 (19.5) and 52% were female. Hospital factors explained 10% of the variation (ICC = 0.10), and the median odds ratio (MOR) was 2.03. We identified marked variation across hospitals, with HAAR ranging from 6.9 to 65.9%. After adjusting for hospital variables, there was still marked variation between hospitals with similar characteristics. Conclusion We found substantial variation in hospital admissions following a presentation to the ED due to LBP even after controlling by case-mix and hospital characteristics. Given the substantial costs associated with these admissions, our findings indicate the need to investigate sources of variation and to determine instances where the observed variation is warranted or unwarranted. Supplementary information The online version contains supplementary material available at 10.1186/s12913-022-08134-8.
Collapse
Affiliation(s)
- Giovanni Ferreira
- Institute for Musculoskeletal Health, Sydney Local Health District, Sydney, Australia. .,School of Public Health, Faculty of Medicine and Health, The University of Sydney, Sydney, NSW, Australia. .,, Camperdown, Australia.
| | - Marina Lobo
- Center for Health Technology and Services Research (CINTESIS), Department of Community Medicine, Information and Health Decision Sciences (MEDCIDS), Faculty of Medicine, University of Porto, Porto, Portugal
| | - Bethan Richards
- Institute for Musculoskeletal Health, Sydney Local Health District, Sydney, Australia
| | - Michael Dinh
- The RPA Green Light Institute for Emergency Care, Royal Prince Alfred Hospital, Sydney, Australia
| | - Chris Maher
- Institute for Musculoskeletal Health, Sydney Local Health District, Sydney, Australia.,School of Public Health, Faculty of Medicine and Health, The University of Sydney, Sydney, NSW, Australia
| |
Collapse
|
7
|
Azizi PM, Wijeysundera DN, Wijeysundera HC, Austin PC, Jerath A, Han L, Koh M, Ko DT. Troponin testing after noncardiac surgery: a population-based historical cohort study on variation and factors associated with testing in Ontario. Can J Anaesth 2022; 69:572-581. [DOI: 10.1007/s12630-022-02219-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2021] [Revised: 12/12/2021] [Accepted: 12/29/2021] [Indexed: 10/18/2022] Open
|
8
|
Simmonds KP, Burke J, Kozlowski AJ, Andary M, Luo Z, Reeves MJ. Emulating Three Clinical Trials that Compare Stroke Rehabilitation at Inpatient Rehabilitation Facilities to Skilled Nursing Facilities. Arch Phys Med Rehabil 2022; 103:1311-1319. [PMID: 35245481 DOI: 10.1016/j.apmr.2021.12.029] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/23/2021] [Accepted: 12/12/2021] [Indexed: 11/27/2022]
Abstract
OBJECTIVE To inform the design of a potential future randomized controlled trial, we emulated three trials where patient-level outcomes were compared following stroke rehabilitation at Inpatient Rehabilitation Facilities (IRFs) to Skilled Nursing Facilities (SNFs). DESIGN Trials were emulated using a 1:1 matched propensity score analysis. The three trials differed as facilities from rehabilitation networks with different case-volumes were compared. Rehabilitation network case-volumes were based on the number of stroke patients that each hospital discharged to each specific IRF or SNF. Trial 1 included 60,529 patients from all networks, trial 2 included 34,444 patients from networks with medium- and large case-volumes (i.e., ≥5 patients), trial 3 included 19,161 patients from networks with large case-volumes (i.e., ≥10 patients). E-values were calculated to estimate the minimum strength that an unmeasured confounder would need to be to nullify the results. SETTING A national sample of IRFs and SNFs from across the United States. PARTICIPANTS Acute Fee-for-service Medicare stroke patients who received IRF or SNF based rehabilitation. INTERVENTIONS Not applicable. MAIN OUTCOME MEASURE(S) 1-year successful community discharge (home for >30 consecutive days) and all-cause mortality. RESULTS Overall, 29,500, 15,156, and 7,450 patients were matched for trials 1, 2 and 3. For 1-year successful community discharge, absolute risk differences for IRF patients were 0.21 (95% CI: 0.20, 0.22), 0.17 (95% CI: 0.16, 0.19), and 0.12 (95% CI: 0.10, 0.14) in trials 1, 2 and 3, respectively. For 1-year all-cause mortality, corresponding risk differences were -0.11 (95% CI: -0.12, -0.11), -0.11 (95% CI: -0.12, -0.09), and -0.08 (95% CI: -0.10, -0.06). E-values indicated that a moderately sized unmeasured confounder, with a relative risk of 1.6 to 2.0 would nullify differences in successful community discharge. CONCLUSION(S) IRF patients had superior outcomes, but differences were attenuated when IRFs and SNFs from larger rehabilitation networks were compared. The vulnerability of the findings to unmeasured confounding supports the need for an RCT.
Collapse
Affiliation(s)
- Kent P Simmonds
- Department of Epidemiology and Biostatistics, Michigan State University - College of Human Medicine
| | - James Burke
- Department of Neurology, University of Michigan School of Medicine, Ann Arbor, Mi
| | - Allan J Kozlowski
- Department of Epidemiology and Biostatistics, Michigan State University - College of Human Medicine; John F. Butzer Center for Research and Innovation, Mary Free Bed Rehabilitation Hospital
| | - Michael Andary
- Department of Physical Medicine & Rehabilitation, Michigan State University - College of Osteopathic Medicine
| | - Zhehui Luo
- Department of Epidemiology and Biostatistics, Michigan State University - College of Human Medicine
| | - Mathew J Reeves
- Department of Epidemiology and Biostatistics, Michigan State University - College of Human Medicine.
| |
Collapse
|
9
|
Gu HQ, Wang CJ, Yang X, Liu C, Wang X, Zhao XQ, Wang YL, Liu LP, Meng X, Jiang Y, Li H, Wang YJ, Li ZX. Sex differences in vascular risk factors, in-hospital management, and outcomes of patients with acute ischemic stroke in China. Eur J Neurol 2021; 29:188-198. [PMID: 34564908 DOI: 10.1111/ene.15124] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/13/2021] [Revised: 09/19/2021] [Accepted: 09/21/2021] [Indexed: 02/05/2023]
Abstract
BACKGROUND Previous assessments of sex differences for patients with acute ischemic stroke were limited in a specific region or population, narrow scope, or small sample size. METHODS Patients with acute ischemic stroke hospitalized in the China Stroke Center Alliance hospitals were analyzed. Absolute standardized differences (ASDs) were used to assess sex differences in vascular risk factors, guideline-recommended in-hospital management measures and outcomes, including stroke severity (National Institutes of Health Stroke Scale≥16), death/discharge against medical advice, major adverse cardiovascular events, pneumonia, and disability (modified Rankin Scale≥3). RESULTS Of 838,229 patients analyzed, 524351 (62.6%) were men and 313,878 (37.4%) were women. Compared with men, women were older (68.6 vs. 64.7 years), had higher prevalence of hypertension (67.7% vs. 62.4%), diabetes (24.7% vs. 19.5%), and atrial fibrillation (7.1% vs. 4.3%), but lower prevalence of smoking (4.5% vs. 56.6%) and drinking (2.6% vs 35.8%) (ASDs >10%). No sex differences were seen in guideline-directed management measures, indicated by risk-adjusted individual measures and the all-or-null summary measure (34.5% vs 34.9%, ASD = 1.0%). Compared to men, women tended to have strokes that were more severe at presentation (6.5% vs. 4.5%, ASD = 8.8%) and more disabilities at discharge (34.9% vs 30.5%, ASD =9.4%). However, all sex-related differences in outcomes were attenuated to null after risk adjustments (ASDs<2%). CONCLUSIONS Compared to male patients, female patients had more vascular risk factors and received similar in-hospital care. They had strokes that were more severe at presentation and more disabilities at discharge, both of which may be explained by worse vascular risk profiles.
Collapse
Affiliation(s)
- Hong-Qiu Gu
- China National Clinical Research Center for Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing, China.,National Center for Healthcare Quality Management in Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
| | - Chun-Juan Wang
- China National Clinical Research Center for Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing, China.,National Center for Healthcare Quality Management in Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
| | - Xin Yang
- China National Clinical Research Center for Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing, China.,National Center for Healthcare Quality Management in Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
| | - Chelsea Liu
- Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, USA
| | - Xia Wang
- The George Institute for Global Health, University of New South Wales, Sydney, Australia
| | - Xing-Quan Zhao
- Vascular Neurology, Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
| | - Yi-Long Wang
- Vascular Neurology, Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
| | - Li-Ping Liu
- Neuro-intensive Care Unit, Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
| | - Xia Meng
- China National Clinical Research Center for Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
| | - Yong Jiang
- China National Clinical Research Center for Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
| | - Hao Li
- China National Clinical Research Center for Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
| | - Yong-Jun Wang
- China National Clinical Research Center for Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing, China.,National Center for Healthcare Quality Management in Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing, China.,Vascular Neurology, Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
| | - Zi-Xiao Li
- China National Clinical Research Center for Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing, China.,National Center for Healthcare Quality Management in Neurological Diseases, Beijing Tiantan Hospital, Capital Medical University, Beijing, China.,Vascular Neurology, Department of Neurology, Beijing Tiantan Hospital, Capital Medical University, Beijing, China.,Chinese Institute for Brain Research, Beijing, China
| |
Collapse
|
10
|
van Meenen LCC, den Hartog SJ, Groot AE, Emmer BJ, Smeekes MD, Siegers A, Kommer GJ, Majoie CBLM, Roos YBWEM, van Es ACGM, Dippel DW, van der Worp HB, Lingsma HF, Roozenbeek B, Coutinho JM. Relationship between primary stroke center volume and time to endovascular thrombectomy in acute ischemic stroke. Eur J Neurol 2021; 28:4031-4038. [PMID: 34528335 PMCID: PMC9292965 DOI: 10.1111/ene.15107] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/06/2021] [Revised: 09/03/2021] [Accepted: 09/09/2021] [Indexed: 11/29/2022]
Abstract
Background and purpose We investigated whether the annual volume of patients with acute ischemic stroke referred from a primary stroke center (PSC) for endovascular treatment (EVT) is associated with treatment times and functional outcome. Methods We used data from the Multicenter Randomized Clinical Trial of Endovascular Treatment for Acute Ischemic Stroke in the Netherlands (MR CLEAN) registry (2014–2017). We included patients with acute ischemic stroke of the anterior circulation who were transferred from a PSC to a comprehensive stroke center (CSC) for EVT. We examined the association between EVT referral volume of PSCs and treatment times and functional outcome using multivariable regression modeling. The main outcomes were time from arrival at the PSC to groin puncture (PSC‐door‐to‐groin time), adjusted for estimated ambulance travel times, time from arrival at the CSC to groin puncture (CSC‐door‐to‐groin time), and modified Rankin Scale (mRS) score at 90 days after stroke. Results Of the 3637 patients in the registry, 1541 patients (42%) from 65 PSCs were included. Mean age was 71 years (SD ± 13.3), median National Institutes of Health Stroke Scale score was 16 (interquartile range [IQR]: 12–19), and median time from stroke onset to arrival at the PSC was 53 min (IQR: 38–90). Eighty‐three percent had received intravenous thrombolysis. EVT referral volume was not associated with PSC‐door‐to‐groin time (adjusted coefficient: −0.49 min/annual referral, 95% confidence interval [CI]: −1.27 to 0.29), CSC‐door‐to‐groin time (adjusted coefficient: −0.34 min/annual referral, 95% CI: −0.69 to 0.01) or 90‐day mRS score (adjusted common odds ratio: 0.99, 95% CI: 0.96–1.01). Conclusions In patients transferred from a PSC for EVT, higher PSC volumes do not seem to translate into better workflow metrics or patient outcome.
Collapse
Affiliation(s)
- Laura C C van Meenen
- Department of Neurology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, the Netherlands
| | - Sanne J den Hartog
- Department of Neurology, Erasmus Medical Center, University Medical Center, Rotterdam, the Netherlands.,Department of Radiology & Nuclear Medicine, Erasmus Medical Center, University Medical Center, Rotterdam, the Netherlands.,Department of Public Health, Erasmus Medical Center, University Medical Center, Rotterdam, the Netherlands
| | - Adrien E Groot
- Department of Neurology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, the Netherlands
| | - Bart J Emmer
- Department of Radiology and Nuclear Medicine, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, the Netherlands
| | - Martin D Smeekes
- Emergency Medical Services North-Holland North, Alkmaar, the Netherlands
| | | | - Geert Jan Kommer
- Center for Nutrition, Prevention, and Health Services, National Institute of Public Health and the Environment, Bilthoven, the Netherlands
| | - Charles B L M Majoie
- Department of Radiology and Nuclear Medicine, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, the Netherlands
| | - Yvo B W E M Roos
- Department of Neurology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, the Netherlands
| | - Adriaan C G M van Es
- Department of Radiology and Nuclear Medicine, Leiden University Medical Center, Leiden, the Netherlands
| | - Diederik W Dippel
- Department of Neurology, Erasmus Medical Center, University Medical Center, Rotterdam, the Netherlands
| | - H Bart van der Worp
- Department of Neurology and Neurosurgery, Brain Center, University Medical Center Utrecht, Utrecht, the Netherlands
| | - Hester F Lingsma
- Department of Public Health, Erasmus Medical Center, University Medical Center, Rotterdam, the Netherlands
| | - Bob Roozenbeek
- Department of Neurology, Erasmus Medical Center, University Medical Center, Rotterdam, the Netherlands
| | - Jonathan M Coutinho
- Department of Neurology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, the Netherlands
| | | |
Collapse
|
11
|
Variability in outcomes and quality-of-care indicators across clinics participating in a large smoking-cessation program. J Subst Abuse Treat 2021; 130:108409. [PMID: 34118701 DOI: 10.1016/j.jsat.2021.108409] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2020] [Revised: 04/08/2021] [Accepted: 04/08/2021] [Indexed: 11/20/2022]
Abstract
BACKGROUND The effectiveness of care for substance-related problems varies across providers. Best-known treatments are rarely universally applied, and various process differences can affect participant outcomes. Measuring and understanding this variability can suggest changes that will improve system performance. METHODS We measure variability in 7-day cigarette abstinence at a six-month follow-up; return for a second clinical visit; and receipt of combination nicotine replacement therapy across 223 primary care clinics participating in the Smoking Treatment for Ontario Patients program, a large smoking cessation initiative in Ontario, Canada. We include 41,992 enrolments from April 11, 2016 and May 31, 2019. We risk adjust for demographic and clinical case-mix differences and characterize variability using funnel plots and measures based on clinic-level variance explained. The abstinence outcome is missing for 38% of participants. We adjust for missingness using multiple imputation and inverse probability weighting. RESULTS Abstinence was achieved by 28.0% (95% CI = 27.5%-28.5%) of participants, 63.2% (62.8%-63.7%) received combination NRT, and 72.9% (72.4%-73.3%) returned for a second clinical visit. Variability was moderate for abstinence (median odds ratio (MOR) = 1.16) and pronounced for return visit (MOR = 1.29) and combination therapy (MOR = 1.89). CONCLUSION Outcomes and processes vary significantly across clinics within a program with shared guidelines and standards. Differences across providers may be greater in other contexts. Results underscore the importance of measuring and understanding variability, and of ongoing maintenance and improvement. The existence of high outliers holds out the possibility of identifying practices that might be more widely adopted.
Collapse
|
12
|
Casciani F, Trudeau MT, Asbun HJ, Ball CG, Bassi C, Behrman SW, Berger AC, Bloomston MP, Callery MP, Christein JD, Falconi M, Fernandez-Del Castillo C, Dillhoff ME, Dickson EJ, Dixon E, Fisher WE, House MG, Hughes SJ, Kent TS, Malleo G, Partelli S, Salem RR, Stauffer JA, Wolfgang CL, Zureikat AH, Vollmer CM. Surgeon experience contributes to improved outcomes in pancreatoduodenectomies at high risk for fistula development. Surgery 2021; 169:708-720. [PMID: 33386129 DOI: 10.1016/j.surg.2020.11.022] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2020] [Revised: 11/12/2020] [Accepted: 11/17/2020] [Indexed: 12/31/2022]
Abstract
BACKGROUND Pancreatoduodenectomies at high risk for clinically relevant pancreatic fistula are uncommon, yet intimidating, situations. In such scenarios, the impact of individual surgeon experience on outcomes is poorly understood. METHODS The fistula risk score was applied to identify high-risk patients (fistula risk score 7-10) from 7,706 pancreatoduodenectomies performed at 18 international institutions (2003-2020). For each case, surgeon pancreatoduodenectomy career volume and years of practice were linked to intraoperative fistula mitigation strategy adoption and outcomes. Consequently, best operative approaches for clinically relevant pancreatic fistula prevention and best performer profiles were identified through multivariable analysis models. RESULTS Eight hundred and thirty high-risk pancreatoduodenectomies, performed by 64 surgeons, displayed an overall clinically relevant pancreatic fistula rate of 33.7%. Clinically relevant pancreatic fistula rates decreased with escalating surgeon career pancreatoduodenectomy (-49.7%) and career length (-41.2%; both P < .001), as did transfusion and reoperation rates, postoperative morbidity index, and duration of stay. Great experience (≥400 pancreatoduodenectomies performed or ≥21-year-long career) was a significant predictor of clinically relevant pancreatic fistula prevention (odds ratio 0.52, 95% confidence interval 0.35-0.76) and was more often associated with pancreatojejunostomy reconstruction and prophylactic octreotide omission, which were both independently associated with clinically relevant pancreatic fistula reduction. A risk-adjusted performance analysis also correlated with experience. Moreover, minimizing blood loss (≤400 mL) significantly contributed to clinically relevant pancreatic fistula prevention (odds ratio 0.40, 95% confidence interval 0.22-0.74). CONCLUSION Surgeon experience is a key contributor to achieve better outcomes after high-risk pancreatoduodenectomy. Surgeons can improve their performance in these challenging situations by employing pancreatojejunostomy reconstruction, omitting prophylactic octreotide, and minimizing blood loss.
Collapse
Affiliation(s)
- Fabio Casciani
- Department of Surgery, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA; Unit of General and Pancreatic Surgery, The Pancreas Institute, University of Verona, Italy
| | - Maxwell T Trudeau
- Department of Surgery, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA
| | | | - Chad G Ball
- Department of Surgery, University of Calgary, Canada
| | - Claudio Bassi
- Unit of General and Pancreatic Surgery, The Pancreas Institute, University of Verona, Italy
| | | | - Adam C Berger
- Department of Surgery, Jefferson Medical College, Philadelphia, PA
| | - Mark P Bloomston
- Department of Surgery, The Ohio State University Wexner Medical Center, Columbus, OH
| | - Mark P Callery
- Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA
| | - John D Christein
- Department of Surgery, University of Alabama at Birmingham School of Medicine, AL
| | | | | | - Mary E Dillhoff
- Department of Surgery, The Ohio State University Wexner Medical Center, Columbus, OH
| | - Euan J Dickson
- West of Scotland Pancreatic Unit, Glasgow Royal Infirmary, United Kingdom
| | - Elijah Dixon
- Department of Surgery, University of Calgary, Canada
| | | | - Michael G House
- Department of Surgery, Indiana University School of Medicine, Indianapolis, IN
| | - Steven J Hughes
- Department of Surgery, University of Florida College of Medicine, Gainesville, FL
| | - Tara S Kent
- Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA
| | - Giuseppe Malleo
- Unit of General and Pancreatic Surgery, The Pancreas Institute, University of Verona, Italy
| | | | - Ronald R Salem
- Department of Surgery, Yale School of Medicine, New Haven, CT
| | | | | | - Amer H Zureikat
- Department of Surgery, University of Pittsburgh Medical Center, Pittsburgh, PA
| | - Charles M Vollmer
- Department of Surgery, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA.
| |
Collapse
|
13
|
Gilmartin HM, Hess E, Mueller C, Plomondon ME, Waldo SW, Battaglia C. A pilot study to assess the learning environment and use of reliability enhancing work practices in VHA cardiac catheterization laboratories. Learn Health Syst 2021; 5:e10227. [PMID: 33889736 PMCID: PMC8051348 DOI: 10.1002/lrh2.10227] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/11/2019] [Revised: 02/26/2020] [Accepted: 03/15/2020] [Indexed: 12/02/2022] Open
Abstract
INTRODUCTION A learning health system (LHS) harnesses data and analytics to learn from clinical encounters to implement the best care with high reliability. The 81 Veterans Health Administration (VHA) cardiac catheterization laboratories (cath lab) are a model LHS. The quality and safety of coronary procedures are monitored and reported by the Clinical Assessment, Reporting and Tracking (CART) Program, which has identified variation in care across cath labs. This variation may be due to underappreciated aspects of LHSs, the learning environment and reliability enhancing work practices (REWPs). Learning environments are the educational approaches, context, and settings in which learning occurs. REWPs are the organizational practices found in high reliability organizations. High learning environments and use of REWPs are associated with improved outcomes. This study assessed the learning environments and use of REWPs in VHA cath labs to examine factors supportive of learning and high reliability. METHODS In 2018, the learning organization survey-27 and the REWP survey were administered to 732 cath lab staff. Factor analysis and linear models were computed. Unit-level analyses and site ranking (high, low) were conducted on cath labs with >40% response rate using Bayesian methods. RESULTS Surveys from 40% of cath lab staff (n = 294) at 84% of cath labs (n = 68) were included. Learning environment and REWP strengths across cath labs include the presence of training programs, openness to new ideas, and respectful interaction. Learning environment and REWP gaps include lack of structured knowledge transfer (eg, checklists) and low use of forums for improvement. Survey dimensions matched established factor structures and demonstrated high reliability (Cronbach's alpha >.76). Unit-level analyses were conducted for 29 cath labs. One ranked as high and four as low learning environments. CONCLUSIONS This work demonstrates an approach to assess local learning environments and use of REWPs, providing insights for systems working to become a LHS.
Collapse
Affiliation(s)
- Heather M. Gilmartin
- Denver/Seattle Center of Innovation for Veteran‐Centered and Value Driven CareVHA Eastern Colorado Healthcare SystemAuroraColoradoUSA
- Health Systems, Management, and PolicyUniversity of Colorado, School of Public HealthAuroraColoradoUSA
| | - Edward Hess
- Denver/Seattle Center of Innovation for Veteran‐Centered and Value Driven CareVHA Eastern Colorado Healthcare SystemAuroraColoradoUSA
| | - Candice Mueller
- Denver/Seattle Center of Innovation for Veteran‐Centered and Value Driven CareVHA Eastern Colorado Healthcare SystemAuroraColoradoUSA
- Clinical Assessment Reporting and Tracking ProgramVHA Eastern Colorado Healthcare SystemAuroraColoradoUSA
| | - Mary E. Plomondon
- Health Systems, Management, and PolicyUniversity of Colorado, School of Public HealthAuroraColoradoUSA
- Clinical Assessment Reporting and Tracking ProgramVHA Eastern Colorado Healthcare SystemAuroraColoradoUSA
| | - Stephen W. Waldo
- Denver/Seattle Center of Innovation for Veteran‐Centered and Value Driven CareVHA Eastern Colorado Healthcare SystemAuroraColoradoUSA
- Clinical Assessment Reporting and Tracking ProgramVHA Eastern Colorado Healthcare SystemAuroraColoradoUSA
| | - Catherine Battaglia
- Denver/Seattle Center of Innovation for Veteran‐Centered and Value Driven CareVHA Eastern Colorado Healthcare SystemAuroraColoradoUSA
- Health Systems, Management, and PolicyUniversity of Colorado, School of Public HealthAuroraColoradoUSA
| |
Collapse
|
14
|
Austin PC, Leckie G. Bootstrapped inference for variance parameters, measures of heterogeneity and random effects in multilevel logistic regression models. J STAT COMPUT SIM 2020. [DOI: 10.1080/00949655.2020.1797738] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
Affiliation(s)
- Peter C. Austin
- ICES, Toronto, Canada
- Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada
- Schulich Heart Research Program, Sunnybrook Research Institute, Toronto, Canada
| | - George Leckie
- Centre for Multilevel Modeling, University of Bristol, Bristol, UK
| |
Collapse
|
15
|
Grunwald GK, Arnett JA, Liu W, Ho PM. Bayesian profiling for cost with zeros to decompose total cost into probability of cost and mean nonzero cost. Biom J 2020; 62:1631-1649. [PMID: 32542678 DOI: 10.1002/bimj.201900148] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2019] [Revised: 04/09/2020] [Accepted: 04/16/2020] [Indexed: 11/09/2022]
Abstract
Cost of health care can vary substantially across hospitals, centers, or providers. Data from electronic health records provide information for studying patterns of cost variation and identifying high or low cost centers. Cost data often include zero values when patients receive no care, and joint two-part models have been developed for clustered cost data with zeros. Standard methods for center comparisons, sometimes called profiling, can use these methods to incorporate zero values into total cost. However, zero costs also provide opportunities to further examine sources of cost variation and outliers. For example, a hospital may have high (or low) cost due to frequency of nonzero cost, amount of nonzero cost, or a combination of those. We give methods for decomposing hospital differences in total cost with zeros into components for probability of use (i.e., of nonzero cost) and for cost of use (mean of nonzero cost). The components multiply to total cost and quantify components on the same easily interpreted multiplicative scales. The methods are based on Bayesian hierarchical models and counterfactual arguments, with Markov chain Monte Carlo estimation. We used simulated data to illustrate use, interpretation, and visualization of the methods in diverse situations, and applied the methods to 30,024 patients at 57 US Veterans Administration hospitals to characterize outlier hospitals in one year cost of inpatient care following a cardiac procedure. Twenty eight percent of patients had zero cost. These methods are useful in providing insight into cost variation and outliers for planning future studies or interventions.
Collapse
Affiliation(s)
- Gary K Grunwald
- Department of Biostatistics and Informatics, Colorado School of Public Health, University of Colorado Anschutz Medical Campus, Aurora, CO, USA.,VA Center of Innovation for Veteran-Centered and Value Driven Care, VA Eastern Colorado Health Care System, Aurora, CO, USA
| | - James A Arnett
- Medical Economics, Contessa Health Inc., Nashville, TN, USA
| | - Wenhui Liu
- VA Center of Innovation for Veteran-Centered and Value Driven Care, VA Eastern Colorado Health Care System, Aurora, CO, USA
| | - P Michael Ho
- VA Center of Innovation for Veteran-Centered and Value Driven Care, VA Eastern Colorado Health Care System, Aurora, CO, USA.,Division of Cardiology, University of Colorado School of Medicine, Aurora, CO, USA
| |
Collapse
|
16
|
Thompson MP, Luo Z, Gardiner J, Burke JF, Nickles A, Reeves MJ. Impact of Missing Stroke Severity Data on the Accuracy of Hospital Ischemic Stroke Mortality Profiling. Circ Cardiovasc Qual Outcomes 2019; 11:e004951. [PMID: 30354572 DOI: 10.1161/circoutcomes.118.004951] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/10/2023]
Abstract
BACKGROUND The Centers for Medicare and Medicaid Services have proposed 30-day ischemic stroke risk-standardized mortality rates that include adjustment for stroke severity using the National Institute of Health Stroke Scale (NIHSS), which is often undocumented. We used simulations to quantify the effect of missing NIHSS data on the accuracy of hospital-level ischemic stroke risk-standardized mortality rate profiling for 100 hypothetical hospitals with different case volumes. METHODS AND RESULTS We generated simulated data sets of patients with NIHSS scores and other predictors of 30-day mortality based on empirical analysis of data from 7654 patients with ischemic stroke in the Michigan Stroke Registry. We assigned and rank-ordered a true (known) 30-day mortality rate to each hospital in the simulated data sets of N=100 hospitals with either low (100 patients with stroke), medium (300), or high (500) case volumes. We then estimated and rank-ordered 30-day risk-standardized mortality rates for the N=100 hospitals in each simulated data set using hierarchical logistic regression models. In each data set, we systematically varied the rate of missing NIHSS data and whether missing NIHSS data was independent (missing completely at random) or dependent (missing not at random) on the NIHSS score. With no missing NIHSS data, the Spearman correlation between the true hospital performance rank order assigned during data set generation and the estimated 30-day risk-standardized mortality rate rank order was 0.72, 0.88, and 0.91 in low, medium, and high volume hospitals, respectively. However, this fell to as low as 0.50, 0.71, and 0.79 as missing NIHSS data reached 70%. CONCLUSIONS Missing NIHSS data had substantial detrimental effects on the accuracy of profiling of ischemic stroke mortality, especially in lower volume hospitals. Even with complete NIHSS documentation, significant limitations in ischemic stroke mortality profiling remain.
Collapse
Affiliation(s)
- Michael P Thompson
- Department of Epidemiology and Biostatistics, Michigan State University, East Lansing, MI (M.P.T., Z.L., J.G., M.J.R.).,Department of Cardiac Surgery, University of Michigan Medical School, Ann Arbor, MI (M.P.T.)
| | - Zhehui Luo
- Department of Epidemiology and Biostatistics, Michigan State University, East Lansing, MI (M.P.T., Z.L., J.G., M.J.R.)
| | - Joseph Gardiner
- Department of Epidemiology and Biostatistics, Michigan State University, East Lansing, MI (M.P.T., Z.L., J.G., M.J.R.)
| | - James F Burke
- Department of Neurology, University of Michigan Medical School, Ann Arbor, MI (J.F.B.)
| | | | - Mathew J Reeves
- Department of Epidemiology and Biostatistics, Michigan State University, East Lansing, MI (M.P.T., Z.L., J.G., M.J.R.)
| |
Collapse
|
17
|
Daignault K, Lawson KA, Finelli A, Saarela O. Causal Mediation Analysis for Standardized Mortality Ratios. Epidemiology 2019; 30:532-540. [PMID: 31166215 DOI: 10.1097/ede.0000000000001015] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
Indirectly standardized mortality ratios (SMR) are often used to compare patient outcomes between health care providers as indicators of quality of care. Observed differences in the outcomes raise the question of whether these could be causally attributable to earlier processes or outcomes in the pathway of care that the patients received. Such pathways can be naturally addressed in a causal mediation analysis framework. Adopting causal mediation models allows the total provider effect on outcome to be decomposed into direct and indirect (mediated) effects. This in turn enables quantification of the improvement in patient outcomes due to a hypothetical intervention on the mediator. We formulate the effect decomposition for the indirectly standardized SMR when comparing to a health care system-wide average performance, propose novel model-based and semiparametric estimators for the decomposition, study the properties of these through simulations, and demonstrate their use through application to Ontario kidney cancer data.
Collapse
Affiliation(s)
- Katherine Daignault
- From the Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada
| | - Keith A Lawson
- Division of Urology, Departments of Surgery and Surgical Oncology, Princess Margaret Cancer Centre - University Health Network, Toronto, ON, Canada
| | - Antonio Finelli
- Division of Urology, Departments of Surgery and Surgical Oncology, Princess Margaret Cancer Centre - University Health Network, Toronto, ON, Canada
| | - Olli Saarela
- From the Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
18
|
Chen Y, Şentürk D, Estes JP, Campos LF, Rhee CM, Dalrymple LS, Kalantar-Zadeh K, Nguyen DV. Performance Characteristics of Profiling Methods and the Impact of Inadequate Case-mix Adjustment. COMMUN STAT-SIMUL C 2019; 2019. [PMID: 33311840 DOI: 10.1080/03610918.2019.1595649] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/27/2022]
Abstract
Profiling or evaluation of health care providers involves the application of statistical models to compare each provider's performance with respect to a patient outcome, such as unplanned 30-day hospital readmission, adjusted for patient case-mix characteristics. The nationally adopted method is based on random effects (RE) hierarchical logistic regression models. Although RE models are sensible for modeling hierarchical data, novel high dimensional fixed effects (FE) models have been proposed which may be well-suited for the objective of identifying sub-standard performance. However, there are limited comparative studies. Thus, we examine their relative performance, including the impact of inadequate case-mix adjustment.
Collapse
Affiliation(s)
- Yanjun Chen
- Institute for Clinical and Translational Science, University of California, Irvine, CA 92687, U.S.A
| | - Damla Şentürk
- Department of Biostatistics, University of California, Los Angeles, CA 90095, U.S.A
| | - Jason P Estes
- Research, Pratt & Whitney, East Hartford, CT 06118, U.S.A
| | - Luis F Campos
- Department of Statistics, Harvard University, Cambridge, MA 02138, U.S.A
| | - Connie M Rhee
- Harold Simmons Center for Chronic Disease Research and Epidemiology, University of California Irvine School of Medicine, Orange, CA 92868, U.S.A
| | - Lorien S Dalrymple
- Epidemiology and Research, Fresenius Medical Care, Waltham, MA 02451, U.S.A
| | - Kamyar Kalantar-Zadeh
- Harold Simmons Center for Chronic Disease Research and Epidemiology, University of California Irvine School of Medicine, Orange, CA 92868, U.S.A
| | - Danh V Nguyen
- Department of Medicine, University of California Irvine, Orange, CA 92868, U.S.A
| |
Collapse
|
19
|
Şentürk D, Chen Y, Estes JP, Campos LF, Rhee CM, Kalantar-Zadeh K, Nguyen DV. Impact of Case-Mix Measurement Error on Estimation and Inference in Profiling of Health Care Providers. COMMUN STAT-SIMUL C 2018; 49:2206-2224. [PMID: 33311842 PMCID: PMC7731965 DOI: 10.1080/03610918.2018.1515360] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2018] [Revised: 07/25/2018] [Accepted: 08/04/2018] [Indexed: 10/27/2022]
Abstract
Profiling analysis aims to evaluate health care providers by modeling each provider's performance with respect to a patient outcome, such as unplanned hospital readmission. High-dimensional regression models are used in profiling to risk-adjust for patient case-mix covariates. Case-mix covariates typically ascertained from administrative databases are inherently error-prone. We examine the impact of case-mix measurement error (ME) on profiling models. The results show that even though the models' coefficient estimates are biased, this does not affect the estimation of standardized readmission ratio (SRR). However, ME leads to increased variation in SRR estimates and degrades the ability to identify under-performing providers.
Collapse
Affiliation(s)
- Damla Şentürk
- Department of Biostatistics, University of California, Los Angeles, CA 90095, U.S.A
| | - Yanjun Chen
- Institute for Clinical and Translational Science, University of California, Irvine, CA 92687, U.S.A
| | - Jason P. Estes
- Research, Pratt & Whitney, East Hartford, CT 06118, U.S.A
| | - Luis F. Campos
- Department of Statistics, Harvard University, Cambridge, MA 02138, U.S.A
| | - Connie M. Rhee
- Harold Simmons Center for Chronic Disease Research and Epidemiology, University of California Irvine School of Medicine, Orange, CA 92868, U.S.A
- Department of Medicine, University of California Irvine, Orange, CA 92868, U.S.A
| | - Kamyar Kalantar-Zadeh
- Harold Simmons Center for Chronic Disease Research and Epidemiology, University of California Irvine School of Medicine, Orange, CA 92868, U.S.A
- Department of Medicine, University of California Irvine, Orange, CA 92868, U.S.A
| | - Danh V. Nguyen
- Harold Simmons Center for Chronic Disease Research and Epidemiology, University of California Irvine School of Medicine, Orange, CA 92868, U.S.A
| |
Collapse
|
20
|
Brakenhoff TB, Moons KG, Kluin J, Groenwold RH. Investigating Risk Adjustment Methods for Health Care Provider Profiling When Observations are Scarce or Events Rare. Health Serv Insights 2018; 11:1178632918785133. [PMID: 30083056 PMCID: PMC6069022 DOI: 10.1177/1178632918785133] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2018] [Accepted: 05/24/2018] [Indexed: 12/03/2022] Open
Abstract
Background: When profiling health care providers, adjustment for case-mix is essential. However, conventional risk adjustment methods may perform poorly, especially when provider volumes are small or events rare. Propensity score (PS) methods, commonly used in observational studies of binary treatments, have been shown to perform well when the amount of observations and/or events are low and can be extended to a multiple provider setting. The objective of this study was to evaluate the performance of different risk adjustment methods when profiling multiple health care providers that perform highly protocolized procedures, such as coronary artery bypass grafting. Methods: In a simulation study, provider effects estimated using PS adjustment, PS weighting, PS matching, and multivariable logistic regression were compared in terms of bias, coverage and mean squared error (MSE) when varying the event rate, sample size, provider volumes, and number of providers. An empirical example from the field of cardiac surgery was used to demonstrate the different methods. Results: Overall, PS adjustment, PS weighting, and logistic regression resulted in provider effects with low amounts of bias and good coverage. The PS matching and PS weighting with trimming led to biased effects and high MSE across several scenarios. Moreover, PS matching is not practical to implement when the number of providers surpasses three. Conclusions: None of the PS methods clearly outperformed logistic regression, except when sample sizes were relatively small. Propensity score matching performed worse than the other PS methods considered.
Collapse
Affiliation(s)
- Timo B Brakenhoff
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Karel Gm Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Jolanda Kluin
- Heart Center, Academic Medical Center, Amsterdam, The Netherlands
| | - Rolf Hh Groenwold
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands
| |
Collapse
|
21
|
Bottle A, Honeyford K, Chowdhury F, Bell D, Aylin P. Factors associated with hospital emergency readmission and mortality rates in patients with heart failure or chronic obstructive pulmonary disease: a national observational study. HEALTH SERVICES AND DELIVERY RESEARCH 2018. [PMID: 30044581 DOI: 10.3310/hsdr06260] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023] Open
Abstract
BackgroundHeart failure (HF) and chronic obstructive pulmonary disease (COPD) lead to unplanned hospital activity, but our understanding of what drives this is incomplete.ObjectivesTo model patient, primary care and hospital factors associated with readmission and mortality for patients with HF and COPD, to assess the statistical performance of post-discharge emergency department (ED) attendance compared with readmission metrics and to compare all the results for the two conditions.DesignObservational study.SettingEnglish NHS.ParticipantsAll patients admitted to acute non-specialist hospitals as an emergency for HF or COPD.InterventionsNone.Main outcome measuresOne-year mortality and 30-day emergency readmission following the patient’s first unplanned admission (‘index admission’) for HF or COPD.Data sourcesPatient-level data from Hospital Episodes Statistics were combined with publicly available practice- and hospital-level data on performance, patient and staff experience and rehabilitation programme website information.ResultsOne-year mortality rates were 39.6% for HF and 24.1% for COPD and 30-day readmission rates were 19.8% for HF and 16.5% for COPD. Most patients were elderly with multiple comorbidities. Patient factors predicting mortality included older age, male sex, white ethnicity, prior missed outpatient appointments, (long) index length of hospital stay (LOS) and several comorbidities. Older age, missed appointments, (short) LOS and comorbidities also predicted readmission. Of the practice and hospital factors we considered, only more doctors per 10 beds [odds ratio (OR) 0.95 per doctor;p < 0.001] was significant for both cohorts for mortality, with staff recommending to friends and family (OR 0.80 per unit increase;p < 0.001) and number of general practitioners (GPs) per 1000 patients (OR 0.89 per extra GP;p = 0.004) important for COPD. For readmission, only hospital size [OR per 100 beds = 2.16, 95% confidence interval (CI) 1.34 to 3.48 for HF, and 2.27, 95% CI 1.40 to 3.66 for COPD] and doctors per 10 beds (OR 0.98;p < 0.001) were significantly associated. Some factors, such as comorbidities, varied in importance depending on the readmission diagnosis. ED visits were common after the index discharge, with 75% resulting in admission. Many predictors of admission at this visit were as for readmission minus comorbidities and plus attendance outside the day shift and numbers of admissions that hour. Hospital-level rates for ED attendance varied much more than those for readmission, but the omega statistics favoured them as a performance indicator.LimitationsData lacked direct information on disease severity and ED attendance reasons; NHS surveys were not specific to HF or COPD patients; and some data sets were aggregated.ConclusionsFollowing an index admission for HF or COPD, older age, prior missed outpatient appointments, LOS and many comorbidities predict both mortality and readmission. Of the aggregated practice and hospital information, only doctors per bed and numbers of hospital beds were strongly associated with either outcome (both negatively). The 30-day ED visits and diagnosis-specific readmission rates seem to be useful performance indicators.Future workHospital variations in ED visits could be investigated using existing data despite coding limitations. Primary care management could be explored using individual-level linked databases.FundingThe National Institute for Health Research Health Services and Delivery Research programme.
Collapse
Affiliation(s)
- Alex Bottle
- Dr Foster Unit, Department of Primary Care and Public Health, Imperial College London, London, UK
| | - Kate Honeyford
- Dr Foster Unit, Department of Primary Care and Public Health, Imperial College London, London, UK
| | - Faiza Chowdhury
- Department of Acute Medicine, Chelsea and Westminster Hospital, Imperial College London, London, UK
- National Institute for Health Research under the Collaborations for Leadership in Applied Health Research and Care Programme North West London, Imperial College London, London, UK
| | - Derek Bell
- Department of Acute Medicine, Chelsea and Westminster Hospital, Imperial College London, London, UK
- National Institute for Health Research under the Collaborations for Leadership in Applied Health Research and Care Programme North West London, Imperial College London, London, UK
- Royal College of Physicians, Edinburgh, UK
| | - Paul Aylin
- Dr Foster Unit, Department of Primary Care and Public Health, Imperial College London, London, UK
| |
Collapse
|
22
|
Brakenhoff TB, Roes KCB, Moons KGM, Groenwold RHH. Outlier classification performance of risk adjustment methods when profiling multiple providers. BMC Med Res Methodol 2018; 18:54. [PMID: 29902975 PMCID: PMC6003201 DOI: 10.1186/s12874-018-0510-1] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2017] [Accepted: 05/15/2018] [Indexed: 12/25/2022] Open
Abstract
BACKGROUND When profiling multiple health care providers, adjustment for case-mix is essential to accurately classify the quality of providers. Unfortunately, misclassification of provider performance is not uncommon and can have grave implications. Propensity score (PS) methods have been proposed as viable alternatives to conventional multivariable regression. The objective was to assess the outlier classification performance of risk adjustment methods when profiling multiple providers. METHODS In a simulation study based on empirical data, the classification performance of logistic regression (fixed and random effects), PS adjustment, and three PS weighting methods was evaluated when varying parameters such as the number of providers, the average incidence of the outcome, and the percentage of outliers. Traditional classification accuracy measures were considered, including sensitivity and specificity. RESULTS Fixed effects logistic regression consistently had the highest sensitivity and negative predictive value, yet a low specificity and positive predictive value. Of the random effects methods, PS adjustment and random effects logistic regression performed equally well or better than all the remaining PS methods for all classification accuracy measures across the studied scenarios. CONCLUSIONS Of the evaluated PS methods, only PS adjustment can be considered a viable alternative to random effects logistic regression when profiling multiple providers in different scenarios.
Collapse
Affiliation(s)
- Timo B. Brakenhoff
- Julius Center for Health Sciences and Primary CareUniversity Medical Center Utrecht, PO Box 85500, Utrecht, 3508 GA the Netherlands
| | - Kit C. B. Roes
- Julius Center for Health Sciences and Primary CareUniversity Medical Center Utrecht, PO Box 85500, Utrecht, 3508 GA the Netherlands
| | - Karel G. M. Moons
- Julius Center for Health Sciences and Primary CareUniversity Medical Center Utrecht, PO Box 85500, Utrecht, 3508 GA the Netherlands
| | - Rolf H. H. Groenwold
- Julius Center for Health Sciences and Primary CareUniversity Medical Center Utrecht, PO Box 85500, Utrecht, 3508 GA the Netherlands
| |
Collapse
|
23
|
Gutacker N, Bloor K, Bojke C, Walshe K. Should interventions to reduce variation in care quality target doctors or hospitals? Health Policy 2018; 122:660-666. [PMID: 29703654 PMCID: PMC6022214 DOI: 10.1016/j.healthpol.2018.04.004] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2017] [Revised: 10/16/2017] [Accepted: 04/06/2018] [Indexed: 11/29/2022]
Abstract
Performance management initiatives are increasingly targeting individual doctors as well as hospitals. Less than 25% of variation in clinical outcomes is attributable to providers. More variation in clinical outcomes is associated with doctors than with hospitals. Performance estimates for individual doctors are unreliable due to small samples.
Interventions to reduce variation in care quality are increasingly targeted at both individual doctors and the organisations in which they work. Concerns remain about the scope and consequences for such performance management, the relative contribution of individuals and organisations to observed variation, and whether performance can be measured reliably. This study explores these issues in the context of the English National Health Service by analysing comprehensive administrative data for all patients treated for four clinical conditions (acute myocardial infarction, hip fracture, pneumonia, ischemic stroke) and two surgical procedures (coronary artery bypass, hip replacement) during April 2010–February 2013. Performance indicators are defined as 30-day mortality, 28-day emergency readmission and inpatient length of stay. Three-level hierarchical generalised linear mixed models are estimated to attribute variation in case-mix adjusted indicators to individual doctors and hospital organisations. Except for length of stay after hip replacement, no more than 11% of variation in case-mix adjusted performance indicators can be attributed to doctors and organisations with the rest reflecting random chance and unobserved patient factors. Doctor variation exceeds hospital variation by a factor of 1.2 or more. However, identifying poor performance amongst doctors is hampered by insufficient numbers of cases per doctor to reliably estimate their individual performances. Policy makers and regulators should therefore be cautious when targeting individual doctors in performance improvement initiatives.
Collapse
Affiliation(s)
- Nils Gutacker
- Centre for Health Economics, University of York, Heslington, York, YO10 5DD, United Kingdom.
| | - Karen Bloor
- Department of Health Sciences, University of York, United Kingdom.
| | - Chris Bojke
- Centre for Health Economics, University of York, United Kingdom.
| | - Kieran Walshe
- Alliance Manchester Business School, University of Manchester, United Kingdom.
| |
Collapse
|
24
|
Gutacker N, Street A. Multidimensional performance assessment of public sector organisations using dominance criteria. HEALTH ECONOMICS 2018; 27:e13-e27. [PMID: 28833902 PMCID: PMC5900921 DOI: 10.1002/hec.3554] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/24/2016] [Revised: 04/05/2017] [Accepted: 06/12/2017] [Indexed: 05/21/2023]
Abstract
Public sector organisations pursue multiple objectives and serve a number of stakeholders. But stakeholders are rarely explicit about the valuations they attach to different objectives, nor are these valuations likely to be identical. This complicates the assessment of their performance because no single set of weights can be chosen legitimately to aggregate outputs into unidimensional composite scores. We propose the use of dominance criteria in a multidimensional performance assessment framework to identify best practice and poor performance under relatively weak assumptions about stakeholders' preferences. We use as an example providers of hip replacement surgery in the English National Health Service and estimate multivariate multilevel models to study their performance in terms of length of stay, readmission rates, post-operative patient-reported health status and waiting time. We find substantial correlation between objectives and demonstrate that ignoring the correlation can lead to incorrect assessments of performance.
Collapse
|
25
|
Measurement of faculty anesthesiologists’ quality of clinical supervision has greater reliability when controlling for the leniency of the rating anesthesia resident: a retrospective cohort study. Can J Anaesth 2017; 64:643-655. [DOI: 10.1007/s12630-017-0866-4] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2016] [Revised: 01/30/2017] [Accepted: 03/13/2017] [Indexed: 11/27/2022] Open
|
26
|
Mixed effects logistic regression modeling of daily evaluations of nurse anesthetists’ work habits adjusting for leniency of the rating anesthesiologists. ACTA ACUST UNITED AC 2017. [DOI: 10.1016/j.pcorm.2017.02.001] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
|
27
|
Chalkley M, McCormick B, Anderson R, Aragon MJ, Nessa N, Nicodemo C, Redding S, Wittenberg R. Elective hospital admissions: secondary data analysis and modelling with an emphasis on policies to moderate growth. HEALTH SERVICES AND DELIVERY RESEARCH 2017. [DOI: 10.3310/hsdr05070] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022] Open
Abstract
BackgroundThe English NHS faces financial pressures that may render the growth rates of elective admissions seen between 2001/2 and 2011/12 unsustainable. A better understanding of admissions growth, and the influence of policy, are needed to minimise the impact on health gain for patients.ObjectivesThis project had several objectives: (1) to better understand the determinants of elective activity and policy to moderate growth at minimum health loss for patients; (2) to build a rich data set integrating health, practice and local area data to study general practitioner (GP) referrals and resulting admissions; (3) to predict patients whose treatment is unlikely to be cost-effective using patient-reported outcomes and to examine variation in provider performance; and (4) to study how policies that aim to reduce elective admissions may change demand for emergency care. The main drivers of elective admissions growth have increased either supply of or demand for care, and could include, for example, technical innovations or increased awareness of treatment benefits. Of the factors studied, neither system reform nor population ageing appears to be a key driver. The introduction of the prospective payment tariff ‘Payment by Results’ appears to have led to primary care trusts (PCTs) having increasingly similar lengths of stay. In deprived areas, increasing GP supply appears to moderate elective admissions. Reducing the incidence of single-handed practices tends to reduce referrals and admissions. Policies to reduce referrals are likely to reduce admissions but treatments may be particularly reduced in the lowest referring practices, in which resulting health loss may be greatest. In this model, per full-time equivalent, female and highly experienced GPs identify more patients admitted by specialists.ResultsIt appears from our studies that some patient characteristics are associated with not achieving sufficient patient gain to warrant cost-effective treatment. The introduction of independent sector treatment centres is estimated to have caused an increase in emergency activity rates at local PCTs. The explanations offered for increasing elective admissions indicate that they are manageable by health policy.ConclusionsFurther work is required to understand some of the results identified, such as whether or not high-volume Clinical Commissioning Groups are fulfilling unmet need; why some practices refer at low rates relative to admissions; why the period effect, which results from factors that equally affect all in the study at a point in time, dominates in the age–period–cohort analysis; and exactly how the emergency and elective sections of hospital treatment interact. This project relies on the analysis of secondary data. This type of research does not easily facilitate the important input of clinical experts or service users. It would be beneficial if other methods, including surveys and consultation with key stakeholders, could be incorporated into future research now that we have uncovered important questions.FundingThe National Institute for Health Research Health Services and Delivery Research programme.
Collapse
Affiliation(s)
| | - Barry McCormick
- Centre for Health Service Economics and Organisation, Nuffield Department of Primary Care, University of Oxford, Oxford, UK
| | - Robert Anderson
- Centre for Health Service Economics and Organisation, Nuffield Department of Primary Care, University of Oxford, Oxford, UK
| | | | - Nazma Nessa
- Centre for Health Service Economics and Organisation, Nuffield Department of Primary Care, University of Oxford, Oxford, UK
- Department for Business, Innovation and Skills, London, UK
| | - Catia Nicodemo
- Centre for Health Service Economics and Organisation, Nuffield Department of Primary Care, University of Oxford, Oxford, UK
| | - Stuart Redding
- Centre for Health Service Economics and Organisation, Nuffield Department of Primary Care, University of Oxford, Oxford, UK
| | - Raphael Wittenberg
- Centre for Health Service Economics and Organisation, Nuffield Department of Primary Care, University of Oxford, Oxford, UK
| |
Collapse
|
28
|
Rikhtehgaran R, Kazemi I, Verbeke G. A comparative study on estimation methods to deal with the endogeneity in linear random-intercept models with an extension. J STAT COMPUT SIM 2017. [DOI: 10.1080/00949655.2016.1196689] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
Affiliation(s)
- Reyhaneh Rikhtehgaran
- Department of Mathematical Sciences, Isfahan University of Technology, Isfahan 84156-83111, Iran
| | - Iraj Kazemi
- Department of Statistics, University of Isfahan, Isfahan 81746, Iran
| | - Geert Verbeke
- KU Leuven, B-3000 Leuven, Belgium
- U Hasselt, B-3590 Diepenbeek, Belgium
| |
Collapse
|
29
|
Abstract
Abstract
Background
The validity of basing healthcare reimbursement policy on pay-for-performance is grounded in the accuracy of performance measurement.
Methods
Monte Carlo simulation was used to examine the accuracy of performance profiling as a function of statistical methodology, case volume, and the extent to which hospital or physician performance deviates from the average.
Results
There is extensive variation in the true-positive rate and false discovery rate as a function of model specification, hospital quality, and hospital case volume. Hierarchical and nonhierarchical modeling are both highly accurate at very high case volumes for very low-quality hospitals. At equivalent case volumes and hospital effect sizes, the true-positive rate is higher for nonhierarchical modeling than for hierarchical modeling, but the false discovery rate is generally much lower for hierarchical modeling than for nonhierarchical modeling. At low hospital case volumes (200) that are typical for many procedures, and for hospitals with twice the rate of death or major complications for patients undergoing isolated coronary artery bypass graft surgery at the average hospital, hierarchical modeling missed 90.6% of low-quality hospitals, whereas nonhierarchical modeling missed 65.3%. However, at low case volumes, 38.9% of hospitals classified as low-quality outliers using nonhierarchical modeling were actually average quality, compared to 5.3% using hierarchical modeling.
Conclusions
Nonhierarchical modeling frequently misclassified average-quality hospitals as low quality. Hierarchical modeling commonly misclassified low-quality hospitals as average. Assuming that the consequences of misclassifying an average-quality hospital as low quality outweigh the consequences of misclassifying a low-quality hospital as average, hierarchical modeling may be the better choice for quality measurement.
Collapse
|
30
|
Abstract
Purpose. Hospital report cards provide information designed to help patients and providers to make decisions. The purpose of this study was to place the design of hospital report cards into a decision-theoretic framework. The authors’ objectives were 2-fold: 1st, to determine what the choice of significance level implies about the relative value of the different types of misclassifications that can arise. Second, to determine optimal significance levels for specific cost functions describing the relative costs associated with different types of misclassifications. Methods. Using a previously published theoretical model for hospital mortality, the authors computed false positive (i.e., falsely classified as providing poor-quality care) and false negative (falsely classified as providing good-quality care) rates. First, they determined the cost functions for false negatives and false positives that are implicitly associated with the use of significance levels of 0.05 and 0.01 for identifying hospitals with higher than average mortality. Second, they determined the levels of statistical significance that should be chosen to minimize predefined cost functions, thus minimizing costs associated with misclassifying hospitals. Results. The lower the statistical significance level required for identifying hospitals with higher than average mortality, the lower the implicit cost of false negatives compared to false positives. For a given significance level, the greater the number of patients treated at each hospital or the greater the proportion of truly poorly performing hospitals, the lower the value of the implicit cost incurred by a false negative compared to that for a false positive. For cost functions that put a high relative penalty on false negatives compared to false positives, the use of significance levels of 0.05 or 0.01 does not result in optimal decisions across expected number of patients treated at each hospital or proportions of truly poor-quality care. Conclusions. Hospital report cards that use significance levels of either 0.05 or 0.01 to identify hospitals that have statistically significantly higher than average mortality make implicit assumptions about cost functions, and the values of the optimal cost function vary across scenarios.
Collapse
Affiliation(s)
- Peter C Austin
- Institute for Clinical Evaluative Sciences, Toronto, Ontario, Canada.
| | | |
Collapse
|
31
|
The Importance of Integrating Clinical Relevance and Statistical Significance in the Assessment of Quality of Care--Illustrated Using the Swedish Stroke Register. PLoS One 2016; 11:e0153082. [PMID: 27054326 PMCID: PMC4824466 DOI: 10.1371/journal.pone.0153082] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2015] [Accepted: 03/23/2016] [Indexed: 11/19/2022] Open
Abstract
Background When profiling hospital performance, quality inicators are commonly evaluated through hospital-specific adjusted means with confidence intervals. When identifying deviations from a norm, large hospitals can have statistically significant results even for clinically irrelevant deviations while important deviations in small hospitals can remain undiscovered. We have used data from the Swedish Stroke Register (Riksstroke) to illustrate the properties of a benchmarking method that integrates considerations of both clinical relevance and level of statistical significance. Methods The performance measure used was case-mix adjusted risk of death or dependency in activities of daily living within 3 months after stroke. A hospital was labeled as having outlying performance if its case-mix adjusted risk exceeded a benchmark value with a specified statistical confidence level. The benchmark was expressed relative to the population risk and should reflect the clinically relevant deviation that is to be detected. A simulation study based on Riksstroke patient data from 2008–2009 was performed to investigate the effect of the choice of the statistical confidence level and benchmark value on the diagnostic properties of the method. Results Simulations were based on 18,309 patients in 76 hospitals. The widely used setting, comparing 95% confidence intervals to the national average, resulted in low sensitivity (0.252) and high specificity (0.991). There were large variations in sensitivity and specificity for different requirements of statistical confidence. Lowering statistical confidence improved sensitivity with a relatively smaller loss of specificity. Variations due to different benchmark values were smaller, especially for sensitivity. This allows the choice of a clinically relevant benchmark to be driven by clinical factors without major concerns about sufficiently reliable evidence. Conclusions The study emphasizes the importance of combining clinical relevance and level of statistical confidence when profiling hospital performance. To guide the decision process a web-based tool that gives ROC-curves for different scenarios is provided.
Collapse
|
32
|
Varewyck M, Vansteelandt S, Eriksson M, Goetghebeur E. On the practice of ignoring center-patient interactions in evaluating hospital performance. Stat Med 2015; 35:227-38. [PMID: 26303843 PMCID: PMC5049670 DOI: 10.1002/sim.6634] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2015] [Accepted: 08/07/2015] [Indexed: 12/11/2022]
Abstract
We evaluate the performance of medical centers based on a continuous or binary patient outcome (e.g., 30‐day mortality). Common practice adjusts for differences in patient mix through outcome regression models, which include patient‐specific baseline covariates (e.g., age and disease stage) besides center effects. Because a large number of centers may need to be evaluated, the typical model postulates that the effect of a center on outcome is constant over patient characteristics. This may be violated, for example, when some centers are specialized in children or geriatric patients. Including interactions between certain patient characteristics and the many fixed center effects in the model increases the risk for overfitting, however, and could imply a loss of power for detecting centers with deviating mortality. Therefore, we assess how the common practice of ignoring such interactions impacts the bias and precision of directly and indirectly standardized risks. The reassuring conclusion is that the common practice of working with the main effects of a center has minor impact on hospital evaluation, unless some centers actually perform substantially better on a specific group of patients and there is strong confounding through the corresponding patient characteristic. The bias is then driven by an interplay of the relative center size, the overlap between covariate distributions, and the magnitude of the interaction effect. Interestingly, the bias on indirectly standardized risks is smaller than on directly standardized risks. We illustrate our findings by simulation and in an analysis of 30‐day mortality on Riksstroke. © 2015 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.
Collapse
Affiliation(s)
- Machteld Varewyck
- Department of Applied Mathematics, Computer Science and Statistics, Ghent University, Ghent, 9000, Belgium
| | - Stijn Vansteelandt
- Department of Applied Mathematics, Computer Science and Statistics, Ghent University, Ghent, 9000, Belgium
| | - Marie Eriksson
- Department of Statistics, Umeå University, 901 87, Umeå, Sweden
| | - Els Goetghebeur
- Department of Applied Mathematics, Computer Science and Statistics, Ghent University, Ghent, 9000, Belgium
| |
Collapse
|
33
|
Abstract
BACKGROUND Identifying unwarranted variation in health care can highlight opportunities to reduce harm. One often discretionary process in oncology is use of implanted ports to administer intravenous chemotherapy. While there are benefits, ports carry risks. This study's objective was to assess provider-driven variation in port use among cancer patients receiving chemotherapy. RESEARCH DESIGN Retrospective assessment using population-based SEER-Medicare data to assess differences in port use across health care providers of older adults with cancer. Participants included over 18,000 patients ages 66 and older diagnosed with breast, colorectal, lung, or pancreatic cancer in 2005-2007, treated by approximately 2900 providers. We identified port use for patients receiving treatment from hospital outpatient facilities versus physicians' offices. Our main analysis assessed the likelihood of a patient receiving a port given port use by the provider's last patient. For a subset of high-use providers, we examined individual provider-level variation by estimating the risk-adjusted likelihood of insertion. RESULTS Patients receiving chemotherapy in hospital outpatient facilities were significantly less likely to receive a port than those treated in physicians' offices, with adjusted odds ratios (AOR) varying from 0.50 to 0.75 across cancer sites. Implanting a port was associated with increased likelihood of port insertion in the provider's next patient (AOR varied from 1.71 to 2.25). Significant between-provider variation was found among providers with at least 10 patients. CONCLUSIONS Our findings support the idea that there is provider-driven variation in the use of ports for chemotherapy administration. This variation highlights an opportunity to standardize practice and reduce unnecessary use.
Collapse
|
34
|
Bradley SM, Hess E, Winchester DE, Sussman JB, Aggarwal V, Maddox TM, Barón AE, Rumsfeld JS, Ho PM. Stress Testing After Percutaneous Coronary Intervention in the Veterans Affairs HealthCare System: Insights from the Veterans Affairs Clinical Assessment, Reporting, and Tracking Program. Circ Cardiovasc Qual Outcomes 2015. [PMID: 26198400 DOI: 10.1161/circoutcomes.114.001561] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
BACKGROUND Stress testing after percutaneous coronary intervention (PCI) in fee-for-service settings is common and rates vary by hospital. Rates of stress testing after PCI within integrated healthcare systems, such as the Veterans Affairs (VA) are unknown. METHODS AND RESULTS We evaluated all VA patients who underwent PCI from October 2007 through June 2010. To avoid the influence of Medicare eligibility on rates of stress testing use in the VA, we excluded Medicare eligible patients during the follow-up period. Hospital-level variation in risk-standardized rates of stress testing and the association with 1-year mortality and myocardial infarction was determined from Markov chain Monte Carlo methods. Among 10 293 patients undergoing PCI at 55 VA hospitals, 2239 (21.8%) had a stress test performed within 1 year of PCI and 3902 (37.9%) within 2 years. Most stress tests after PCI were performed with nuclear imaging (79.8%). The hospital-level risk-standardized rate of stress testing differed significantly from the average at 14 hospitals, with 8 (14.5%) hospitals significantly below and 6 (10.9%) hospitals significantly above the average stress testing rate. Hospital-level risk-standardized stress testing rates were not significantly correlated with risk-standardized mortality (Spearman ρ=-0.24; P=0.08) or myocardial infarction rates (Spearman ρ=0.20; P=0.14). CONCLUSIONS In the VA, nearly 40% of patients underwent stress testing in the 2 years after PCI, which is a third less than published studies from other healthcare systems. However, stress testing rates varied across VA hospitals, suggesting opportunities to optimize the use of stress testing are still present in integrated healthcare systems.
Collapse
Affiliation(s)
- Steven M Bradley
- From the VA Eastern Colorado Health Care System, Denver (S.M.B., E.H., T.M.M., A.E.B., J.S.R., P.M.H.); University of Colorado School of Medicine, Aurora (S.M.B., T.M.M., A.E.B., J.S.R., P.M.H.); Colorado Cardiovascular Outcomes Research Consortium, Denver (S.M.B., E.H., T.M.M., J.S.R., P.M.H.); Malcom Randall VA Medical Center, Gainesville, FL (D.E.W.); University of Florida College of Medicine, Gainesville (D.E.W.); VA Ann Arbor Healthcare System, Ann Arbor, MI (J.B.S.); and Temple University School of Medicine, Philadelphia, PA (V.A.).
| | - Edward Hess
- From the VA Eastern Colorado Health Care System, Denver (S.M.B., E.H., T.M.M., A.E.B., J.S.R., P.M.H.); University of Colorado School of Medicine, Aurora (S.M.B., T.M.M., A.E.B., J.S.R., P.M.H.); Colorado Cardiovascular Outcomes Research Consortium, Denver (S.M.B., E.H., T.M.M., J.S.R., P.M.H.); Malcom Randall VA Medical Center, Gainesville, FL (D.E.W.); University of Florida College of Medicine, Gainesville (D.E.W.); VA Ann Arbor Healthcare System, Ann Arbor, MI (J.B.S.); and Temple University School of Medicine, Philadelphia, PA (V.A.)
| | - David E Winchester
- From the VA Eastern Colorado Health Care System, Denver (S.M.B., E.H., T.M.M., A.E.B., J.S.R., P.M.H.); University of Colorado School of Medicine, Aurora (S.M.B., T.M.M., A.E.B., J.S.R., P.M.H.); Colorado Cardiovascular Outcomes Research Consortium, Denver (S.M.B., E.H., T.M.M., J.S.R., P.M.H.); Malcom Randall VA Medical Center, Gainesville, FL (D.E.W.); University of Florida College of Medicine, Gainesville (D.E.W.); VA Ann Arbor Healthcare System, Ann Arbor, MI (J.B.S.); and Temple University School of Medicine, Philadelphia, PA (V.A.)
| | - Jeremy B Sussman
- From the VA Eastern Colorado Health Care System, Denver (S.M.B., E.H., T.M.M., A.E.B., J.S.R., P.M.H.); University of Colorado School of Medicine, Aurora (S.M.B., T.M.M., A.E.B., J.S.R., P.M.H.); Colorado Cardiovascular Outcomes Research Consortium, Denver (S.M.B., E.H., T.M.M., J.S.R., P.M.H.); Malcom Randall VA Medical Center, Gainesville, FL (D.E.W.); University of Florida College of Medicine, Gainesville (D.E.W.); VA Ann Arbor Healthcare System, Ann Arbor, MI (J.B.S.); and Temple University School of Medicine, Philadelphia, PA (V.A.)
| | - Vikas Aggarwal
- From the VA Eastern Colorado Health Care System, Denver (S.M.B., E.H., T.M.M., A.E.B., J.S.R., P.M.H.); University of Colorado School of Medicine, Aurora (S.M.B., T.M.M., A.E.B., J.S.R., P.M.H.); Colorado Cardiovascular Outcomes Research Consortium, Denver (S.M.B., E.H., T.M.M., J.S.R., P.M.H.); Malcom Randall VA Medical Center, Gainesville, FL (D.E.W.); University of Florida College of Medicine, Gainesville (D.E.W.); VA Ann Arbor Healthcare System, Ann Arbor, MI (J.B.S.); and Temple University School of Medicine, Philadelphia, PA (V.A.)
| | - Thomas M Maddox
- From the VA Eastern Colorado Health Care System, Denver (S.M.B., E.H., T.M.M., A.E.B., J.S.R., P.M.H.); University of Colorado School of Medicine, Aurora (S.M.B., T.M.M., A.E.B., J.S.R., P.M.H.); Colorado Cardiovascular Outcomes Research Consortium, Denver (S.M.B., E.H., T.M.M., J.S.R., P.M.H.); Malcom Randall VA Medical Center, Gainesville, FL (D.E.W.); University of Florida College of Medicine, Gainesville (D.E.W.); VA Ann Arbor Healthcare System, Ann Arbor, MI (J.B.S.); and Temple University School of Medicine, Philadelphia, PA (V.A.)
| | - Anna E Barón
- From the VA Eastern Colorado Health Care System, Denver (S.M.B., E.H., T.M.M., A.E.B., J.S.R., P.M.H.); University of Colorado School of Medicine, Aurora (S.M.B., T.M.M., A.E.B., J.S.R., P.M.H.); Colorado Cardiovascular Outcomes Research Consortium, Denver (S.M.B., E.H., T.M.M., J.S.R., P.M.H.); Malcom Randall VA Medical Center, Gainesville, FL (D.E.W.); University of Florida College of Medicine, Gainesville (D.E.W.); VA Ann Arbor Healthcare System, Ann Arbor, MI (J.B.S.); and Temple University School of Medicine, Philadelphia, PA (V.A.)
| | - John S Rumsfeld
- From the VA Eastern Colorado Health Care System, Denver (S.M.B., E.H., T.M.M., A.E.B., J.S.R., P.M.H.); University of Colorado School of Medicine, Aurora (S.M.B., T.M.M., A.E.B., J.S.R., P.M.H.); Colorado Cardiovascular Outcomes Research Consortium, Denver (S.M.B., E.H., T.M.M., J.S.R., P.M.H.); Malcom Randall VA Medical Center, Gainesville, FL (D.E.W.); University of Florida College of Medicine, Gainesville (D.E.W.); VA Ann Arbor Healthcare System, Ann Arbor, MI (J.B.S.); and Temple University School of Medicine, Philadelphia, PA (V.A.)
| | - P Michael Ho
- From the VA Eastern Colorado Health Care System, Denver (S.M.B., E.H., T.M.M., A.E.B., J.S.R., P.M.H.); University of Colorado School of Medicine, Aurora (S.M.B., T.M.M., A.E.B., J.S.R., P.M.H.); Colorado Cardiovascular Outcomes Research Consortium, Denver (S.M.B., E.H., T.M.M., J.S.R., P.M.H.); Malcom Randall VA Medical Center, Gainesville, FL (D.E.W.); University of Florida College of Medicine, Gainesville (D.E.W.); VA Ann Arbor Healthcare System, Ann Arbor, MI (J.B.S.); and Temple University School of Medicine, Philadelphia, PA (V.A.)
| |
Collapse
|
35
|
Facility-Level Variation in Hospitalization, Mortality, and Costs in the 30 Days After Percutaneous Coronary Intervention. Circulation 2015; 132:101-8. [DOI: 10.1161/circulationaha.115.015351] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/06/2015] [Accepted: 05/01/2015] [Indexed: 11/16/2022]
Abstract
Background—
Policies to reduce unnecessary hospitalizations after percutaneous coronary intervention (PCI) are intended to improve healthcare value by reducing costs while maintaining patient outcomes. Whether facility-level hospitalization rates after PCI are associated with cost of care is unknown.
Methods and Results—
We studied 32 080 patients who received PCI at any 1 of 62 Veterans Affairs hospitals from 2008 to 2011. We identified facility outliers for 30-day risk-standardized hospitalization, mortality, and cost. Compared with the risk-standardized average, 2 hospitals (3.2%) had a lower-than-expected hospitalization rate, and 2 hospitals (3.2%) had a higher-than-expected hospitalization rate. We observed no statistically significant variation in facility-level risk-standardized mortality. The facility-level unadjusted median per patient 30-day total cost was $23 820 (interquartile range, $19 604–$29 958). Compared with the risk-standardized average, 17 hospitals (27.4%) had lower-than-expected costs, and 14 hospitals (22.6%) had higher-than-expected costs. At the facility level, the index PCI accounted for 83.1% of the total cost (range, 60.3%–92.2%), whereas hospitalization after PCI accounted for only 5.8% (range, 2.0%–12.7%) of the 30-day total cost. Facilities with higher hospitalization rates were not more expensive (Spearman ρ=0.16; 95% confidence interval, −0.09 to 0.39;
P
=0.21).
Conclusions—
In this national study, hospitalizations in the 30 day after PCI accounted for only 5.8% of 30-day cost, and facility-level cost was not correlated with hospitalization rates. This challenges the focus on reducing hospitalizations after PCI as an effective means of improving healthcare value. Opportunities remain to improve PCI value by reducing the variation in total cost of PCI without compromising patient outcomes.
Collapse
|
36
|
MacKenzie TA, Grunkemeier GL, Grunwald GK, O'Malley AJ, Bohn C, Wu Y, Malenka DJ. A primer on using shrinkage to compare in-hospital mortality between centers. Ann Thorac Surg 2015; 99:757-61. [PMID: 25742812 DOI: 10.1016/j.athoracsur.2014.11.039] [Citation(s) in RCA: 31] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/30/2014] [Revised: 11/10/2014] [Accepted: 11/17/2014] [Indexed: 12/01/2022]
Abstract
Outcomes of cardiothoracic surgery are usually compared among hospitals or physicians by reporting the frequency of in-hospital mortality. Although there is agreement that these frequencies should be adjusted for case mix, there remains uncertainty about the value of using a statistical model that represents hospitals as random effects as opposed to the conventional approach of fixed effects. For years, the Northern New England Cardiovascular Disease Study Group has compared in-hospital mortality after coronary artery bypass graft surgery among centers using a fixed effects approach. An alternative method using random effects has become increasingly popular, and is the method used by cardiothoracic surgery registries such as the Massachusetts Data Analysis Center. The purpose of this report is to provide a short background on fixed versus random effects modeling, describe the use of shrinkage estimators including empirical Bayes, and illustrate them using data from the Northern New England Cardiovascular Disease Study Group. We conclude that both are acceptable approaches to hospital profiling if done in combination with appropriate risk adjustment.
Collapse
Affiliation(s)
- Todd A MacKenzie
- Department of Biomedical Data Science, Dartmouth College, Geisel School of Medicine, Hanover, New Hampshire.
| | | | - Gary K Grunwald
- Department of Biostatistics and Bioinformatics, University of Colorado at Denver, Denver, Colorado
| | - A James O'Malley
- Department of Biomedical Data Science, Dartmouth College, Geisel School of Medicine, Hanover, New Hampshire
| | - Chad Bohn
- Department of Medicine, Dartmouth College, Geisel School of Medicine, Hanover, New Hampshire
| | | | - David J Malenka
- Department of Medicine, Dartmouth College, Geisel School of Medicine, Hanover, New Hampshire
| |
Collapse
|
37
|
|
38
|
Fischer C, Lingsma HF, van Leersum N, Tollenaar RAEM, Wouters MW, Steyerberg EW. Comparing colon cancer outcomes: The impact of low hospital case volume and case-mix adjustment. Eur J Surg Oncol 2015; 41:1045-53. [PMID: 26067372 DOI: 10.1016/j.ejso.2015.04.009] [Citation(s) in RCA: 18] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/05/2014] [Revised: 04/01/2015] [Accepted: 04/16/2015] [Indexed: 12/19/2022] Open
Abstract
OBJECTIVE When comparing performance across hospitals it is essential to consider the noise caused by low hospital case volume and to perform adequate case-mix adjustment. We aimed to quantify the role of noise and case-mix adjustment on standardized postoperative mortality and anastomotic leakage (AL) rates. METHODS We studied 13,120 patients who underwent colon cancer resection in 85 Dutch hospitals. We addressed differences between hospitals in postoperative mortality and AL, using fixed (ignoring noise) and random effects (incorporating noise) logistic regression models with general and additional, disease specific, case-mix adjustment. RESULTS Adding disease specific variables improved the performance of the case-mix adjustment models for postoperative mortality (c-statistic increased from 0.77 to 0.81). The overall variation in standardized mortality ratios was similar, but some individual hospitals changed considerably. For the standardized AL rates the performance of the adjustment models was poor (c-statistic 0.59 and 0.60) and overall variation was small. Most of the observed variation between hospitals was actually noise. CONCLUSION Noise had a larger effect on hospital performance than extended case-mix adjustment, although some individual hospital outcome rates were affected by more detailed case-mix adjustment. To compare outcomes between hospitals it is crucial to consider noise due to low hospital case volume with a random effects model.
Collapse
Affiliation(s)
- C Fischer
- Department of Public Health, Centre for Medical Decision Making, Erasmus MC, Rotterdam, The Netherlands.
| | - H F Lingsma
- Department of Public Health, Centre for Medical Decision Making, Erasmus MC, Rotterdam, The Netherlands.
| | - N van Leersum
- Department of Surgery, Leiden University Medical Centre, Leiden, The Netherlands; Dutch Institute for Clinical Auditing, Leiden, The Netherlands.
| | - R A E M Tollenaar
- Department of Surgery, Leiden University Medical Centre, Leiden, The Netherlands; Dutch Institute for Clinical Auditing, Leiden, The Netherlands.
| | - M W Wouters
- Department of Surgical Oncology, Netherlands Cancer Institute, Amsterdam, The Netherlands; Dutch Institute for Clinical Auditing, Leiden, The Netherlands.
| | - E W Steyerberg
- Department of Public Health, Centre for Medical Decision Making, Erasmus MC, Rotterdam, The Netherlands.
| |
Collapse
|
39
|
Yang H, Zhang C, Zhou Q, Wang Y, Chen L. Clinical outcomes with alternative dosing strategies for piperacillin/tazobactam: a systematic review and meta-analysis. PLoS One 2015; 10:e0116769. [PMID: 25575030 PMCID: PMC4289069 DOI: 10.1371/journal.pone.0116769] [Citation(s) in RCA: 41] [Impact Index Per Article: 4.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2014] [Accepted: 12/12/2014] [Indexed: 12/03/2022] Open
Abstract
Objectives A better dosing strategy can improve clinical outcomes for patients. We sought to compare the extended or continuous infusion with conventional intermittent infusion of piperacillin/tazobactam, investigating which approach is better and worthy of recommendation for clinical use. Methods Articles were gathered from PubMed, Web of Science, ProQuest, Science Direct, Cochrane, two Chinese literature databases (CNKI, Wan Fang Data) and related ICAAC and ACCP conferences. Randomized controlled and observational studies that compared extended or continuous infusion with conventional intermittent infusion of piperacillin/tazobactam were identified from the databases above and analyzed. Two reviewers independently extracted and investigated the data. A meta-analysis was performed using Revman 5.2 software. The quality of each study was assessed. Sensitivity analysis and publication bias were evaluated. Results Five randomized controlled trials and nine observational studies were included in this study. All included studies had high quality and no publication bias was found. Compared to the conventional intermittent infusion approach, the extended or continuous infusion group had a significantly higher clinical cure rate (OR 1.88, 95% CI 1.29-2.73, P = 0.0009) and a lower mortality rate (OR 0.67, 95% CI 0.50-0.89, P = 0.005). No statistical difference was observed for bacteriologic cure (OR 1.40, 95% CI 0.82-2.37, P = 0.22) between the two dosing regimens. The sensitivity analysis showed the results were stable. Conclusions Our systematic review and meta-analysis suggested that the extended or continuous infusion strategy of piperacillin/tazobactam should be recommended for clinical use considering its higher clinical cure rate and lower mortality rate in comparison with conventional intermittent strategy. Data from this study could be extrapolated for other β-lactam antimicrobials. Therefore, this dosing strategy could be considered in clinical practice.
Collapse
Affiliation(s)
- Hui Yang
- Department of pharmacy, Peking University Third Hospital, Beijing, China
- School of pharmaceutical sciences, Peking University, Beijing, China
| | - Chao Zhang
- Department of pharmacy, Peking University Third Hospital, Beijing, China
- * E-mail:
| | - Quanyu Zhou
- Department of pharmacy, The First People’s Hospital of Lianyungang, Jiangsu, China
| | - Yike Wang
- School of pharmaceutical sciences, Peking University, Beijing, China
| | - Lujia Chen
- School of pharmaceutical sciences, Peking University, Beijing, China
| |
Collapse
|
40
|
Varagunam M, Hutchings A, Black N. Do patient-reported outcomes offer a more sensitive method for comparing the outcomes of consultants than mortality? A multilevel analysis of routine data. BMJ Qual Saf 2014; 24:195-202. [DOI: 10.1136/bmjqs-2014-003551] [Citation(s) in RCA: 23] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
|
41
|
Bottle A, Gaudoin R, Goudie R, Jones S, Aylin P. Can valid and practical risk-prediction or casemix adjustment models, including adjustment for comorbidity, be generated from English hospital administrative data (Hospital Episode Statistics)? A national observational study. HEALTH SERVICES AND DELIVERY RESEARCH 2014. [DOI: 10.3310/hsdr02400] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/02/2023] Open
Abstract
BackgroundNHS hospitals collect a wealth of administrative data covering accident and emergency (A&E) department attendances, inpatient and day case activity, and outpatient appointments. Such data are increasingly being used to compare units and services, but adjusting for risk is difficult.ObjectivesTo derive robust risk-adjustment models for various patient groups, including those admitted for heart failure (HF), acute myocardial infarction, colorectal and orthopaedic surgery, and outcomes adjusting for available patient factors such as comorbidity, using England’s Hospital Episode Statistics (HES) data. To assess if more sophisticated statistical methods based on machine learning such as artificial neural networks (ANNs) outperform traditional logistic regression (LR) for risk prediction. To update and assess for the NHS the Charlson index for comorbidity. To assess the usefulness of outpatient data for these models.Main outcome measuresMortality, readmission, return to theatre, outpatient non-attendance. For HF patients we considered various readmission measures such as diagnosis-specific and total within a year.MethodsWe systematically reviewed studies comparing two or more comorbidity indices. Logistic regression, ANNs, support vector machines and random forests were compared for mortality and readmission. Models were assessed using discrimination and calibration statistics. Competing risks proportional hazards regression and various count models were used for future admissions and bed-days.ResultsOur systematic review and empirical analysis suggested that for general purposes comorbidity is currently best described by the set of 30 Elixhauser comorbidities plus dementia. Model discrimination was often high for mortality and poor, or at best moderate, for other outcomes, for examplec = 0.62 for readmission andc = 0.73 for death following stroke. Calibration was often good for procedure groups but poorer for diagnosis groups, with overprediction of low risk a common cause. The machine learning methods we investigated offered little beyond LR for their greater complexity and implementation difficulties. For HF, some patient-level predictors differed by primary diagnosis of readmission but not by length of follow-up. Prior non-attendance at outpatient appointments was a useful, strong predictor of readmission. Hospital-level readmission rates for HF did not correlate with readmission rates for non-HF; hospital performance on national audit process measures largely correlated only with HF readmission rates.ConclusionsMany practical risk-prediction or casemix adjustment models can be generated from HES data using LR, though an extra step is often required for accurate calibration. Including outpatient data in readmission models is useful. The three machine learning methods we assessed added little with these data. Readmission rates for HF patients should be divided by diagnosis on readmission when used for quality improvement.Future workAs HES data continue to develop and improve in scope and accuracy, they can be used more, for instance A&E records. The return to theatre metric appears promising and could be extended to other index procedures and specialties. While our data did not warrant the testing of a larger number of machine learning methods, databases augmented with physiological and pathology information, for example, might benefit from methods such as boosted trees. Finally, one could apply the HF readmissions analysis to other chronic conditions.FundingThe National Institute for Health Research Health Services and Delivery Research programme.
Collapse
Affiliation(s)
- Alex Bottle
- Dr Foster Unit at Imperial, Department of Primary Care and Public Health, Imperial College London, London, UK
| | - Rene Gaudoin
- Dr Foster Unit at Imperial, Department of Primary Care and Public Health, Imperial College London, London, UK
| | - Rosalind Goudie
- Dr Foster Unit at Imperial, Department of Primary Care and Public Health, Imperial College London, London, UK
| | - Simon Jones
- Department of Health Care Management and Policy, University of Surrey, Surrey, UK
| | - Paul Aylin
- Dr Foster Unit at Imperial, Department of Primary Care and Public Health, Imperial College London, London, UK
| |
Collapse
|
42
|
Hubbard RA, Benjamin-Johnson R, Onega T, Smith-Bindman R, Zhu W, Fenton JJ. Classification accuracy of claims-based methods for identifying providers failing to meet performance targets. Stat Med 2014; 34:93-105. [PMID: 25302935 DOI: 10.1002/sim.6318] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/21/2013] [Revised: 08/15/2014] [Accepted: 09/14/2014] [Indexed: 11/09/2022]
Abstract
Quality assessment is critical for healthcare reform, but data sources are lacking for measurement of many important healthcare outcomes. With over 49 million people covered by Medicare as of 2010, Medicare claims data offer a potentially valuable source that could be used in targeted health care quality improvement efforts. However, little is known about the operating characteristics of provider profiling methods using claims-based outcome measures that may estimate provider performance with error. Motivated by the example of screening mammography performance, we compared approaches to identifying providers failing to meet guideline targets using Medicare claims data. We used data from the Breast Cancer Surveillance Consortium and linked Medicare claims to compare claims-based and clinical estimates of cancer detection rate. We then demonstrated the performance of claim-based estimates across a broad range of operating characteristics using simulation studies. We found that identification of poor performing providers was extremely sensitive to algorithm specificity, with no approach identifying more than 65% of poor performing providers when claims-based measures had specificity of 0.995 or less. We conclude that claims have the potential to contribute important information on healthcare outcomes to quality improvement efforts. However, to achieve this potential, development of highly accurate claims-based outcome measures should remain a priority.
Collapse
Affiliation(s)
- Rebecca A Hubbard
- Group Health Research Institute, Seattle, WA, U.S.A.; Department of Biostatistics, University of Washington, Seattle, WA, U.S.A
| | | | | | | | | | | |
Collapse
|
43
|
Moran JL, Solomon PJ. Fixed effects modelling for provider mortality outcomes: Analysis of the Australia and New Zealand Intensive Care Society (ANZICS) Adult Patient Data-base. PLoS One 2014; 9:e102297. [PMID: 25029164 PMCID: PMC4100889 DOI: 10.1371/journal.pone.0102297] [Citation(s) in RCA: 29] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2014] [Accepted: 06/17/2014] [Indexed: 11/19/2022] Open
Abstract
BACKGROUND Risk adjusted mortality for intensive care units (ICU) is usually estimated via logistic regression. Random effects (RE) or hierarchical models have been advocated to estimate provider risk-adjusted mortality on the basis that standard estimators increase false outlier classification. The utility of fixed effects (FE) estimators (separate ICU-specific intercepts) has not been fully explored. METHODS Using a cohort from the Australian and New Zealand Intensive Care Society Adult Patient Database, 2009-2010, the model fit of different logistic estimators (FE, random-intercept and random-coefficient) was characterised: Bayesian Information Criterion (BIC; lower values better), receiver-operator characteristic curve area (AUC) and Hosmer-Lemeshow (H-L) statistic. ICU standardised hospital mortality ratios (SMR) and 95%CI were compared between models. ICU site performance (FE), relative to the grand observation-weighted mean (GO-WM) on odds ratio (OR), risk ratio (RR) and probability scales were assessed using model-based average marginal effects (AME). RESULTS The data set consisted of 145355 patients in 128 ICUs, years 2009 (47.5%) & 2010 (52.5%), with mean(SD) age 60.9(18.8) years, 56% male and ICU and hospital mortalities of 7.0% and 10.9% respectively. The FE model had a BIC = 64058, AUC = 0.90 and an H-L statistic P-value = 0.22. The best-fitting random-intercept model had a BIC = 64457, AUC = 0.90 and H-L statistic P-value = 0.32 and random-coefficient model, BIC = 64556, AUC = 0.90 and H-L statistic P-value = 0.28. Across ICUs and over years no outliers (SMR 95% CI excluding null-value = 1) were identified and no model difference in SMR spread or 95%CI span was demonstrated. Using AME (OR and RR scale), ICU site-specific estimates diverged from the GO-WM, and the effect spread decreased over calendar years. On the probability scale, a majority of ICUs demonstrated calendar year decrease, but in the for-profit sector, this trend was reversed. CONCLUSIONS The FE estimator had model advantage compared with conventional RE models. Using AME, between and over-year ICU site-effects were easily characterised.
Collapse
Affiliation(s)
- John L. Moran
- Department of Intensive Care Medicine, The Queen Elizabeth Hospital, Woodville, South Australia, Australia
- * E-mail:
| | - Patricia J. Solomon
- School of Mathematical Sciences, University of Adelaide, Adelaide, South Australia, Australia
| | | |
Collapse
|
44
|
Ohl ME, Richardson KK, Goto M, Vaughan-Sarrazin M, Schweizer ML, Perencevich EN. HIV quality report cards: impact of case-mix adjustment and statistical methods. Clin Infect Dis 2014; 59:1160-7. [PMID: 25034427 DOI: 10.1093/cid/ciu551] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Abstract
BACKGROUND There will be increasing pressure to publicly report and rank the performance of healthcare systems on human immunodeficiency virus (HIV) quality measures. To inform discussion of public reporting, we evaluated the influence of case-mix adjustment when ranking individual care systems on the viral control quality measure. METHODS We used data from the Veterans Health Administration (VHA) HIV Clinical Case Registry and administrative databases to estimate case-mix adjusted viral control for 91 local systems caring for 12 368 patients. We compared results using 2 adjustment methods, the observed-to-expected estimator and the risk-standardized ratio. RESULTS Overall, 10 913 patients (88.2%) achieved viral control (viral load ≤400 copies/mL). Prior to case-mix adjustment, system-level viral control ranged from 51% to 100%. Seventeen (19%) systems were labeled as low outliers (performance significantly below the overall mean) and 11 (12%) as high outliers. Adjustment for case mix (patient demographics, comorbidity, CD4 nadir, time on therapy, and income from VHA administrative databases) reduced the number of low outliers by approximately one-third, but results differed by method. The adjustment model had moderate discrimination (c statistic = 0.66), suggesting potential for unadjusted risk when using administrative data to measure case mix. CONCLUSIONS Case-mix adjustment affects rankings of care systems on the viral control quality measure. Given the sensitivity of rankings to selection of case-mix adjustment methods-and potential for unadjusted risk when using variables limited to current administrative databases-the HIV care community should explore optimal methods for case-mix adjustment before moving forward with public reporting.
Collapse
Affiliation(s)
- Michael E Ohl
- Center for Comprehensive Access and Delivery Research and Evaluation, Iowa City Veterans Affairs Medical Center Department of Internal Medicine, University of Iowa Carver College of Medicine, Iowa City
| | - Kelly K Richardson
- Center for Comprehensive Access and Delivery Research and Evaluation, Iowa City Veterans Affairs Medical Center Department of Internal Medicine, University of Iowa Carver College of Medicine, Iowa City
| | - Michihiko Goto
- Center for Comprehensive Access and Delivery Research and Evaluation, Iowa City Veterans Affairs Medical Center Department of Internal Medicine, University of Iowa Carver College of Medicine, Iowa City
| | - Mary Vaughan-Sarrazin
- Center for Comprehensive Access and Delivery Research and Evaluation, Iowa City Veterans Affairs Medical Center Department of Internal Medicine, University of Iowa Carver College of Medicine, Iowa City
| | - Marin L Schweizer
- Center for Comprehensive Access and Delivery Research and Evaluation, Iowa City Veterans Affairs Medical Center Department of Internal Medicine, University of Iowa Carver College of Medicine, Iowa City
| | - Eli N Perencevich
- Center for Comprehensive Access and Delivery Research and Evaluation, Iowa City Veterans Affairs Medical Center Department of Internal Medicine, University of Iowa Carver College of Medicine, Iowa City
| |
Collapse
|
45
|
Varewyck M, Goetghebeur E, Eriksson M, Vansteelandt S. On shrinkage and model extrapolation in the evaluation of clinical center performance. Biostatistics 2014; 15:651-64. [PMID: 24812420 PMCID: PMC4173104 DOI: 10.1093/biostatistics/kxu019] [Citation(s) in RCA: 21] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/27/2023] Open
Abstract
We consider statistical methods for benchmarking clinical centers based on a dichotomous outcome indicator. Borrowing ideas from the causal inference literature, we aim to reveal how the entire study population would have fared under the current care level of each center. To this end, we evaluate direct standardization based on fixed versus random center effects outcome models that incorporate patient-specific baseline covariates to adjust for differential case-mix. We explore fixed effects (FE) regression with Firth correction and normal mixed effects (ME) regression to maintain convergence in the presence of very small centers. Moreover, we study doubly robust FE regression to avoid outcome model extrapolation. Simulation studies show that shrinkage following standard ME modeling can result in substantial power loss relative to the considered alternatives, especially for small centers. Results are consistent with findings in the analysis of 30-day mortality risk following acute stroke across 90 centers in the Swedish Stroke Register.
Collapse
Affiliation(s)
- Machteld Varewyck
- Department of Applied Mathematics, Computer Science and Statistics, Ghent University, 9000 Ghent, Belgium
| | - Els Goetghebeur
- Department of Applied Mathematics, Computer Science and Statistics, Ghent University, 9000 Ghent, Belgium
| | - Marie Eriksson
- Department of Statistics, Umeå University, 901 87 Umeå, Sweden
| | - Stijn Vansteelandt
- Department of Applied Mathematics, Computer Science and Statistics, Ghent University, 9000 Ghent, Belgium
| |
Collapse
|
46
|
Classifying hospitals as mortality outliers: logistic versus hierarchical logistic models. J Med Syst 2014; 38:29. [PMID: 24711175 DOI: 10.1007/s10916-014-0029-x] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2013] [Accepted: 03/10/2014] [Indexed: 10/25/2022]
Abstract
The use of hierarchical logistic regression for provider profiling has been recommended due to the clustering of patients within hospitals, but has some associated difficulties. We assess changes in hospital outlier status based on standard logistic versus hierarchical logistic modelling of mortality. The study population consisted of all patients admitted to acute, non-specialist hospitals in England between 2007 and 2011 with a primary diagnosis of acute myocardial infarction, acute cerebrovascular disease or fracture of neck of femur or a primary procedure of coronary artery bypass graft or repair of abdominal aortic aneurysm. We compared standardised mortality ratios (SMRs) from non-hierarchical models with SMRs from hierarchical models, without and with shrinkage estimates of the predicted probabilities (Model 1 and Model 2). The SMRs from standard logistic and hierarchical models were highly statistically significantly correlated (r > 0.91, p = 0.01). More outliers were recorded in the standard logistic regression than hierarchical modelling only when using shrinkage estimates (Model 2): 21 hospitals (out of a cumulative number of 565 pairs of hospitals under study) changed from a low outlier and 8 hospitals changed from a high outlier based on the logistic regression to a not-an-outlier based on shrinkage estimates. Both standard logistic and hierarchical modelling have identified nearly the same hospitals as mortality outliers. The choice of methodological approach should, however, also consider whether the modelling aim is judgment or improvement, as shrinkage may be more appropriate for the former than the latter.
Collapse
|
47
|
Austin PC, Reeves MJ. Effect of Provider Volume on the Accuracy of Hospital Report Cards. Circ Cardiovasc Qual Outcomes 2014; 7:299-305. [DOI: 10.1161/circoutcomes.113.000685] [Citation(s) in RCA: 16] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Affiliation(s)
- Peter C. Austin
- From the Institute for Clinical Evaluative Sciences, Toronto, Ontario, Canada (P.C.A.); Institute of Health Management, Policy and Evaluation, University of Toronto (P.C.A.); Schulich Heart Research Program, Sunnybrook Research Institute, Toronto, Canada (P.C.A.); and Department of Epidemiology and Biostatistics, Michigan State University, East Lansing, MI (M.J.R.)
| | - Mathew J. Reeves
- From the Institute for Clinical Evaluative Sciences, Toronto, Ontario, Canada (P.C.A.); Institute of Health Management, Policy and Evaluation, University of Toronto (P.C.A.); Schulich Heart Research Program, Sunnybrook Research Institute, Toronto, Canada (P.C.A.); and Department of Epidemiology and Biostatistics, Michigan State University, East Lansing, MI (M.J.R.)
| |
Collapse
|
48
|
Yu J, Liu L, Collins RL, Vincent PC, Epstein LH. Analytical Problems and Suggestions in the Analysis of Behavioral Economic Demand Curves. MULTIVARIATE BEHAVIORAL RESEARCH 2014; 49:178-192. [PMID: 26741176 DOI: 10.1080/00273171.2013.862491] [Citation(s) in RCA: 33] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/05/2023]
Abstract
Behavioral economic demand curves (Hursh, Raslear, Shurtleff, Bauman, & Simmons, 1988) are innovative approaches to characterize the relationships between consumption of a substance and its price. In this article, we investigate common analytical issues in the use of behavioral economic demand curves, which can cause inconsistent interpretations of demand curves, and then we provide methodological suggestions to address those analytical issues. We first demonstrate that log transformation with different added values for handling zeros changes model parameter estimates dramatically. Second, demand curves are often analyzed using an overparameterized model that results in an inefficient use of the available data and a lack of assessment of the variability among individuals. To address these issues, we apply a nonlinear mixed effects model based on multivariate error structures that has not been used previously to analyze behavioral economic demand curves in the literature. We also propose analytical formulas for the relevant standard errors of derived values such as P max, O max, and elasticity. The proposed model stabilizes the derived values regardless of using different added increments and provides substantially smaller standard errors. We illustrate the data analysis procedure using data from a relative reinforcement efficacy study of simulated marijuana purchasing.
Collapse
Affiliation(s)
- Jihnhee Yu
- a Department of Biostatistics , University at Buffalo, State University of New York
| | - Liu Liu
- a Department of Biostatistics , University at Buffalo, State University of New York
| | - R Lorraine Collins
- b Department of Community Health and Health Behavior , University at Buffalo, State University of New York
| | - Paula C Vincent
- b Department of Community Health and Health Behavior , University at Buffalo, State University of New York
| | - Leonard H Epstein
- c Departments of Pediatrics and Social and Community Health and Health Behavior , University at Buffalo, State University of New York
| |
Collapse
|
49
|
Katzan IL, Spertus J, Bettger JP, Bravata DM, Reeves MJ, Smith EE, Bushnell C, Higashida RT, Hinchey JA, Holloway RG, Howard G, King RB, Krumholz HM, Lutz BJ, Yeh RW. Risk adjustment of ischemic stroke outcomes for comparing hospital performance: a statement for healthcare professionals from the American Heart Association/American Stroke Association. Stroke 2014; 45:918-44. [PMID: 24457296 DOI: 10.1161/01.str.0000441948.35804.77] [Citation(s) in RCA: 84] [Impact Index Per Article: 8.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/22/2022]
Abstract
BACKGROUND AND PURPOSE Stroke is the fourth-leading cause of death and a leading cause of long-term major disability in the United States. Measuring outcomes after stroke has important policy implications. The primary goals of this consensus statement are to (1) review statistical considerations when evaluating models that define hospital performance in providing stroke care; (2) discuss the benefits, limitations, and potential unintended consequences of using various outcome measures when evaluating the quality of ischemic stroke care at the hospital level; (3) summarize the evidence on the role of specific clinical and administrative variables, including patient preferences, in risk-adjusted models of ischemic stroke outcomes; (4) provide recommendations on the minimum list of variables that should be included in risk adjustment of ischemic stroke outcomes for comparisons of quality at the hospital level; and (5) provide recommendations for further research. METHODS AND RESULTS This statement gives an overview of statistical considerations for the evaluation of hospital-level outcomes after stroke and provides a systematic review of the literature for the following outcome measures for ischemic stroke at 30 days: functional outcomes, mortality, and readmissions. Data on outcomes after stroke have primarily involved studies conducted at an individual patient level rather than a hospital level. On the basis of the available information, the following factors should be included in all hospital-level risk-adjustment models: age, sex, stroke severity, comorbid conditions, and vascular risk factors. Because stroke severity is the most important prognostic factor for individual patients and appears to be a significant predictor of hospital-level performance for 30-day mortality, inclusion of a stroke severity measure in risk-adjustment models for 30-day outcome measures is recommended. Risk-adjustment models that do not include stroke severity or other recommended variables must provide comparable classification of hospital performance as models that include these variables. Stroke severity and other variables that are included in risk-adjustment models should be standardized across sites, so that their reliability and accuracy are equivalent. There is a pressing need for research in multiple areas to better identify methods and metrics to evaluate outcomes of stroke care. CONCLUSIONS There are a number of important methodological challenges in undertaking risk-adjusted outcome comparisons to assess the quality of stroke care in different hospitals. It is important for stakeholders to recognize these challenges and for there to be a concerted approach to improving the methods for quality assessment and improvement.
Collapse
|
50
|
Ieva F, Paganoni AM. Detecting and visualizing outliers in provider profiling via funnel plots and mixed effect models. Health Care Manag Sci 2014; 18:166-72. [PMID: 24402171 DOI: 10.1007/s10729-013-9264-9] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/28/2013] [Accepted: 12/16/2013] [Indexed: 11/26/2022]
Abstract
In this work we propose the use of a graphical diagnostic tool (the funnel plot) to detect outliers among hospitals that treat patients affected by Acute Myocardial Infarction (AMI). We consider an application to data on AMI hospitalizations recorded in the administrative databases of our regional district. The outcome of interest is the in-hospital mortality, a variable indicating if the patient has been discharged dead or alive. We then compare the results obtained by graphical diagnostic tools with those arising from fitting parametric mixed effects models to the same data.
Collapse
Affiliation(s)
- Francesca Ieva
- MOX - Modellistica e Calcolo Scientifico, Dipartimento di Matematica, Politecnico di Milano, via Bonardi 9, 20133, Milan, Italy,
| | | |
Collapse
|