1
|
Rosati D, Palmieri M, Brunelli G, Morrione A, Iannelli F, Frullanti E, Giordano A. Differential gene expression analysis pipelines and bioinformatic tools for the identification of specific biomarkers: A review. Comput Struct Biotechnol J 2024; 23:1154-1168. [PMID: 38510977 PMCID: PMC10951429 DOI: 10.1016/j.csbj.2024.02.018] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2023] [Revised: 02/20/2024] [Accepted: 02/20/2024] [Indexed: 03/22/2024] Open
Abstract
In recent years, the role of bioinformatics and computational biology together with omics techniques and transcriptomics has gained tremendous importance in biomedicine and healthcare, particularly for the identification of biomarkers for precision medicine and drug discovery. Differential gene expression (DGE) analysis is one of the most used techniques for RNA-sequencing (RNA-seq) data analysis. This tool, which is typically used in various RNA-seq data processing applications, allows the identification of differentially expressed genes across two or more sample sets. Functional enrichment analyses can then be performed to annotate and contextualize the resulting gene lists. These studies provide valuable information about disease-causing biological processes and can help in identifying molecular targets for novel therapies. This review focuses on differential gene expression (DGE) analysis pipelines and bioinformatic techniques commonly used to identify specific biomarkers and discuss the advantages and disadvantages of these techniques.
Collapse
Affiliation(s)
- Diletta Rosati
- Department of Medical Biotechnologies, University of Siena, 53100 Siena, Italy
- Cancer Genomics & Systems Biology Lab, Dept. of Medical Biotechnologies, University of Siena, 53100 Siena, Italy
- Med Biotech Hub and Competence Center, Department of Medical Biotechnologies, University of Siena, Italy
| | - Maria Palmieri
- Cancer Genomics & Systems Biology Lab, Dept. of Medical Biotechnologies, University of Siena, 53100 Siena, Italy
- Med Biotech Hub and Competence Center, Department of Medical Biotechnologies, University of Siena, Italy
| | - Giulia Brunelli
- Med Biotech Hub and Competence Center, Department of Medical Biotechnologies, University of Siena, Italy
| | - Andrea Morrione
- Sbarro Institute for Cancer Research and Molecular Medicine, Center for Biotechnology, Department of Biology, College of Science and Technology, Temple University, Philadelphia, PA 19122, USA
| | - Francesco Iannelli
- Laboratory of Molecular Microbiology and Biotechnology, Department of Medical Biotechnologies, University of Siena, Siena, Italy
| | - Elisa Frullanti
- Cancer Genomics & Systems Biology Lab, Dept. of Medical Biotechnologies, University of Siena, 53100 Siena, Italy
- Med Biotech Hub and Competence Center, Department of Medical Biotechnologies, University of Siena, Italy
| | - Antonio Giordano
- Department of Medical Biotechnologies, University of Siena, 53100 Siena, Italy
- Sbarro Institute for Cancer Research and Molecular Medicine, Center for Biotechnology, Department of Biology, College of Science and Technology, Temple University, Philadelphia, PA 19122, USA
| |
Collapse
|
2
|
Al-Sahab B, Leviton A, Loddenkemper T, Paneth N, Zhang B. Biases in Electronic Health Records Data for Generating Real-World Evidence: An Overview. JOURNAL OF HEALTHCARE INFORMATICS RESEARCH 2024; 8:121-139. [PMID: 38273982 PMCID: PMC10805748 DOI: 10.1007/s41666-023-00153-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2023] [Revised: 09/05/2023] [Accepted: 11/07/2023] [Indexed: 01/27/2024]
Abstract
Electronic Health Records (EHR) are increasingly being perceived as a unique source of data for clinical research as they provide unprecedentedly large volumes of real-time data from real-world settings. In this review of the secondary uses of EHR, we identify the anticipated breadth of opportunities, pointing out the data deficiencies and potential biases that are likely to limit the search for true causal relationships. This paper provides a comprehensive overview of the types of biases that arise along the pathways that generate real-world evidence and the sources of these biases. We distinguish between two levels in the production of EHR data where biases are likely to arise: (i) at the healthcare system level, where the principal source of bias resides in access to, and provision of, medical care, and in the acquisition and documentation of medical and administrative data; and (ii) at the research level, where biases arise from the processes of extracting, analyzing, and interpreting these data. Due to the plethora of biases, mainly in the form of selection and information bias, we conclude with advising extreme caution about making causal inferences based on secondary uses of EHRs.
Collapse
Affiliation(s)
- Ban Al-Sahab
- Department of Family Medicine, College of Human Medicine, Michigan State University, B100 Clinical Center, 788 Service Road, East Lansing, MI USA
| | - Alan Leviton
- Department of Neurology, Harvard Medical School, Boston, MA USA
- Department of Neurology, Boston Children’s Hospital, Boston, MA USA
| | - Tobias Loddenkemper
- Department of Neurology, Harvard Medical School, Boston, MA USA
- Department of Neurology, Boston Children’s Hospital, Boston, MA USA
| | - Nigel Paneth
- Department of Epidemiology and Biostatistics, College of Human Medicine, Michigan State University, East Lansing, MI USA
- Department of Pediatrics and Human Development, College of Human Medicine, Michigan State University, East Lansing, MI USA
| | - Bo Zhang
- Department of Neurology, Boston Children’s Hospital, Boston, MA USA
- Biostatistics and Research Design, Institutional Centers of Clinical and Translational Research, Boston Children’s Hospital, Boston, MA USA
- Harvard Medical School, Boston, MA USA
| |
Collapse
|
3
|
Kollmann NP, Langenberger B, Busse R, Pross C. Stability of hospital quality indicators over time: A multi-year observational study of German hospital data. PLoS One 2023; 18:e0293723. [PMID: 37934753 PMCID: PMC10629650 DOI: 10.1371/journal.pone.0293723] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2022] [Accepted: 10/19/2023] [Indexed: 11/09/2023] Open
Abstract
BACKGROUND Retrospective hospital quality indicators can only be useful if they are trustworthy signals of current or future quality. Despite extensive longitudinal quality indicator data and many hospital quality public reporting initiatives, research on quality indicator stability over time is scarce and skepticism about their usefulness widespread. OBJECTIVE Based on aggregated, widely available hospital-level quality indicators, this paper sought to determine whether quality indicators are stable over time. Implications for health policy were drawn and the limited methodological foundation for stability assessments of hospital-level quality indicators enhanced. METHODS Two longitudinal datasets (self-reported and routine data), including all hospitals in Germany and covering the period from 2004 to 2017, were analysed. A logistic regression using Generalized Estimating Equations, a time-dependent, graphic quintile representation of risk-adjusted rates and Spearman's rank correlation coefficient were used. RESULTS For a total of eight German quality indicators significant stability over time was demonstrated. The probability of remaining in the best quality cluster in the future across all hospitals reached from 46.9% (CI: 42.4-51.6%) for hip replacement reoperations to 80.4% (CI: 76.4-83.8%) for decubitus. Furthermore, graphical descriptive analysis showed that the difference in adverse event rates for the 20% top performing compared to the 20% worst performing hospitals in the two following years is on average between 30% for stroke and AMI and 79% for decubitus. Stability over time has been shown to vary strongly between indicators and treatment areas. CONCLUSION Quality indicators were found to have sufficient stability over time for public reporting. Potentially, increasing case volumes per hospital, centralisation of medical services and minimum-quantity regulations may lead to more stable and reliable quality of care indicators. Finally, more robust policy interventions such as outcome-based payment, should only be applied to outcome indicators with a higher level of stability over time. This should be subject to future research.
Collapse
Affiliation(s)
| | - Benedikt Langenberger
- Department of Health Care Management, Berlin University of Technology, Berlin, Germany
| | - Reinhard Busse
- Department of Health Care Management, Berlin University of Technology, Berlin, Germany
| | - Christoph Pross
- Department of Health Care Management, Berlin University of Technology, Berlin, Germany
| |
Collapse
|
4
|
Chu WM, Kuo WY, Tung YC. Effects of different palliative care models on decedents with kidney failure receiving maintenance dialysis: a nationwide population-based retrospective observational study in Taiwan. BMJ Open 2023; 13:e069835. [PMID: 37429693 DOI: 10.1136/bmjopen-2022-069835] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 07/12/2023] Open
Abstract
OBJECTIVES Patients with kidney failure receiving maintenance dialysis are a particularly important population and carry a heavy disease burden. However, evidence related to palliative care for patients with kidney failure receiving maintenance dialysis remains scarce, especially in regard to palliative care consultation services and palliative home care. This study aimed to evaluate the effects of different palliative care models on aggressive treatment among patients with kidney failure receiving maintenance dialysis during the end of life. DESIGN A population-based retrospective observational study. SETTING This study used a population database maintained by Taiwan's Ministry of Health and Welfare in combination with Taiwan's National Health Research Insurance Database. PARTICIPANTS We enrolled all decedents who were patients with kidney failure receiving maintenance dialysis from the period 1 January 2017 to 31 December 2017 in Taiwan. MAIN EXPOSURE MEASURE Hospice care during the 1-year period before death. MAIN OUTCOME MEASURES Eight aggressive treatments within 30 days before death, more than one emergency department visit, more than one admission, a longer than 14-day admission, admission to an intensive care unit, death in hospital, endotracheal tube use, ventilator use and need for cardiopulmonary resuscitation. RESULTS A total of 10 083 patients were enrolled, including 1786 (17.7%) patients with kidney failure who received palliative care 1 year before death. Compared with patients without palliative care, patients with palliative care had significantly less aggressive treatments within 30 days before death (Estimates: -0.09, CI: -0.10 to -0.08). Patients with inpatient palliative care, palliative home care or a mixed model experienced significantly lower treatment aggressiveness within 30 days before death. CONCLUSIONS Palliative care, particularly use of a mixed care model, inpatient palliative care and palliative home care in patients with kidney failure receiving dialysis, could all significantly reduce the aggressiveness of treatment within 30 days before death.
Collapse
Affiliation(s)
- Wei-Min Chu
- Department of Family Medicine, Taichung Veterans General Hospital, Taichung, Taiwan
- School of Medicine, National Yang Ming Chiao Tung University, Hsinchu, Taiwan
- Department of Post-Baccalaureate Medicine, College of Medicine, National Chung Hsing University, Taichung, Taiwan
- Geriatrics and Gerontology Research Center, College of Medicine, National Chung Hsing University, Taichung, Taiwan
- Department of Epidemiology on Aging, National Center for Geriatrics and Gerontology, Aichi, Japan
| | - Wen-Yu Kuo
- Institute of Health Policy and Management, National Taiwan University, Taipei, Taiwan
| | - Yu-Chi Tung
- Institute of Health Policy and Management, National Taiwan University, Taipei, Taiwan
- Population Health Research Center, National Taiwan University, Taipei, Taiwan
| |
Collapse
|
5
|
Ye S, Li D, Yu T, Caroff DA, Guy J, Poland RE, Sands KE, Septimus EJ, Huang SS, Platt R, Wang R. The impact of surgical volume on hospital ranking using the standardized infection ratio. Sci Rep 2023; 13:7624. [PMID: 37165033 PMCID: PMC10172297 DOI: 10.1038/s41598-023-33937-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2022] [Accepted: 04/21/2023] [Indexed: 05/12/2023] Open
Abstract
The Centers for Medicare and Medicaid Services require hospitals to report on quality metrics which are used to financially penalize those that perform in the lowest quartile. Surgical site infections (SSIs) are a critical component of the quality metrics that target healthcare-associated infections. However, the accuracy of such hospital profiling is highly affected by small surgical volumes which lead to a large amount of uncertainty in estimating standardized hospital-specific infection rates. Currently, hospitals with less than one expected SSI are excluded from rankings, but the effectiveness of this exclusion criterion is unknown. Tools that can quantify the classification accuracy and can determine the minimal surgical volume required for a desired level of accuracy are lacking. We investigate the effect of surgical volume on the accuracy of identifying poorly performing hospitals based on the standardized infection ratio and develop simulation-based algorithms for quantifying the classification accuracy. We apply our proposed method to data from HCA Healthcare (2014-2016) on SSIs in colon surgery patients. We estimate that for a procedure like colon surgery with an overall SSI rate of 3%, to rank hospitals in the HCA colon SSI dataset, hospitals that perform less than 200 procedures have a greater than 10% chance of being incorrectly assigned to the worst performing quartile. Minimum surgical volumes and predicted events criteria are required to make evaluating hospitals reliable, and these criteria vary by overall prevalence and between-hospital variability.
Collapse
Affiliation(s)
- Shangyuan Ye
- Department of Population Medicine, Harvard Pilgrim Health Care Institute and Harvard Medical School, Boston, MA, 02215, USA
- Biostatistics Shared Resource, Knight Cancer Institute, Oregon Health and Science University, Portland, OR, 97201, USA
| | - Daniel Li
- Department of Biostatistics, Harvard T. H. Chan School of Public Health, Boston, MA, 02215, USA
| | - Tingting Yu
- Department of Population Medicine, Harvard Pilgrim Health Care Institute and Harvard Medical School, Boston, MA, 02215, USA
| | - Daniel A Caroff
- Department of Infectious Diseases, Lahey Hospital and Medical Center, Burlington, MA, 01805, USA
| | - Jeffrey Guy
- Clinical Operations Group, HCA Healthcare, Nashville, TN, 37203, USA
| | - Russell E Poland
- Department of Population Medicine, Harvard Pilgrim Health Care Institute and Harvard Medical School, Boston, MA, 02215, USA
- Clinical Operations Group, HCA Healthcare, Nashville, TN, 37203, USA
| | - Kenneth E Sands
- Department of Population Medicine, Harvard Pilgrim Health Care Institute and Harvard Medical School, Boston, MA, 02215, USA
- Clinical Operations Group, HCA Healthcare, Nashville, TN, 37203, USA
| | - Edward J Septimus
- Department of Population Medicine, Harvard Pilgrim Health Care Institute and Harvard Medical School, Boston, MA, 02215, USA
- Texas A &M College of Medicine, Houston, TX, 77030, USA
| | - Susan S Huang
- Department of Population Medicine, Harvard Pilgrim Health Care Institute and Harvard Medical School, Boston, MA, 02215, USA
- University of California Irvine School of Medicine, Irvine, CA, 92617, USA
| | - Richard Platt
- Department of Population Medicine, Harvard Pilgrim Health Care Institute and Harvard Medical School, Boston, MA, 02215, USA
| | - Rui Wang
- Department of Population Medicine, Harvard Pilgrim Health Care Institute and Harvard Medical School, Boston, MA, 02215, USA.
- Department of Biostatistics, Harvard T. H. Chan School of Public Health, Boston, MA, 02215, USA.
| |
Collapse
|
6
|
Austin PC. The iterative bisection procedure: a useful tool for determining parameter values in data-generating processes in Monte Carlo simulations. BMC Med Res Methodol 2023; 23:45. [PMID: 36800931 PMCID: PMC9936690 DOI: 10.1186/s12874-023-01836-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2022] [Accepted: 01/06/2023] [Indexed: 02/19/2023] Open
Abstract
BACKGROUND Data-generating processes are key to the design of Monte Carlo simulations. It is important for investigators to be able to simulate data with specific characteristics. METHODS We described an iterative bisection procedure that can be used to determine the numeric values of parameters of a data-generating process to produce simulated samples with specified characteristics. We illustrated the application of the procedure in four different scenarios: (i) simulating binary outcome data from a logistic model such that the prevalence of the outcome is equal to a specified value; (ii) simulating binary outcome data from a logistic model based on treatment status and baseline covariates so that the simulated outcomes have a specified treatment relative risk; (iii) simulating binary outcome data from a logistic model so that the model c-statistic has a specified value; (iv) simulating time-to-event outcome data from a Cox proportional hazards model so that treatment induces a specified marginal or population-average hazard ratio. RESULTS In each of the four scenarios the bisection procedure converged rapidly and identified parameter values that resulted in the simulated data having the desired characteristics. CONCLUSION An iterative bisection procedure can be used to identify numeric values for parameters in data-generating processes to generate data with specified characteristics.
Collapse
Affiliation(s)
- Peter C. Austin
- grid.418647.80000 0000 8849 1617ICES, 2075 Bayview Avenue, Toronto, ON G106M4N 3M5 Canada ,grid.17063.330000 0001 2157 2938Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON Canada ,grid.17063.330000 0001 2157 2938Sunnybrook Research Institute, Toronto, ON Canada
| |
Collapse
|
7
|
Schouten AM, Flipse SM, van Nieuwenhuizen KE, Jansen FW, van der Eijk AC, van den Dobbelsteen JJ. Operating Room Performance Optimization Metrics: a Systematic Review. J Med Syst 2023; 47:19. [PMID: 36738376 PMCID: PMC9899172 DOI: 10.1007/s10916-023-01912-9] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2022] [Accepted: 11/26/2022] [Indexed: 02/05/2023]
Abstract
Literature proposes numerous initiatives for optimization of the Operating Room (OR). Despite multiple suggested strategies for the optimization of workflow on the OR, its patients and (medical) staff, no uniform description of 'optimization' has been adopted. This makes it difficult to evaluate the proposed optimization strategies. In particular, the metrics used to quantify OR performance are diverse so that assessing the impact of suggested approaches is complex or even impossible. To secure a higher implementation success rate of optimisation strategies in practice we believe OR optimisation and its quantification should be further investigated. We aim to provide an inventory of the metrics and methods used to optimise the OR by the means of a structured literature study. We observe that several aspects of OR performance are unaddressed in literature, and no studies account for possible interactions between metrics of quality and efficiency. We conclude that a systems approach is needed to align metrics across different elements of OR performance, and that the wellbeing of healthcare professionals is underrepresented in current optimisation approaches.
Collapse
Affiliation(s)
- Anne M Schouten
- Biomedical Engineering Department, Technical University of Delft, Mekelweg 5, 2628 CD, Delft, the Netherlands.
| | - Steven M Flipse
- Science Education and Communication Department, Technical University of Delft, Mekelweg 5, 2628 CD, Delft, the Netherlands
| | - Kim E van Nieuwenhuizen
- Gynecology Department, Leiden University Medical Center, Albinusdreef 2, 2333 ZA, Leiden, the Netherlands
| | - Frank Willem Jansen
- Biomedical Engineering Department, Technical University of Delft, Mekelweg 5, 2628 CD, Delft, the Netherlands
- Gynecology Department, Leiden University Medical Center, Albinusdreef 2, 2333 ZA, Leiden, the Netherlands
| | - Anne C van der Eijk
- Operation Room Centre, Leiden University Medical Center, Albinusdreef 2, 2333 ZA, Leiden, the Netherlands
| | - John J van den Dobbelsteen
- Biomedical Engineering Department, Technical University of Delft, Mekelweg 5, 2628 CD, Delft, the Netherlands
- Gynecology Department, Leiden University Medical Center, Albinusdreef 2, 2333 ZA, Leiden, the Netherlands
| |
Collapse
|
8
|
Sangji NF, Cain-Nielsen AH, Jakubus JL, Mikhail JN, Lussiez A, Neiman P, Montgomery JR, Oliphant BW, Scott JW, Hemmila MR. Application of power analysis to determine the optimal reporting time frame for use in statewide trauma system quality reporting. Surgery 2022; 172:1015-1020. [PMID: 35811165 DOI: 10.1016/j.surg.2022.05.028] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2022] [Revised: 04/27/2022] [Accepted: 05/30/2022] [Indexed: 11/19/2022]
Abstract
BACKGROUND Meaningful reporting of quality metrics relies on detecting a statistical difference when a true difference in performance exists. Larger cohorts and longer time frames can produce higher rates of statistical differences. However, older data are less relevant when attempting to enact change in the clinical setting. The selection of time frames must reflect a balance between being too small (type II errors) and too long (stale data). We explored the use of power analysis to optimize time frame selection for trauma quality reporting. METHODS Using data from 22 Level III trauma centers, we tested for differences in 4 outcomes within 4 cohorts of patients. With bootstrapping, we calculated the power for rejecting the null hypothesis that no difference exists amongst the centers for different time frames. From the entire sample for each site, we simulated randomly generated datasets. Each simulated dataset was tested for whether a difference was observed from the average. Power was calculated as the percentage of simulated datasets where a difference was observed. This process was repeated for each outcome. RESULTS The power calculations for the 4 cohorts revealed that the optimal time frame for Level III trauma centers to assess whether a single site's outcomes are different from the overall average was 2 years based on an 80% cutoff. CONCLUSION Power analysis with simulated datasets allows testing of different time frames to assess outcome differences. This type of analysis allows selection of an optimal time frame for benchmarking of Level III trauma center data.
Collapse
Affiliation(s)
- Naveen F Sangji
- Department of Surgery, University of Michigan, Ann Arbor, MI; Center for Healthcare Outcomes and Policy, University of Michigan, Ann Arbor, MI.
| | - Anne H Cain-Nielsen
- Center for Healthcare Outcomes and Policy, University of Michigan, Ann Arbor, MI
| | - Jill L Jakubus
- Center for Healthcare Outcomes and Policy, University of Michigan, Ann Arbor, MI
| | - Judy N Mikhail
- Center for Healthcare Outcomes and Policy, University of Michigan, Ann Arbor, MI
| | - Alisha Lussiez
- National Clinician Scholars Program, University of Michigan, Ann Arbor, MI
| | - Pooja Neiman
- Center for Healthcare Outcomes and Policy, University of Michigan, Ann Arbor, MI; National Clinician Scholars Program, University of Michigan, Ann Arbor, MI; Department of Surgery, Brigham and Women's Hospital, Boston, MA
| | - John R Montgomery
- Department of Surgery, University of Michigan, Ann Arbor, MI; Center for Healthcare Outcomes and Policy, University of Michigan, Ann Arbor, MI
| | - Bryant W Oliphant
- Center for Healthcare Outcomes and Policy, University of Michigan, Ann Arbor, MI; Department of Orthopaedic Surgery, University of Michigan, Ann Arbor, MI. https://twitter.com/BonezNQuality
| | - John W Scott
- Department of Surgery, University of Michigan, Ann Arbor, MI; Center for Healthcare Outcomes and Policy, University of Michigan, Ann Arbor, MI. https://twitter.com/DrJohnScott
| | - Mark R Hemmila
- Department of Surgery, University of Michigan, Ann Arbor, MI; Center for Healthcare Outcomes and Policy, University of Michigan, Ann Arbor, MI
| |
Collapse
|
9
|
Kara P, Valentin JB, Mainz J, Johnsen SP. Composite measures of quality of health care: Evidence mapping of methodology and reporting. PLoS One 2022; 17:e0268320. [PMID: 35552561 PMCID: PMC9098058 DOI: 10.1371/journal.pone.0268320] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2021] [Accepted: 04/27/2022] [Indexed: 11/19/2022] Open
Abstract
Background Quality indicators are used to quantify the quality of care. A large number of quality indicators makes assessment of overall quality difficult, time consuming and impractical. There is consequently an increasing interest for composite measures based on a combination of multiple indicators. Objective To examine the use of different approaches to construct composite measures of quality of care and to assess the use of methodological considerations and justifications. Methods We conducted a literature search on PubMed and EMBASE databases (latest update 1 December 2020). For each publication, we extracted information on the weighting and aggregation methodology that had been used to construct composite indicator(s). Results A total of 2711 publications were identified of which 145 were included after a screening process. Opportunity scoring with equal weights was the most used approach (86/145, 59%) followed by all-or-none scoring (48/145, 33%). Other approaches regarding aggregation or weighting of individual indicators were used in 32 publications (22%). The rationale for selecting a specific type of composite measure was reported in 36 publications (25%), whereas 22 papers (15%) addressed limitations regarding the composite measure. Conclusion Opportunity scoring and all-or-none scoring are the most frequently used approaches when constructing composite measures of quality of care. The attention towards the rationale and limitations of the composite measures appears low. Discussion Considering the widespread use and the potential implications for decision-making of composite measures, a high level of transparency regarding the construction process of the composite and the functionality of the measures is crucial.
Collapse
Affiliation(s)
- Pinar Kara
- Danish Center for Clinical Health Services Research (DACS), Department of Clinical Medicine, Aalborg University, Aalborg, Denmark
- Psychiatry, Aalborg University Hospital, Aalborg, Denmark
- * E-mail:
| | - Jan Brink Valentin
- Danish Center for Clinical Health Services Research (DACS), Department of Clinical Medicine, Aalborg University, Aalborg, Denmark
| | - Jan Mainz
- Danish Center for Clinical Health Services Research (DACS), Department of Clinical Medicine, Aalborg University, Aalborg, Denmark
- Psychiatry, Aalborg University Hospital, Aalborg, Denmark
- Department for Community Mental Health, University of Haifa, Haifa, Israel
- Department of Health Economics, University of Southern Denmark, Odense, Denmark
| | - Søren Paaske Johnsen
- Danish Center for Clinical Health Services Research (DACS), Department of Clinical Medicine, Aalborg University, Aalborg, Denmark
| |
Collapse
|
10
|
VAN SCHIE P, VAN BODEGOM-VOS L, VAN STEENBERGEN LN, NELISSEN RGHH, MARANG-VAN DE MHEEN PJ. A more comprehensive evaluation of quality of care after total hip and knee arthroplasty: combining 4 indicators in an ordered composite outcome. Acta Orthop 2022; 93:138-145. [PMID: 34984484 PMCID: PMC8815379 DOI: 10.2340/17453674.2021.861] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/07/2021] [Indexed: 01/31/2023] Open
Abstract
Background and purpose - Most arthroplasty registers give hospital-specific feedback on revision rates after total hip and knee arthroplasties (THA/TKA). However, due to the low number of events per hospital, multiple years of data are required to reliably detect worsening performance, and any single indicator provides only part of the quality of care delivered. Therefore, we developed an ordered composite outcome including revision, readmission, complications, and long length-of-stay (LOS) for a more comprehensive view on quality of care and assessed the ability to reliably differentiate between hospitals in their performance (rankability) with fewer years of data. Methods - All THA and TKA performed between 2017 and 2019 in 20 Dutch hospitals were included. All combinations of the 4 indicators were ranked from best to worst to create the ordinal composite outcome for THA and TKA separately. Between-hospital variation for the composite outcome was compared with individual indicators standardized for case-mix differences, and we calculated the statistical rankability using fixed and random effects models. Results - 22,908 THA and 20,423 TKA were included. Between-hospital variation for the THA and TKA composite outcomes was larger when compared with revision, readmission, and complications, and similar to long LOS. Rankabilities for the composite outcomes were above 80% even with 1 year of data, meaning that largely true hospital differences were detected rather than random variation. Interpretation - The ordinal composite outcome gives a more comprehensive overview of quality of delivered care and can reliably differentiate between hospitals in their performance using 1 year of data, thereby allowing earlier introduction of quality improvement initiatives.
Collapse
Affiliation(s)
- Peter VAN SCHIE
- Department of Orthopedics, Leiden University Medical Centre, Leiden,Department of Biomedical Data Sciences, Medical Decision Making, Leiden University Medical Centre, Leiden
| | - Leti VAN BODEGOM-VOS
- Department of Biomedical Data Sciences, Medical Decision Making, Leiden University Medical Centre, Leiden
| | | | | | | | | |
Collapse
|
11
|
Greenberg JK, Olsen MA, Dibble CF, Zhang JK, Pennicooke BH, Yamaguchi K, Kelly MP, Hall BL, Ray WZ. Comparison of cost and complication rates for profiling hospital performance in lumbar fusion for spondylolisthesis. Spine J 2021; 21:2026-2034. [PMID: 34161844 PMCID: PMC8720504 DOI: 10.1016/j.spinee.2021.06.014] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/25/2021] [Revised: 05/04/2021] [Accepted: 06/11/2021] [Indexed: 02/03/2023]
Abstract
BACKGROUND CONTEXT There is growing interest among payers in profiling hospital value and quality-of-care, including both the cost and safety of common surgeries, such as lumbar fusion. Nonetheless, there is sparse evidence describing the statistical reliability of such measures when applied to lumbar fusion for spondylolisthesis. PURPOSE To evaluate the reliability of 90-day inpatient hospital costs, overall complications, and rates of serious complications for profiling hospital performance in lumbar fusion surgery for spondylolisthesis. STUDY DESIGN/SETTING Data for this analysis came from State Inpatient Databases from nine states made available through the Healthcare Cost and Utilization Project. PATIENT SAMPLE Patients undergoing elective lumbar spine fusion for spondylolisthesis from 2010 to 2017 in participating states. OUTCOME MEASURES Statistical reliability, defined as the ability to distinguish true performance differences across hospitals relative to statistical noise. Reliability was assessed separately for 90-day inpatient costs (standardized across years to 2019 dollars), overall complications, and serious complication rates. METHODS Statistical reliability was measured as the amount of variation between hospitals relative to the total amount of variation for each measure. Total variation includes both between-hospital variation ("signal") and within-hospital variation ("statistical noise"). Thus, reliability equals signal over (signal plus noise) and ranges from 0 to 1. To adjust for differences in patient-level risk and procedural characteristics, hierarchical linear and logistic regression models were created for the cost and complication outcomes. Random hospital intercepts were used to assess between-hospital variation. We evaluated the reliability of each measure by study year and examined the number of hospitals meeting different thresholds of reliability by year. RESULTS We included a total of 66,571 elective lumbar fusion surgeries for spondylolisthesis performed at 244 hospitals during the study period. The mean 90-day hospital cost was $30,827 (2019 dollars). 12.0% of patients experienced a complication within 90 days of surgery, including 7.8% who had a serious complication. The median reliability of 90-day cost ranged from 0.97to 0.99 across study years, and there was a narrow distribution of reliability values. By comparison, the median reliability for the overall complication metric ranged from 0.22 to 0.44, and the reliability of the serious complication measure ranged from 0.30 to 0.49 across the study years. At least 96% of hospitals had high (> 0.7) reliability for cost in any year, whereas only 0-9% and 0-11% of hospitals reached this cutoff for the overall and serious complication rate in any year, respectively. By comparison, 10%-69% of hospitals per year achieved a more moderate threshold of 0.4 reliability for overall complications, compared to 21%-80% of hospitals who achieved this lower reliability threshold for serious complications. CONCLUSIONS 90-day inpatient costs are highly reliable for assessing variation across hospitals, whereas overall and serious complications are only moderately reliable for profiling performance. These results support the viability of emerging bundled payment programs that assume true differences in costs of care exist across hospitals.
Collapse
Affiliation(s)
| | | | | | | | | | - Ken Yamaguchi
- Department of Orthopaedic Surgery,Washington University in St. Louis, St. Louis, MO. Centene Corporation, St. Louis, MO
| | | | | | | |
Collapse
|
12
|
Greenberg JK, Olsen MA, Poe J, Dibble CF, Yamaguchi K, Kelly MP, Hall BL, Ray WZ. Administrative Data Are Unreliable for Ranking Hospital Performance Based on Serious Complications After Spine Fusion. Spine (Phila Pa 1976) 2021; 46:1181-1190. [PMID: 33826589 PMCID: PMC8363514 DOI: 10.1097/brs.0000000000004017] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/01/2023]
Abstract
STUDY DESIGN Retrospective analysis of administrative billing data. OBJECTIVE To evaluate the extent to which a metric of serious complications determined from administrative data can reliably profile hospital performance in spine fusion surgery. SUMMARY OF BACKGROUND DATA While payers are increasingly focused on implementing pay-for-performance measures, quality metrics must reliably reflect true differences in performance among the hospitals profiled. METHODS We used State Inpatient Databases from nine states to characterize serious complications after elective cervical and thoracolumbar fusion. Hierarchical logistic regression was used to risk-adjust differences in case mix, along with variability from low case volumes. The reliability of this risk-stratified complication rate (RSCR) was assessed as the variation between hospitals that was not due to chance alone, calculated separately by fusion type and year. Finally, we estimated the proportion of hospitals that had sufficient case volumes to obtain reliable (>0.7) complication estimates. RESULTS From 2010 to 2017 we identified 154,078 cervical and 213,133 thoracolumbar fusion surgeries. 4.2% of cervical fusion patients had a serious complication, and the median RSCR increased from 4.2% in 2010 to 5.5% in 2017. The reliability of the RSCR for cervical fusion was poor and varied substantially by year (range 0.04-0.28). Overall, 7.7% of thoracolumbar fusion patients experienced a serious complication, and the RSCR varied from 6.8% to 8.0% during the study period. Although still modest, the RSCR reliability was higher for thoracolumbar fusion (range 0.16-0.43). Depending on the study year, 0% to 4.5% of hospitals had sufficient cervical fusion case volume to report reliable (>0.7) estimates, whereas 15% to 36% of hospitals reached this threshold for thoracolumbar fusion. CONCLUSION A metric of serious complications was unreliable for benchmarking cervical fusion outcomes and only modestly reliable for thoracolumbar fusion. When assessed using administrative datasets, these measures appear inappropriate for high-stakes applications, such as public reporting or pay-for-performance.Level of Evidence: 3.
Collapse
Affiliation(s)
- Jacob K. Greenberg
- Department of Neurological Surgery, Washington University in St. Louis, St. Louis, MO
| | - Margaret A. Olsen
- Department of Medicine, Washington University in St. Louis, St. Louis, MO
| | - John Poe
- Department of Internal Medicine, University of Michigan Medical School, Ann Arbor, MI
| | - Christopher F Dibble
- Department of Neurological Surgery, Washington University in St. Louis, St. Louis, MO
| | - Ken Yamaguchi
- Department of Orthopaedic Surgery, Washington University in St. Louis, St. Louis, MO
- Centene Corporation, St. Louis, MO
| | - Michael P Kelly
- Department of Orthopaedic Surgery, Washington University in St. Louis, St. Louis, MO
| | - Bruce L Hall
- Department of Surgery, Washington University in St. Louis, St. Louis, MO
| | - Wilson Z. Ray
- Department of Neurological Surgery, Washington University in St. Louis, St. Louis, MO
| |
Collapse
|
13
|
Warps AK, Detering R, Tollenaar RAEM, Tanis PJ, Dekker JWT. Textbook outcome after rectal cancer surgery as a composite measure for quality of care: A population-based study. Eur J Surg Oncol 2021; 47:2821-2829. [PMID: 34120807 DOI: 10.1016/j.ejso.2021.05.045] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2021] [Revised: 04/09/2021] [Accepted: 05/28/2021] [Indexed: 10/21/2022] Open
Abstract
BACKGROUND Textbook outcome is a composite measure of combined outcome indicators, which has been suggested to be of additional value over single outcome parameters in clinical auditing of surgical treatment. This study aimed to assess textbook outcome after rectal cancer surgery as short-term marker for quality of care. MATERIALS AND METHODS Patients who underwent elective rectal cancer surgery between 2012 and 2019 and registered in the Dutch ColoRectal Audit were included. Textbook outcome was achieved when the following criteria were met: 30-day and primary hospital admission survival, no reintervention, tumour-free margins, no postoperative complications, a hospital stay of less than 14 days and no readmission. Hospital variation was evaluated in case-mix corrected funnel-plots. A multilevel logistic regression analysis was performed to identify associated factors with textbook outcome. RESULTS The study population consisted of 20,521 patients who underwent primary rectal cancer surgery, of whom 56.3% achieved textbook outcome. Postoperative complications were the main contributor to not achieving textbook outcome. Case-mix corrected funnel plots demonstrated that underperforming hospitals in 2012-2015 were no underperformers in 2016-2019 anymore. Female sex, laparoscopic surgery, and rectal resection without defunctioning stoma creation were positively associated with textbook outcome. CONCLUSION Textbook outcome after rectal cancer resection is mainly driven by postoperative complications. Although textbook outcome showed some discriminating value for identifying underperforming hospitals, it does not fit the plan-do-check-act cycle of clinical auditing. In our opinion, textbook outcome has little added value to the current outcome indicators for rectal cancer surgery.
Collapse
Affiliation(s)
- A K Warps
- Leiden University Medical Centre, Department of Surgery, Albinusdreef 2, 2333 ZA, Leiden, the Netherlands; Dutch Institute for Clinical Auditing, Rijnsburgerweg 10, 2333 AA, Leiden, the Netherlands
| | - R Detering
- Amsterdam University Medical Centres, Department of Surgery, Meibergdreef 9, 1105 AZ, Amsterdam, the Netherlands
| | - R A E M Tollenaar
- Leiden University Medical Centre, Department of Surgery, Albinusdreef 2, 2333 ZA, Leiden, the Netherlands; Dutch Institute for Clinical Auditing, Rijnsburgerweg 10, 2333 AA, Leiden, the Netherlands
| | - P J Tanis
- Amsterdam University Medical Centres, Department of Surgery, University of Amsterdam, Cancer Center Amsterdam, De Boelelaan 1117, 1081 HV, Amsterdam, the Netherlands
| | - J W T Dekker
- Reinier de Graaf Groep, Department of Surgery, Reinier de Graafweg 5, 2625 AD, Delft, the Netherlands.
| | | |
Collapse
|
14
|
Favez L, Zúñiga F, Sharma N, Blatter C, Simon M. Assessing Nursing Homes Quality Indicators' Between-Provider Variability and Reliability: A Cross-Sectional Study Using ICCs and Rankability. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2020; 17:ijerph17249249. [PMID: 33321952 PMCID: PMC7764139 DOI: 10.3390/ijerph17249249] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/16/2020] [Revised: 12/07/2020] [Accepted: 12/08/2020] [Indexed: 01/14/2023]
Abstract
Nursing home quality indicators are often used to publicly report the quality of nursing home care. In Switzerland, six national nursing home quality indicators covering four clinical domains (polypharmacy, pain, use of physical restraints and weight loss) were recently developed. To allow for meaningful comparisons, these indicators must reliably show differences in quality of care levels between nursing homes. This study’s objectives were to assess nursing home quality indicators’ between-provider variability and reliability using intraclass correlations and rankability. This approach has not yet been used in long-term care contexts but presents methodological advantages. This cross-sectional multicenter study uses data of 11,412 residents from a convenience sample of 152 Swiss nursing homes. After calculating intraclass correlation 1 (ICC1) and rankability, we describe between-provider variability for each quality indicator using empirical Bayes estimate-based caterpillar plots. To assess reliability, we used intraclass correlation 2 (ICC2). Overall, ICC1 values were high, ranging from 0.068 (95% confidence interval (CI) 0.047–0.086) for polypharmacy to 0.396 (95% CI 0.297–0.474) for physical restraints, with quality indicator caterpillar plots showing sufficient between-provider variability. However, testing for rankability produced mixed results, with low figures for two indicators (0.144 for polypharmacy; 0.471 for self-reported pain) and moderate to high figures for the four others (from 0.692 for observed pain to 0.976 for physical restraints). High ICC2 figures, ranging from 0.896 (95% CI 0.852–0.917) (self-reported pain) to 0.990 (95% CI 0.985–0.993) (physical restraints), indicated good reliability for all six quality indicators. Intraclass correlations and rankability can be used to assess nursing home quality indicators’ between-provider variability and reliability. The six selected quality indicators reliably distinguish care differences between nursing homes and can be recommended for use, although the variability of two—polypharmacy and self-reported pain—is substantially chance-driven, limiting their utility.
Collapse
Affiliation(s)
- Lauriane Favez
- Institute of Nursing Science, University of Basel, Bernoullistrasse 28, 4056 Basel, Switzerland; (L.F.); (N.S.); (C.B.); (M.S.)
| | - Franziska Zúñiga
- Institute of Nursing Science, University of Basel, Bernoullistrasse 28, 4056 Basel, Switzerland; (L.F.); (N.S.); (C.B.); (M.S.)
- Correspondence: ; Tel.: +41-61-207-09-13
| | - Narayan Sharma
- Institute of Nursing Science, University of Basel, Bernoullistrasse 28, 4056 Basel, Switzerland; (L.F.); (N.S.); (C.B.); (M.S.)
| | - Catherine Blatter
- Institute of Nursing Science, University of Basel, Bernoullistrasse 28, 4056 Basel, Switzerland; (L.F.); (N.S.); (C.B.); (M.S.)
| | - Michael Simon
- Institute of Nursing Science, University of Basel, Bernoullistrasse 28, 4056 Basel, Switzerland; (L.F.); (N.S.); (C.B.); (M.S.)
- Nursing and Midwifery Research Unit, Inselspital Bern University Hospital, Freiburgstrasse, 3010 Bern, Switzerland
| |
Collapse
|
15
|
Lawrence H, Lim WS, McKeever TM. Variation in clinical outcomes and process of care measures in community acquired pneumonia: a systematic review. Pneumonia (Nathan) 2020; 12:10. [PMID: 32999854 PMCID: PMC7517805 DOI: 10.1186/s41479-020-00073-4] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2020] [Accepted: 08/11/2020] [Indexed: 12/23/2022] Open
Abstract
Background Variation in outcomes of patients with community acquired pneumonia (CAP) has been reported in some, but not all, studies. Although some variation is expected, unwarranted variation in healthcare impacts patient outcomes and equity of care. The aim of this systematic review was to: i) summarise current evidence on regional and inter-hospital variation in the clinical outcomes and process of care measures of patients hospitalised with CAP and ii) assess the strength of this evidence. Methods Databases were systematically searched from inception to February 2018 for relevant studies and data independently extracted by two investigators in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement. Included studies enrolled adults hospitalised with CAP and reported a measure of variation between two or more units in healthcare outcomes or process of care measures. Outcomes of interest were mortality, length of hospital stay (LOS) and re-admission rates. A structured synthesis of the studies was performed. Results Twenty-two studies were included in the analysis. The median number of units compared across studies was five (IQR 4–15). Evidence for variation in mortality between units was inconsistent; of eleven studies that performed statistical significance testing, five found significant variation. For LOS, of nine relevant studies, all found statistically significant variation. Four studies reported site of admission accounted for 1–24% of the total observed variation in LOS. A shorter LOS was not associated with increased mortality or readmission rates. For readmission, evidence was mixed; of seven studies, 4 found statistically significant variation. There was consistent evidence for variation in the use of intensive care, obtaining blood cultures on admission, receiving antibiotics within 8 h of admission and duration of intravenous antibiotics. Across all outcome measures, only one study accounted for natural variation between units in their analysis. Conclusion There is consistent evidence of moderate quality for significant variation in length of stay and process of care measures but not for in-patient mortality or hospital re-admission. Evidence linking variation in outcomes with variation in process of care measures was limited; where present no difference in mortality was detected despite POC variation. Adjustment for natural variation within studies was lacking; the proportion of observed variation due to chance is not quantified by existing evidence.
Collapse
Affiliation(s)
- H Lawrence
- Nottingham University Hospitals NHS Trust, Hucknall Road, Nottingham, UK.,Division of Epidemiology and Public Health, School of Medicine, Clinical Sciences Building, Nottingham City Hospital, University of Nottingham, Nottingham, UK
| | - W S Lim
- Nottingham University Hospitals NHS Trust, Hucknall Road, Nottingham, UK.,NIHR Nottingham Biomedical Research Centre/Nottingham Clinical Research Facilities, Nottingham, UK
| | - T M McKeever
- Division of Epidemiology and Public Health, School of Medicine, Clinical Sciences Building, Nottingham City Hospital, University of Nottingham, Nottingham, UK.,NIHR Nottingham Biomedical Research Centre/Nottingham Clinical Research Facilities, Nottingham, UK
| |
Collapse
|
16
|
Vos EL, Lingsma HF, Jager A, Schreuder K, Spronk P, Vrancken Peeters MJTFD, Siesling S, Koppert LB. Effect of Case-Mix and Random Variation on Breast Cancer Care Quality Indicators and Their Rankability. VALUE IN HEALTH : THE JOURNAL OF THE INTERNATIONAL SOCIETY FOR PHARMACOECONOMICS AND OUTCOMES RESEARCH 2020; 23:1191-1199. [PMID: 32940237 DOI: 10.1016/j.jval.2019.12.014] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/04/2019] [Revised: 11/11/2019] [Accepted: 12/15/2019] [Indexed: 06/11/2023]
Abstract
OBJECTIVES Hospital comparisons to improve quality of care require valid and reliable quality indicators. We aimed to test the validity and reliability of 6 breast cancer indicators by quantifying the influence of case-mix and random variation. METHODS The nationwide population-based database included 79 690 patients with breast cancer from 91 Dutch hospitals between 2011 and 2016. The indicator-scores calculated were: (1) irradical breast-conserving surgery (BCS) for invasive disease, (2) irradical BCS for ductal carcinoma-in-situ, (3) breast contour-preserving treatment, (4) magnetic resonance imaging (MRI) before neo-adjuvant chemotherapy, (5) radiotherapy for locally advanced disease, and (6) surgery within 5 weeks from diagnosis. Case-mix and random variation adjustments were performed by multivariable fixed and random effect logistic regression models. Rankability quantified the between-hospital variation, representing unexplained differences that might be the result of the level of quality of care, as low (<50%), moderate (50%-75%), or high (>75%). RESULTS All of the indicators showed between-hospital variation with wide (interquartile) ranges. Case-mix adjustment reduced variation in indicators 1 and 3 to 5. Random variation adjustment (further) reduced the variation for all indicators. Case-mix and random variation adjustments influenced the indicator-scores of individual hospitals and their ranking. Rankability was poor for indicator 1, 2, and 5, and moderate for 3, 4, and 6. CONCLUSIONS The 6 indicators lacked validity and/or reliability to a certain extent. Although measuring quality indicators may stimulate quality improvement in general, comparisons and judgments of individual hospital performance should be made with caution if based on indicators that have not been tested or adjusted for validity and reliability, especially in benchmarking.
Collapse
Affiliation(s)
- Elvira L Vos
- Department of Surgery, Erasmus MC Cancer Institute, Rotterdam, The Netherlands
| | - Hester F Lingsma
- Department of Public Health, Erasmus MC Cancer Institute, Rotterdam, The Netherlands
| | - Agnes Jager
- Department of Medical Oncology, Erasmus MC Cancer Institute, Rotterdam, The Netherlands
| | - Kay Schreuder
- Department of Research, Netherlands Comprehensive Cancer Organisation, Utrecht, The Netherlands; Department of Health Technology and Services Research, Technical Medical Centre, University of Twente, Enschede, The Netherlands
| | - Pauline Spronk
- Department of Plastic Surgery, Erasmus MC Cancer Institute, Rotterdam, The Netherlands
| | | | - Sabine Siesling
- Department of Research, Netherlands Comprehensive Cancer Organisation, Utrecht, The Netherlands; Department of Health Technology and Services Research, Technical Medical Centre, University of Twente, Enschede, The Netherlands
| | - Linetta B Koppert
- Department of Surgery, Erasmus MC Cancer Institute, Rotterdam, The Netherlands.
| |
Collapse
|