601
|
Gynnild MN, Hageman SHJ, Dorresteijn JAN, Spigset O, Lydersen S, Wethal T, Saltvedt I, Visseren FLJ, Ellekjær H. Risk Stratification in Patients with Ischemic Stroke and Residual Cardiovascular Risk with Current Secondary Prevention. Clin Epidemiol 2021; 13:813-823. [PMID: 34566434 PMCID: PMC8456548 DOI: 10.2147/clep.s322779] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2021] [Accepted: 08/20/2021] [Indexed: 11/28/2022] Open
Abstract
PURPOSE Suboptimal secondary prevention in patients with stroke causes a remaining cardiovascular risk desirable to reduce. We have validated a prognostic model for secondary preventive settings and estimated future cardiovascular risk and theoretical benefit of reaching guideline recommended risk factor targets. PATIENTS AND METHODS The SMART-REACH (Secondary Manifestations of Arterial Disease-Reduction of Atherothrombosis for Continued Health) model for 10-year and lifetime risk of cardiovascular events was applied to 465 patients in the Norwegian Cognitive Impairment After Stroke (Nor-COAST) study, a multicenter observational study with two-year follow-up by linkage to national registries for cardiovascular disease and mortality. The residual risk when reaching recommended targets for blood pressure, low-density lipoprotein cholesterol, smoking cessation and antithrombotics was estimated. RESULTS In total, 11.2% had a new event. Calibration plots showed adequate agreement between estimated and observed 2-year prognosis (C-statistics 0.63, 95% confidence interval 0.55-0.71). Median estimated 10-year risk of recurrent cardiovascular events was 42% (Interquartile range (IQR) 32-54%) and could be reduced to 32% by optimal guideline-based therapy. The corresponding numbers for lifetime risk were 70% (IQR 63-76%) and 61%. We estimated an overall median gain of 1.4 (IQR 0.2-3.4) event-free life years if guideline targets were met. CONCLUSION Secondary prevention was suboptimal and residual risk remains elevated even after optimization according to current guidelines. Considerable interindividual variation in risk exists, with a corresponding variation in benefit from intensification of treatment. The SMART-REACH model can be used to identify patients with the largest benefit from more intensive treatment and follow-up.
Collapse
Affiliation(s)
- Mari Nordbø Gynnild
- Department of Neuromedicine and Movement Science, Faculty of Medicine and Health Science, NTNU – Norwegian University of Science and Technology, Trondheim, Norway
- Department of Stroke, Clinic of Medicine, St. Olavs Hospital, Trondheim University Hospital, Trondheim, Norway
| | - Steven H J Hageman
- Department of Vascular Medicine, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Jannick A N Dorresteijn
- Department of Vascular Medicine, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Olav Spigset
- Department of Clinical Pharmacology, St. Olavs Hospital, Trondheim University Hospital, Trondheim, Norway
- Department of Clinical and Molecular Medicine, NTNU – Norwegian University of Science and Technology, Trondheim, Norway
| | - Stian Lydersen
- Department of Mental Health, Faculty of Medicine and Health Sciences, NTNU – Norwegian University of Science and Technology, Trondheim, Norway
| | - Torgeir Wethal
- Department of Neuromedicine and Movement Science, Faculty of Medicine and Health Science, NTNU – Norwegian University of Science and Technology, Trondheim, Norway
- Department of Stroke, Clinic of Medicine, St. Olavs Hospital, Trondheim University Hospital, Trondheim, Norway
| | - Ingvild Saltvedt
- Department of Neuromedicine and Movement Science, Faculty of Medicine and Health Science, NTNU – Norwegian University of Science and Technology, Trondheim, Norway
- Department of Geriatrics, Clinic of Medicine, St. Olavs Hospital, Trondheim University Hospital, Trondheim, Norway
| | - Frank L J Visseren
- Department of Vascular Medicine, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Hanne Ellekjær
- Department of Neuromedicine and Movement Science, Faculty of Medicine and Health Science, NTNU – Norwegian University of Science and Technology, Trondheim, Norway
- Department of Stroke, Clinic of Medicine, St. Olavs Hospital, Trondheim University Hospital, Trondheim, Norway
| |
Collapse
|
602
|
Abstract
OBJECTIVES The purpose of this scoping review is to: (1) identify existing supervised machine learning (ML) approaches on the prediction of cancer in asymptomatic adults; (2) to compare the performance of ML models with each other and (3) to identify potential gaps in research. DESIGN Scoping review using the population, concept and context approach. SEARCH STRATEGY PubMed search engine was used from inception to 10 November 2020 to identify literature meeting following inclusion criteria: (1) a general adult (≥18 years) population, either sex, asymptomatic (population); (2) any study using ML techniques to derive predictive models for future cancer risk using clinical and/or demographic and/or basic laboratory data (concept) and (3) original research articles conducted in all settings in any region of the world (context). RESULTS The search returned 627 unique articles, of which 580 articles were excluded because they did not meet the inclusion criteria, were duplicates or were related to benign neoplasm. Full-text reviews were conducted for 47 articles and a final set of 10 articles were included in this scoping review. These 10 very heterogeneous studies used ML to predict future cancer risk in asymptomatic individuals. All studies reported area under the receiver operating characteristics curve (AUC) values as metrics of model performance, but no study reported measures of model calibration. CONCLUSIONS Research gaps that must be addressed in order to deliver validated ML-based models to assist clinical decision-making include: (1) establishing model generalisability through validation in independent cohorts, including those from low-income and middle-income countries; (2) establishing models for all cancer types; (3) thorough comparisons of ML models with best available clinical tools to ensure transparency of their potential clinical utility; (4) reporting of model calibration performance and (5) comparisons of different methods on the same cohort to reveal important information about model generalisability and performance.
Collapse
Affiliation(s)
- Asma Abdullah Alfayez
- Institute of Health Informatics, University College London, London, UK
- King Abdullah International Medical Research Center, Riyadh, Saudi Arabia
| | - Holger Kunz
- Institute of Health Informatics, University College London, London, UK
| | - Alvina Grace Lai
- Institute of Health Informatics, University College London, London, UK
| |
Collapse
|
603
|
Affiliation(s)
- Tony Antoniou
- Li Ka Shing Centre for Healthcare Analytics Research & Training (Antoniou, Mamdani), Unity Health Toronto; Li Ka Shing Knowledge Institute (Antoniou, Mamdani), Unity Health Toronto; Department of Family and Community Medicine (Antoniou), Unity Health Toronto and University of Toronto; Temerty Faculty of Medicine (Mamdani) and Leslie Dan Faculty of Pharmacy (Mamdani), University of Toronto; Institute of Health Policy, Management, and Evaluation (Mamdani), University of Toronto, Toronto, Ont.
| | - Muhammad Mamdani
- Li Ka Shing Centre for Healthcare Analytics Research & Training (Antoniou, Mamdani), Unity Health Toronto; Li Ka Shing Knowledge Institute (Antoniou, Mamdani), Unity Health Toronto; Department of Family and Community Medicine (Antoniou), Unity Health Toronto and University of Toronto; Temerty Faculty of Medicine (Mamdani) and Leslie Dan Faculty of Pharmacy (Mamdani), University of Toronto; Institute of Health Policy, Management, and Evaluation (Mamdani), University of Toronto, Toronto, Ont
| |
Collapse
|
604
|
Sadatsafavi M, Adibi A, Puhan M, Gershon A, Aaron SD, Sin DD. Moving beyond AUC: decision curve analysis for quantifying net benefit of risk prediction models. Eur Respir J 2021; 58:13993003.01186-2021. [PMID: 34503984 DOI: 10.1183/13993003.01186-2021] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2021] [Accepted: 08/31/2021] [Indexed: 11/05/2022]
Affiliation(s)
- Mohsen Sadatsafavi
- Respiratory Evaluation Sciences Program, Collaboration for Outcomes Research and Evaluation, Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC, Canada
| | - Amin Adibi
- Respiratory Evaluation Sciences Program, Collaboration for Outcomes Research and Evaluation, Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, BC, Canada
| | - Milo Puhan
- Epidemiology, Biostatistics and Prevention Institute, University of Zurich, Zurich, Switzerland
| | - Andrea Gershon
- Institute of Clinical Evaluation Sciences, University of Toronto, Toronto, ON, Canada
| | - Shawn D Aaron
- The Ottawa Hospital Research Institute, University of Ottawa, Ottawa, ON, Canada
| | - Don D Sin
- Centre for Heart Lung Innovation, University of British Columbia, Vancouver, BC, Canada
| |
Collapse
|
605
|
Cai A, Zhu Y, Clarkson SA, Feng Y. The Use of Machine Learning for the Care of Hypertension and Heart Failure. JACC. ASIA 2021; 1:162-172. [PMID: 36338169 PMCID: PMC9627876 DOI: 10.1016/j.jacasi.2021.07.005] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/03/2021] [Revised: 06/22/2021] [Accepted: 07/19/2021] [Indexed: 06/12/2023]
Abstract
Machine learning (ML) is a branch of artificial intelligence that combines computer science, statistics, and decision theory to learn complex patterns from voluminous data. In the last decade, accumulating evidence has shown the utility of ML for prediction, diagnosis, and classification of hypertension and heart failure (HF). In addition, ML-enabled image analysis has potential value in assessing cardiac structure and function in an accurate, scalable, and efficient way. Considering the high burden of hypertension and HF in China and worldwide, ML may help address these challenges from different aspects. Indeed, prior studies have shown that ML can enhance each stage of patient care, from research and development, to daily clinical practice and population health. Through reviewing the published literature, the aims of the current systemic review are to summarize the utilities of ML for the care of those with hypertension and HF.
Collapse
Key Words
- ANN, artificial neural network
- AUC, area under the curve
- CNN, convolutional neural network
- HFpEF, heart failure with preserved ejection fraction
- LRM, linear or logistic regression model
- LVDD, left ventricular diastolic dysfunction
- LVH, left ventricular hypertrophy
- ML, machine learning
- RF, random forest
- SVM, support vector machine
- algorithms
- heart failure
- hypertension machine learning
Collapse
Affiliation(s)
- Anping Cai
- Department of Cardiology, Guangdong Cardiovascular Institute, Hypertension Research Laboratory, Guangdong Provincial People’s Hospital, Guangdong Academy of Medical Sciences, Guangzhou, China
| | - Yicheng Zhu
- Department of Cardiology, Guangdong Cardiovascular Institute, Hypertension Research Laboratory, Guangdong Provincial People’s Hospital, Guangdong Academy of Medical Sciences, Guangzhou, China
| | - Stephen A. Clarkson
- Division of Cardiovascular Disease, Department of Medicine, University of Alabama at Birmingham, Birmingham, Alabama, USA
| | - Yingqing Feng
- Department of Cardiology, Guangdong Cardiovascular Institute, Hypertension Research Laboratory, Guangdong Provincial People’s Hospital, Guangdong Academy of Medical Sciences, Guangzhou, China
| |
Collapse
|
606
|
Riley RD, Debray TPA, Collins GS, Archer L, Ensor J, van Smeden M, Snell KIE. Minimum sample size for external validation of a clinical prediction model with a binary outcome. Stat Med 2021; 40:4230-4251. [PMID: 34031906 DOI: 10.1002/sim.9025] [Citation(s) in RCA: 116] [Impact Index Per Article: 38.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2020] [Revised: 02/01/2021] [Accepted: 03/22/2021] [Indexed: 12/22/2022]
Abstract
In prediction model research, external validation is needed to examine an existing model's performance using data independent to that for model development. Current external validation studies often suffer from small sample sizes and consequently imprecise predictive performance estimates. To address this, we propose how to determine the minimum sample size needed for a new external validation study of a prediction model for a binary outcome. Our calculations aim to precisely estimate calibration (Observed/Expected and calibration slope), discrimination (C-statistic), and clinical utility (net benefit). For each measure, we propose closed-form and iterative solutions for calculating the minimum sample size required. These require specifying: (i) target SEs (confidence interval widths) for each estimate of interest, (ii) the anticipated outcome event proportion in the validation population, (iii) the prediction model's anticipated (mis)calibration and variance of linear predictor values in the validation population, and (iv) potential risk thresholds for clinical decision-making. The calculations can also be used to inform whether the sample size of an existing (already collected) dataset is adequate for external validation. We illustrate our proposal for external validation of a prediction model for mechanical heart valve failure with an expected outcome event proportion of 0.018. Calculations suggest at least 9835 participants (177 events) are required to precisely estimate the calibration and discrimination measures, with this number driven by the calibration slope criterion, which we anticipate will often be the case. Also, 6443 participants (116 events) are required to precisely estimate net benefit at a risk threshold of 8%. Software code is provided.
Collapse
Affiliation(s)
- Richard D Riley
- Centre for Prognosis Research, School of Medicine, Keele University, Staffordshire, UK
| | - Thomas P A Debray
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
- NIHR Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford, UK
| | - Lucinda Archer
- Centre for Prognosis Research, School of Medicine, Keele University, Staffordshire, UK
| | - Joie Ensor
- Centre for Prognosis Research, School of Medicine, Keele University, Staffordshire, UK
| | - Maarten van Smeden
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Kym I E Snell
- Centre for Prognosis Research, School of Medicine, Keele University, Staffordshire, UK
| |
Collapse
|
607
|
Chichareon P, Nilmoje T, Suriyaamorn W, Preechawettayakul I, Suwanugsorn S. Predicting mortality in cardiac care unit patients: external validation of the Mayo cardiac intensive care unit admission risk score. EUROPEAN HEART JOURNAL-ACUTE CARDIOVASCULAR CARE 2021; 10:1065-1073. [PMID: 34448824 DOI: 10.1093/ehjacc/zuab070] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/05/2021] [Revised: 07/15/2021] [Accepted: 08/05/2021] [Indexed: 11/12/2022]
Abstract
AIMS The Mayo Cardiac Intensive Care Unit Admission Risk Score (M-CARS) had excellent performance in predicting in-hospital mortality in the US population. We sought to validate the M-CARS for in-hospital and post-discharge mortality in Asian patients admitted to the cardiac care unit (CCU). METHODS AND RESULTS Patients admitted to the CCU of a tertiary care centre between July 2015 and December 2019 were included into the study. Patients with intra-hospital transfer to the CCU due to intensive care unit overflow, postoperative cardiac surgery, or for monitoring after elective procedures were excluded. Cardiac arrest, cardiogenic shock, respiratory failure, Braden skin score, blood urea nitrogen, anion gap, and red cell distribution width, were used to calculate the M-CARS. Patients were stratified into three groups, according to the M-CARS (<2, 2-6, >6). Of 1988 patients in the study, 30.1% were female with a median age of 65 years. Prevalence of cardiogenic shock and respiratory failure at admission were 2.8% and 4.5%, respectively. One hundred and seventeen patients died during the admission (mortality rate of 5.9%). The in-hospital mortality rate in patients with M-CARS of <2, 2-6, and >6 was 1.1%, 9.8%, and 35.5%, respectively. C-statistic of M-CARS for in-hospital mortality was 0.840 (95% CI 0.805-0.873); whereas, it was 0.727 (95% CI 0.690-0.761) for 1-year post-discharge mortality. Calibration plot showed good agreement between predicted and observed in-hospital mortality in the majority of patients. CONCLUSIONS The M-CARS was useful in our study, in terms of discrimination and calibration. M-CARS identified high-risk patients in CCU, who had unacceptably high mortality rate during hospital stay and thereafter.
Collapse
Affiliation(s)
- Ply Chichareon
- Cardiology Unit, Division of Internal Medicine, Faculty of Medicine, Prince of Songkla University, Songkhla 90110, Thailand.,Naradhiwas Rajanagarindra Heart Center, Songklanagarind Hospital, Songkhla 90110, Thailand
| | - Thanapon Nilmoje
- Cardiology Unit, Division of Internal Medicine, Faculty of Medicine, Prince of Songkla University, Songkhla 90110, Thailand
| | - Wisanuwee Suriyaamorn
- Cardiology Unit, Division of Internal Medicine, Faculty of Medicine, Prince of Songkla University, Songkhla 90110, Thailand
| | - Ittipon Preechawettayakul
- Cardiology Unit, Division of Internal Medicine, Faculty of Medicine, Prince of Songkla University, Songkhla 90110, Thailand
| | - Saranyou Suwanugsorn
- Cardiology Unit, Division of Internal Medicine, Faculty of Medicine, Prince of Songkla University, Songkhla 90110, Thailand
| |
Collapse
|
608
|
Li B, Zhao Y, Cai W, Ming A, Li H. Validation and update of a multivariable prediction model for the identification and management of patients at risk for hepatocellular carcinoma. Clin Proteomics 2021; 18:21. [PMID: 34412596 PMCID: PMC8374120 DOI: 10.1186/s12014-021-09326-w] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2021] [Accepted: 08/08/2021] [Indexed: 02/07/2023] Open
Abstract
Background A hepatocellular carcinoma (HCC) prediction model (ASAP), including age, sex, and the biomarkers alpha-fetoprotein and prothrombin induced by vitamin K absence-II, showed potential clinical value in the early detection of HCC. We validated and updated the model in a real-world cohort and promoted its transferability to daily clinical practice. Methods This retrospective cohort analysis included 1012 of the 2479 eligible patients aged 35 years or older undergoing surveillance for HCC. The data were extracted from the electronic medical records. Biomarker values within the test-to-diagnosis interval were used to validate the ASAP model. Due to its unsatisfactory calibration, three logistic regression models were constructed to recalibrate and update the model. Their discrimination, calibration, and clinical utility were compared. The performance statistics of the final updated model at several risk thresholds are presented. The outcomes of 855 non-HCC patients were further assessed during a median of 10.2 months of follow-up. Statistical analyses were performed using packages in R software. Results The ASAP model had superior discriminative performance in the validation cohort [C-statistic = 0.982, (95% confidence interval 0.972–0.992)] but significantly overestimated the risk of HCC (intercept − 3.243 and slope 1.192 in the calibration plot), reducing its clinical usefulness. Recalibration-in-the-large, which exhibited performance comparable to that of the refitted model revision, led to the retention of the excellent discrimination and substantial improvements in the calibration and clinical utility, achieving a sensitivity of 100% at the median prediction probability of the absence of HCC (1.3%). The probability threshold of 1.3% and the incidence of HCC in the cohort (15.5%) were used to stratify the patients into low-, medium-, and high-risk groups. The cumulative HCC incidences in the non-HCC patients significantly differed among the risk groups (log-rank test, p-value < 0.001). The 3-month, 6-month and 18-month cumulative incidences in the low-risk group were 0.6%, 0.9% and 0.9%, respectively. Conclusions The ASAP model is an accurate tool for HCC risk estimation that requires recalibration before use in a new region because calibration varies with clinical environments. Additionally, rational risk stratification and risk-based management decision-making, e.g., 3-month follow-up recommendations for targeted individuals, helped improve HCC surveillance, which warrants assessment in larger cohorts. Supplementary Information The online version contains supplementary material available at 10.1186/s12014-021-09326-w.
Collapse
Affiliation(s)
- Bo Li
- Clinical Laboratory, Hubei Provincial Hospital of Traditional Chinese Medicine, Affiliated Hospital of Hubei University of Traditional Chinese Medicine, Hubei Province Academy of Traditional Chinese Medicine, Wuhan, China
| | - Youyun Zhao
- Clinical Laboratory, Hubei Provincial Hospital of Traditional Chinese Medicine, Affiliated Hospital of Hubei University of Traditional Chinese Medicine, Hubei Province Academy of Traditional Chinese Medicine, Wuhan, China
| | - Wangxi Cai
- Clinical Laboratory, Hubei Provincial Hospital of Traditional Chinese Medicine, Affiliated Hospital of Hubei University of Traditional Chinese Medicine, Hubei Province Academy of Traditional Chinese Medicine, Wuhan, China
| | - Anping Ming
- Clinical Laboratory, Hubei Provincial Hospital of Traditional Chinese Medicine, Affiliated Hospital of Hubei University of Traditional Chinese Medicine, Hubei Province Academy of Traditional Chinese Medicine, Wuhan, China
| | - Hanmin Li
- Institute of Hepatology, Hubei Provincial Hospital of Traditional Chinese Medicine, Affiliated Hospital of Hubei University of Traditional Chinese Medicine, Hubei Province Academy of Traditional Chinese Medicine, Wuhan, China. .,Theory and Application Research of Liver and Kidney in Traditional Chinese Medicine, Hubei Provincial Key Laboratory, Cell Molecular Biology Laboratory, Level 3 Laboratory of Traditional Chinese Medicine Research, State Administration of Traditional Chinese Medicine, 4 Huayuanshan, Yanzhi Road, Liangdao Street, Wuchang District, Hubei, 430061, Wuhan, China.
| |
Collapse
|
609
|
Preparing Laboratories for Interconnected Health Care. Diagnostics (Basel) 2021; 11:diagnostics11081487. [PMID: 34441421 PMCID: PMC8391810 DOI: 10.3390/diagnostics11081487] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/11/2021] [Revised: 07/26/2021] [Accepted: 07/28/2021] [Indexed: 12/01/2022] Open
Abstract
In an increasingly interconnected health care system, laboratory medicine can facilitate diagnosis and treatment of patients effectively. This article describes necessary changes and points to potential challenges on a technical, content, and organizational level. As a technical precondition, electronic laboratory reports have to become machine-readable and interpretable. Terminologies such as Logical Observation Identifiers Names and Codes (LOINC), Nomenclature for Properties and Units (NPU), Unified Code for Units of Measure (UCUM), and SNOMED-CT can lead to the necessary semantic interoperability. Even if only single “atomized” results of the whole report are extracted, the necessary information for correct interpretation must be available. Therefore, interpretive comments, e.g., concerns about an increased measurement uncertainty must be electronically attached to every affected measurement result. Standardization of laboratory analyses with traceable standards and reference materials will enable knowledge transfer and safe interpretation of laboratory analyses from multiple laboratories. In an interconnected health care system, laboratories should strive to transform themselves into a data hub that not only receives samples but also extensive information about the patient. On that basis, they can return measurement results enriched with high-quality interpretive comments tailored to the individual patient and unlock the full potential of laboratory medicine.
Collapse
|
610
|
Quah J, Liew CJY, Zou L, Koh XH, Alsuwaigh R, Narayan V, Lu TY, Ngoh C, Wang Z, Koh JZ, Ang C, Fu Z, Goh HL. Chest radiograph-based artificial intelligence predictive model for mortality in community-acquired pneumonia. BMJ Open Respir Res 2021; 8:8/1/e001045. [PMID: 34376402 PMCID: PMC8354266 DOI: 10.1136/bmjresp-2021-001045] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2021] [Accepted: 07/21/2021] [Indexed: 12/15/2022] Open
Abstract
Background Chest radiograph (CXR) is a basic diagnostic test in community-acquired pneumonia (CAP) with prognostic value. We developed a CXR-based artificial intelligence (AI) model (CAP AI predictive Engine: CAPE) and prospectively evaluated its discrimination for 30-day mortality. Methods Deep-learning model using convolutional neural network (CNN) was trained with a retrospective cohort of 2235 CXRs from 1966 unique adult patients admitted for CAP from 1 January 2019 to 31 December 2019. A single-centre prospective cohort between 11 May 2020 and 15 June 2020 was analysed for model performance. CAPE mortality risk score based on CNN analysis of the first CXR performed for CAP was used to determine the area under the receiver operating characteristic curve (AUC) for 30-day mortality. Results 315 inpatient episodes for CAP occurred, with 30-day mortality of 19.4% (n=61/315). Non-survivors were older than survivors (mean (SD)age, 80.4 (10.3) vs 69.2 (18.7)); more likely to have dementia (n=27/61 vs n=58/254) and malignancies (n=16/61 vs n=18/254); demonstrate higher serum C reactive protein (mean (SD), 109 mg/L (98.6) vs 59.3 mg/L (69.7)) and serum procalcitonin (mean (SD), 11.3 (27.8) μg/L vs 1.4 (5.9) μg/L). The AUC for CAPE mortality risk score for 30-day mortality was 0.79 (95% CI 0.73 to 0.85, p<0.001); Pneumonia Severity Index (PSI) 0.80 (95% CI 0.74 to 0.86, p<0.001); Confusion of new onset, blood Urea nitrogen, Respiratory rate, Blood pressure, 65 (CURB-65) score 0.76 (95% CI 0.70 to 0.81, p<0.001), respectively. CAPE combined with CURB-65 model has an AUC of 0.83 (95% CI 0.77 to 0.88, p<0.001). The best performing model was CAPE incorporated with PSI, with an AUC of 0.84 (95% CI 0.79 to 0.89, p<0.001). Conclusion CXR-based CAPE mortality risk score was comparable to traditional pneumonia severity scores and improved its discrimination when combined.
Collapse
Affiliation(s)
- Jessica Quah
- Department of Respiratory and Critical Care Medicine, Changi General Hospital, Singapore
| | | | - Lin Zou
- Integrated Health Information Systems Pte Ltd, Singapore
| | - Xuan Han Koh
- Health Services Research, Changi General Hospital, Singapore
| | - Rayan Alsuwaigh
- Department of Respiratory and Critical Care Medicine, Changi General Hospital, Singapore
| | | | - Tian Yi Lu
- Integrated Health Information Systems Pte Ltd, Singapore
| | - Clarence Ngoh
- Integrated Health Information Systems Pte Ltd, Singapore
| | - Zhiyu Wang
- Integrated Health Information Systems Pte Ltd, Singapore
| | - Juan Zhen Koh
- Integrated Health Information Systems Pte Ltd, Singapore
| | - Christine Ang
- Integrated Health Information Systems Pte Ltd, Singapore
| | - Zhiyan Fu
- Integrated Health Information Systems Pte Ltd, Singapore
| | - Han Leong Goh
- Integrated Health Information Systems Pte Ltd, Singapore
| |
Collapse
|
611
|
Isci S, Kalender DSY, Bayraktar F, Yaman A. Machine Learning Models for Classification of Cushing's Syndrome Using Retrospective Data. IEEE J Biomed Health Inform 2021; 25:3153-3162. [PMID: 33513119 DOI: 10.1109/jbhi.2021.3054592] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
Accurate classification of Cushing's Syndrome (CS) plays a critical role in providing the early and correct diagnosis of CS that may facilitate treatment and improve patient outcomes. Diagnosis of CS is a complex process, which requires careful and concurrent interpretation of signs and symptoms, multiple biochemical test results, and findings of medical imaging by physicians with a high degree of specialty and knowledge to make correct judgments. In this article, we explore the state of the art machine learning algorithms to demonstrate their potential as a clinical decision support system to analyze and classify CS to facilitate the diagnosis, prognosis, and treatment of CS. Prominent algorithms are compared using nested cross-validation and various class comparison strategies including multiclass, one vs. all, and one vs. one binary classification. Our findings show that Random Forest (RF) algorithm is most suitable for the classification of CS. We demonstrate that the proposed approach can classify CS with an average accuracy of 92% and an average F1 score of 91.5%, depending on the class comparison strategy and selected features. RF-based one vs. all binary classification model achieves sensitivity of 97.6%, precision of 91.1%, and specificity of 87.1% to discriminate CS from non-CS on the test dataset. RF-based multiclass classification model achieves average per class sensitivity of 91.8%, average per class specificity of 97.1%, and average per class precision of 92.1% to classify different subtypes of CS on the test dataset. Clinical performance evaluation suggests that the developed models can help improve physicians' judgment in diagnosing CS.
Collapse
|
612
|
A Multivariable Predictive Model for Success of External Cephalic Version. Obstet Gynecol 2021; 138:426-433. [PMID: 34352823 DOI: 10.1097/aog.0000000000004518] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/04/2021] [Accepted: 05/20/2021] [Indexed: 11/26/2022]
Abstract
OBJECTIVE To create a prediction model for external cephalic version (ECV) success using objective patient characteristics. METHODS This retrospective study included pregnant individuals of at least 18 years of age with a nonanomalous, singleton gestation who underwent an ECV attempt between 2006 and 2016 at a single quaternary care hospital. Variables assessed included maternal age, height, weight, body mass index (BMI), parity, fetal sex, gestational age, estimated fetal weight, type of fetal malpresentation, placental location, and amniotic fluid volume. Univariable and multivariable logistic regression models were used to determine the association of patient characteristics with ECV success. Estimated odds ratios and corresponding 95% CIs were calculated for each variable, and backward elimination and bootstrapping were used to find a parsimonious model for ECV success with the highest discriminatory capacity (as determined by the area under the receiver operating characteristic curve [AUC]). This model was evaluated with a calibration curve across deciles of success. RESULTS A total of 1,138 individuals underwent an ECV attempt and were included in this analysis. The overall ECV success frequency was 40.6%. Factors significantly associated with ECV success were maternal age, parity, placental location, estimated fetal weight, and type of fetal malpresentation. A final model with BMI, parity, placental location, and type of fetal malpresentation had the highest AUC (0.667 [95% CI 0.634-0.701]), resulted in good calibration, and is represented by the following equation: 1/[1+e-x] where x=1.1726-0.0314 (BMI)-0.9299 (nulliparity)+1.0218 (transverse or oblique presentation at ECV)-0.5113 (anterior placenta). An interactive version of this equation was created and can be accessed at www.ecvcalculator.com. CONCLUSION A prediction model that estimates the probability of ECV success was created and internally validated. This model incorporates easily obtainable and objective patient factors known before ECV and may be used in decision making and patient counseling about ECV.
Collapse
|
613
|
Meid AD, Gonzalez-Gonzalez AI, Dinh TS, Blom J, van den Akker M, Elders P, Thiem U, Küllenberg de Gaudry D, Swart KMA, Rudolf H, Bosch-Lenders D, Trampisch HJ, Meerpohl JJ, Gerlach FM, Flaig B, Kom G, Snell KIE, Perera R, Haefeli WE, Glasziou P, Muth C. Predicting hospital admissions from individual patient data (IPD): an applied example to explore key elements driving external validity. BMJ Open 2021; 11:e045572. [PMID: 34348947 PMCID: PMC8340284 DOI: 10.1136/bmjopen-2020-045572] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/18/2022] Open
Abstract
OBJECTIVE To explore factors that potentially impact external validation performance while developing and validating a prognostic model for hospital admissions (HAs) in complex older general practice patients. STUDY DESIGN AND SETTING Using individual participant data from four cluster-randomised trials conducted in the Netherlands and Germany, we used logistic regression to develop a prognostic model to predict all-cause HAs within a 6-month follow-up period. A stratified intercept was used to account for heterogeneity in baseline risk between the studies. The model was validated both internally and by using internal-external cross-validation (IECV). RESULTS Prior HAs, physical components of the health-related quality of life comorbidity index, and medication-related variables were used in the final model. While achieving moderate discriminatory performance, internal bootstrap validation revealed a pronounced risk of overfitting. The results of the IECV, in which calibration was highly variable even after accounting for between-study heterogeneity, agreed with this finding. Heterogeneity was equally reflected in differing baseline risk, predictor effects and absolute risk predictions. CONCLUSIONS Predictor effect heterogeneity and differing baseline risk can explain the limited external performance of HA prediction models. With such drivers known, model adjustments in external validation settings (eg, intercept recalibration, complete updating) can be applied more purposefully. TRIAL REGISTRATION NUMBER PROSPERO id: CRD42018088129.
Collapse
Affiliation(s)
- Andreas Daniel Meid
- Department of Clinical Pharmacology & Pharmacoepidemiology, Heidelberg University, Heidelberg, Baden-Württemberg, Germany
| | - Ana Isabel Gonzalez-Gonzalez
- Institute of General Practice, Goethe University, Frankfurt am Main, Hessen, Germany
- Red de Investigación en Servicios de Salud en Enfermedades Crónicas (REDISSEC), Madrid, Spain
| | - Truc Sophia Dinh
- Institute of General Practice, Goethe University, Frankfurt am Main, Hessen, Germany
| | - Jeanet Blom
- Department of Public Health and Primary Care, Leiden University Medical Center, Leiden, The Netherlands
| | - Marjan van den Akker
- Institute of General Practice, Goethe University, Frankfurt am Main, Hessen, Germany
- School of CAPHRI, Department of Family Medicine, Maastricht University, Maastricht, The Netherlands
| | - Petra Elders
- Department of General Practice and Elderly Care Medicine, Amsterdam UMC, Vrije Universiteit, Amstedarm Public Health Research Institute, Amsterdam, The Netherlands
| | - Ulrich Thiem
- Chair of Geriatrics and Gerontology, University Clinic Eppendorf, Hamburg, Germany
| | - Daniela Küllenberg de Gaudry
- Institute for Evidence in Medicine (for Cochrane Germany Foundation), Medical Center-University of Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany
| | - Karin M A Swart
- Department of General Practice and Elderly Care Medicine, Amsterdam UMC, Vrije Universiteit, Amstedarm Public Health Research Institute, Amsterdam, The Netherlands
| | - Henrik Rudolf
- Department of Medical Informatics, Biometry and Epidemiology, Ruhr University Bochum, Bochum, Nordrhein-Westfalen, Germany
| | - Donna Bosch-Lenders
- School of CAPHRI, Department of Family Medicine, Maastricht University, Maastricht, The Netherlands
| | - Hans J Trampisch
- Department of Medical Informatics, Biometry and Epidemiology, Ruhr University Bochum, Bochum, Nordrhein-Westfalen, Germany
| | - Joerg J Meerpohl
- Institute for Evidence in Medicine (for Cochrane Germany Foundation), Medical Center-University of Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany
| | - Ferdinand M Gerlach
- Institute of General Practice, Goethe University, Frankfurt am Main, Hessen, Germany
| | - Benno Flaig
- Institute of General Practice, Goethe University, Frankfurt am Main, Hessen, Germany
| | | | - Kym I E Snell
- Centre for Prognosis Research, School of Primary Care Research, Community and Social Care, Keele University, Keele, UK
| | - Rafael Perera
- Nuffield Department of Primary Care, University of Oxford, Oxford, UK
| | - Walter Emil Haefeli
- Department of Clinical Pharmacology & Pharmacoepidemiology, Heidelberg University, Heidelberg, Baden-Württemberg, Germany
| | - Paul Glasziou
- Centre for Research in Evidence-Based Practice, Bond University, Robina, Queensland, Australia
| | - Christiane Muth
- Institute of General Practice, Goethe University, Frankfurt am Main, Hessen, Germany
- Department of General Practice and Family Medicine, Medical Faculty OWL, University of Bielefeld, Bielefeld, Germany
| |
Collapse
|
614
|
Wessler BS, Nelson J, Park JG, McGinnes H, Gulati G, Brazil R, Van Calster B, van Klaveren D, Venema E, Steyerberg E, Paulus JK, Kent DM. External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature. Circ Cardiovasc Qual Outcomes 2021; 14:e007858. [PMID: 34340529 PMCID: PMC8366535 DOI: 10.1161/circoutcomes.121.007858] [Citation(s) in RCA: 28] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
Abstract
BACKGROUND There are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested, and how well they generally perform has not been broadly evaluated. METHODS A SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts Predictive Analytics and Comparative Effectiveness CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data. RESULTS We identified 2030 external validations of 1382 CPMs. Eight hundred seven (58%) of the CPMs in the Registry have never been externally validated. On average, there were 1.5 validations per CPM (range, 0-94). The median external validation area under the receiver operating characteristic curve was 0.73 (25th-75th percentile [interquartile range (IQR)], 0.66-0.79), representing a median percent decrease in discrimination of -11.1% (IQR, -32.4% to +2.7%) compared with performance on derivation data. 81% (n=1333) of validations reporting area under the receiver operating characteristic curve showed discrimination below that reported in the derivation dataset. 53% (n=983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was -3.7% (IQR, -13.2 to 3.1) for closely related validations (n=123), -9.0 (IQR, -27.6 to 3.9) for related validations (n=862), and -17.2% (IQR, -42.3 to 0) for distantly related validations (n=717; P<0.001). CONCLUSIONS Many published cardiovascular CPMs have never been externally validated, and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings.
Collapse
Affiliation(s)
- Benjamin S Wessler
- Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA.,Division of Cardiology (B.S.W., G.G.), Tufts Medical Center, Boston, MA
| | - Jason Nelson
- Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
| | - Jinny G Park
- Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
| | - Hannah McGinnes
- Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
| | - Gaurav Gulati
- Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA.,Division of Cardiology (B.S.W., G.G.), Tufts Medical Center, Boston, MA
| | - Riley Brazil
- Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
| | - Ben Van Calster
- KU Leuven, Department of Development and Regeneration, Belgium (B.V.C.)
| | - David van Klaveren
- Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA.,Department of Biomedical Data Sciences (D.v.K.), Leiden University Medical Centre, Netherlands
| | - Esmee Venema
- Department of Public Health (E.V., E.S.), Erasmus MC University Medical Center, Rotterdam, the Netherlands.,Department of Neurology (E.V.), Erasmus MC University Medical Center, Rotterdam, the Netherlands
| | - Ewout Steyerberg
- Department of Biomedical Data Sciences (E.S.), Leiden University Medical Centre, Netherlands.,Department of Public Health (E.V., E.S.), Erasmus MC University Medical Center, Rotterdam, the Netherlands
| | - Jessica K Paulus
- Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
| | - David M Kent
- Predictive Analytics and Comparative Effectiveness (PACE) (B.S.W., J.N., J.G.P., H.G., G.G., R.B., D.v.K., J.K.P., D.M.K.), Tufts Medical Center, Boston, MA
| |
Collapse
|
615
|
Van Calster B, Collins GS, Wilkinson J. Methodological Issues in the Development of a Prediction Tool for First-Trimester Outcomes. JOURNAL OF ULTRASOUND IN MEDICINE : OFFICIAL JOURNAL OF THE AMERICAN INSTITUTE OF ULTRASOUND IN MEDICINE 2021; 40:1731-1732. [PMID: 33098327 DOI: 10.1002/jum.15543] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/04/2020] [Accepted: 09/08/2020] [Indexed: 06/11/2023]
Affiliation(s)
- Ben Van Calster
- Department of Development and Regeneration, Katholieke Universiteit Leuven, Leuven, Belgium
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands
| | - Gary S Collins
- Nuffield Department of Orthopedics, Rheumatology, and Musculoskeletal Sciences, Centre for Statistics in Medicine, University of Oxford, Oxford, UK
| | - Jack Wilkinson
- Center for Biostatistics, Division of Population Health, Health Services Research, and Primary Care, University of Manchester, Manchester, UK
| |
Collapse
|
616
|
Tseregounis IE, Henry SG. Assessing opioid overdose risk: a review of clinical prediction models utilizing patient-level data. Transl Res 2021; 234:74-87. [PMID: 33762186 PMCID: PMC8217215 DOI: 10.1016/j.trsl.2021.03.012] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/18/2020] [Revised: 02/24/2021] [Accepted: 03/16/2021] [Indexed: 12/23/2022]
Abstract
Drug, and specifically opioid-related, overdoses remain a major public health problem in the United States. Multiple studies have examined individual risk factors associated with overdose risk, but research developing clinical risk prediction tools for overdose has only emerged in the last few years. We conducted a comprehensive review of the literature on patient-level factors associated with opioid-related overdose risk, with an emphasis on clinical risk prediction models for opioid-related overdose in the United States. Studies that developed and/or validated clinical prediction models were closely reviewed and evaluated to determine the state of the field. We identified 12 studies that reported risk prediction models for opioid-related overdose risk. Published models were developed from a variety of data sources, including Veterans Health Administration data, Medicare data, commercial insurance data, and statewide linked datasets. Studies reported model performance using measures of discrimination, usually at good-to-excellent levels, though they did not always assess calibration. C-statistics were better for models that included clinical predictors (c-statistics: 0.75-0.95) compared to models without them (c-statistics: 0.69-0.82). External validation of models was rare, and we found no studies evaluating implementation of models or risk prediction tools into clinical practice. A common feature of these models was a high rate of false positives, largely because opioid-related overdose is rare in the general population. Thus, efforts to implement prediction models into practice should take into account that published models overestimate overdose risk for many low-risk patients. Future prediction models assessing overdose risk should employ external validation and address model calibration. In order to translate findings from prediction models into clinical public health benefit, future studies should focus on developing clinical prediction tools based on prediction models, implementing these tools into clinical practice, and evaluating the impact of these models on treatment decisions, patient outcomes, and, ultimately, opioid overdose rates.
Collapse
Affiliation(s)
- Iraklis Erik Tseregounis
- Center for Healthcare Policy and Research, University of California Davis, Sacramento, California, USA
| | - Stephen G Henry
- Center for Healthcare Policy and Research, University of California Davis, Sacramento, California, USA; Department of Internal Medicine, University of California Davis, Sacramento, California, USA.
| |
Collapse
|
617
|
Yun H, Choi J, Park JH. XGBoost Algorithm Prediction of Critical Care Outcome for Adult Patients Presenting to Emergency Department Using Initial Triage Information. JMIR Med Inform 2021; 9:e30770. [PMID: 34346889 PMCID: PMC8491120 DOI: 10.2196/30770] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/28/2021] [Revised: 06/27/2021] [Accepted: 07/27/2021] [Indexed: 12/23/2022] Open
Abstract
Background The emergency department (ED) triage system to classify and prioritize patients from high risk to less urgent continues to be a challenge. Objective This study, comprising 80,433 patients, aims to develop a machine learning algorithm prediction model of critical care outcomes for adult patients using information collected during ED triage and compare the performance with that of the baseline model using the Korean Triage and Acuity Scale (KTAS). Methods To predict the need for critical care, we used 13 predictors from triage information: age, gender, mode of ED arrival, the time interval between onset and ED arrival, reason of ED visit, chief complaints, systolic blood pressure, diastolic blood pressure, pulse rate, respiratory rate, body temperature, oxygen saturation, and level of consciousness. The baseline model with KTAS was developed using logistic regression, and the machine learning model with 13 variables was generated using extreme gradient boosting (XGB) and deep neural network (DNN) algorithms. The discrimination was measured by the area under the receiver operating characteristic (AUROC) curve. The ability of calibration with Hosmer–Lemeshow test and reclassification with net reclassification index were evaluated. The calibration plot and partial dependence plot were used in the analysis. Results The AUROC of the model with the full set of variables (0.833-0.861) was better than that of the baseline model (0.796). The XGB model of AUROC 0.861 (95% CI 0.848-0.874) showed a higher discriminative performance than the DNN model of 0.833 (95% CI 0.819-0.848). The XGB and DNN models proved better reclassification than the baseline model with a positive net reclassification index. The XGB models were well-calibrated (Hosmer-Lemeshow test; P>.05); however, the DNN showed poor calibration power (Hosmer-Lemeshow test; P<.001). We further interpreted the nonlinear association between variables and critical care prediction. Conclusions Our study demonstrated that the performance of the XGB model using initial information at ED triage for predicting patients in need of critical care outperformed the conventional model with KTAS.
Collapse
Affiliation(s)
- Hyoungju Yun
- Interdisciplinary Program of Medical Informatics, College of Medicine, Seoul National University, Seoul, KR
| | - Jinwook Choi
- Interdisciplinary Program of Medical Informatics, College of Medicine, Seoul National University, Seoul, KR.,Department of Biomedical Engineering, College of Medicine, Seoul National University, Seoul, KR.,Institute of Medical and Biological Engineering,, Seoul National University Medical Research Center, 103 Daehak-Ro, Jongno-Gu, Seoul, KR
| | - Jeong Ho Park
- Department of Emergency Medicine, College of Medicine, Seoul National University, Seoul, KR.,Laboratory of Emergency Medical Services, Seoul National University Hospital Biomedical Research Institute, Seoul, KR
| |
Collapse
|
618
|
Althouse AD, Below JE, Claggett BL, Cox NJ, de Lemos JA, Deo RC, Duval S, Hachamovitch R, Kaul S, Keith SW, Secemsky E, Teixeira-Pinto A, Roger VL. Recommendations for Statistical Reporting in Cardiovascular Medicine: A Special Report From the American Heart Association. Circulation 2021; 144:e70-e91. [PMID: 34032474 DOI: 10.1161/circulationaha.121.055393] [Citation(s) in RCA: 39] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/01/2023]
Abstract
Statistical analyses are a crucial component of the biomedical research process and are necessary to draw inferences from biomedical research data. The application of sound statistical methodology is a prerequisite for publication in the American Heart Association (AHA) journal portfolio. The objective of this document is to summarize key aspects of statistical reporting that might be most relevant to the authors, reviewers, and readership of AHA journals. The AHA Scientific Publication Committee convened a task force to inventory existing statistical standards for publication in biomedical journals and to identify approaches suitable for the AHA journal portfolio. The experts on the task force were selected by the AHA Scientific Publication Committee, who identified 12 key topics that serve as the section headers for this document. For each topic, the members of the writing group identified relevant references and evaluated them as a resource to make the standards summarized herein. Each section was independently reviewed by an expert reviewer who was not part of the task force. Expert reviewers were also permitted to comment on other sections if they chose. Differences of opinion were adjudicated by consensus. The standards presented in this report are intended to serve as a guide for high-quality reporting of statistical analyses methods and results.
Collapse
Affiliation(s)
- Andrew D Althouse
- Center for Research on Health Care Data Center, Division of General Internal Medicine, University of Pittsburgh, PA (A.D.A.)
| | - Jennifer E Below
- Vanderbilt Genetics Institute, Vanderbilt University Medical Center, Nashville, TN (J.E.B., N.J.C.)
| | - Brian L Claggett
- Division of Cardiovascular Medicine, Brigham and Women's Hospital, Boston, MA (B.L.C., R.C.D.)
| | - Nancy J Cox
- Vanderbilt Genetics Institute, Vanderbilt University Medical Center, Nashville, TN (J.E.B., N.J.C.)
| | - James A de Lemos
- Division of Cardiology, University of Texas Southwestern Medical Center, Dallas (J.A.d.L.)
| | - Rahul C Deo
- Division of Cardiovascular Medicine, Brigham and Women's Hospital, Boston, MA (B.L.C., R.C.D.)
| | - Sue Duval
- Cardiovascular Division, University of Minnesota Medical School, Minneapolis (S.D.)
| | - Rory Hachamovitch
- Department of Cardiovascular Medicine, Heart and Vascular Institute, Cleveland Clinic Foundation, OH (R.H.)
| | - Sanjay Kaul
- Department of Cardiology, Cedars-Sinai Medical Center, and the David Geffen School of Medicine, University of California, Los Angeles (S.K.)
| | - Scott W Keith
- Division of Biostatistics, Department of Pharmacology and Experimental Therapeutics, Sidney Kimmel Medical College of Thomas Jefferson University, Philadelphia, PA (S.W.K.)
| | - Eric Secemsky
- Richard A. and Susan F. Smith Center for Outcomes Research in Cardiology, Division of Cardiology, Department of Medicine, Beth Israel Deaconess Medical Center, Boston, MA (E.S.)
| | - Armando Teixeira-Pinto
- School of Public Health, Faculty of Medicine and Health, University of Sydney, Australia (A.T.-P.)
| | - Veronique L Roger
- Department of Cardiovascular Diseases Medicine, Mayo Clinic College of Medicine, Rochester, MN (V.L.R.)
- now with Epidemiology and Community Health Branch National Heart, Lung and Blood Institute, National Institutes of Health, Bethesda, MD (V.L.R.)
| |
Collapse
|
619
|
Abstract
Machine learning can be used to make sense of healthcare data. Probabilistic machine learning models help provide a complete picture of observed data in healthcare. In this review, we examine how probabilistic machine learning can advance healthcare. We consider challenges in the predictive model building pipeline where probabilistic models can be beneficial, including calibration and missing data. Beyond predictive models, we also investigate the utility of probabilistic machine learning models in phenotyping, in generative models for clinical use cases, and in reinforcement learning.
Collapse
Affiliation(s)
- Irene Y Chen
- Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139, USA;
| | | | - Marzyeh Ghassemi
- Vector Institute, Toronto, Ontario M5G 1M1, Canada; .,Institute for Medical and Evaluative Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139, USA
| | - Rajesh Ranganath
- Department of Computer Science, Courant Institute, New York University, New York, NY 10012, USA.,Center for Data Science, New York University, New York, NY 10012, USA.,Department of Population Health, New York University Grossman School of Medicine, New York, NY 10016, USA
| |
Collapse
|
620
|
Rinderknecht MD, Klopfenstein Y. Predicting critical state after COVID-19 diagnosis: model development using a large US electronic health record dataset. NPJ Digit Med 2021; 4:113. [PMID: 34285316 PMCID: PMC8292360 DOI: 10.1038/s41746-021-00482-9] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2020] [Accepted: 06/21/2021] [Indexed: 01/08/2023] Open
Abstract
As the COVID-19 pandemic is challenging healthcare systems worldwide, early identification of patients with a high risk of complication is crucial. We present a prognostic model predicting critical state within 28 days following COVID-19 diagnosis trained on data from US electronic health records (IBM Explorys), including demographics, comorbidities, symptoms, and hospitalization. Out of 15753 COVID-19 patients, 2050 went into critical state or deceased. Non-random train-test splits by time were repeated 100 times and led to a ROC AUC of 0.861 [0.838, 0.883] and a precision-recall AUC of 0.434 [0.414, 0.485] (median and interquartile range). The interpretability analysis confirmed evidence on major risk factors (e.g., older age, higher BMI, male gender, diabetes, and cardiovascular disease) in an efficient way compared to clinical studies, demonstrating the model validity. Such personalized predictions could enable fine-graded risk stratification for optimized care management.
Collapse
|
621
|
Park SY, Park JE, Kim H, Park SH. Review of Statistical Methods for Evaluating the Performance of Survival or Other Time-to-Event Prediction Models (from Conventional to Deep Learning Approaches). Korean J Radiol 2021; 22:1697-1707. [PMID: 34269532 PMCID: PMC8484151 DOI: 10.3348/kjr.2021.0223] [Citation(s) in RCA: 24] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2021] [Revised: 04/29/2021] [Accepted: 05/17/2021] [Indexed: 11/15/2022] Open
Abstract
The recent introduction of various high-dimensional modeling methods, such as radiomics and deep learning, has created a much greater diversity in modeling approaches for survival prediction (or, more generally, time-to-event prediction). The newness of the recent modeling approaches and unfamiliarity with the model outputs may confuse some researchers and practitioners about the evaluation of the performance of such models. Methodological literacy to critically appraise the performance evaluation of the models and, ideally, the ability to conduct such an evaluation would be needed for those who want to develop models or apply them in practice. This article intends to provide intuitive, conceptual, and practical explanations of the statistical methods for evaluating the performance of survival prediction models with minimal usage of mathematical descriptions. It covers from conventional to deep learning methods, and emphasis has been placed on recent modeling approaches. This review article includes straightforward explanations of C indices (Harrell's C index, etc.), time-dependent receiver operating characteristic curve analysis, calibration plot, other methods for evaluating the calibration performance, and Brier score.
Collapse
Affiliation(s)
- Seo Young Park
- Department of Statistics and Data Science, Korea National Open University, Seoul, Korea
| | - Ji Eun Park
- Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
| | - Hyungjin Kim
- Department of Radiology, Seoul National University College of Medicine, Seoul National University Hospital, Seoul, Korea
| | - Seong Ho Park
- Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea.
| |
Collapse
|
622
|
Sun Y, Zhang L, Dong D, Li X, Wang J, Yin C, Poon LC, Tian J, Wu Q. Application of an individualized nomogram in first-trimester screening for trisomy 21. ULTRASOUND IN OBSTETRICS & GYNECOLOGY : THE OFFICIAL JOURNAL OF THE INTERNATIONAL SOCIETY OF ULTRASOUND IN OBSTETRICS AND GYNECOLOGY 2021; 58:56-66. [PMID: 32438493 PMCID: PMC8362158 DOI: 10.1002/uog.22087] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/20/2019] [Revised: 05/03/2020] [Accepted: 05/07/2020] [Indexed: 05/17/2023]
Abstract
OBJECTIVES To develop and validate a nomogram based on fetal nuchal translucency thickness (NT) and ultrasonographic facial markers for screening for trisomy 21 in the first trimester of pregnancy. METHODS This was a retrospective case-control study using stored two-dimensional midsagittal fetal profile images captured at 11 + 0 to 13 + 6 weeks' gestation in singleton pregnancies. We included images from 302 trisomy-21 pregnancies and 322 euploid pregnancies. Cases were divided into a training set (200 euploid + 200 with trisomy 21) and a validation set (122 euploid + 102 with trisomy 21) at a ratio of approximately 2:1. For each, the maternal age, gestational age, fetal NT and karyotype were noted, and 12 ultrasonographic fetal facial markers were measured. The least absolute shrinkage and selection operator (LASSO) method and multivariable analysis were used to select automatically the discriminative markers. Logistic regression was used to develop a LASSO model, based on the selected markers, to screen for trisomy 21 in the first trimester of pregnancy. Furthermore, 60 of the 624 images were selected randomly as a retest set to evaluate the model's robustness. The predictive performance of screening for trisomy 21 of a model based on fetal NT and maternal age and of the LASSO model was assessed using the area under the receiver-operating-characteristics curve (AUC). A nomogram was developed as an individualized tool to predict patient-specific probability for trisomy 21, which is a more visual presentation of the LASSO model. The performance of the nomogram was assessed using the C-index and calibration curve. RESULTS Into the LASSO model were incorporated eight markers, including fetal NT, prenasal-thickness-to-nasal-bone-length ratio, facial profile line, frontomaxillary facial angle, frontonasal facial angle, mandibulomaxillary facial angle, maxilla-nasion-mandible angle and d2 (distance between the anterior edge of the prefrontal skin and the mandibulomaxillary line) (all P < 0.05). The AUCs of the LASSO model for screening for trisomy 21 were 0.983 (95% CI, 0.971-0.994) in the training set and 0.979 (95% CI, 0.966-0.993) in the validation set, and these were higher than the AUCs of all eight individual ultrasonographic markers included in the model. The AUC of the LASSO model in the retest set was 0.997 (95% CI, 0.990-1.000), indicating good robustness of the LASSO model. The AUC of the LASSO model was significantly higher than that of the model based on fetal NT and maternal age in both training and validation sets (P < 0.001 for both). The nomogram of the LASSO model showed good discrimination of trisomy 21, with C-indices of 0.983 in the training set and 0.981 in the validation set. CONCLUSIONS We present an individualized nomogram which incorporates fetal NT and a series of ultrasonographic facial profile markers selected by the LASSO method and multivariable analysis. This nomogram can potentially be utilized as a convenient and effective tool in screening for trisomy 21 in the first trimester of pregnancy. © 2020 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.
Collapse
Affiliation(s)
- Y. Sun
- Beijing Obstetrics and Gynecology HospitalCapital Medical UniversityBeijingChina
- Beijing Maternal and Child Health Care HospitalBeijingChina
| | - L. Zhang
- School of Artificial IntelligenceUniversity of Chinese Academy of SciencesBeijingChina
- CAS Key Laboratory of Molecular Imaging, Institute of AutomationChinese Academy of SciencesBeijingChina
| | - D. Dong
- School of Artificial IntelligenceUniversity of Chinese Academy of SciencesBeijingChina
- CAS Key Laboratory of Molecular Imaging, Institute of AutomationChinese Academy of SciencesBeijingChina
| | - X. Li
- Beijing Obstetrics and Gynecology HospitalCapital Medical UniversityBeijingChina
| | - J. Wang
- Beijing Obstetrics and Gynecology HospitalCapital Medical UniversityBeijingChina
- Beijing Maternal and Child Health Care HospitalBeijingChina
| | - C. Yin
- Beijing Obstetrics and Gynecology HospitalCapital Medical UniversityBeijingChina
- Beijing Maternal and Child Health Care HospitalBeijingChina
| | - L. C. Poon
- Department of Obstetrics and GynaecologyChinese University of Hong KongHong Kong
| | - J. Tian
- CAS Key Laboratory of Molecular Imaging, Institute of AutomationChinese Academy of SciencesBeijingChina
- Beijing Advanced Innovation Center for Big Data‐Based Precision MedicineSchool of Medicine, Beihang UniversityBeijingChina
| | - Q. Wu
- Beijing Obstetrics and Gynecology HospitalCapital Medical UniversityBeijingChina
- Beijing Maternal and Child Health Care HospitalBeijingChina
| |
Collapse
|
623
|
Nguyen TV. Personalized fracture risk assessment: where are we at? Expert Rev Endocrinol Metab 2021; 16:191-200. [PMID: 33982611 DOI: 10.1080/17446651.2021.1924672] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/07/2021] [Accepted: 04/26/2021] [Indexed: 10/21/2022]
Abstract
Introduction: Osteoporotic fracture imposes a significant health care burden globally. Personalized assessment of fracture risk can potentially guide treatment decisions. Over the past decade, a number of risk prediction models, including the Garvan Fracture Risk Calculator (Garvan) and FRAX®, have been developed and implemented in clinical practice. Areas covered: This article reviews recent development and validation results concerning the prognostic performance of the two tools. The main areas of review are the need for personalized fracture risk prediction, purposes of risk prediction, predictive performance in terms of discrimination and calibration, concordance between the Garvan and FRAX tools, genetic profiling for improving predictive performance, and treatment thresholds. In some validation studies, FRAX tended to underestimate fracture by as high as 50%. Studies have shown that the predicted risk from the Garvan tool is highly concordant with clinical decision. Expert opinion: Although there are some discrepancy in fracture risk prediction between Garvan and FRAX, both tools are valid and can aid patients and doctors communicate about risk and make informed decision. The ideal of personalized risk assessment for osteoporosis patients will be realized through the incorporation of genetic profiling into existing fracture risk assessment tools.
Collapse
Affiliation(s)
- Tuan V Nguyen
- Healthy Ageing Theme, Garvan Institute of Medical Research Darlinghurst Australia
- St Vincent's Clinical School, UNSW Sydney, Sydney Australia
- School of Biomedical Engineering, University of Technology Sydney Sydney Australia
| |
Collapse
|
624
|
Zhu Y, Zhang J, Wang G, Yao R, Ren C, Chen G, Jin X, Guo J, Liu S, Zheng H, Chen Y, Guo Q, Li L, Du B, Xi X, Li W, Huang H, Li Y, Yu Q. Machine Learning Prediction Models for Mechanically Ventilated Patients: Analyses of the MIMIC-III Database. Front Med (Lausanne) 2021; 8:662340. [PMID: 34277655 PMCID: PMC8280779 DOI: 10.3389/fmed.2021.662340] [Citation(s) in RCA: 23] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2021] [Accepted: 06/01/2021] [Indexed: 01/27/2023] Open
Abstract
Background: Mechanically ventilated patients in the intensive care unit (ICU) have high mortality rates. There are multiple prediction scores, such as the Simplified Acute Physiology Score II (SAPS II), Oxford Acute Severity of Illness Score (OASIS), and Sequential Organ Failure Assessment (SOFA), widely used in the general ICU population. We aimed to establish prediction scores on mechanically ventilated patients with the combination of these disease severity scores and other features available on the first day of admission. Methods: A retrospective administrative database study from the Medical Information Mart for Intensive Care (MIMIC-III) database was conducted. The exposures of interest consisted of the demographics, pre-ICU comorbidity, ICU diagnosis, disease severity scores, vital signs, and laboratory test results on the first day of ICU admission. Hospital mortality was used as the outcome. We used the machine learning methods of k-nearest neighbors (KNN), logistic regression, bagging, decision tree, random forest, Extreme Gradient Boosting (XGBoost), and neural network for model establishment. A sample of 70% of the cohort was used for the training set; the remaining 30% was applied for testing. Areas under the receiver operating characteristic curves (AUCs) and calibration plots would be constructed for the evaluation and comparison of the models' performance. The significance of the risk factors was identified through models and the top factors were reported. Results: A total of 28,530 subjects were enrolled through the screening of the MIMIC-III database. After data preprocessing, 25,659 adult patients with 66 predictors were included in the model analyses. With the training set, the models of KNN, logistic regression, decision tree, random forest, neural network, bagging, and XGBoost were established and the testing set obtained AUCs of 0.806, 0.818, 0.743, 0.819, 0.780, 0.803, and 0.821, respectively. The calibration curves of all the models, except for the neural network, performed well. The XGBoost model performed best among the seven models. The top five predictors were age, respiratory dysfunction, SAPS II score, maximum hemoglobin, and minimum lactate. Conclusion: The current study indicates that models with the risk of factors on the first day could be successfully established for predicting mortality in ventilated patients. The XGBoost model performs best among the seven machine learning models.
Collapse
Affiliation(s)
- Yibing Zhu
- Medical Research and Biometrics Center, National Center for Cardiovascular Diseases, Fuwai Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China.,Department of Emergency, Guang'anmen Hospital, China Academy of Chinese Medical Sciences, Beijing, China
| | - Jin Zhang
- School of Economics and Management, Beijing Institute of Technology, Beijing, China
| | - Guowei Wang
- School of Computer Science and Technology, Wuhan University of Technology, Wuhan, China
| | - Renqi Yao
- Department of Burn Surgery, The First Affiliated Hospital of Naval Medical University, Shanghai, China.,Translational Medicine Research Center, Fourth Medical Center and Medical Innovation Research Division of the Chinese People's Liberation Army (PLA) General Hospital, Beijing, China
| | - Chao Ren
- Translational Medicine Research Center, Fourth Medical Center and Medical Innovation Research Division of the Chinese People's Liberation Army (PLA) General Hospital, Beijing, China
| | - Ge Chen
- Medical Research and Biometrics Center, National Center for Cardiovascular Diseases, Fuwai Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Xin Jin
- Yidu Cloud Technology Inc., Beijing, China
| | - Junyang Guo
- Beijing Big Eye Xing Tu Culture Media Co., Ltd., Beijing, China
| | - Shi Liu
- School of Information Science and Engineering, Hebei North University, Shijiazhuang, China
| | - Hua Zheng
- Medical ICU, Peking Union Medical College Hospital, Peking Union Medical College and Chinese Academy of Medical Sciences, Beijing, China
| | - Yan Chen
- Medical ICU, Peking Union Medical College Hospital, Peking Union Medical College and Chinese Academy of Medical Sciences, Beijing, China
| | - Qianqian Guo
- Department of Anesthesiology, Peking University Shougang Hospital, Beijing, China
| | - Lin Li
- School of Computer Science and Technology, Wuhan University of Technology, Wuhan, China
| | - Bin Du
- Medical ICU, Peking Union Medical College Hospital, Peking Union Medical College and Chinese Academy of Medical Sciences, Beijing, China
| | - Xiuming Xi
- Department of Critical Care Medicine, Fuxing Hospital, Capital Medical University, Beijing, China
| | - Wei Li
- Medical Research and Biometrics Center, National Center for Cardiovascular Diseases, Fuwai Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Huibin Huang
- Department of Critical Care Medicine, Beijing Tsinghua Changgung Hospital, School of Clinical Medicine, Tsinghua University, Beijing, China
| | - Yang Li
- Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, China
| | - Qian Yu
- Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, China
| |
Collapse
|
625
|
Kragh MF, Karstoft H. Embryo selection with artificial intelligence: how to evaluate and compare methods? J Assist Reprod Genet 2021; 38:1675-1689. [PMID: 34173914 PMCID: PMC8324599 DOI: 10.1007/s10815-021-02254-6] [Citation(s) in RCA: 39] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2021] [Accepted: 06/02/2021] [Indexed: 12/19/2022] Open
Abstract
Embryo selection within in vitro fertilization (IVF) is the process of evaluating qualities of fertilized oocytes (embryos) and selecting the best embryo(s) available within a patient cohort for subsequent transfer or cryopreservation. In recent years, artificial intelligence (AI) has been used extensively to improve and automate the embryo ranking and selection procedure by extracting relevant information from embryo microscopy images. The AI models are evaluated based on their ability to identify the embryo(s) with the highest chance(s) of achieving a successful pregnancy. Whether such evaluations should be based on ranking performance or pregnancy prediction, however, seems to divide studies. As such, a variety of performance metrics are reported, and comparisons between studies are often made on different outcomes and data foundations. Moreover, superiority of AI methods over manual human evaluation is often claimed based on retrospective data, without any mentions of potential bias. In this paper, we provide a technical view on some of the major topics that divide how current AI models are trained, evaluated and compared. We explain and discuss the most common evaluation metrics and relate them to the two separate evaluation objectives, ranking and prediction. We also discuss when and how to compare AI models across studies and explain in detail how a selection bias is inevitable when comparing AI models against current embryo selection practice in retrospective cohort studies.
Collapse
Affiliation(s)
- Mikkel Fly Kragh
- Department of Electrical and Computer Engineering, Aarhus University, Aarhus N, Denmark.
- Vitrolife A/S, Viby J, Denmark.
| | - Henrik Karstoft
- Department of Electrical and Computer Engineering, Aarhus University, Aarhus N, Denmark
| |
Collapse
|
626
|
Snell KIE, Archer L, Ensor J, Bonnett LJ, Debray TPA, Phillips B, Collins GS, Riley RD. External validation of clinical prediction models: simulation-based sample size calculations were more reliable than rules-of-thumb. J Clin Epidemiol 2021; 135:79-89. [PMID: 33596458 PMCID: PMC8352630 DOI: 10.1016/j.jclinepi.2021.02.011] [Citation(s) in RCA: 49] [Impact Index Per Article: 16.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2020] [Revised: 12/14/2020] [Accepted: 02/09/2021] [Indexed: 01/01/2023]
Abstract
INTRODUCTION Sample size "rules-of-thumb" for external validation of clinical prediction models suggest at least 100 events and 100 non-events. Such blanket guidance is imprecise, and not specific to the model or validation setting. We investigate factors affecting precision of model performance estimates upon external validation, and propose a more tailored sample size approach. METHODS Simulation of logistic regression prediction models to investigate factors associated with precision of performance estimates. Then, explanation and illustration of a simulation-based approach to calculate the minimum sample size required to precisely estimate a model's calibration, discrimination and clinical utility. RESULTS Precision is affected by the model's linear predictor (LP) distribution, in addition to number of events and total sample size. Sample sizes of 100 (or even 200) events and non-events can give imprecise estimates, especially for calibration. The simulation-based calculation accounts for the LP distribution and (mis)calibration in the validation sample. Application identifies 2430 required participants (531 events) for external validation of a deep vein thrombosis diagnostic model. CONCLUSION Where researchers can anticipate the distribution of the model's LP (eg, based on development sample, or a pilot study), a simulation-based approach for calculating sample size for external validation offers more flexibility and reliability than rules-of-thumb.
Collapse
Affiliation(s)
- Kym I E Snell
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, Staffordshire, United Kingdom.
| | - Lucinda Archer
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, Staffordshire, United Kingdom
| | - Joie Ensor
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, Staffordshire, United Kingdom
| | - Laura J Bonnett
- Department of Biostatistics, University of Liverpool, Liverpool, United Kingdom
| | - Thomas P A Debray
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Bob Phillips
- Centre for Reviews and Dissemination, University of York, York, United Kingdom
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, United Kingdom; NIHR Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford, United Kingdom
| | - Richard D Riley
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, Staffordshire, United Kingdom
| |
Collapse
|
627
|
Migriño JR, Batangan ARU. Using machine learning to create a decision tree model to predict outcomes of COVID-19 cases in the Philippines. Western Pac Surveill Response J 2021; 12:56-64. [PMID: 34703636 PMCID: PMC8521127 DOI: 10.5365/wpsar.2021.12.3.831] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022] Open
Abstract
OBJECTIVE The aim of this study was to create a decision tree model with machine learning to predict the outcomes of COVID-19 cases from data publicly available in the Philippine Department of Health (DOH) COVID Data Drop. METHODS The study design was a cross-sectional records review of the DOH COVID Data Drop for 25 August 2020. Resolved cases that had either recovered or died were used as the final data set. Machine learning processes were used to generate, train and validate a decision tree model. RESULTS A list of 132 939 resolved COVID-19 cases was used. The notification rates and case fatality rates were higher among males (145.67 per 100 000 and 2.46%, respectively). Most COVID-19 cases were clustered among people of working age, and older cases had higher case fatality rates. The majority of cases were from the National Capital Region (590.20 per 100 000), and the highest case fatality rate (5.83%) was observed in Region VII. The decision tree model prioritized age and history of hospital admission as predictors of mortality. The model had high accuracy (81.42%), sensitivity (81.65%), specificity (81.41%) and area under the curve (0.876) but a poor F-score (16.74%). DISCUSSION The model predicted higher case fatality rates among older people. For cases aged > 51 years, a history of hospital admission increased the probability of COVID-19-related death. We recommend that more comprehensive primary COVID-19 data sets be used to create more robust prognostic models.
Collapse
Affiliation(s)
- Julius R. Migriño
- San Beda University College of Medicine, Manila, Philippines
- School of Medicine and Public Health, Ateneo de Manila University, Pasig City, Philippines
| | | |
Collapse
|
628
|
Imai T, Nishimoto A, Kubota S, Nakazawa M, Uzawa N. Predictive scoring model for inferior alveolar nerve injury after lower third molar removal based on features of cone-beam computed tomography image. JOURNAL OF STOMATOLOGY, ORAL AND MAXILLOFACIAL SURGERY 2021; 123:136-141. [PMID: 34171526 DOI: 10.1016/j.jormas.2021.06.007] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/21/2021] [Revised: 06/04/2021] [Accepted: 06/21/2021] [Indexed: 12/25/2022]
Abstract
INTRODUCTION This study aimed to construct a predictive scoring system for inferior alveolar nerve injury (IANI) following lower third molar (LM3) surgery based on cone-beam computed tomography (CBCT) images. MATERIAL AND METHODS Of the 1573 patients who underwent LM3 removal following the CBCT, 39 with IANI and 457 randomly selected patients without IANI were enrolled. We collected information regarding the demographic characteristics of the patients, surgical situations, and inferior alveolar canal (IAC)-related CBCT factors. The association with IANI-risk was evaluated with a backward stepwise logistic regression model as per the Akaike information criterion. Scoring models' abilities of discrimination (area under the curve) and calibration (Hosmer-Lemeshow test and calibration plots) were assessed, followed by evaluation of the clinical usefulness using decision curve analysis. RESULTS As per the multivariate analysis, the coronal positioned IAC on the enlarged root (odds ratio [OR], 3.78; P = 0.001), the length of perforated IAC (>3.4 mm) (OR, 3.05; P = 0.012), lingual/inter-radicular position of the IAC (OR, 3.96; P = 0.001), multiple roots closed to the perforated IAC (OR, 2.78; P = 0.025), and age >30 y (OR, 2.31; P = 0.076) were identified in the extended scoring model ranging from 0 to 12. This model was compared with our previously constructed baseline model that involved the latter three variables mentioned above, resulting in superior performance than that of the baseline model. CONCLUSION The extended model would be a useful tool for reliable determination of the preoperative probability of IANI.
Collapse
Affiliation(s)
- Tomoaki Imai
- Department of Oral and Maxillofacial Surgery II, Osaka University Graduate School of Dentistry, 1-8 Yamadaoka, Suita, Osaka 565-0871, Japan.
| | - Ayano Nishimoto
- Department of Oral and Maxillofacial Surgery II, Osaka University Graduate School of Dentistry, 1-8 Yamadaoka, Suita, Osaka 565-0871, Japan
| | - Seiko Kubota
- Department of Oral and Maxillofacial Surgery II, Osaka University Graduate School of Dentistry, 1-8 Yamadaoka, Suita, Osaka 565-0871, Japan
| | - Mitsuhiro Nakazawa
- Department of Oral and Maxillofacial Surgery II, Osaka University Graduate School of Dentistry, 1-8 Yamadaoka, Suita, Osaka 565-0871, Japan
| | - Narikazu Uzawa
- Department of Oral and Maxillofacial Surgery II, Osaka University Graduate School of Dentistry, 1-8 Yamadaoka, Suita, Osaka 565-0871, Japan
| |
Collapse
|
629
|
Moran JL, Santamaria JD, Duke GJ. Modelling hospital outcome: problems with endogeneity. BMC Med Res Methodol 2021; 21:124. [PMID: 34154530 PMCID: PMC8215743 DOI: 10.1186/s12874-021-01251-8] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2021] [Accepted: 03/09/2021] [Indexed: 12/23/2022] Open
Abstract
Background Mortality modelling in the critical care paradigm traditionally uses logistic regression, despite the availability of estimators commonly used in alternate disciplines. Little attention has been paid to covariate endogeneity and the status of non-randomized treatment assignment. Using a large registry database, various binary outcome modelling strategies and methods to account for covariate endogeneity were explored. Methods Patient mortality data was sourced from the Australian & New Zealand Intensive Society Adult Patient Database for 2016. Hospital mortality was modelled using logistic, probit and linear probability (LPM) models with intensive care (ICU) providers as fixed (FE) and random (RE) effects. Model comparison entailed indices of discrimination and calibration, information criteria (AIC and BIC) and binned residual analysis. Suspect covariate and ventilation treatment assignment endogeneity was identified by correlation between predictor variable and hospital mortality error terms, using the Stata™ “eprobit” estimator. Marginal effects were used to demonstrate effect estimate differences between probit and “eprobit” models. Results The cohort comprised 92,693 patients from 124 intensive care units (ICU) in calendar year 2016. Patients mean age was 61.8 (SD 17.5) years, 41.6% were female and APACHE III severity of illness score 54.5(25.6); 43.7% were ventilated. Of the models considered in predicting hospital mortality, logistic regression (with or without ICU FE) and RE logistic regression dominated, more so the latter using information criteria indices. The LPM suffered from many predictions outside the unit [0,1] interval and both poor discrimination and calibration. Error terms of hospital length of stay, an independent risk of death score and ventilation status were correlated with the mortality error term. Marked differences in the ventilation mortality marginal effect was demonstrated between the probit and the "eprobit" models which were scenario dependent. Endogeneity was not demonstrated for the APACHE III score. Conclusions Logistic regression accounting for provider effects was the preferred estimator for hospital mortality modelling. Endogeneity of covariates and treatment variables may be identified using appropriate modelling, but failure to do so yields problematic effect estimates. Supplementary Information The online version contains supplementary material available at 10.1186/s12874-021-01251-8.
Collapse
Affiliation(s)
- John L Moran
- Department of Intensive Care Medicine, The Queen Elizabeth Hospital, Woodville, Australia.
| | - John D Santamaria
- Department of Critical Care Medicine, St Vincent's Hospital (Melbourne), Fitzroy, Australia
| | - Graeme J Duke
- Intensive Services, Eastern Health, Box Hill, Australia
| | | |
Collapse
|
630
|
Yan AR, Samarawickrema I, Naunton M, Peterson GM, Yip D, De Rosa S, Mortazavi R. Risk Factors and Prediction Models for Venous Thromboembolism in Ambulatory Patients with Lung Cancer. Healthcare (Basel) 2021; 9:778. [PMID: 34205695 PMCID: PMC8233898 DOI: 10.3390/healthcare9060778] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/21/2021] [Revised: 06/13/2021] [Accepted: 06/17/2021] [Indexed: 12/21/2022] Open
Abstract
Venous thromboembolism (VTE) is a significant cause of mortality in patients with lung cancer. Despite the availability of a wide range of anticoagulants to help prevent thrombosis, thromboprophylaxis in ambulatory patients is a challenge due to its associated risk of haemorrhage. As a result, anticoagulation is only recommended in patients with a relatively high risk of VTE. Efforts have been made to develop predictive models for VTE risk assessment in cancer patients, but the availability of a reliable predictive model for ambulate patients with lung cancer is unclear. We have analysed the latest information on this topic, with a focus on the lung cancer-related risk factors for VTE, and risk prediction models developed and validated in this group of patients. The existing risk models, such as the Khorana score, the PROTECHT score and the CONKO score, have shown poor performance in external validations, failing to identify many high-risk individuals. Some of the newly developed and updated models may be promising, but their further validation is needed.
Collapse
Affiliation(s)
- Ann-Rong Yan
- School of Health Sciences, Faculty of Health, University of Canberra, Canberra 2617, Australia; (A.-R.Y.); (M.N.); (G.M.P.); (D.Y.)
| | - Indira Samarawickrema
- School of Nursing, Midwifery and Public Health, Faculty of Health, University of Canberra, Canberra 2617, Australia;
| | - Mark Naunton
- School of Health Sciences, Faculty of Health, University of Canberra, Canberra 2617, Australia; (A.-R.Y.); (M.N.); (G.M.P.); (D.Y.)
| | - Gregory M. Peterson
- School of Health Sciences, Faculty of Health, University of Canberra, Canberra 2617, Australia; (A.-R.Y.); (M.N.); (G.M.P.); (D.Y.)
- College of Health and Medicine, University of Tasmania, Hobart 7005, Australia
| | - Desmond Yip
- School of Health Sciences, Faculty of Health, University of Canberra, Canberra 2617, Australia; (A.-R.Y.); (M.N.); (G.M.P.); (D.Y.)
- Department of Medical Oncology, The Canberra Hospital, Garran 2605, Australia
- ANU Medical School, Australian National University, Canberra 0200, Australia
| | - Salvatore De Rosa
- Department of Medical and Surgical Science, Magna Graecia University, 88100 Catanzaro, Italy;
| | - Reza Mortazavi
- School of Health Sciences, Faculty of Health, University of Canberra, Canberra 2617, Australia; (A.-R.Y.); (M.N.); (G.M.P.); (D.Y.)
- Prehab Activity Cancer Exercise Survivorship Research Group, Faculty of Health, University of Canberra, Canberra 2617, Australia
| |
Collapse
|
631
|
Elahi C, Adil SM, Sakita F, Mmbaga BT, Rocha TAH, Fuller A, Haglund MM, Vissoci JRN, Staton C. Corticosteroid Randomization after Significant Head Injury and International Mission for Prognosis and Clinical Trialsin Traumatic Brain Injury Models Compared with a Machine Learning-Based Predictive Model from Tanzania. J Neurotrauma 2021; 39:151-158. [PMID: 33980030 DOI: 10.1089/neu.2020.7483] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022] Open
Abstract
Hospitals in low- and middle-income countries (LMICs) could benefit from decision support technologies to reduce time to triage, diagnosis, and surgery for patients with traumatic brain injury (TBI). Corticosteroid Randomization after Significant Head Injury (CRASH) and International Mission for Prognosis and Clinical Trials in Traumatic Brain Injury (IMPACT) are robust examples of TBI prognostic models, although they have yet to be validated in Sub-Saharan Africa (SSA). Moreover, machine learning and improved data quality in LMICs provide an opportunity to develop context-specific, and potentially more accurate, prognostic models. We aim to externally validate CRASH and IMPACT on our TBI registry and compare their performances to that of the locally derived model (from the Kilimanjaro Christian Medical Center [KCMC]). We developed a machine learning-based prognostic model from a TBI registry collected at a regional referral hospital in Moshi, Tanzania. We also used the core CRASH and IMPACT online risk calculators to generate risk scores for each patient. We compared the discrimination (area under the curve [AUC]) and calibration before and after Platt scaling (Brier, Hosmer-Lemeshow Test, and calibration plots) for CRASH, IMPACT, and the KCMC model. The outcome of interest was unfavorable in-hospital outcome defined as a Glasgow Outcome Scale score of 1-3. There were 2972 patients included in the TBI registry, of whom 11% had an unfavorable outcome. The AUCs for the KCMC model, CRASH, and IMPACT were 0.919, 0.876, and 0.821, respectively. Prior to Platt scaling, CRASH was the best calibrated model (χ2 = 68.1) followed by IMPACT (χ2 = 380.9) and KCMC (χ2 = 1025.6). We provide the first SSA validation of the core CRASH and IMPACT models. The KCMC model had better discrimination than either of these. CRASH had the best calibration, although all model predictions could be successfully calibrated. The top performing models, KCMC and CRASH, were both developed using LMIC data, suggesting that locally derived models may outperform imported ones from different contexts of care. Further work is needed to externally validate the KCMC model.
Collapse
Affiliation(s)
- Cyrus Elahi
- Barrow Neurological Institute, Phoenix, Arizona, USA.,Division of Neurosurgery and Neurology, Department of Neurosurgery, and Department of Surgery, Duke University Medical Center, Durham, North Carolina, USA
| | - Syed M Adil
- Division of Neurosurgery and Neurology, Department of Neurosurgery, and Department of Surgery, Duke University Medical Center, Durham, North Carolina, USA
| | - Francis Sakita
- Emergency Department, and Kilimanjaro Christian Medical Center, Moshi, Tanzania
| | - Blandina T Mmbaga
- Kilimanjaro Christian Medical University College, Moshi, Tanzania.,Kilimanjaro Clinical Research Institute, Kilimanjaro Christian Medical Center, Moshi, Tanzania
| | | | - Anthony Fuller
- Division of Neurosurgery and Neurology, Department of Neurosurgery, and Department of Surgery, Duke University Medical Center, Durham, North Carolina, USA.,Duke Global Health Institute, Duke University, Durham, North Carolina, USA
| | - Michael M Haglund
- Division of Neurosurgery and Neurology, Department of Neurosurgery, and Department of Surgery, Duke University Medical Center, Durham, North Carolina, USA.,Duke Global Health Institute, Duke University, Durham, North Carolina, USA
| | - João Ricardo Nickenig Vissoci
- Duke Global Health Institute, Duke University, Durham, North Carolina, USA.,Division of Emergency Medicine, Department of Surgery, Duke University Medical Center, Durham, North Carolina, USA
| | - Catherine Staton
- Duke Global Health Institute, Duke University, Durham, North Carolina, USA.,Division of Emergency Medicine, Department of Surgery, Duke University Medical Center, Durham, North Carolina, USA
| |
Collapse
|
632
|
Cabitza F, Campagner A. The need to separate the wheat from the chaff in medical informatics: Introducing a comprehensive checklist for the (self)-assessment of medical AI studies. Int J Med Inform 2021; 153:104510. [PMID: 34108105 DOI: 10.1016/j.ijmedinf.2021.104510] [Citation(s) in RCA: 116] [Impact Index Per Article: 38.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2021] [Revised: 05/26/2021] [Accepted: 05/27/2021] [Indexed: 12/23/2022]
Abstract
This editorial aims to contribute to the current debate about the quality of studies that apply machine learning (ML) methodologies to medical data to extract value from them and provide clinicians with viable and useful tools supporting everyday care practices. We propose a practical checklist to help authors to self assess the quality of their contribution and to help reviewers to recognize and appreciate high-quality medical ML studies by distinguishing them from the mere application of ML techniques to medical data.
Collapse
Affiliation(s)
- Federico Cabitza
- DISCo, University of Milano-Bicocca, viale Sarca 336, Milano 20126, Italy.
| | - Andrea Campagner
- DISCo, University of Milano-Bicocca, viale Sarca 336, Milano 20126, Italy
| |
Collapse
|
633
|
Rossman H, Shilo S, Barbash-Hazan S, Artzi NS, Hadar E, Balicer RD, Feldman B, Wiznitzer A, Segal E. Prediction of Childhood Obesity from Nationwide Health Records. J Pediatr 2021; 233:132-140.e1. [PMID: 33581105 DOI: 10.1016/j.jpeds.2021.02.010] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/19/2020] [Revised: 02/02/2021] [Accepted: 02/04/2021] [Indexed: 11/28/2022]
Abstract
OBJECTIVE To evaluate body mass index (BMI) acceleration patterns in children and to develop a prediction model targeted to identify children at high risk for obesity before the critical time window in which the largest increase in BMI percentile occurs. STUDY DESIGN We analyzed electronic health records of children from Israel's largest healthcare provider from 2002 to 2018. Data included demographics, anthropometric measurements, medications, diagnoses, and laboratory tests of children and their families. Obesity was defined as BMI ≥95th percentile for age and sex. To identify the time window in which the largest annual increases in BMI z score occurs during early childhood, we first analyzed childhood BMI acceleration patterns among 417 915 adolescents. Next, we devised a model targeted to identify children at high risk before this time window, predicting obesity at 5-6 years of age based on data from the first 2 years of life of 132 262 children. RESULTS Retrospective BMI analysis revealed that among adolescents with obesity, the greatest acceleration in BMI z score occurred between 2 and 4 years of age. Our model, validated temporally and geographically, accurately predicted obesity at 5-6 years old (area under the receiver operating characteristic curve of 0.803). Discrimination results on subpopulations demonstrated its robustness across the pediatric population. The model's most influential predictors included anthropometric measurements of the child and family. Other impactful predictors included ancestry and pregnancy glucose. CONCLUSIONS Rapid rise in the prevalence of childhood obesity warrant the development of better prevention strategies. Our model may allow an accurate identification of children at high risk of obesity.
Collapse
Affiliation(s)
- Hagai Rossman
- Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Rehovot, Israel; Department of Molecular Cell Biology, Weizmann Institute of Science, Rehovot, Israel
| | - Smadar Shilo
- Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Rehovot, Israel; Department of Molecular Cell Biology, Weizmann Institute of Science, Rehovot, Israel; Pediatric Diabetes Unit, Ruth Rappaport Children's Hospital of Haifa, Rambam Healthcare Campus, Haifa, Israel
| | - Shiri Barbash-Hazan
- Helen Schneider Hospital for Women, Rabin Medical Center, Petach Tikva, Israel; Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel
| | - Nitzan Shalom Artzi
- Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Rehovot, Israel; Department of Molecular Cell Biology, Weizmann Institute of Science, Rehovot, Israel
| | - Eran Hadar
- Helen Schneider Hospital for Women, Rabin Medical Center, Petach Tikva, Israel; Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel
| | - Ran D Balicer
- Clalit Health Services, Clalit Research Institute, Tel Aviv, Israel; Department of Public Health, Faculty of Health Sciences, Ben Gurion University, Beer Sheva, Israel
| | - Becca Feldman
- Clalit Health Services, Clalit Research Institute, Tel Aviv, Israel
| | - Arnon Wiznitzer
- Helen Schneider Hospital for Women, Rabin Medical Center, Petach Tikva, Israel; Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel
| | - Eran Segal
- Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Rehovot, Israel; Department of Molecular Cell Biology, Weizmann Institute of Science, Rehovot, Israel.
| |
Collapse
|
634
|
Deniffel D, Healy GM, Dong X, Ghai S, Salinas-Miranda E, Fleshner N, Hamilton R, Kulkarni G, Toi A, van der Kwast T, Zlotta A, Finelli A, Perlis N, Haider MA. Avoiding Unnecessary Biopsy: MRI-based Risk Models versus a PI-RADS and PSA Density Strategy for Clinically Significant Prostate Cancer. Radiology 2021; 300:369-379. [PMID: 34032510 DOI: 10.1148/radiol.2021204112] [Citation(s) in RCA: 40] [Impact Index Per Article: 13.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
Background In validation studies, risk models for clinically significant prostate cancer (csPCa; Gleason score ≥3+4) combining multiparametric MRI and clinical factors have demonstrated poor calibration (over- and underprediction) and limited use in avoiding unnecessary prostate biopsies. Purpose MRI-based risk models following local recalibration were compared with a strategy that combined Prostate Imaging Data and Reporting System (PI-RADS; version 2) and prostate-specific antigen density (PSAd) to assess the potential reduction of unnecessary prostate biopsies. Materials and Methods This retrospective study included 385 patients without prostate cancer diagnosis who underwent multipara-metric MRI (PI-RADS category ≥3) and MRI-targeted biopsy between 2015 and 2019. Recalibration and selection of the best-performing MRI model (MRI-European Randomized Study of Screening for Prostate Cancer [ERSPC], van Leeuwen, Radtke, and Mehralivand models) were undertaken in cohort C1 (n = 242; 2015-2017). The impact on biopsy decisions was compared with an alternative strategy (no biopsy for PI-RADS category 3 plus PSAd < 0.1 ng/mL per milliliter) in cohort C2 (n = 143; 2018-2019). Discrimination, calibration, and clinical utility were assessed by using the area under the receiver operating characteristic curve (AUC), calibration plots, and decision curve analysis, respectively. Results The prevalence of csPCa was 38% (93 of 242 patients) and 45% (64 of 143 patients) in cohorts C1 and C2, respectively. Decision curve analysis demonstrated the highest net benefit for the van Leeuwen and Mehralivand models in C1. Used for biopsy decisions in C2, van Leeuwen (AUC, 0.84; 95% CI: 0.77, 0.9) and Mehralivand (AUC, 0.79; 95% CI: 0.72, 0.86) enabled no net benefit at a risk threshold of 10%. Up to a risk threshold of 15%, net benefit remained inferior to the PI-RADS plus PSAd strategy, which avoided biopsy in 63 per 1000 men, without missing csPCa. Without prior recalibration in C1, three of four models (MRIERSPC, Radtke, Mehralivand) were poorly calibrated and not clinically useful in C2. Conclusion The number of unnecessary prostate biopsies in men with positive MRI may be safely reduced by using a prostate-specific antigen density-based strategy. In a risk-averse scenario, this strategy enabled better biopsy decisions compared with MRI-based risk models. ©RSNA, 2021 Online supplemental material is available for this article.
Collapse
Affiliation(s)
- Dominik Deniffel
- From the Department of Diagnostic and Interventional Radiology, Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany (D.D.); Lunenfeld-Tanenbaum Research Institute, Sinai Health System, 600 University Ave, Toronto, ON, Canada M5G 1X5 (D.D., G.M.H., X.D., E.S.M., A.Z., M.A.H.); Joint Department of Medical Imaging, University Health Network, Sinai Health System and University of Toronto, Toronto, ON, Canada (D.D., G.M.H., S.G., E.S.M., A.T., M.A.H.); Division of Urology, Department of Surgical Oncology, Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada (N.F., R.H., G.K., A.Z., A.F., N.P.); Department of Pathology, Laboratory Medicine Program, University Health Network, Toronto, ON, Canada (T.v.d.K.); and Department of Surgery, Division of Urology, Mount Sinai Hospital, Toronto, ON, Canada (A.Z.)
| | - Gerard M Healy
- From the Department of Diagnostic and Interventional Radiology, Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany (D.D.); Lunenfeld-Tanenbaum Research Institute, Sinai Health System, 600 University Ave, Toronto, ON, Canada M5G 1X5 (D.D., G.M.H., X.D., E.S.M., A.Z., M.A.H.); Joint Department of Medical Imaging, University Health Network, Sinai Health System and University of Toronto, Toronto, ON, Canada (D.D., G.M.H., S.G., E.S.M., A.T., M.A.H.); Division of Urology, Department of Surgical Oncology, Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada (N.F., R.H., G.K., A.Z., A.F., N.P.); Department of Pathology, Laboratory Medicine Program, University Health Network, Toronto, ON, Canada (T.v.d.K.); and Department of Surgery, Division of Urology, Mount Sinai Hospital, Toronto, ON, Canada (A.Z.)
| | - Xin Dong
- From the Department of Diagnostic and Interventional Radiology, Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany (D.D.); Lunenfeld-Tanenbaum Research Institute, Sinai Health System, 600 University Ave, Toronto, ON, Canada M5G 1X5 (D.D., G.M.H., X.D., E.S.M., A.Z., M.A.H.); Joint Department of Medical Imaging, University Health Network, Sinai Health System and University of Toronto, Toronto, ON, Canada (D.D., G.M.H., S.G., E.S.M., A.T., M.A.H.); Division of Urology, Department of Surgical Oncology, Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada (N.F., R.H., G.K., A.Z., A.F., N.P.); Department of Pathology, Laboratory Medicine Program, University Health Network, Toronto, ON, Canada (T.v.d.K.); and Department of Surgery, Division of Urology, Mount Sinai Hospital, Toronto, ON, Canada (A.Z.)
| | - Sangeet Ghai
- From the Department of Diagnostic and Interventional Radiology, Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany (D.D.); Lunenfeld-Tanenbaum Research Institute, Sinai Health System, 600 University Ave, Toronto, ON, Canada M5G 1X5 (D.D., G.M.H., X.D., E.S.M., A.Z., M.A.H.); Joint Department of Medical Imaging, University Health Network, Sinai Health System and University of Toronto, Toronto, ON, Canada (D.D., G.M.H., S.G., E.S.M., A.T., M.A.H.); Division of Urology, Department of Surgical Oncology, Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada (N.F., R.H., G.K., A.Z., A.F., N.P.); Department of Pathology, Laboratory Medicine Program, University Health Network, Toronto, ON, Canada (T.v.d.K.); and Department of Surgery, Division of Urology, Mount Sinai Hospital, Toronto, ON, Canada (A.Z.)
| | - Emmanuel Salinas-Miranda
- From the Department of Diagnostic and Interventional Radiology, Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany (D.D.); Lunenfeld-Tanenbaum Research Institute, Sinai Health System, 600 University Ave, Toronto, ON, Canada M5G 1X5 (D.D., G.M.H., X.D., E.S.M., A.Z., M.A.H.); Joint Department of Medical Imaging, University Health Network, Sinai Health System and University of Toronto, Toronto, ON, Canada (D.D., G.M.H., S.G., E.S.M., A.T., M.A.H.); Division of Urology, Department of Surgical Oncology, Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada (N.F., R.H., G.K., A.Z., A.F., N.P.); Department of Pathology, Laboratory Medicine Program, University Health Network, Toronto, ON, Canada (T.v.d.K.); and Department of Surgery, Division of Urology, Mount Sinai Hospital, Toronto, ON, Canada (A.Z.)
| | - Neil Fleshner
- From the Department of Diagnostic and Interventional Radiology, Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany (D.D.); Lunenfeld-Tanenbaum Research Institute, Sinai Health System, 600 University Ave, Toronto, ON, Canada M5G 1X5 (D.D., G.M.H., X.D., E.S.M., A.Z., M.A.H.); Joint Department of Medical Imaging, University Health Network, Sinai Health System and University of Toronto, Toronto, ON, Canada (D.D., G.M.H., S.G., E.S.M., A.T., M.A.H.); Division of Urology, Department of Surgical Oncology, Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada (N.F., R.H., G.K., A.Z., A.F., N.P.); Department of Pathology, Laboratory Medicine Program, University Health Network, Toronto, ON, Canada (T.v.d.K.); and Department of Surgery, Division of Urology, Mount Sinai Hospital, Toronto, ON, Canada (A.Z.)
| | - Robert Hamilton
- From the Department of Diagnostic and Interventional Radiology, Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany (D.D.); Lunenfeld-Tanenbaum Research Institute, Sinai Health System, 600 University Ave, Toronto, ON, Canada M5G 1X5 (D.D., G.M.H., X.D., E.S.M., A.Z., M.A.H.); Joint Department of Medical Imaging, University Health Network, Sinai Health System and University of Toronto, Toronto, ON, Canada (D.D., G.M.H., S.G., E.S.M., A.T., M.A.H.); Division of Urology, Department of Surgical Oncology, Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada (N.F., R.H., G.K., A.Z., A.F., N.P.); Department of Pathology, Laboratory Medicine Program, University Health Network, Toronto, ON, Canada (T.v.d.K.); and Department of Surgery, Division of Urology, Mount Sinai Hospital, Toronto, ON, Canada (A.Z.)
| | - Girish Kulkarni
- From the Department of Diagnostic and Interventional Radiology, Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany (D.D.); Lunenfeld-Tanenbaum Research Institute, Sinai Health System, 600 University Ave, Toronto, ON, Canada M5G 1X5 (D.D., G.M.H., X.D., E.S.M., A.Z., M.A.H.); Joint Department of Medical Imaging, University Health Network, Sinai Health System and University of Toronto, Toronto, ON, Canada (D.D., G.M.H., S.G., E.S.M., A.T., M.A.H.); Division of Urology, Department of Surgical Oncology, Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada (N.F., R.H., G.K., A.Z., A.F., N.P.); Department of Pathology, Laboratory Medicine Program, University Health Network, Toronto, ON, Canada (T.v.d.K.); and Department of Surgery, Division of Urology, Mount Sinai Hospital, Toronto, ON, Canada (A.Z.)
| | - Ants Toi
- From the Department of Diagnostic and Interventional Radiology, Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany (D.D.); Lunenfeld-Tanenbaum Research Institute, Sinai Health System, 600 University Ave, Toronto, ON, Canada M5G 1X5 (D.D., G.M.H., X.D., E.S.M., A.Z., M.A.H.); Joint Department of Medical Imaging, University Health Network, Sinai Health System and University of Toronto, Toronto, ON, Canada (D.D., G.M.H., S.G., E.S.M., A.T., M.A.H.); Division of Urology, Department of Surgical Oncology, Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada (N.F., R.H., G.K., A.Z., A.F., N.P.); Department of Pathology, Laboratory Medicine Program, University Health Network, Toronto, ON, Canada (T.v.d.K.); and Department of Surgery, Division of Urology, Mount Sinai Hospital, Toronto, ON, Canada (A.Z.)
| | - Theodorus van der Kwast
- From the Department of Diagnostic and Interventional Radiology, Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany (D.D.); Lunenfeld-Tanenbaum Research Institute, Sinai Health System, 600 University Ave, Toronto, ON, Canada M5G 1X5 (D.D., G.M.H., X.D., E.S.M., A.Z., M.A.H.); Joint Department of Medical Imaging, University Health Network, Sinai Health System and University of Toronto, Toronto, ON, Canada (D.D., G.M.H., S.G., E.S.M., A.T., M.A.H.); Division of Urology, Department of Surgical Oncology, Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada (N.F., R.H., G.K., A.Z., A.F., N.P.); Department of Pathology, Laboratory Medicine Program, University Health Network, Toronto, ON, Canada (T.v.d.K.); and Department of Surgery, Division of Urology, Mount Sinai Hospital, Toronto, ON, Canada (A.Z.)
| | - Alexandre Zlotta
- From the Department of Diagnostic and Interventional Radiology, Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany (D.D.); Lunenfeld-Tanenbaum Research Institute, Sinai Health System, 600 University Ave, Toronto, ON, Canada M5G 1X5 (D.D., G.M.H., X.D., E.S.M., A.Z., M.A.H.); Joint Department of Medical Imaging, University Health Network, Sinai Health System and University of Toronto, Toronto, ON, Canada (D.D., G.M.H., S.G., E.S.M., A.T., M.A.H.); Division of Urology, Department of Surgical Oncology, Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada (N.F., R.H., G.K., A.Z., A.F., N.P.); Department of Pathology, Laboratory Medicine Program, University Health Network, Toronto, ON, Canada (T.v.d.K.); and Department of Surgery, Division of Urology, Mount Sinai Hospital, Toronto, ON, Canada (A.Z.)
| | - Antonio Finelli
- From the Department of Diagnostic and Interventional Radiology, Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany (D.D.); Lunenfeld-Tanenbaum Research Institute, Sinai Health System, 600 University Ave, Toronto, ON, Canada M5G 1X5 (D.D., G.M.H., X.D., E.S.M., A.Z., M.A.H.); Joint Department of Medical Imaging, University Health Network, Sinai Health System and University of Toronto, Toronto, ON, Canada (D.D., G.M.H., S.G., E.S.M., A.T., M.A.H.); Division of Urology, Department of Surgical Oncology, Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada (N.F., R.H., G.K., A.Z., A.F., N.P.); Department of Pathology, Laboratory Medicine Program, University Health Network, Toronto, ON, Canada (T.v.d.K.); and Department of Surgery, Division of Urology, Mount Sinai Hospital, Toronto, ON, Canada (A.Z.)
| | - Nathan Perlis
- From the Department of Diagnostic and Interventional Radiology, Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany (D.D.); Lunenfeld-Tanenbaum Research Institute, Sinai Health System, 600 University Ave, Toronto, ON, Canada M5G 1X5 (D.D., G.M.H., X.D., E.S.M., A.Z., M.A.H.); Joint Department of Medical Imaging, University Health Network, Sinai Health System and University of Toronto, Toronto, ON, Canada (D.D., G.M.H., S.G., E.S.M., A.T., M.A.H.); Division of Urology, Department of Surgical Oncology, Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada (N.F., R.H., G.K., A.Z., A.F., N.P.); Department of Pathology, Laboratory Medicine Program, University Health Network, Toronto, ON, Canada (T.v.d.K.); and Department of Surgery, Division of Urology, Mount Sinai Hospital, Toronto, ON, Canada (A.Z.)
| | - Masoom A Haider
- From the Department of Diagnostic and Interventional Radiology, Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany (D.D.); Lunenfeld-Tanenbaum Research Institute, Sinai Health System, 600 University Ave, Toronto, ON, Canada M5G 1X5 (D.D., G.M.H., X.D., E.S.M., A.Z., M.A.H.); Joint Department of Medical Imaging, University Health Network, Sinai Health System and University of Toronto, Toronto, ON, Canada (D.D., G.M.H., S.G., E.S.M., A.T., M.A.H.); Division of Urology, Department of Surgical Oncology, Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada (N.F., R.H., G.K., A.Z., A.F., N.P.); Department of Pathology, Laboratory Medicine Program, University Health Network, Toronto, ON, Canada (T.v.d.K.); and Department of Surgery, Division of Urology, Mount Sinai Hospital, Toronto, ON, Canada (A.Z.)
| |
Collapse
|
635
|
Valente F, Henriques J, Paredes S, Rocha T, de Carvalho P, Morais J. A new approach for interpretability and reliability in clinical risk prediction: Acute coronary syndrome scenario. Artif Intell Med 2021; 117:102113. [PMID: 34127242 DOI: 10.1016/j.artmed.2021.102113] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2020] [Revised: 04/18/2021] [Accepted: 05/11/2021] [Indexed: 12/22/2022]
Abstract
INTRODUCTION The risk prediction of the occurrence of a clinical event is often based on conventional statistical procedures, through the implementation of risk score models. Recently, approaches based on more complex machine learning (ML) methods have been developed. Despite the latter usually have a better predictive performance, they obtain little approval from the physicians, as they lack interpretability and, therefore, clinical confidence. One clinical issue where both types of models have received great attention is the mortality risk prediction after acute coronary syndromes (ACS). OBJECTIVE We intend to create a new risk assessment methodology that combines the best characteristics of both risk score and ML models. More specifically, we aim to develop a method that, besides having a good performance, offers a personalized model and outcome for each patient, presents high interpretability, and incorporates an estimation of the prediction reliability which is not usually available. By combining these features in the same approach we expect that it can boost the confidence of physicians to use such a tool in their daily activity. METHODS In order to achieve the mentioned goals, a three-step methodology was developed: several rules were created by dichotomizing risk factors; such rules were trained with a machine learning classifier to predict the acceptance degree of each rule (the probability that the rule is correct) for each patient; that information was combined and used to compute the risk of mortality and the reliability of such prediction. The methodology was applied to a dataset of 1111 patients admitted with any type of ACS (myocardial infarction and unstable angina) in two Portuguese hospitals, to assess the 30-days all-cause mortality risk, being validated through a Monte-Carlo cross-validation technique. The performance was compared with state-of-the-art approaches: logistic regression (LR), artificial neural network (ANN), and clinical risk score model (namely the Global Registry of Acute Coronary Events - GRACE). RESULTS For the scenario being analyzed, the performance of the proposed approach and the comparison models was assessed through discrimination and calibration. The ability to rank the patients was evaluated through the area under the ROC curve (AUC), and the ability to stratify the patients into low or high-risk groups was determined using the geometric mean (GM) of specificity and sensitivity, the negative predictive value (NPV) and the positive predictive value (PPV). The validation calibration curves were also inspected. The proposed approach (AUC = 81%, GM = 74%, PPV = 17%, NPV = 99%) achieved testing results identical to the standard LR model (AUC = 83%, GM = 73%, PPV = 16%, NPV=99%), but offers superior interpretability and personalization; it also significantly outperforms the GRACE risk model (AUC = 79%, GM = 47%, PPV = 13%, NPV = 98%) and the standard ANN model (AUC = 78%, GM = 70%, PPV = 13%, NPV = 98%). The calibration curve also suggests a very good generalization ability of the obtained model as it approaches the ideal curve (slope = 0.96). Finally, the reliability estimation of individual predictions presented a great correlation with the misclassifications rate. CONCLUSION We developed and described a new tool that showed great potential to guide the clinical staff in the risk assessment and decision-making process, and to obtain their wide acceptance due to its interpretability and reliability estimation properties. The methodology presented a good performance when applied to ACS events, but those properties may have a beneficial application in other clinical scenarios as well.
Collapse
Affiliation(s)
- Francisco Valente
- Center for Informatics and Systems of University of Coimbra, University of Coimbra, Pólo II, 3030-290 Coimbra, Portugal.
| | - Jorge Henriques
- Center for Informatics and Systems of University of Coimbra, University of Coimbra, Pólo II, 3030-290 Coimbra, Portugal.
| | - Simão Paredes
- Center for Informatics and Systems of University of Coimbra, University of Coimbra, Pólo II, 3030-290 Coimbra, Portugal; Polytechnic of Coimbra, Department of Systems and Computer Engineering, Rua Pedro Nunes - Quinta da Nora, 3030-199 Coimbra, Portugal.
| | - Teresa Rocha
- Center for Informatics and Systems of University of Coimbra, University of Coimbra, Pólo II, 3030-290 Coimbra, Portugal; Polytechnic of Coimbra, Department of Systems and Computer Engineering, Rua Pedro Nunes - Quinta da Nora, 3030-199 Coimbra, Portugal.
| | - Paulo de Carvalho
- Center for Informatics and Systems of University of Coimbra, University of Coimbra, Pólo II, 3030-290 Coimbra, Portugal.
| | - João Morais
- Cardiology Department, Leiria Hospital Centre, Leiria, Portugal.
| |
Collapse
|
636
|
Beachler DC, Taylor DH, Anthony MS, Yin R, Li L, Saltus CW, Li L, Shaunik A, Walsh KE, Rothman KJ, Johannes CB, Aroda VR, Carr W, Goldberg P, Accardi A, O'Shura JS, Sharma K, Juhaeri J, Lanes S, Wu C. Development and validation of a predictive model algorithm to identify anaphylaxis in adults with type 2 diabetes in U.S. administrative claims data. Pharmacoepidemiol Drug Saf 2021; 30:918-926. [PMID: 33899314 DOI: 10.1002/pds.5257] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2020] [Revised: 04/19/2021] [Accepted: 04/21/2021] [Indexed: 11/11/2022]
Abstract
PURPOSE To use medical record adjudication and predictive modeling methods to develop and validate an algorithm to identify anaphylaxis among adults with type 2 diabetes (T2D) in administrative claims. METHODS A conventional screening algorithm that prioritized sensitivity to identify potential anaphylaxis cases was developed and consisted of diagnosis codes for anaphylaxis or relevant signs and symptoms. This algorithm was applied to adults with T2D in the HealthCore Integrated Research Database (HIRD) from 2016 to 2018. Clinical experts adjudicated anaphylaxis case status from redacted medical records. We used confirmed case status as an outcome for predictive models developed using lasso regression with 10-fold cross-validation to identify predictors and estimate the probability of confirmed anaphylaxis. RESULTS Clinical adjudicators reviewed medical records with sufficient information from 272 adults identified by the anaphylaxis screening algorithm, which had an estimated Positive Predictive Value (PPV) of 65% (95% confidence interval [CI]: 60%-71%). The predictive model algorithm had a c-statistic of 0.95. The model's probability threshold of 0.60 excluded 89% (84/94) of false positives identified by the screening algorithm, with a PPV of 94% (95% CI: 91%-98%). The model excluded very few true positives (15 of 178), and identified 92% (95% CI: 87%-96%) of the cases selected by the screening algorithm. CONCLUSIONS Predictive modeling techniques yielded an accurate algorithm with high PPV and sensitivity for identifying anaphylaxis in administrative claims. This algorithm could be considered in future safety studies using similar claims data to reduce potential outcome misclassification.
Collapse
Affiliation(s)
| | | | | | - Ruihua Yin
- Anthem, Inc., Indianapolis, Indiana, USA
| | - Ling Li
- HealthCore, Inc., Wilmington, Delaware, USA
| | | | | | | | - Kathleen E Walsh
- Division of General Pediatrics, Department of Pediatrics, Harvard Medical School, Boston Children's Hospital, Boston, Massachusetts, USA
| | | | | | | | - Warner Carr
- Allergy & Asthma Associates of Southern California, San Jose, California, USA
| | - Pinkus Goldberg
- Allergy Partners of Central Indiana, Indianapolis, Indiana, USA
| | | | | | | | | | | | | |
Collapse
|
637
|
Ten Haaf K, van der Aalst CM, de Koning HJ, Kaaks R, Tammemägi MC. Personalising lung cancer screening: An overview of risk-stratification opportunities and challenges. Int J Cancer 2021; 149:250-263. [PMID: 33783822 PMCID: PMC8251929 DOI: 10.1002/ijc.33578] [Citation(s) in RCA: 37] [Impact Index Per Article: 12.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2020] [Revised: 03/04/2021] [Accepted: 03/12/2021] [Indexed: 12/17/2022]
Abstract
Randomised clinical trials have shown the efficacy of computed tomography lung cancer screening, initiating discussions on whether and how to implement population‐based screening programs. Due to smoking behaviour being the primary risk‐factor for lung cancer and part of the criteria for determining screening eligibility, lung cancer screening is inherently risk‐based. In fact, the selection of high‐risk individuals has been shown to be essential in implementing lung cancer screening in a cost‐effective manner. Furthermore, studies have shown that further risk‐stratification may improve screening efficiency, allow personalisation of the screening interval and reduce health disparities. However, implementing risk‐based lung cancer screening programs also requires overcoming a number of challenges. There are indications that risk‐based approaches can negatively influence the trade‐off between individual benefits and harms if not applied thoughtfully. Large‐scale implementation of targeted, risk‐based screening programs has been limited thus far. Consequently, questions remain on how to efficiently identify and invite high‐risk individuals from the general population. Finally, while risk‐based approaches may increase screening program efficiency, efficiency should be balanced with the overall impact of the screening program. In this review, we will address the opportunities and challenges in applying risk‐stratification in different aspects of lung cancer screening programs, as well as the balance between screening program efficiency and impact.
Collapse
Affiliation(s)
- Kevin Ten Haaf
- Department of Public Health, Erasmus MC-University Medical Center Rotterdam, Rotterdam, The Netherlands
| | - Carlijn M van der Aalst
- Department of Public Health, Erasmus MC-University Medical Center Rotterdam, Rotterdam, The Netherlands
| | - Harry J de Koning
- Department of Public Health, Erasmus MC-University Medical Center Rotterdam, Rotterdam, The Netherlands
| | - Rudolf Kaaks
- Division of Cancer Epidemiology, German Cancer Research Center (DKFZ), Heidelberg, Germany.,Translational Lung Research Center (TLRC) Heidelberg, Member of the German Center for Lung Research (DZL), Heidelberg, Germany
| | - Martin C Tammemägi
- Department of Health Sciences, Brock University, St. Catharines, Ontario, Canada
| |
Collapse
|
638
|
Ravaut M, Harish V, Sadeghi H, Leung KK, Volkovs M, Kornas K, Watson T, Poutanen T, Rosella LC. Development and Validation of a Machine Learning Model Using Administrative Health Data to Predict Onset of Type 2 Diabetes. JAMA Netw Open 2021; 4:e2111315. [PMID: 34032855 PMCID: PMC8150694 DOI: 10.1001/jamanetworkopen.2021.11315] [Citation(s) in RCA: 31] [Impact Index Per Article: 10.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/20/2020] [Accepted: 04/01/2021] [Indexed: 11/14/2022] Open
Abstract
Importance Systems-level barriers to diabetes care could be improved with population health planning tools that accurately discriminate between high- and low-risk groups to guide investments and targeted interventions. Objective To develop and validate a population-level machine learning model for predicting type 2 diabetes 5 years before diabetes onset using administrative health data. Design, Setting, and Participants This decision analytical model study used linked administrative health data from the diverse, single-payer health system in Ontario, Canada, between January 1, 2006, and December 31, 2016. A gradient boosting decision tree model was trained on data from 1 657 395 patients, validated on 243 442 patients, and tested on 236 506 patients. Costs associated with each patient were estimated using a validated costing algorithm. Data were analyzed from January 1, 2006, to December 31, 2016. Exposures A random sample of 2 137 343 residents of Ontario without type 2 diabetes was obtained at study start time. More than 300 features from data sets capturing demographic information, laboratory measurements, drug benefits, health care system interactions, social determinants of health, and ambulatory care and hospitalization records were compiled over 2-year patient medical histories to generate quarterly predictions. Main Outcomes and Measures Discrimination was assessed using the area under the receiver operating characteristic curve statistic, and calibration was assessed visually using calibration plots. Feature contribution was assessed with Shapley values. Costs were estimated in 2020 US dollars. Results This study trained a gradient boosting decision tree model on data from 1 657 395 patients (12 900 257 instances; 6 666 662 women [51.7%]). The developed model achieved a test area under the curve of 80.26 (range, 80.21-80.29), demonstrated good calibration, and was robust to sex, immigration status, area-level marginalization with regard to material deprivation and race/ethnicity, and low contact with the health care system. The top 5% of patients predicted as high risk by the model represented 26% of the total annual diabetes cost in Ontario. Conclusions and Relevance In this decision analytical model study, a machine learning model approach accurately predicted the incidence of diabetes in the population using routinely collected health administrative data. These results suggest that the model could be used to inform decision-making for population health planning and diabetes prevention.
Collapse
Affiliation(s)
- Mathieu Ravaut
- Layer 6 AI, Toronto, Ontario, Canada
- Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
| | - Vinyas Harish
- Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
- Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada
- Temerty Centre for Artificial Intelligence Research and Education in Medicine, University of Toronto, Toronto, Ontario, Canada
- Vector Institute, Toronto, Ontario, Canada
| | | | | | | | - Kathy Kornas
- Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
| | - Tristan Watson
- Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
- Institute of Clinical Evaluative Sciences (ICES), Toronto, Ontario, Canada
| | | | - Laura C. Rosella
- Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
- Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada
- Temerty Centre for Artificial Intelligence Research and Education in Medicine, University of Toronto, Toronto, Ontario, Canada
- Vector Institute, Toronto, Ontario, Canada
- Institute of Clinical Evaluative Sciences (ICES), Toronto, Ontario, Canada
- Institute for Better Health, Trillium Health Partners, Mississauga, Ontario, Canada
| |
Collapse
|
639
|
Kunze KN, Polce EM, Nwachukwu BU, Chahla J, Nho SJ. Development and Internal Validation of Supervised Machine Learning Algorithms for Predicting Clinically Significant Functional Improvement in a Mixed Population of Primary Hip Arthroscopy. Arthroscopy 2021; 37:1488-1497. [PMID: 33460708 DOI: 10.1016/j.arthro.2021.01.005] [Citation(s) in RCA: 24] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/03/2020] [Revised: 12/30/2020] [Accepted: 01/03/2021] [Indexed: 02/06/2023]
Abstract
PURPOSE To (1) develop and validate a machine learning algorithm to predict clinically significant functional improvements after hip arthroscopy for femoroacetabular impingement syndrome and to (2) develop a digital application capable of providing patients with individual risk profiles to determine their propensity to gain clinically significant improvements in function. METHODS A retrospective review of consecutive hip arthroscopy patients who underwent cam/pincer correction, labral preservation, and capsular closure between January 2012 and 2017 from 1 large academic and 3 community hospitals operated on by a single high-volume hip arthroscopist was performed. The primary outcome was the minimal clinically important difference (MCID) for the Hip Outcome Score (HOS)-Activities of Daily Living (ADL) at 2 years postoperatively, which was calculated using a distribution-based method. A total of 21 demographic, radiographic, and patient-reported outcome measures were considered as potential covariates. An 80:20 random split was used to create training and testing sets from the patient cohort. Five supervised machine learning algorithms were developed using 3 iterations of 10-fold cross-validation on the training set and assessed by discrimination, calibration, Brier score, and decision curve analysis on an independent testing set of patients. RESULTS A total of 818 patients with a median (interquartile range) age of 32.0 (22.0-42.0) and 69.2% female were included, of whom 74.3% achieved the MCID for the HOS-ADL. The best-performing algorithm was the stochastic gradient boosting model (c-statistic = 0.84, calibration intercept = 0.20, calibration slope = 0.83, and Brier score = 0.13). Of the initial 21 candidate variables, the 8 most important features for predicting the MCID for the HOS-ADL included in model training were body mass index, age, preoperative HOS-ADL score, preoperative pain level, sex, Tönnis grade, symptom duration, and drug allergies. The algorithm was subsequently transformed into a digital application using local explanations to provide customized risk assessment: https://orthoapps.shinyapps.io/HPRG_ADL/. CONCLUSIONS The stochastic boosting gradient model conferred excellent predictive ability for propensity to gain clinically significant improvements in function after hip arthroscopy. An open-access digital application was created, which may augment shared decision-making and allow for preoperative risk stratification. External validation of this model is warranted to confirm the performance of these algorithms, as the generalizability is currently unknown. LEVEL OF EVIDENCE IV, Case series.
Collapse
Affiliation(s)
- Kyle N Kunze
- Department of Orthopedic Surgery, Division of Sports Medicine, Hospital for Special Surgery, New York, New York, U.S.A..
| | - Evan M Polce
- Section of Young Adult Hip Surgery, Division of Sports Medicine, Department of Orthopedic Surgery, Rush Medical College of Rush University, Rush University Medical Center, Chicago, Illinois
| | - Benedict U Nwachukwu
- Department of Orthopedic Surgery, Division of Sports Medicine, Hospital for Special Surgery, New York, New York, U.S.A
| | - Jorge Chahla
- Section of Young Adult Hip Surgery, Division of Sports Medicine, Department of Orthopedic Surgery, Rush Medical College of Rush University, Rush University Medical Center, Chicago, Illinois
| | - Shane J Nho
- Section of Young Adult Hip Surgery, Division of Sports Medicine, Department of Orthopedic Surgery, Rush Medical College of Rush University, Rush University Medical Center, Chicago, Illinois
| |
Collapse
|
640
|
Chee ML, Ong MEH, Siddiqui FJ, Zhang Z, Lim SL, Ho AFW, Liu N. Artificial Intelligence Applications for COVID-19 in Intensive Care and Emergency Settings: A Systematic Review. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2021; 18:ijerph18094749. [PMID: 33947006 PMCID: PMC8125462 DOI: 10.3390/ijerph18094749] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 03/25/2021] [Revised: 04/20/2021] [Accepted: 04/21/2021] [Indexed: 02/07/2023]
Abstract
Background: Little is known about the role of artificial intelligence (AI) as a decisive technology in the clinical management of COVID-19 patients. We aimed to systematically review and critically appraise the current evidence on AI applications for COVID-19 in intensive care and emergency settings. Methods: We systematically searched PubMed, Embase, Scopus, CINAHL, IEEE Xplore, and ACM Digital Library databases from inception to 1 October 2020, without language restrictions. We included peer-reviewed original studies that applied AI for COVID-19 patients, healthcare workers, or health systems in intensive care, emergency, or prehospital settings. We assessed predictive modelling studies and critically appraised the methodology and key findings of all other studies. Results: Of fourteen eligible studies, eleven developed prognostic or diagnostic AI predictive models, all of which were assessed to be at high risk of bias. Common pitfalls included inadequate sample sizes, poor handling of missing data, failure to account for censored participants, and weak validation of models. Conclusions: Current AI applications for COVID-19 are not ready for deployment in acute care settings, given their limited scope and poor quality. Our findings underscore the need for improvements to facilitate safe and effective clinical adoption of AI applications, for and beyond the COVID-19 pandemic.
Collapse
Affiliation(s)
- Marcel Lucas Chee
- Faculty of Medicine, Nursing and Health Sciences, Monash University, Victoria 3800, Australia;
| | - Marcus Eng Hock Ong
- Duke-NUS Medical School, National University of Singapore, Singapore 169857, Singapore; (M.E.H.O.); (F.J.S.); (A.F.W.H.)
- Department of Emergency Medicine, Singapore General Hospital, Singapore 169608, Singapore
| | - Fahad Javaid Siddiqui
- Duke-NUS Medical School, National University of Singapore, Singapore 169857, Singapore; (M.E.H.O.); (F.J.S.); (A.F.W.H.)
| | - Zhongheng Zhang
- Department of Emergency Medicine, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou 310016, China;
| | - Shir Lynn Lim
- Department of Cardiology, National University Heart Centre, Singapore 119074, Singapore;
| | - Andrew Fu Wah Ho
- Duke-NUS Medical School, National University of Singapore, Singapore 169857, Singapore; (M.E.H.O.); (F.J.S.); (A.F.W.H.)
- Department of Emergency Medicine, Singapore General Hospital, Singapore 169608, Singapore
| | - Nan Liu
- Duke-NUS Medical School, National University of Singapore, Singapore 169857, Singapore; (M.E.H.O.); (F.J.S.); (A.F.W.H.)
- Health Service Research Centre, Singapore Health Services, Singapore 169856, Singapore
- Institute of Data Science, National University of Singapore, Singapore 117602, Singapore
- Correspondence:
| |
Collapse
|
641
|
Alhmoud B, Bonnici T, Patel R, Melley D, Williams B, Banerjee A. Performance of universal early warning scores in different patient subgroups and clinical settings: a systematic review. BMJ Open 2021; 11:e045849. [PMID: 36044371 PMCID: PMC8039269 DOI: 10.1136/bmjopen-2020-045849] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/13/2020] [Revised: 03/01/2021] [Accepted: 03/04/2021] [Indexed: 12/23/2022] Open
Abstract
OBJECTIVE To assess predictive performance of universal early warning scores (EWS) in disease subgroups and clinical settings. DESIGN Systematic review. DATA SOURCES Medline, CINAHL, Embase and Cochrane database of systematic reviews from 1997 to 2019. INCLUSION CRITERIA Randomised trials and observational studies of internal or external validation of EWS to predict deterioration (mortality, intensive care unit (ICU) transfer and cardiac arrest) in disease subgroups or clinical settings. RESULTS We identified 770 studies, of which 103 were included. Study designs and methods were inconsistent, with significant risk of bias (high: n=16 and unclear: n=64 and low risk: n=28). There were only two randomised trials. There was a high degree of heterogeneity in all subgroups and in national early warning score (I2=72%-99%). Predictive accuracy (mean area under the curve; 95% CI) was highest in medical (0.74; 0.74 to 0.75) and surgical (0.77; 0.75 to 0.80) settings and respiratory diseases (0.77; 0.75 to 0.80). Few studies evaluated EWS in specific diseases, for example, cardiology (n=1) and respiratory (n=7). Mortality and ICU transfer were most frequently studied outcomes, and cardiac arrest was least examined (n=8). Integration with electronic health records was uncommon (n=9). CONCLUSION Methodology and quality of validation studies of EWS are insufficient to recommend their use in all diseases and all clinical settings despite good performance of EWS in some subgroups. There is urgent need for consistency in methods and study design, following consensus guidelines for predictive risk scores. Further research should consider specific diseases and settings, using electronic health record data, prior to large-scale implementation. PROSPERO REGISTRATION NUMBER PROSPERO CRD42019143141.
Collapse
Affiliation(s)
- Baneen Alhmoud
- Institute of Health Informatics, University College London, London, UK
| | - Timothy Bonnici
- Institute of Health Informatics, University College London, London, UK
- University College London Hospitals NHS Trust, London, UK
| | - Riyaz Patel
- University College London Hospitals NHS Trust, London, UK
- Institute of Cardiovascular Science, University College London, London, UK
- Barts Health NHS Trust, London, UK
| | | | - Bryan Williams
- University College London Hospitals NHS Trust, London, UK
- Institute of Cardiovascular Science, University College London, London, UK
| | - Amitava Banerjee
- Institute of Health Informatics, University College London, London, UK
- University College London Hospitals NHS Trust, London, UK
- Barts Health NHS Trust, London, UK
| |
Collapse
|
642
|
van Smeden M, Reitsma JB, Riley RD, Collins GS, Moons KG. Clinical prediction models: diagnosis versus prognosis. J Clin Epidemiol 2021; 132:142-145. [PMID: 33775387 DOI: 10.1016/j.jclinepi.2021.01.009] [Citation(s) in RCA: 50] [Impact Index Per Article: 16.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/05/2020] [Revised: 01/13/2021] [Accepted: 01/14/2021] [Indexed: 12/19/2022]
Abstract
Clinical prediction models play an increasingly important role in contemporary clinical care, by informing healthcare professionals, patients and their relatives about outcome risks, with the aim to facilitate (shared) medical decision making and improve health outcomes. Diagnostic prediction models aim to calculate an individual's risk that a disease is already present, whilst prognostic prediction models aim to calculate the risk of particular heath states occurring in the future. This article serves as a primer for diagnostic and prognostic clinical prediction models, by discussing the basic terminology, some of the inherent challenges, and the need for validation of predictive performance and the evaluation of impact of these models in clinical care.
Collapse
Affiliation(s)
- Maarten van Smeden
- Julius Center for Health Science and Primary Care, University Medical Center Utrecht, University of Utrecht, Utrecht, the Netherlands.
| | - Johannes B Reitsma
- Julius Center for Health Science and Primary Care, University Medical Center Utrecht, University of Utrecht, Utrecht, the Netherlands
| | - Richard D Riley
- Centre for Prognosis Research, School of Medicine, Keele University, Staffordshire, UK
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Botnar Research Centre, Oxford, UK; NIHR Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford, UK
| | - Karel Gm Moons
- Julius Center for Health Science and Primary Care, University Medical Center Utrecht, University of Utrecht, Utrecht, the Netherlands
| |
Collapse
|
643
|
Dong W, Wan EYF, Fong DYT, Kwok RLP, Chao DVK, Tan KCB, Hui EMT, Tsui WWS, Chan KH, Fung CSC, Lam CLK. Prediction models and nomograms for 10-year risk of end-stage renal disease in Chinese type 2 diabetes mellitus patients in primary care. Diabetes Obes Metab 2021; 23:897-909. [PMID: 33319467 DOI: 10.1111/dom.14292] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/09/2020] [Revised: 11/28/2020] [Accepted: 12/07/2020] [Indexed: 12/30/2022]
Abstract
AIMS To develop and validate 10-year risk prediction models, nomograms and charts for end-stage renal disease (ESRD) in Chinese patients with type 2 diabetes mellitus (T2DM) in primary care, in order to guide individualized treatment. MATERIALS AND METHODS This was a 10-year population-based observational cohort study. A total of 141 516 Chinese T2DM patients without history of cardiovascular disease or ESRD who were managed in public primary care clinics in 2008 were included and followed up until December 2017. Two-thirds of these patients were randomly selected to develop sex-specific ESRD risk prediction models using Cox regressions. The validity and accuracy of the models were tested on the remaining third of patients using Harrell's C-index. We selected variables based on their clinical and statistical importance to construct the nomograms and charts. RESULTS The median follow-up period was 9.75 years. The cumulative incidence of ESRD was 6.0% (men: 6.1%, women: 5.9%). Age, diabetes duration, systolic blood pressure (SBP), SBP variability, diastolic blood pressure, triglycerides, glycated haemoglobin (HbA1c), HbA1c variability, urine albumin to creatinine ratio (UACR), and estimated glomerular filtration rate (eGFR) were significant predictors for both sexes. Smoking and total cholesterol to HDL cholesterol ratio were additional significant predictors for men and women, respectively. The models showed Harrell's C-statistics of 0.889/0.889 (women/men). Age, eGFR, UACR, SBP and HbA1c were selected for both sexes to develop nomograms and charts. CONCLUSIONS Using routinely available variables, the 10-year ESRD risk of Chinese T2DM patients in primary care can be predicted with approximately 90% accuracy. We have developed different tools to facilitate routine ESRD risk prediction in primary care, so that individualized care can be provided to prevent or delay ESRD in T2DM patients.
Collapse
Affiliation(s)
- Weinan Dong
- Department of Family Medicine and Primary Care, University of Hong Kong, Hong Kong
| | - Eric Yuk Fai Wan
- Department of Family Medicine and Primary Care, University of Hong Kong, Hong Kong
- Department of Pharmacology and Pharmacy, University of Hong Kong, Hong Kong
| | | | - Ruby Lai Ping Kwok
- Department of Primary and Community Services, Hospital Authority, Hong Kong
| | - David Vai Kiong Chao
- Department of Family Medicine and Primary Health Care, Kowloon East Cluster, Hospital Authority, Hong Kong
| | | | - Eric Ming Tung Hui
- Department of Family Medicine, New Territories East Cluster, Hospital Authority, Hong Kong
| | - Wendy Wing Sze Tsui
- Family Medicine and Primary Healthcare, QMH, Hong Kong West Cluster, Hospital Authority, Hong Kong
| | | | | | - Cindy Lo Kuen Lam
- Department of Family Medicine and Primary Care, University of Hong Kong, Hong Kong
| |
Collapse
|
644
|
Langerak AJ, McCambridge AB, Stubbs PW, Fabricius J, Rogers K, Quel de Oliveira C, Nielsen JF, Verhagen AP. Externally validated model predicting gait independence after stroke showed fair performance and improved after updating. J Clin Epidemiol 2021; 137:73-82. [PMID: 33812010 DOI: 10.1016/j.jclinepi.2021.03.022] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2021] [Revised: 03/21/2021] [Accepted: 03/25/2021] [Indexed: 12/23/2022]
Abstract
OBJECTIVE To externally validate recent prognostic models that predict independent gait following stroke. STUDY DESIGN AND SETTING A systematic search identified recent models (<10 years) that predicted independent gait in adult stroke patients, using easily obtainable predictors. Predictors from the original models were assigned proxies when required, and model performance was evaluated in the validation cohort (n = 957). Models were updated to determine if performance could be improved. RESULTS Three prognostic models met our criteria, all with high Risk of Bias. Validation data was only available for the Australian model. This model used National Institute of Health Stroke Scale (NIHSS) and age to predict independent gait, using Motor Assessment Scale (MAS) walking item. For validation, Scandinavian Stroke Scale (SSS) was a proxy for NIHSS, and Functional Independence Measure (FIM) locomotion item was a proxy for MAS. The Area Under the Curve was 0.77 (0.74-0.80) and had good calibration in the validation dataset. Adjustment of the intercept and regression coefficients slightly improved discrimination. By adding paretic leg strength, the model further improved (AUC 0.82). CONCLUSION External validation of the Australian model with proxies showed fair discrimination and good calibration. Updating the model by adding paretic leg strength further improved model performance.
Collapse
Affiliation(s)
- Anthonia J Langerak
- University of Technology Sydney, Graduate School of Health, Discipline of Physiotherapy, Sydney, Australia; Utrecht University, University Medical Center Utrecht, Physical Therapy Sciences, program in Clinical Health Sciences, Utrecht, the Netherlands
| | - Alana B McCambridge
- University of Technology Sydney, Graduate School of Health, Discipline of Physiotherapy, Sydney, Australia
| | - Peter W Stubbs
- University of Technology Sydney, Graduate School of Health, Discipline of Physiotherapy, Sydney, Australia
| | - Jesper Fabricius
- Hammel Neurorehabilitation Centre and University Research Clinic, Aarhus University, Hammel, Denmark
| | - Kris Rogers
- University of Technology Sydney, Graduate School of Health, Discipline of Physiotherapy, Sydney, Australia
| | - Camila Quel de Oliveira
- University of Technology Sydney, Graduate School of Health, Discipline of Physiotherapy, Sydney, Australia
| | - Jørgen F Nielsen
- Hammel Neurorehabilitation Centre and University Research Clinic, Aarhus University, Hammel, Denmark
| | - Arianne P Verhagen
- University of Technology Sydney, Graduate School of Health, Discipline of Physiotherapy, Sydney, Australia.
| |
Collapse
|
645
|
Mark DG, Huang J, Kene MV, Sax DR, Cotton DM, Lin JS, Bouvet SC, Chettipally UK, Anderson ML, McLachlan ID, Simon LE, Shan J, Rauchwerger AS, Vinson DR, Ballard DW, Reed ME. Prospective Validation and Comparative Analysis of Coronary Risk Stratification Strategies Among Emergency Department Patients With Chest Pain. J Am Heart Assoc 2021; 10:e020082. [PMID: 33787290 PMCID: PMC8174350 DOI: 10.1161/jaha.120.020082] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
Abstract
Background Coronary risk stratification is recommended for emergency department patients with chest pain. Many protocols are designed as “rule‐out” binary classification strategies, while others use graded‐risk stratification. The comparative performance of competing approaches at varying levels of risk tolerance has not been widely reported. Methods and Results This is a prospective cohort study of adult patients with chest pain presenting between January 2018 and December 2019 to 13 medical center emergency departments within an integrated healthcare delivery system. Using an electronic clinical decision support interface, we externally validated and assessed the net benefit (at varying risk thresholds) of several coronary risk scores (History, ECG, Age, Risk Factors, and Troponin [HEART] score, HEART pathway, Emergency Department Assessment of Chest Pain Score Accelerated Diagnostic Protocol), troponin‐only strategies (fourth‐generation assay), unstructured physician gestalt, and a novel risk algorithm (RISTRA‐ACS). The primary outcome was 60‐day major adverse cardiac event defined as myocardial infarction, cardiac arrest, cardiogenic shock, coronary revascularization, or all‐cause mortality. There were 13 192 patient encounters included with a 60‐day major adverse cardiac event incidence of 3.7%. RISTRA‐ACS and HEART pathway had the lowest negative likelihood ratios (0.06, 95% CI, 0.03–0.10 and 0.07, 95% CI, 0.04–0.11, respectively) and the greatest net benefit across a range of low‐risk thresholds. RISTRA‐ACS demonstrated the highest discrimination for 60‐day major adverse cardiac event (area under the receiver operating characteristic curve 0.92, 95% CI, 0.91–0.94, P<0.0001). Conclusions RISTRA‐ACS and HEART pathway were the optimal rule‐out approaches, while RISTRA‐ACS was the best‐performing graded‐risk approach. RISTRA‐ACS offers promise as a versatile single approach to emergency department coronary risk stratification. Registration URL: https://www.clinicaltrials.gov; Unique identifier: NCT03286179.
Collapse
Affiliation(s)
- Dustin G Mark
- Department of Emergency Medicine Kaiser Permanente Oakland Medical Center Oakland CA.,Department of Critical Care Medicine Kaiser Permanente Oakland Medical Center Oakland CA.,Division of Research Kaiser Permanente Northern California Oakland CA
| | - Jie Huang
- Division of Research Kaiser Permanente Northern California Oakland CA
| | - Mamata V Kene
- Department of Emergency Medicine Kaiser Permanente San Leandro Medical Center San Leandro CA
| | - Dana R Sax
- Department of Emergency Medicine Kaiser Permanente Oakland Medical Center Oakland CA.,Division of Research Kaiser Permanente Northern California Oakland CA
| | - Dale M Cotton
- Department of Emergency Medicine Kaiser Permanente South Sacramento Medical Center Sacramento CA
| | - James S Lin
- Department of Emergency Medicine Kaiser Permanente Santa Clara Medical Center Santa Clara CA
| | - Sean C Bouvet
- Department of Emergency Medicine Kaiser Permanente Walnut Creek Medical Center Walnut Creek CA
| | - Uli K Chettipally
- Department of Emergency Medicine Kaiser Permanente South San Francisco Medical Center South San Francisco CA
| | - Megan L Anderson
- Department of Emergency Medicine Kaiser Permanente Roseville Medical Center Roseville CA
| | - Ian D McLachlan
- Department of Emergency Medicine Kaiser Permanente San Francisco Medical Center San Francisco CA
| | - Laura E Simon
- University of California San Diego School of Medicine San Diego CA
| | - Judy Shan
- Division of Research Kaiser Permanente Northern California Oakland CA
| | | | - David R Vinson
- Division of Research Kaiser Permanente Northern California Oakland CA.,Department of Emergency Medicine Kaiser Permanente Roseville Medical Center Roseville CA
| | - Dustin W Ballard
- Division of Research Kaiser Permanente Northern California Oakland CA.,Department of Emergency Medicine Kaiser Permanente San Rafael Medical Center San Rafael CA
| | - Mary E Reed
- Division of Research Kaiser Permanente Northern California Oakland CA
| | | |
Collapse
|
646
|
Burns KEA, Rizvi L, Cook DJ, Lebovic G, Dodek P, Villar J, Slutsky AS, Jones A, Kapadia FN, Gattas DJ, Epstein SK, Pelosi P, Kefala K, Meade MO. Ventilator Weaning and Discontinuation Practices for Critically Ill Patients. JAMA 2021; 325:1173-1184. [PMID: 33755077 PMCID: PMC7988370 DOI: 10.1001/jama.2021.2384] [Citation(s) in RCA: 50] [Impact Index Per Article: 16.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/28/2020] [Accepted: 02/09/2021] [Indexed: 01/31/2023]
Abstract
Importance Although most critically ill patients receive invasive mechanical ventilation (IMV), few studies have characterized how IMV is discontinued in practice. Objective To describe practice variation in IMV discontinuation internationally, associations between initial discontinuation events and outcomes, and factors associated with the use of select discontinuation strategies and failed initial spontaneous breathing trials (SBTs). Design, Setting, and Participants Prospective, multinational, observational study of critically ill adults who received IMV for at least 24 hours from 142 intensive care units (ICUs) in 19 countries within 6 regions (27 in Canada, 23 in India, 22 in the UK, 26 in Europe, 21 in Australia/New Zealand, and 23 in the US). Exposures Receiving IMV. Main Outcomes and Measures Primary analyses characterized types of initial IMV discontinuation events (extubation, SBT, or tracheostomy) and associations with clinical outcomes (including duration of ventilation, ICU and hospital mortality, and ICU and hospital length of stay). Secondary analyses examined the associations between SBT outcome and SBT timing and clinical outcomes. Results Among 1868 patients (median [interquartile range] age, 61.8 [48.9-73.1] years; 1173 [62.8%] men) 424 (22.7%) underwent direct extubation, 930 (49.8%) had an initial SBT (761 [81.8%] successful), 150 (8.0%) underwent direct tracheostomy, and 364 (19.5%) died before a weaning attempt. Across regions, there was variation in the use of written directives to guide care, daily screening, SBT techniques, ventilator modes, and the roles played by clinicians involved in weaning. Compared with initial direct extubation, patients who had an initial SBT had higher ICU mortality (20 [4.7%] vs 96 [10.3%]; absolute difference, 5.6% [95% CI, 2.6%-8.6%]), longer duration of ventilation (median of 2.9 vs 4.1 days; absolute difference, 1.2 days [95% CI, 0.7-1.6]), and longer ICU stay (median of 6.7 vs 8.1 days; absolute difference, 1.4 days [95% CI, 0.8-2.4]). Patients whose initial SBT failed (vs passed) had higher ICU mortality (29 [17.2%] vs 67 [8.8%]; absolute difference, 8.4% [95% CI, 2.0%-14.7%]), longer duration of ventilation (median of 6.1 vs 3.5 days; absolute difference, 2.6 days [95% CI, 1.6-3.6]), and longer ICU stay (median of 10.6 vs 7.7 days; absolute difference, 2.8 days [95% CI, 1.1-5.2]). Compared with patients who underwent early initial SBTs, patients who underwent late initial SBTs (>2.3 days after intubation) had longer duration of ventilation (median of 2.1 vs 6.1 days; absolute difference, 4.0 days [95% CI, 3.7-4.5]), longer ICU stay (median of 5.9 vs 10.8 days; absolute difference, 4.9 days [95% CI, 4.0-6.3]), and longer hospital stay (median of 14.3 vs 22.8 days; absolute difference, 8.5 days [95% CI, 6.0-11.0]). Conclusions and Relevance In this observational study of invasive mechanical ventilation discontinuation in 142 ICUs in Canada, India, the UK, Europe, Australia/New Zealand, and the US from 2013 to 2016, weaning practices varied internationally. Trial Registration ClinicalTrials.gov Identifier: NCT03955874.
Collapse
Affiliation(s)
- Karen E. A. Burns
- Interdepartmental Division of Critical Care, University of Toronto, Toronto, Ontario, Canada
- Division of Critical Care Medicine, Department of Medicine, Unity Health Toronto, St Michael's Hospital, Toronto, Ontario, Canada
- Li Ka Shing Knowledge Institute, St Michael’s Hospital, Toronto, Ontario, Canada
| | - Leena Rizvi
- Division of Critical Care Medicine, Department of Medicine, Unity Health Toronto, St Michael's Hospital, Toronto, Ontario, Canada
| | - Deborah J. Cook
- Division of Critical Care Medicine, St Joseph’s Hospital, Hamilton, Ontario, Canada
- Departments of Medicine and Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada
| | - Gerald Lebovic
- Applied Health Research Centre, St Michael’s Hospital, Toronto, Ontario, Canada
| | - Peter Dodek
- Centre for Health Evaluation and Outcome Sciences, Division of Critical Care Medicine, St Paul’s Hospital, University of British Columbia, Vancouver, British Columbia, Canada
| | - Jesús Villar
- Li Ka Shing Knowledge Institute, St Michael’s Hospital, Toronto, Ontario, Canada
- CIBER de Enfermedades Respiratorias, Instituto de Salud Carlos III, Madrid, Spain
- Multidisciplinary Organ Dysfunction Evaluation Research Network, Research Unit, Hospital Universitario Dr Negrin, Las Palmas de Gran Canaria, Spain
| | - Arthur S. Slutsky
- Interdepartmental Division of Critical Care, University of Toronto, Toronto, Ontario, Canada
- Division of Critical Care Medicine, Department of Medicine, Unity Health Toronto, St Michael's Hospital, Toronto, Ontario, Canada
- Li Ka Shing Knowledge Institute, St Michael’s Hospital, Toronto, Ontario, Canada
| | - Andrew Jones
- Department of Critical Care Medicine, Guy’s and St Thomas’ NHS Foundation Trust, London, United Kingdom
| | - Farhad N. Kapadia
- Department of Intensive Care, Hinduja National Hospital, Bombay, India
| | - David J. Gattas
- Intensive Care Unit, Royal Prince Alfred Hospital, University of Sydney, Camperdown, New South Wales, Australia
- The George Institute for Global Health, Sydney Australia
| | | | - Paolo Pelosi
- Department of Surgical Sciences and Integrated Diagnostics, University of Genoa, Genoa, Italy
- San Martino Policlinico Hospital, IRCCS for Oncology and Neurosciences, Genoa, Italy
| | - Kallirroi Kefala
- Anaesthesia, Critical Care and Pain Medicine, Edinburgh Royal Infirmary, Edinburgh, Scotland, United Kingdom
| | - Maureen O. Meade
- Departments of Medicine and Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada
| |
Collapse
|
647
|
Bullock GS, Hughes T, Sergeant JC, Callaghan MJ, Collins GS, Riley RD. Improving prediction model systematic review methodology: Letter to the Editor. TRANSLATIONAL SPORTS MEDICINE 2021. [DOI: 10.1002/tsm2.240] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
Affiliation(s)
- Garrett S. Bullock
- Nuffield Department of Rheumatology and Musculoskeletal Sciences University of Oxford Oxford UK
| | - Tom Hughes
- Manchester United Football Club Manchester UK
| | | | - Michael J. Callaghan
- Manchester United Football Club Manchester UK
- Centre for Biostatistics University of Manchester Manchester UK
- Department of Health Professions Manchester Metropolitan University Manchester UK
| | - Gary S. Collins
- Centre for Statistics in Medicine University of Oxford Oxford UK
| | - Richard D. Riley
- Centre for Prognosis Research School of Medicine Keele University Keele UK
| |
Collapse
|
648
|
Machine Learning, Predictive Analytics, and the Emperor's New Clothes: Why Artificial Intelligence Has Not Yet Replaced Conventional Approaches. Can J Cardiol 2021; 37:1156-1158. [PMID: 33711476 DOI: 10.1016/j.cjca.2021.03.003] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2021] [Revised: 03/02/2021] [Accepted: 03/02/2021] [Indexed: 12/23/2022] Open
|
649
|
Qu J, Li M, Zeng X, Zhang X, Wei W, Zuo X, Zhu P, Ye S, Zhang W, Zheng Y, Qi W, Li Y, Zhang Z, Ding F, Gu J, Liu Y, Zhang M, Qian J, Huang C, Zhao J, Wang Q, Liu Y, Tian Z, Wang Y. Validation of the REVEAL Prognostic Models in Systemic Lupus Erythematosus-Associated Pulmonary Arterial Hypertension. Front Med (Lausanne) 2021; 8:618486. [PMID: 33748158 PMCID: PMC7969505 DOI: 10.3389/fmed.2021.618486] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2020] [Accepted: 02/09/2021] [Indexed: 02/05/2023] Open
Abstract
No previous studies have investigated the predictive performance of the Registry to Evaluate Early and Long-term Pulmonary Arterial Hypertension Disease Management (REVEAL) prognostic equation and simplified risk score calculator in patients with systemic lupus erythematosus-associated pulmonary arterial hypertension (SLE-PAH). We aimed to validate these prediction tools in an external cohort of patients with SLE-PAH. In this study, the validation cohort consisted of patients with SLE-PAH registered in a prospective, multicenter, nationwide database between November 2006 and May2016. The follow-up of patients was censored at 1 year. Discrimination, calibration, model fit, and risk stratification of the REVEAL prognostic equation and simplified risk score calculator were validated. As a result, a total of 306 patients with SLE-PAH were included. The 1-year overall survival rate was 91.5%. The C-index of the prognostic equation was 0.736, demonstrating reasonably good discrimination, and it was greater than that for the simplified risk score calculator (0.710). The overall calibration slope was 0.83, and the Brier score was 0.079. The risk of renal insufficiency and World Health Organization Functional Class III (WHO FC III) were underestimated, and the risk assigned to a heart rate >92 bpm in the REVEAL prognostic models was not observed in our validation cohort. Both model discrimination and calibration were poor in the very high-risk group. In conclusion, the REVEAL models exhibit good discriminatory ability when predicting 1-year overall survival in patients with SLE-PAH. Findings from both models should be interpreted with caution in cases of renal insufficiency, WHO FC III, and heart rate >92 bpm.
Collapse
Affiliation(s)
- Jingge Qu
- Department of Rheumatology and Clinical Immunology, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital (PUMCH), Chinese Academy of Medical Sciences and Peking Union Medical College, National Clinical Research Center for Dermatologic and Immunologic Diseases (NCRC-DID), Ministry of Science and Technology, Key Laboratory of Rheumatology and Clinical Immunology, Ministry of Education, Beijing, China
| | - Mengtao Li
- Department of Rheumatology and Clinical Immunology, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital (PUMCH), Chinese Academy of Medical Sciences and Peking Union Medical College, National Clinical Research Center for Dermatologic and Immunologic Diseases (NCRC-DID), Ministry of Science and Technology, Key Laboratory of Rheumatology and Clinical Immunology, Ministry of Education, Beijing, China
| | - Xiaofeng Zeng
- Department of Rheumatology and Clinical Immunology, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital (PUMCH), Chinese Academy of Medical Sciences and Peking Union Medical College, National Clinical Research Center for Dermatologic and Immunologic Diseases (NCRC-DID), Ministry of Science and Technology, Key Laboratory of Rheumatology and Clinical Immunology, Ministry of Education, Beijing, China
| | - Xiao Zhang
- Department of Rheumatology, Guangdong General Hosptal, Guangzhou, China
| | - Wei Wei
- Department of Rheumatology, Tianjin Medical University General Hospital, Tianjin, China
| | - Xiaoxia Zuo
- Department of Rheumatology, Xiangya Hospital, Central South University, Changsha, China
| | - Ping Zhu
- Department of Clinical Immunology, People's Liberation Army Specialized Research Institute of Rheumatology and Immunology, Xijing Hospital, Fourth Military Medical University, Xi'an, China
| | - Shuang Ye
- Department of Rheumatology, Ren Ji Hospital, School of Medicine, Shanghai Jiao Tong University, Shanghai, China
| | - Wei Zhang
- Department of Rheumatology, Ren Ji Hospital, School of Medicine, Shanghai Jiao Tong University, Shanghai, China
| | - Yi Zheng
- Department of Rheumatology, Beijing Chao-Yang Hospital, Capital Medical University, Beijing, China
| | - Wufang Qi
- Department of Rheumatology, The First Central Hospital, Tianjin, China
| | - Yang Li
- Department of Rheumatology, The Second Affiliated Hospital of Harbin Medical University, Harbin, China
| | - Zhuoli Zhang
- Department of Rheumatology and Clinical Immunology, Peking University First Hospital, Beijing, China
| | - Feng Ding
- Department of Rheumatology, Qilu Hospital of Shandong University, Jinan, China
| | - Jieruo Gu
- Department of Rheumatology, The Third Affiliated Hospital of Sun Yat-sen University, Guangzhou, China
| | - Yi Liu
- Department of Rheumatology and Immunology, West China Hospital, Sichuan University, Chengdu, China
| | - Miaojia Zhang
- Department of Rheumatology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China
| | - Junyan Qian
- Department of Rheumatology and Clinical Immunology, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital (PUMCH), Chinese Academy of Medical Sciences and Peking Union Medical College, National Clinical Research Center for Dermatologic and Immunologic Diseases (NCRC-DID), Ministry of Science and Technology, Key Laboratory of Rheumatology and Clinical Immunology, Ministry of Education, Beijing, China
| | - Can Huang
- Department of Rheumatology and Clinical Immunology, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital (PUMCH), Chinese Academy of Medical Sciences and Peking Union Medical College, National Clinical Research Center for Dermatologic and Immunologic Diseases (NCRC-DID), Ministry of Science and Technology, Key Laboratory of Rheumatology and Clinical Immunology, Ministry of Education, Beijing, China
| | - Jiuliang Zhao
- Department of Rheumatology and Clinical Immunology, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital (PUMCH), Chinese Academy of Medical Sciences and Peking Union Medical College, National Clinical Research Center for Dermatologic and Immunologic Diseases (NCRC-DID), Ministry of Science and Technology, Key Laboratory of Rheumatology and Clinical Immunology, Ministry of Education, Beijing, China
| | - Qian Wang
- Department of Rheumatology and Clinical Immunology, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital (PUMCH), Chinese Academy of Medical Sciences and Peking Union Medical College, National Clinical Research Center for Dermatologic and Immunologic Diseases (NCRC-DID), Ministry of Science and Technology, Key Laboratory of Rheumatology and Clinical Immunology, Ministry of Education, Beijing, China
| | - Yongtai Liu
- Department of Cardiology, Peking Union Medical College Hospital, Peking Union Medical College and Chinese Academy of Medical Sciences, Beijing, China
| | - Zhuang Tian
- Department of Cardiology, Peking Union Medical College Hospital, Peking Union Medical College and Chinese Academy of Medical Sciences, Beijing, China
| | - Yanhong Wang
- Department of Epidemiology and Bio-Statistics, Institute of Basic Medical Sciences, China Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| |
Collapse
|
650
|
González Maldonado S, Hynes LC, Motsch E, Heussel CP, Kauczor HU, Robbins HA, Delorme S, Kaaks R. Validation of multivariable lung cancer risk prediction models for the personalized assignment of optimal screening frequency: a retrospective analysis of data from the German Lung Cancer Screening Intervention Trial (LUSI). Transl Lung Cancer Res 2021; 10:1305-1317. [PMID: 33889511 PMCID: PMC8044498 DOI: 10.21037/tlcr-20-1173] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/05/2020] [Accepted: 01/25/2021] [Indexed: 12/23/2022]
Abstract
BACKGROUND Current guidelines for lung cancer screening via low-dose computed tomography recommend annual screening for all candidates meeting basic eligibility criteria. However, lung cancer risk of eligible screening participants can vary widely, and further risk stratification could be used to individually optimize screening intervals in view of expected benefits, possible harms and financial costs. To this effect, models have been developed in the US National Lung Screening Trial based on self-reported lung cancer risk factors and imaging data. We evaluated these models using data from an independent screening trial in Germany. METHODS We examined the Polynomial model by Schreuder et al., the Lung Cancer Risk Assessment Tool extended by CT characteristics (LCRAT + CT) by Robbins et al., and a criterion of presence vs. absence of pulmonary nodules ≥4 mm (Patz et al.), applied to sub-sets of screening participants according to eligibility criteria. Discrimination was evaluated via the receiver operating characteristic curve. Delayed diagnoses and false positive results were calculated at various thresholds of predicted risk. Model calibration was assessed by comparing mean predicted risk versus observed incidence. RESULTS One thousand five hundred and six participants were eligible for the validation of the LCRAT + CT model, and 1,889 for the validation of the Polynomial model and Patz criterion, yielding areas under the receiver operating characteristic curve of 0.73 (95% CI: 0.63, 0.82), 0.75 (0.67, 0.83), and 0.56 (0.53, 0.72) respectively. Skipping 50% annual screenings (participants within the 5 lowest risk deciles by LCRAT + CT in any round or by the Polynomial model; baseline screening round), would have avoided 75% (21.9%, 98.7%) and 40% (21.8%, 61.1%) false positive screen tests and delayed 10% (1.8%, 33.1%) or no (0%, 32.1%) diagnoses, respectively. Using the Patz criterion, referring 63.2% (61.0% to 65.4%) of participants to biennial screening would have avoided 4% (0.2% to 22.3%) of false positive screen tests but delayed 55% (24.6% to 81.9%) diagnoses. CONCLUSIONS In this German trial, the LCRAT + CT and Polynomial models showed useful discrimination of screening participants for one-year lung cancer risk following CT examination. Our results illustrate the remaining heterogeneity in risk within screening-eligible subjects and the trade-off between a low-frequency screening approach and delayed detection.
Collapse
Affiliation(s)
- Sandra González Maldonado
- Division of Cancer Epidemiology (C020), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 581, 69120 Heidelberg, Germany
- Translational Lung Research Center Heidelberg (TLRC-H), Member of the German Center for Lung Research, Heidelberg, Germany
| | - Lucas Cory Hynes
- Division of Cancer Epidemiology (C020), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 581, 69120 Heidelberg, Germany
- Translational Lung Research Center Heidelberg (TLRC-H), Member of the German Center for Lung Research, Heidelberg, Germany
| | - Erna Motsch
- Division of Cancer Epidemiology (C020), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 581, 69120 Heidelberg, Germany
| | - Claus-Peter Heussel
- Translational Lung Research Center Heidelberg (TLRC-H), Member of the German Center for Lung Research, Heidelberg, Germany
- Department of Diagnostic and Interventional Radiology with Nuclear Medicine, Thoraxklinik Heidelberg, Heidelberg University, Heidelberg, Germany
| | - Hans-Ulrich Kauczor
- Translational Lung Research Center Heidelberg (TLRC-H), Member of the German Center for Lung Research, Heidelberg, Germany
- Department of Diagnostic and Interventional Radiology, Heidelberg University Clinic, Heidelberg, Germany
| | | | - Stefan Delorme
- Department of Radiology, German Cancer Research Center (DKFZ), Im Neuenheimer Feld 280, 69120 Heidelberg, Germany
| | - Rudolf Kaaks
- Division of Cancer Epidemiology (C020), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 581, 69120 Heidelberg, Germany
- Translational Lung Research Center Heidelberg (TLRC-H), Member of the German Center for Lung Research, Heidelberg, Germany
| |
Collapse
|