1
|
Zhuo XY, Lei SH, Sun L, Bai YW, Wu J, Zheng YJ, Liu KX, Liu WF, Zhao BC. Preoperative risk prediction models for acute kidney injury after noncardiac surgery: an independent external validation cohort study. Br J Anaesth 2024; 133:508-518. [PMID: 38527923 DOI: 10.1016/j.bja.2024.02.018] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2023] [Revised: 02/24/2024] [Accepted: 02/27/2024] [Indexed: 03/27/2024] Open
Abstract
BACKGROUND Numerous models have been developed to predict acute kidney injury (AKI) after noncardiac surgery, yet there is a lack of independent validation and comparison among them. METHODS We conducted a systematic literature search to review published risk prediction models for AKI after noncardiac surgery. An independent external validation was performed using a retrospective surgical cohort at a large Chinese hospital from January 2019 to October 2022. The cohort included patients undergoing a wide range of noncardiac surgeries with perioperative creatinine measurements. Postoperative AKI was defined according to the Kidney Disease Improving Global Outcomes creatinine criteria. Model performance was assessed in terms of discrimination (area under the receiver operating characteristic curve, AUROC), calibration (calibration plot), and clinical utility (net benefit), before and after model recalibration through intercept and slope updates. A sensitivity analysis was conducted by including patients without postoperative creatinine measurements in the validation cohort and categorising them as non-AKI cases. RESULTS Nine prediction models were evaluated, each with varying clinical and methodological characteristics, including the types of surgical cohorts used for model development, AKI definitions, and predictors. In the validation cohort involving 13,186 patients, 650 (4.9%) developed AKI. Three models demonstrated fair discrimination (AUROC between 0.71 and 0.75); other models had poor or failed discrimination. All models exhibited some miscalibration; five of the nine models were well-calibrated after intercept and slope updates. Decision curve analysis indicated that the three models with fair discrimination consistently provided a positive net benefit after recalibration. The results were confirmed in the sensitivity analysis. CONCLUSIONS We identified three models with fair discrimination and potential clinical utility after recalibration for assessing the risk of acute kidney injury after noncardiac surgery.
Collapse
Affiliation(s)
- Xiao-Yu Zhuo
- Department of Anaesthesiology, Nanfang Hospital, Southern Medical University, Guangzhou, China; Guangdong Provincial Key Laboratory of Precision Anaesthesia and Perioperative Organ Protection, Guangzhou, China
| | - Shao-Hui Lei
- Department of Anaesthesiology, Nanfang Hospital, Southern Medical University, Guangzhou, China; Guangdong Provincial Key Laboratory of Precision Anaesthesia and Perioperative Organ Protection, Guangzhou, China; College of Anaesthesiology, Southern Medical University, Guangzhou, China
| | - Lan Sun
- Department of Anaesthesiology, Nanfang Hospital, Southern Medical University, Guangzhou, China; Department of Biostatistics, Lejiu Healthcare Technology Co., Ltd, Hangzhou, China
| | - Ya-Wen Bai
- College of Anaesthesiology, Southern Medical University, Guangzhou, China
| | - Jiao Wu
- College of Anaesthesiology, Southern Medical University, Guangzhou, China
| | - Yong-Jia Zheng
- College of Anaesthesiology, Southern Medical University, Guangzhou, China
| | - Ke-Xuan Liu
- Department of Anaesthesiology, Nanfang Hospital, Southern Medical University, Guangzhou, China; Guangdong Provincial Key Laboratory of Precision Anaesthesia and Perioperative Organ Protection, Guangzhou, China; College of Anaesthesiology, Southern Medical University, Guangzhou, China; Outcomes Research Consortium, Cleveland, OH, USA.
| | - Wei-Feng Liu
- Department of Anaesthesiology, Nanfang Hospital, Southern Medical University, Guangzhou, China; Guangdong Provincial Key Laboratory of Precision Anaesthesia and Perioperative Organ Protection, Guangzhou, China; College of Anaesthesiology, Southern Medical University, Guangzhou, China.
| | - Bing-Cheng Zhao
- Department of Anaesthesiology, Nanfang Hospital, Southern Medical University, Guangzhou, China; Guangdong Provincial Key Laboratory of Precision Anaesthesia and Perioperative Organ Protection, Guangzhou, China; College of Anaesthesiology, Southern Medical University, Guangzhou, China; Outcomes Research Consortium, Cleveland, OH, USA.
| |
Collapse
|
2
|
Kingsley V, Fox L, Simm D, Martin GP, Thompson W, Faisal M. External validation of the computer aided risk scoring system in predicting in-hospital mortality following emergency medical admissions. Int J Med Inform 2024; 188:105497. [PMID: 38781886 DOI: 10.1016/j.ijmedinf.2024.105497] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2023] [Revised: 05/10/2024] [Accepted: 05/17/2024] [Indexed: 05/25/2024]
Abstract
BACKGROUND Clinical prediction models have the potential to improve the quality of care and enhance patient safety outcomes. A Computer-aided Risk Scoring system (CARSS) was previously developed to predict in-hospital mortality following emergency admissions based on routinely collected blood tests and vitals. We aimed to externally validate the CARSS model. METHODS In this retrospective external validation study, we considered all adult (≥18 years) emergency medical admissions discharged between 11/11/2020 and 11/11/2022 from The Rotherham Foundation Trust (TRFT), UK. We assessed the predictive performance of the CARSS model based on its discriminative (c-statistic) and calibration characteristics (calibration slope and calibration plots). RESULTS Out of 32,774 admissions, 20,422 (62.3 %) admissions were included. The TRFT sample had similar demographic characteristics to the development sample but had higher mortality (6.1 % versus 5.7 %). The CARSS model demonstrated good discrimination (c-statistic 0.87 [95 % CI 0.86-0.88]) and good calibration to the TRFT dataset (slope = 1.03 [95 % CI 0.98-1.08] intercept = 0 [95 % CI -0.06-0.07]) after re-calibrating for differences in baseline mortality (intercept = 0.96 [95 % CI 0.90-1.03] before re-calibration). CONCLUSION In summary, the CARSS model is externally validated after correcting the baseline risk of death between development and validation datasets. External validation of the CARSS model showed that it under-predicted in-hospital mortality. Re-calibration of this model showed adequate performance in the TRFT dataset.
Collapse
Affiliation(s)
- Viveck Kingsley
- Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK; The Rotherham NHS Foundation Trust, Rotherham General Hospital, Rotherham, South Yorkshire, UK.
| | - Lisa Fox
- The Rotherham NHS Foundation Trust, Rotherham General Hospital, Rotherham, South Yorkshire, UK.
| | - David Simm
- The Rotherham NHS Foundation Trust, Rotherham General Hospital, Rotherham, South Yorkshire, UK.
| | - Glen P Martin
- Division of Informatics, Imaging & Data Sciences, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK.
| | - Wendy Thompson
- Division of Dentistry, University of Manchester, Manchester, UK.
| | - Muhammad Faisal
- Centre for Digital Innovations in Health & Social Care, Faculty of Health Studies, University of Bradford, Bradford, UK; Wolfson Centre for Applied Health Research, Bradford, UK.
| |
Collapse
|
3
|
Li Z, Lu J, Liu M, Ma L, Quan K, Zhang H, Liu P, Shi Y, Dong X, You C, Tian R, Zhu W. Development and Validation of a Supplementary Grading Scale for Outcomes of Brainstem Cavernous Malformations. Stroke 2024; 55:1991-2002. [PMID: 38881452 DOI: 10.1161/strokeaha.123.045943] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2023] [Accepted: 05/21/2024] [Indexed: 06/18/2024]
Abstract
BACKGROUND Surgical risk assessment is intriguing for clinical decision-making of brainstem cavernous malformation (BSCM) treatment. While the BSCM grading scale, encompassing size, developmental venous anomaly, crossing axial midpoint, age, and timing of intervention, is increasingly utilized, the clinical relevance of neurological fluctuation and recurrent hemorrhage has not been incorporated. This study aimed to propose a supplementary grading scale with enhanced predictive efficacy. METHODS Using a retrospective nationwide registry of consecutive patients with BSCMs undergoing surgery in China from March 2011 to May 2023, a new supplementary BSCM grading scale was developed from a derivative cohort of 260 patients and validated in an independent concurrent cohort of 67 patients. The primary outcome was unfavorable neurological function (modified Rankin Scale score >2) at the latest follow-up. The performance of the supplementary grading system was evaluated for discrimination, calibration, and clinical utility and further compared with its original counterpart. RESULTS Over a follow-up of at least 6 months after surgery, the unfavorable outcomes were 31% in the overall cohort (101/327 patients). A preoperative motor deficit (odds ratio, 3.13; P=0.001), recurrent hemorrhage (odds ratio, 3.05; P<0.001), timing of intervention (odds ratio, 7.08; P<0.001), and crossing the axial midpoint (odds ratio, 2.57; P=0.006) were associated with the unfavorable outcomes and composed the initial Huashan grading variables. A supplementary BSCM grading system was subsequently developed by incorporating the Huashan grading variables into the original BSCM grading scale. The predictive capability of the supplementary scale was consistently superior to the original counterpart in either the derivative cohort (area under the receiver operating characteristic curve, 0.74 [95% CI, 0.68-0.80] for the supplementary versus 0.68 [95% CI, 0.61-0.74] for the original) or the validation cohort (0.75 [95% CI, 0.62-0.87] versus 0.64 [95% CI, 0.48-0.81]). CONCLUSIONS This study highlights the neurological relevance of BSCM hemorrhage in surgical risk assessment. Via compositing preoperative motor function and recurrent hemorrhages, a supplementary grading scale may improve a dynamic risk assessment for clinical decisions in the management of BSCMs.
Collapse
Affiliation(s)
- Zongze Li
- Department of Neurosurgery, Huashan Hospital, Shanghai Medical College, Fudan University, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- National Center for Neurological Disorders, Shanghai, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Shanghai Key Laboratory of Brain Function and Restoration and Neural Regeneration, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Neurosurgical Institute of Fudan University, Shanghai, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Shanghai Clinical Medical Center of Neurosurgery, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
| | - Junlin Lu
- Department of Neurosurgery, West China Hospital, Sichuan University, Chengdu (J.L., C.Y., R.T.)
| | - Mingjian Liu
- Department of Neurosurgery, Huashan Hospital, Shanghai Medical College, Fudan University, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- National Center for Neurological Disorders, Shanghai, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Shanghai Key Laboratory of Brain Function and Restoration and Neural Regeneration, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Neurosurgical Institute of Fudan University, Shanghai, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Shanghai Clinical Medical Center of Neurosurgery, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
| | - Li Ma
- Department of Neurological Surgery, University of Pittsburgh Medical Center, University of Pittsburgh School of Medicine, PA (L.M.)
| | - Kai Quan
- Department of Neurosurgery, Huashan Hospital, Shanghai Medical College, Fudan University, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- National Center for Neurological Disorders, Shanghai, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Shanghai Key Laboratory of Brain Function and Restoration and Neural Regeneration, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Neurosurgical Institute of Fudan University, Shanghai, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Shanghai Clinical Medical Center of Neurosurgery, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
| | - Hongfei Zhang
- Department of Neurosurgery, Huashan Hospital, Shanghai Medical College, Fudan University, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- National Center for Neurological Disorders, Shanghai, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Shanghai Key Laboratory of Brain Function and Restoration and Neural Regeneration, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Neurosurgical Institute of Fudan University, Shanghai, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Shanghai Clinical Medical Center of Neurosurgery, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
| | - Peixi Liu
- Department of Neurosurgery, Huashan Hospital, Shanghai Medical College, Fudan University, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- National Center for Neurological Disorders, Shanghai, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Shanghai Key Laboratory of Brain Function and Restoration and Neural Regeneration, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Neurosurgical Institute of Fudan University, Shanghai, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Shanghai Clinical Medical Center of Neurosurgery, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
| | - Yuan Shi
- Department of Neurosurgery, Huashan Hospital, Shanghai Medical College, Fudan University, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- National Center for Neurological Disorders, Shanghai, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Shanghai Key Laboratory of Brain Function and Restoration and Neural Regeneration, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Neurosurgical Institute of Fudan University, Shanghai, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Shanghai Clinical Medical Center of Neurosurgery, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
| | - Xuchen Dong
- Department of Neurosurgery, Huashan Hospital, Shanghai Medical College, Fudan University, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- National Center for Neurological Disorders, Shanghai, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Shanghai Key Laboratory of Brain Function and Restoration and Neural Regeneration, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Neurosurgical Institute of Fudan University, Shanghai, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Shanghai Clinical Medical Center of Neurosurgery, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
| | - Chao You
- Department of Neurosurgery, West China Hospital, Sichuan University, Chengdu (J.L., C.Y., R.T.)
| | - Rui Tian
- Department of Neurosurgery, West China Hospital, Sichuan University, Chengdu (J.L., C.Y., R.T.)
| | - Wei Zhu
- Department of Neurosurgery, Huashan Hospital, Shanghai Medical College, Fudan University, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- National Center for Neurological Disorders, Shanghai, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Shanghai Key Laboratory of Brain Function and Restoration and Neural Regeneration, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Neurosurgical Institute of Fudan University, Shanghai, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
- Shanghai Clinical Medical Center of Neurosurgery, China (Z.L., M.L., K.Q., H.Z., P.L., Y.S., X.D., W.Z.)
| |
Collapse
|
4
|
Li Y, Zhang H, Sun Y, Fan Q, Wang L, Ji C, HuiGu, Chen B, Zhao S, Wang D, Yu P, Li J, Yang S, Zhang C, Wang X. Deep learning-based platform performs high detection sensitivity of intracranial aneurysms in 3D brain TOF-MRA: An external clinical validation study. Int J Med Inform 2024; 188:105487. [PMID: 38761459 DOI: 10.1016/j.ijmedinf.2024.105487] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/25/2023] [Revised: 05/06/2024] [Accepted: 05/15/2024] [Indexed: 05/20/2024]
Abstract
PURPOSE To evaluate the diagnostic efficacy of a developed artificial intelligence (AI) platform incorporating deep learning algorithms for the automated detection of intracranial aneurysms in time-of-flight (TOF) magnetic resonance angiography (MRA). METHOD This retrospective study encompassed 3D TOF MRA images acquired between January 2023 and June 2023, aiming to validate the presence of intracranial aneurysms via our developed AI platform. The manual segmentation results by experienced neuroradiologists served as the "gold standard". Following annotation of MRA images by neuroradiologists using InferScholar software, the AI platform conducted automatic segmentation of intracranial aneurysms. Various metrics including accuracy (ACC), balanced ACC, area under the curve (AUC), sensitivity (SE), specificity (SP), F1 score, Brier Score, and Net Benefit were utilized to evaluate the generalization of AI platform. Comparison of intracranial aneurysm identification performance was conducted between the AI platform and six radiologists with experience ranging from 3 to 12 years in interpreting MR images. Additionally, a comparative analysis was carried out between radiologists' detection performance based on independent visual diagnosis and AI-assisted diagnosis. Subgroup analyses were also performed based on the size and location of the aneurysms to explore factors impacting aneurysm detectability. RESULTS 510 patients were enrolled including 215 patients (42.16 %) with intracranial aneurysms and 295 patients (57.84 %) without aneurysms. Compared with six radiologists, the AI platform showed competitive discrimination power (AUC, 0.96), acceptable calibration (Brier Score loss, 0.08), and clinical utility (Net Benefit, 86.96 %). The AI platform demonstrated superior performance in detecting aneurysms with an overall SE, SP, ACC, balanced ACC, and F1 score of 91.63 %, 92.20 %, 91.96 %, 91.92 %, and 90.57 % respectively, outperforming the detectability of the two resident radiologists. For subgroup analysis based on aneurysm size and location, we observed that the SE of the AI platform for identifying tiny (diameter<3mm), small (3 mm ≤ diameter<5mm), medium (5 mm ≤ diameter<7mm) and large aneurysms (diameter ≥ 7 mm) was 87.80 %, 93.14 %, 95.45 %, and 100 %, respectively. Furthermore, the SE for detecting aneurysms in the anterior circulation was higher than that in the posterior circulation. Utilizing the AI assistance, six radiologists (i.e., two residents, two attendings and two professors) achieved statistically significant improvements in mean SE (residents: 71.40 % vs. 88.37 %; attendings: 82.79 % vs. 93.26 %; professors: 90.07 % vs. 97.44 %; P < 0.05) and ACC (residents: 85.29 % vs. 94.12 %; attendings: 91.76 % vs. 97.06 %; professors: 95.29 % vs. 98.82 %; P < 0.05) while no statistically significant change was observed in SP. Overall, radiologists' mean SE increased by 11.40 %, mean SP increased by 1.86 %, and mean ACC increased by 5.88 %, mean balanced ACC promoted by 6.63 %, mean F1 score grew by 7.89 %, and Net Benefit rose by 12.52 %, with a concurrent decrease in mean Brier score declined by 0.06. CONCLUSIONS The deep learning algorithms implemented in the AI platform effectively detected intracranial aneurysms on TOF-MRA and notably enhanced the diagnostic capabilities of radiologists. This indicates that the AI-based auxiliary diagnosis model can provide dependable and precise prediction to improve the diagnostic capacity of radiologists.
Collapse
Affiliation(s)
- Yuanyuan Li
- Department of Radiology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, China; Department of Radiology, Liaocheng People's Hospital, Shandong First Medical University & Shandong Academy of Medical Sciences, China; Department of Radiology, Shandong Provincial Hospital, Cheeloo College of Medicine, Shandong University, China
| | - Huiling Zhang
- Institute of Research, Infervision Medical Technology Co., Ltd, China
| | - Yun Sun
- Department of Radiology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, China
| | - Qianrui Fan
- Institute of Research, Infervision Medical Technology Co., Ltd, China
| | - Long Wang
- Department of Cardiovascular Surgery, Shandong Provincial Hospital Affiliated to Shandong First Medical University, China
| | - Congshan Ji
- Department of Radiology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, China
| | - HuiGu
- Department of Radiology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, China; Department of Radiology, Liaocheng People's Hospital, Shandong First Medical University & Shandong Academy of Medical Sciences, China; Department of Radiology, Shandong Provincial Hospital, Cheeloo College of Medicine, Shandong University, China
| | - Baojin Chen
- Department of Radiology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, China
| | - Shuo Zhao
- Department of Radiology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, China; Department of Radiology, Liaocheng People's Hospital, Shandong First Medical University & Shandong Academy of Medical Sciences, China; Department of Radiology, Shandong Provincial Hospital, Cheeloo College of Medicine, Shandong University, China
| | - Dawei Wang
- Institute of Research, Infervision Medical Technology Co., Ltd, China
| | - Pengxin Yu
- Institute of Research, Infervision Medical Technology Co., Ltd, China
| | - Junchen Li
- Department of Radiology, Changshu Hospital Affiliated to Nanjing University of Chinese Medicine, China
| | - Shifeng Yang
- Department of Radiology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, China.
| | - Chuanchen Zhang
- Department of Radiology, Liaocheng People's Hospital, Shandong First Medical University & Shandong Academy of Medical Sciences, China.
| | - Ximing Wang
- Department of Radiology, Shandong Provincial Hospital Affiliated to Shandong First Medical University, China; Department of Radiology, Shandong Provincial Hospital, Cheeloo College of Medicine, Shandong University, China.
| |
Collapse
|
5
|
Valsamis EM, Jensen ML, Coward G, Sayers A, Pinedo-Villanueva R, Rasmussen JV, Collins GS, Rees JL. Risk of serious adverse events after primary shoulder replacement: development and external validation of a prediction model using linked national data from England and Denmark. THE LANCET. RHEUMATOLOGY 2024:S2665-9913(24)00149-8. [PMID: 39096919 DOI: 10.1016/s2665-9913(24)00149-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/23/2024] [Revised: 05/15/2024] [Accepted: 05/16/2024] [Indexed: 08/05/2024]
Abstract
BACKGROUND Despite a rising rate of serious medical complications after shoulder replacement surgery, there are no prediction models in widespread use to guide surgeons in identifying patients at high risk and to provide patients with personalised risk estimates to support shared decision making. Our aim was to develop and externally validate a prediction model for serious adverse events within 90 days of primary shoulder replacement surgery. METHODS Linked data from the National Joint Registry, National Health Service Hospital Episode Statistics Admitted Patient Care of England, and Civil Registration Mortality databases and Danish Shoulder Arthroplasty Registry and National Patient Register were used for our modelling study. Patients aged 18-100 years who had a primary shoulder replacement between April 1, 2012, and Oct 2, 2020, in England, and April 1, 2012, and Oct 2, 2018, in Denmark, were included. We developed a multivariable logistic regression model using the English dataset to predict the risk of 90-day serious adverse events, which were defined as medical complications requiring admission to hospital and all-cause death. We undertook internal validation using bootstrapping, and internal-external cross-validation across different geographical regions of England. The English model was externally validated on the Danish dataset. FINDINGS Data for 40 631 patients undergoing primary shoulder replacement (mean age 72·5 years [SD 9·9]; 28 709 [70·7%] women and 11 922 [29·3%] men) were used for model development, of whom 2270 (5·6%) had a 90-day serious adverse event. On internal validation, the model had a C-statistic of 0·717 (95% CI 0·707-0·728) and was well calibrated. Internal-external cross-validation showed consistent model performance across all regions in England. Upon external validation on the Danish dataset (n=6653; mean age 70·5 years [SD 10·3]; 4503 [67·7%] women and 2150 [32·3%] men), the model had a C-statistic of 0·750 (95% CI 0·723-0·776). Decision curve analysis showed clinical utility, with net benefit across all risk thresholds. INTERPRETATION This externally validated prediction model uses commonly available clinical variables to accurately predict the risk of serious medical complications after primary shoulder replacement surgery. The model is generalisable and applicable to most patients in need of a shoulder replacement. Its use offers support to clinicians and could inform and empower patients in the shared decision-making process. FUNDING National Institute for Health and Care Research and the Department of Orthopaedic Surgery, Herlev and Gentofte Hospital, Denmark.
Collapse
Affiliation(s)
- Epaminondas Markos Valsamis
- Botnar Research Centre, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK; NIHR Oxford Biomedical Research Centre, Oxford, UK.
| | - Marie Louise Jensen
- Department of Orthopaedic surgery, Herlev and Gentofte Hospital, University of Copenhagen, Copenhagen, Denmark
| | - Gillian Coward
- National Joint Registry Research Committee, National Joint Registry, London, UK
| | - Adrian Sayers
- Musculoskeletal Research Unit, Bristol Medical School, Southmead Hospital, University of Bristol, Bristol, UK
| | - Rafael Pinedo-Villanueva
- Botnar Research Centre, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK; NIHR Oxford Biomedical Research Centre, Oxford, UK
| | - Jeppe V Rasmussen
- Department of Orthopaedic surgery, Herlev and Gentofte Hospital, University of Copenhagen, Copenhagen, Denmark
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK; NIHR Oxford Biomedical Research Centre, Oxford, UK
| | - Jonathan L Rees
- Botnar Research Centre, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK; NIHR Oxford Biomedical Research Centre, Oxford, UK
| |
Collapse
|
6
|
Gehringer CK, Martin GP, Van Calster B, Hyrich KL, Verstappen SMM, Sergeant JC. How to develop, validate, and update clinical prediction models using multinomial logistic regression. J Clin Epidemiol 2024; 174:111481. [PMID: 39067542 DOI: 10.1016/j.jclinepi.2024.111481] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2023] [Revised: 03/14/2024] [Accepted: 07/19/2024] [Indexed: 07/30/2024]
Abstract
OBJECTIVES Multicategory prediction models (MPMs) can be used in health care when the primary outcome of interest has more than two categories. The application of MPMs is scarce, possibly due to added methodological complexities compared to binary outcome models. We provide a guide of how to develop, validate, and update clinical prediction models based on multinomial logistic regression. STUDY DESIGN AND SETTING We present guidance and recommendations based on recent methodological literature, illustrated by a previously developed and validated MPM for treatment outcomes in rheumatoid arthritis. Prediction models using multinomial logistic regression can be developed for nominal outcomes, but also for ordinal outcomes. This article is intended to supplement existing general guidance on prediction model research. RESULTS This guide is split into three parts: 1) outcome definition and variable selection, 2) model development, and 3) model evaluation (including performance assessment, internal and external validation, and model recalibration). We outline how to evaluate and interpret the predictive performance of MPMs. R code is provided. CONCLUSION We recommend the application of MPMs in clinical settings where the prediction of a multicategory outcome is of interest. Future methodological research could focus on MPM-specific considerations for variable selection and sample size criteria for external validation.
Collapse
Affiliation(s)
- Celina K Gehringer
- Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, Division of Musculoskeletal and Dermatological Sciences, University of Manchester, Manchester, UK; Centre for Biostatistics, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK.
| | - Glen P Martin
- Division of Informatics, Imaging and Data Sciences, Centre for Health Informatics, University of Manchester, Manchester, UK
| | - Ben Van Calster
- Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, The Netherlands; Department of Development & Regeneration, KU Leuven, Leuven, Belgium
| | - Kimme L Hyrich
- Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, Division of Musculoskeletal and Dermatological Sciences, University of Manchester, Manchester, UK; NIHR Manchester Biomedical Research Centre, Manchester University NHS Foundation Trust, Manchester Academic Health Science Centre, Manchester, UK
| | - Suzanne M M Verstappen
- Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, Division of Musculoskeletal and Dermatological Sciences, University of Manchester, Manchester, UK; NIHR Manchester Biomedical Research Centre, Manchester University NHS Foundation Trust, Manchester Academic Health Science Centre, Manchester, UK
| | - Jamie C Sergeant
- Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, Division of Musculoskeletal and Dermatological Sciences, University of Manchester, Manchester, UK; Centre for Biostatistics, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK
| |
Collapse
|
7
|
Campagner A, Milella F, Banfi G, Cabitza F. Second opinion machine learning for fast-track pathway assignment in hip and knee replacement surgery: the use of patient-reported outcome measures. BMC Med Inform Decis Mak 2024; 24:203. [PMID: 39044277 PMCID: PMC11267678 DOI: 10.1186/s12911-024-02602-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2024] [Accepted: 07/09/2024] [Indexed: 07/25/2024] Open
Abstract
BACKGROUND The frequency of hip and knee arthroplasty surgeries has been rising steadily in recent decades. This trend is attributed to an aging population, leading to increased demands on healthcare systems. Fast Track (FT) surgical protocols, perioperative procedures designed to expedite patient recovery and early mobilization, have demonstrated efficacy in reducing hospital stays, convalescence periods, and associated costs. However, the criteria for selecting patients for FT procedures have not fully capitalized on the available patient data, including patient-reported outcome measures (PROMs). METHODS Our study focused on developing machine learning (ML) models to support decision making in assigning patients to FT procedures, utilizing data from patients' self-reported health status. These models are specifically designed to predict the potential health status improvement in patients initially selected for FT. Our approach focused on techniques inspired by the concept of controllable AI. This includes eXplainable AI (XAI), which aims to make the model's recommendations comprehensible to clinicians, and cautious prediction, a method used to alert clinicians about potential control losses, thereby enhancing the models' trustworthiness and reliability. RESULTS Our models were trained and tested using a dataset comprising 899 records from individual patients admitted to the FT program at IRCCS Ospedale Galeazzi-Sant'Ambrogio. After training and selecting hyper-parameters, the models were assessed using a separate internal test set. The interpretable models demonstrated performance on par or even better than the most effective 'black-box' model (Random Forest). These models achieved sensitivity, specificity, and positive predictive value (PPV) exceeding 70%, with an area under the curve (AUC) greater than 80%. The cautious prediction models exhibited enhanced performance while maintaining satisfactory coverage (over 50%). Further, when externally validated on a separate cohort from the same hospital-comprising patients from a subsequent time period-the models showed no pragmatically notable decline in performance. CONCLUSIONS Our results demonstrate the effectiveness of utilizing PROMs as basis to develop ML models for planning assignments to FT procedures. Notably, the application of controllable AI techniques, particularly those based on XAI and cautious prediction, emerges as a promising approach. These techniques provide reliable and interpretable support, essential for informed decision-making in clinical processes.
Collapse
Affiliation(s)
| | - Frida Milella
- Department of Computer Science, Systems and Communication, University of Milano-Bicocca, Milan, Italy
| | - Giuseppe Banfi
- IRCCS Ospedale Galeazzi Sant'Ambrogio, Milan, Italy
- Faculty of Medicine and Surgery, Universitá Vita-Salute San Raffaele, Milan, Italy
| | - Federico Cabitza
- IRCCS Ospedale Galeazzi Sant'Ambrogio, Milan, Italy
- Department of Computer Science, Systems and Communication, University of Milano-Bicocca, Milan, Italy
| |
Collapse
|
8
|
Liu X, Liu X, Jin C, Luo Y, Yang L, Ning X, Zhuo C, Xiao F. Prediction models for diagnosis and prognosis of the colonization or infection of multidrug-resistant organisms in adults: a systematic review, critical appraisal, and meta-analysis. Clin Microbiol Infect 2024:S1198-743X(24)00316-1. [PMID: 38992430 DOI: 10.1016/j.cmi.2024.07.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2024] [Revised: 05/02/2024] [Accepted: 07/04/2024] [Indexed: 07/13/2024]
Abstract
BACKGROUND Prediction models help to target patients at risk of multidrug-resistant organism (MDRO) colonization or infection and could serve as tools informing clinical practices to prevent MDRO transmission and inappropriate empiric antibiotic therapy. However, there is limited evidence to identify which among the available models are of low risk of bias and suitable for clinical application. OBJECTIVES To identify, describe, appraise, and summarise the performance of all prognostic and diagnostic models developed or validated for predicting MDRO colonization or infection. DATA SOURCES Six electronic literature databases and clinical registration databases were searched until April 2022. STUDY ELIGIBILITY CRITERIA Development and validation studies of any multivariable prognostic and diagnostic models to predict MDRO colonization or infection in adults. PARTICIPANTS Adults (≥ 18 years old) without MDRO colonization or infection (in prognostic models) or with unknown or suspected MDRO colonization or infection (in diagnostic models). ASSESSMENT OF RISK OF BIAS The Prediction Model Risk of Bias Assessment Tool was used to assess the risk of bias. Evidence certainty was assessed using the Grading of Recommendations Assessment, Development, and Evaluation approach. METHODS OF DATA SYNTHESIS Meta-analyses were conducted to summarize the discrimination and calibration of the models' external validations conducted in at least two non-overlapping datasets. RESULTS We included 162 models (108 studies) developed for diagnosing (n = 135) and predicting (n = 27) MDRO colonization or infection. Models exhibited a high-risk of bias, especially in statistical analysis. High-frequency predictors were age, recent invasive procedures, antibiotic usage, and prior hospitalization. Less than 25% of the models underwent external validations, with only seven by independent teams. Meta-analyses for one diagnostic and two prognostic models only produced very low to low certainty of evidence. CONCLUSIONS The review comprehensively described the models for identifying patients at risk of MDRO colonization or infection. We cannot recommend which models are ready for application because of the high-risk of bias, limited validations, and low certainty of evidence from meta-analyses, indicating a clear need to improve the conducting and reporting of model development and external validation studies to facilitate clinical application.
Collapse
Affiliation(s)
- Xu Liu
- Department of Infectious Diseases, The Fifth Affiliated Hospital, Sun Yat-sen University, Zhuhai, China; Guangdong-Hong Kong-Macao University Joint Laboratory of Interventional Medicine, The Fifth Affiliated Hospital, Sun Yat-sen University, Zhuhai, China
| | - Xi Liu
- Department of Infectious Diseases, The Fifth Affiliated Hospital, Sun Yat-sen University, Zhuhai, China; Guangdong-Hong Kong-Macao University Joint Laboratory of Interventional Medicine, The Fifth Affiliated Hospital, Sun Yat-sen University, Zhuhai, China
| | - Chenyue Jin
- Department of Infectious Diseases, The Fifth Affiliated Hospital, Sun Yat-sen University, Zhuhai, China
| | - Yuting Luo
- Department of Infectious Diseases, The Fifth Affiliated Hospital, Sun Yat-sen University, Zhuhai, China; Department of Infectious Diseases, Liuzhou People's Hospital, Liuzhou, China
| | - Lianping Yang
- School of Public Health, Sun Yat-sen University, Guangzhou, China
| | - Xinjiao Ning
- Department of Infectious Diseases, The Fifth Affiliated Hospital, Sun Yat-sen University, Zhuhai, China
| | - Chao Zhuo
- State Key Laboratory of Respiratory Disease, The First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China.
| | - Fei Xiao
- Department of Infectious Diseases, The Fifth Affiliated Hospital, Sun Yat-sen University, Zhuhai, China; Guangdong-Hong Kong-Macao University Joint Laboratory of Interventional Medicine, The Fifth Affiliated Hospital, Sun Yat-sen University, Zhuhai, China; Guangdong Provincial Engineering Research Center of Molecular Imaging, The Fifth Affiliated Hospital, Sun Yat-sen University, Zhuhai, China; Kashi Guangdong Institute of Science and Technology, The First People's Hospital of Kashi, Kashi, China; State Key Laboratory of Anti-Infective Drug Development, School of Pharmaceutical Sciences, Sun Yat-sen University, Guangzhou, China.
| |
Collapse
|
9
|
Tran DNT, Ducher M, Fouque D, Fauvel JP. External validation of a 2-year all-cause mortality prediction tool developed using machine learning in patients with stage 4-5 chronic kidney disease. J Nephrol 2024:10.1007/s40620-024-02011-9. [PMID: 38965199 DOI: 10.1007/s40620-024-02011-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2024] [Accepted: 06/13/2024] [Indexed: 07/06/2024]
Abstract
BACKGROUND Chronic kidney disease (CKD) is associated with increased mortality. Individual mortality prediction could be of interest to improve individual clinical outcomes. Using an independent regional dataset, the aim of the present study was to externally validate the recently published 2-year all-cause mortality prediction tool developed using machine learning. METHODS A validation dataset of stage 4 or 5 CKD outpatients was used. External validation performance of the prediction tool at the optimal cutoff-point was assessed by the area under the receiver operating characteristic curve (AUC-ROC), accuracy, sensitivity, and specificity. A survival analysis was then performed using the Kaplan-Meier method. RESULTS Data of 527 outpatients with stage 4 or 5 CKD were analyzed. During the 2 years of follow-up, 91 patients died and 436 survived. Compared to the learning dataset, patients in the validation dataset were significantly younger, and the ratio of deceased patients in the validation dataset was significantly lower. The performance of the prediction tool at the optimal cutoff-point was: AUC-ROC = 0.72, accuracy = 63.6%, sensitivity = 72.5%, and specificity = 61.7%. The survival curves of the predicted survived and the predicted deceased groups were significantly different (p < 0.001). CONCLUSION The 2-year all-cause mortality prediction tool for patients with stage 4 or 5 CKD showed satisfactory discriminatory capacity with emphasis on sensitivity. The proposed prediction tool appears to be of clinical interest for further development.
Collapse
Affiliation(s)
- Dung N T Tran
- Laboratoire de Biométrie et Biologie Évolutive, UMR 5558, CNRS Lyon, Université Claude Bernard Lyon 1, 43 Boulevard du 11 Novembre 1918, 69100, Lyon, Villeurbanne, France
- Service de Néphrologie, Hospices Civils de Lyon, Hôpital Edouard Herriot, 69003, Lyon, France
| | - Michel Ducher
- Laboratoire de Biométrie et Biologie Évolutive, UMR 5558, CNRS Lyon, Université Claude Bernard Lyon 1, 43 Boulevard du 11 Novembre 1918, 69100, Lyon, Villeurbanne, France
- EMR3738 Ciblage Thérapeutique en Oncologie, Université Claude Bernard Lyon 1, 43 Boulevard du 11 Novembre 1918, 69100, Lyon, Villeurbanne, France
| | - Denis Fouque
- Dept Nephrology, Nutrition and Dialysis, Faculté de Médecine Lyon-Sud BP 12165 Chemin du Grand Revoyet, Université Claude Bernard Lyon 1, Carmen, 69921, Lyon, Oulllins, France
- Hôpital Lyon Sud, Hospices Civils de Lyon, 69495, Lyon, Pierre-Benite, France
| | - Jean-Pierre Fauvel
- Laboratoire de Biométrie et Biologie Évolutive, UMR 5558, CNRS Lyon, Université Claude Bernard Lyon 1, 43 Boulevard du 11 Novembre 1918, 69100, Lyon, Villeurbanne, France.
- Service de Néphrologie, Hospices Civils de Lyon, Hôpital Edouard Herriot, 69003, Lyon, France.
| |
Collapse
|
10
|
Adeoye J, Su YX. Validity of nomograms for predicting cancer risk in oral leukoplakia and oral lichen planus. Oral Dis 2024; 30:3039-3051. [PMID: 38009867 DOI: 10.1111/odi.14811] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2023] [Revised: 10/27/2023] [Accepted: 11/04/2023] [Indexed: 11/29/2023]
Abstract
OBJECTIVES This study assessed the validity of nomograms for predicting malignant transformation (MT) among patients with oral leukoplakia (OL) and oral lichen planus (OLP). MATERIALS AND METHODS Two nomograms were identified following a systematic search. Variables to interrogate both nomograms were obtained for a retrospective OL/OLP cohort. Then, the nomograms were applied to estimate MT probabilities twice and their average was used to calculate the discriminatory performance, calibration, and potential net benefit of the models. Subgroup analyses were performed for patients with OL, OLP, and oral epithelial dysplasia. RESULTS Predicted probabilities were mostly significantly higher among OL/OLP patients who developed MT compared to those who did not (p = <0.001-0.034). AUC values and Brier scores of the nomograms were 0.644-0.844 and 0.040-0.088 among OL patients and 0.580-0.743 and 0.008-0.072 among OLP patients. Decision curve analysis suggested that the nomograms had some net benefit for risk stratification. However, the models did not best binary dysplasia grading in discriminatory validity and net benefit among patients with OL and oral epithelial dysplasia. CONCLUSION Nomograms for predicting MT may have satisfactory validity among patients with OL than OLP, but they do not outperform binary dysplasia grading in risk stratification of OL.
Collapse
Affiliation(s)
- John Adeoye
- Division of Oral and Maxillofacial Surgery, Faculty of Dentistry, University of Hong Kong, Hong Kong SAR
| | - Yu-Xiong Su
- Division of Oral and Maxillofacial Surgery, Faculty of Dentistry, University of Hong Kong, Hong Kong SAR
| |
Collapse
|
11
|
Strandberg R, Jepsen P, Hagström H. Developing and validating clinical prediction models in hepatology - An overview for clinicians. J Hepatol 2024; 81:149-162. [PMID: 38531493 DOI: 10.1016/j.jhep.2024.03.030] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/30/2023] [Revised: 03/06/2024] [Accepted: 03/12/2024] [Indexed: 03/28/2024]
Abstract
Prediction models are everywhere in clinical medicine. We use them to assign a diagnosis or a prognosis, and there have been continuous efforts to develop better prediction models. It is important to understand the fundamentals of prediction modelling, thus, we herein describe nine steps to develop and validate a clinical prediction model with the intention of implementing it in clinical practice: Determine if there is a need for a new prediction model; define the purpose and intended use of the model; assess the quality and quantity of the data you wish to develop the model on; develop the model using sound statistical methods; generate risk predictions on the probability scale (0-100%); evaluate the performance of the model in terms of discrimination, calibration, and clinical utility; validate the model using bootstrapping to correct for the apparent optimism in performance; validate the model on external datasets to assess the generalisability and transportability of the model; and finally publish the model so that it can be implemented or validated by others.
Collapse
Affiliation(s)
- Rickard Strandberg
- Department of Medicine, Huddinge, Karolinska Institutet, Stockholm, Sweden.
| | - Peter Jepsen
- Department of Hepatology and Gastroenterology, Aarhus University Hospital, Aarhus, Denmark; Department of Clinical Epidemiology, Aarhus University Hospital, Aarhus, Denmark
| | - Hannes Hagström
- Department of Medicine, Huddinge, Karolinska Institutet, Stockholm, Sweden; Division of Hepatology, Department of Upper GI, Karolinska University Hospital, Stockholm, Sweden
| |
Collapse
|
12
|
Sammut-Powell C, Sisk R, Vazquez-Mendez E, Vasnawala H, Goncalves S, Edge M, Cameron R. Global Validation of a Model to Predict Reduced Estimated GFR in People With Type 2 Diabetes Without Diagnosis of CKD. Kidney Int Rep 2024; 9:2047-2055. [PMID: 39081746 PMCID: PMC11284396 DOI: 10.1016/j.ekir.2024.04.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/13/2023] [Revised: 03/31/2024] [Accepted: 04/01/2024] [Indexed: 08/02/2024] Open
Abstract
Introduction A minimal-resource model for predicting reduced kidney function among people with type 2 diabetes and no diagnosis of chronic kidney disease (CKD) stages 3 to 5 was previously developed in a UK population to pre-screen for undiagnosed CKD. This study aims to evaluate the performance of the model on a global population and assess its adequacy with and without regional adjustment. Methods A retrospective observational study was performed using data collected from the iCaReMe global registry (NCT03549754) and the DISCOVER study (NCT02322762 and NCT02226822). Patients were grouped by their World Health Organization classified region. An estimated glomerular filtration rate (eGFR) <60 ml/min per 1.73 m2 was the marker of reduced kidney function. A regional-intercept recalibration was applied to adjust for regional variation. Discrimination and calibration were evaluated for the UK-developed and recalibrated models. Results A total of 14,180 patients (46 countries, 6 regions) were identified with type 2 diabetes, no previous diagnosis of CKD stages 3 to 5, and had a serum creatinine measurement or eGFR recorded. The UK model underestimated risk when applied globally and was deemed inadequate. The model with regional adjustment achieved the target sensitivity (80.5%; 95% confidence interval [CI]: 78.8%-82.3%) and demonstrated a relative improvement of 51.5% (95% CI: 48.1%-55.1%) in the positive predictive value (PPV), compared to a screen-all approach. Conclusion The regional-adjusted model demonstrated adequate performance globally. Incorporating the model within practice could help clinicians to risk-stratify and prioritize patients at high risk. This could enable improved efficiency via risk-tailored screening, particularly in lower-middle-income countries (LMICs).
Collapse
|
13
|
Pate A, Sperrin M, Riley RD, Peek N, Van Staa T, Sergeant JC, Mamas MA, Lip GYH, O'Flaherty M, Barrowman M, Buchan I, Martin GP. Calibration plots for multistate risk predictions models. Stat Med 2024; 43:2830-2852. [PMID: 38720592 DOI: 10.1002/sim.10094] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2023] [Revised: 03/06/2024] [Accepted: 04/17/2024] [Indexed: 06/15/2024]
Abstract
INTRODUCTION There is currently no guidance on how to assess the calibration of multistate models used for risk prediction. We introduce several techniques that can be used to produce calibration plots for the transition probabilities of a multistate model, before assessing their performance in the presence of random and independent censoring through a simulation. METHODS We studied pseudo-values based on the Aalen-Johansen estimator, binary logistic regression with inverse probability of censoring weights (BLR-IPCW), and multinomial logistic regression with inverse probability of censoring weights (MLR-IPCW). The MLR-IPCW approach results in a calibration scatter plot, providing extra insight about the calibration. We simulated data with varying levels of censoring and evaluated the ability of each method to estimate the calibration curve for a set of predicted transition probabilities. We also developed evaluated the calibration of a model predicting the incidence of cardiovascular disease, type 2 diabetes and chronic kidney disease among a cohort of patients derived from linked primary and secondary healthcare records. RESULTS The pseudo-value, BLR-IPCW, and MLR-IPCW approaches give unbiased estimates of the calibration curves under random censoring. These methods remained predominately unbiased in the presence of independent censoring, even if the censoring mechanism was strongly associated with the outcome, with bias concentrated in low-density regions of predicted transition probability. CONCLUSIONS We recommend implementing either the pseudo-value or BLR-IPCW approaches to produce a calibration curve, combined with the MLR-IPCW approach to produce a calibration scatter plot. The methods have been incorporated into the "calibmsm" R package available on CRAN.
Collapse
Affiliation(s)
- Alexander Pate
- Centre for Health Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| | - Matthew Sperrin
- Centre for Health Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
- NIHR Manchester Biomedical Research Centre, University of Manchester, Manchester, UK
| | - Richard D Riley
- Institute of Applied Health Research, University of Birmingham, Birmingham, UK
| | - Niels Peek
- Centre for Health Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
- NIHR Manchester Biomedical Research Centre, University of Manchester, Manchester, UK
| | - Tjeerd Van Staa
- Centre for Health Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| | - Jamie C Sergeant
- Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK
- Centre for Biostatistics, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK
| | - Mamas A Mamas
- Keele Cardiovascular Research Group, Keele University, Stoke-on-Trent, UK
| | - Gregory Y H Lip
- Liverpool Centre for Cardiovascular Science at University of Liverpool, Liverpool John Moores University and Liverpool Heart & Chest Hospital, Liverpool, UK
- Department of Clinical Medicine, Aalborg University, Aalborg, Denmark
| | - Martin O'Flaherty
- NIHR Applied Research Collaboration NW Coast, University of Liverpool, Liverpool, UK
- Independent Researcher, Manchester, UK
| | - Michael Barrowman
- NIHR Applied Research Collaboration NW Coast, University of Liverpool, Liverpool, UK
| | - Iain Buchan
- Independent Researcher, Manchester, UK
- Institute of Population Health, Faculty of Health and Life Sciences, University of Liverpool, Liverpool, UK
| | - Glen P Martin
- Centre for Health Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| |
Collapse
|
14
|
Khadhouri S, Hramyka A, Gallagher K, Light A, Ippoliti S, Edison M, Alexander C, Kulkarni M, Zimmermann E, Nathan A, Orecchia L, Banthia R, Piazza P, Mak D, Pyrgidis N, Narayan P, Abad Lopez P, Nawaz F, Tran TT, Claps F, Hogan D, Gomez Rivas J, Alonso S, Chibuzo I, Gutierrez Hidalgo B, Whitburn J, Teoh J, Marcq G, Szostek A, Bondad J, Sountoulides P, Kelsey T, Kasivisvanathan V. Machine Learning and External Validation of the IDENTIFY Risk Calculator for Patients with Haematuria Referred to Secondary Care for Suspected Urinary Tract Cancer. Eur Urol Focus 2024:S2405-4569(24)00093-2. [PMID: 38906722 DOI: 10.1016/j.euf.2024.06.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2024] [Revised: 05/24/2024] [Accepted: 06/08/2024] [Indexed: 06/23/2024]
Abstract
BACKGROUND The IDENTIFY study developed a model to predict urinary tract cancer using patient characteristics from a large multicentre, international cohort of patients referred with haematuria. In addition to calculating an individual's cancer risk, it proposes thresholds to stratify them into very-low-risk (<1%), low-risk (1-<5%), intermediate-risk (5-<20%), and high-risk (≥20%) groups. OBJECTIVE To externally validate the IDENTIFY haematuria risk calculator and compare traditional regression with machine learning algorithms. DESIGN, SETTING, AND PARTICIPANTS Prospective data were collected on patients referred to secondary care with new haematuria. Data were collected for patient variables included in the IDENTIFY risk calculator, cancer outcome, and TNM staging. Machine learning methods were used to evaluate whether better models than those developed with traditional regression methods existed. OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS The area under the receiver operating characteristic curve (AUC) for the detection of urinary tract cancer, calibration coefficient, calibration in the large (CITL), and Brier score were determined. RESULTS AND LIMITATIONS There were 3582 patients in the validation cohort. The development and validation cohorts were well matched. The AUC of the IDENTIFY risk calculator on the validation cohort was 0.78. This improved to 0.80 on a subanalysis of urothelial cancer prevalent countries alone, with a calibration slope of 1.04, CITL of 0.24, and Brier score of 0.14. The best machine learning model was Random Forest, which achieved an AUC of 0.76 on the validation cohort. There were no cancers stratified to the very-low-risk group in the validation cohort. Most cancers were stratified to the intermediate- and high-risk groups, with more aggressive cancers in higher-risk groups. CONCLUSIONS The IDENTIFY risk calculator performed well at predicting cancer in patients referred with haematuria on external validation. This tool can be used by urologists to better counsel patients on their cancer risks, to prioritise diagnostic resources on appropriate patients, and to avoid unnecessary invasive procedures in those with a very low risk of cancer. PATIENT SUMMARY We previously developed a calculator that predicts patients' risk of cancer when they have blood in their urine, based on their personal characteristics. We have validated this risk calculator, by testing it on a separate group of patients to ensure that it works as expected. Most patients found to have cancer tended to be in the higher-risk groups and had more aggressive types of cancer with a higher risk. This tool can be used by clinicians to fast-track high-risk patients based on the calculator and investigate them more thoroughly.
Collapse
Affiliation(s)
- Sinan Khadhouri
- School of Medicine, University of St Andrews, St Andrews, UK; British Urological Researchers in Surgical Training (BURST), London, UK.
| | - Artsiom Hramyka
- School of Computer Science, University of St Andrews, St Andrews, UK
| | - Kevin Gallagher
- British Urological Researchers in Surgical Training (BURST), London, UK; Institute of Cancer and Genetics, University of Edinburgh, Edinburgh, UK; Division of Surgery and Interventional Science, University College London, London, UK
| | - Alexander Light
- British Urological Researchers in Surgical Training (BURST), London, UK; Department of Surgery and Cancer, Imperial College London, London, UK
| | - Simona Ippoliti
- British Urological Researchers in Surgical Training (BURST), London, UK; Department of Paediatric Surgery, Hull Royal Infirmary, Hull University Teaching Hospitals, Hull, UK
| | - Marie Edison
- British Urological Researchers in Surgical Training (BURST), London, UK; Department of Urology, Chelsea and Westminster Hospital, London, UK
| | - Cameron Alexander
- British Urological Researchers in Surgical Training (BURST), London, UK; Luton and Dunstable University Hospital, Luton, UK
| | - Meghana Kulkarni
- British Urological Researchers in Surgical Training (BURST), London, UK; Department of Urology, St George's University Hospitals NHS Foundation Trust, London, UK
| | - Eleanor Zimmermann
- British Urological Researchers in Surgical Training (BURST), London, UK; Department of Urology, Southmead Hospital, Bristol, UK
| | - Arjun Nathan
- British Urological Researchers in Surgical Training (BURST), London, UK; Division of Surgery and Interventional Science, University College London, London, UK
| | - Luca Orecchia
- AOU Policlinico Tor Vergata University Hospital of Rome, Rome, Italy
| | - Ravi Banthia
- University Hospital Coventry Warwickshire, Coventry, UK
| | - Pietro Piazza
- IRCCS Azienda Ospedaliero-Universitaria di Bologna, Bologna, Italy
| | - David Mak
- Royal Wolverhampton Hospitals, Wolverhampton, UK
| | | | | | | | - Faisal Nawaz
- University Hospitals of Derby and Burton, Derby, UK
| | - Trung-Thanh Tran
- Department of Surgery, Hanoi Medical University, Hanoi, Vietnam; Department of Urology, Hanoi Medical University Hospital, Hanoi, Vietnam
| | | | | | | | | | | | | | | | - Jeremy Teoh
- S.H. Ho Urology Centre, Department of Surgery, The Chinese University of Hong Kong, Hong Kong
| | - Gautier Marcq
- Urology Department, Claude Huriez Hospital, CHU Lille, Lille, France; CNRS, Inserm, CHU Lille, Institut Pasteur de Lille, UMR9020-U1277 - CANTHER - Cancer Heterogeneity Plasticity and Resistance to Therapies, University Lille, Lille, France
| | - Alexandra Szostek
- Urology Department, Claude Huriez Hospital, CHU Lille, Lille, France
| | - Jasper Bondad
- Southend University Hospital, Southend-on-Sea, Essex, UK
| | - Petros Sountoulides
- Department of Urology, Aristotle University of Thessaloniki, Thessaloniki, Greece
| | - Tom Kelsey
- School of Computer Science, University of St Andrews, St Andrews, UK
| | - Veeru Kasivisvanathan
- British Urological Researchers in Surgical Training (BURST), London, UK; University College London, London, UK
| |
Collapse
|
15
|
Diamand R, Guenzel K, Jabbour T, Baudewyns A, Bourgeno HA, Lefebvre Y, Ferriero M, Simone G, Fourcade A, Fournier G, Bui AP, Taha F, Oderda M, Gontero P, Rysankova K, Bernal-Gomez A, Mastrorosa A, Roche JB, Fiard G, Abou Zahr R, Ploussard G, Windisch O, Novello Q, Benamran D, Delavar G, Anract J, Barry Delongchamps N, Halinski A, Dariane C, Vlahopoulos L, Assenmacher G, Roumeguère T, Peltier A. External validation and comparison of magnetic resonance imaging-based risk prediction models for prostate biopsy stratification. World J Urol 2024; 42:372. [PMID: 38866949 DOI: 10.1007/s00345-024-05068-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2024] [Accepted: 05/15/2024] [Indexed: 06/14/2024] Open
Abstract
PURPOSE Magnetic resonance imaging (MRI) is a promising tool for risk assessment, potentially reducing the burden of unnecessary prostate biopsies. Risk prediction models that incorporate MRI data have gained attention, but their external validation and comparison are essential for guiding clinical practice. The aim is to externally validate and compare risk prediction models for the diagnosis of clinically significant prostate cancer (csPCa). METHODS A cohort of 4606 patients across fifteen European tertiary referral centers were identified from a prospective maintained database between January 2016 and April 2023. Transrectal or transperineal image-fusion MRI-targeted and systematic biopsies for PI-RADS score of ≥ 3 or ≥ 2 depending on patient characteristics and physician preferences. Probabilities for csPCa, defined as International Society of Urological Pathology (ISUP) grade ≥ 2, were calculated for each patients using eight models. Performance was characterized by area under the receiver operating characteristic curve (AUC), calibration, and net benefit. Subgroup analyses were performed across various clinically relevant subgroups. RESULTS Overall, csPCa was detected in 2154 (47%) patients. The models exhibited satisfactory performance, demonstrating good discrimination (AUC ranging from 0.75 to 0.78, p < 0.001), adequate calibration, and high net benefit. The model described by Alberts showed the highest clinical utility for threshold probabilities between 10 and 20%. Subgroup analyses highlighted variations in models' performance, particularly when stratified according to PSA level, biopsy technique and PI-RADS version. CONCLUSIONS We report a comprehensive external validation of risk prediction models for csPCa diagnosis in patients who underwent MRI-targeted and systematic biopsies. The model by Alberts demonstrated superior clinical utility and should be favored when determining the need for a prostate biopsy.
Collapse
Affiliation(s)
- Romain Diamand
- Department of Urology, Jules Bordet Institute-Erasme Hospital, Hôpital Universitaire de Bruxelles, Université Libre de Bruxelles, Jules Bordet Institute, HUB, Rue Meylemeersch 90, 1070, Brussels, Belgium.
| | - Karsten Guenzel
- Department of Urology, Vivantes Klinikum Am Urban, Berlin, Germany
| | - Teddy Jabbour
- Department of Urology, Jules Bordet Institute-Erasme Hospital, Hôpital Universitaire de Bruxelles, Université Libre de Bruxelles, Jules Bordet Institute, HUB, Rue Meylemeersch 90, 1070, Brussels, Belgium
| | - Arthur Baudewyns
- Department of Urology, Jules Bordet Institute-Erasme Hospital, Hôpital Universitaire de Bruxelles, Université Libre de Bruxelles, Jules Bordet Institute, HUB, Rue Meylemeersch 90, 1070, Brussels, Belgium
| | - Henri-Alexandre Bourgeno
- Department of Urology, Jules Bordet Institute-Erasme Hospital, Hôpital Universitaire de Bruxelles, Université Libre de Bruxelles, Jules Bordet Institute, HUB, Rue Meylemeersch 90, 1070, Brussels, Belgium
| | - Yolène Lefebvre
- Department of Radiology, Jules Bordet Institute-Erasme Hospital, Hôpital Universitaire de Bruxelles, Université Libre de Bruxelles, Brussels, Belgium
| | | | - Giuseppe Simone
- Department of Urology, IRCCS "Regina Elena" National Cancer Institute, Rome, Italy
| | - Alexandre Fourcade
- Department of Urology, Hôpital Cavale Blanche, CHRU Brest, Brest, France
| | - Georges Fournier
- Department of Urology, Hôpital Cavale Blanche, CHRU Brest, Brest, France
| | | | - Fayek Taha
- Department of Urology, Centre Hospitalier Universitaire de Reims, Reims, France
| | - Marco Oderda
- Department of Urology, Città Della Salute E Della Scienza Di Torino, University of Turin, Turin, Italy
| | - Paolo Gontero
- Department of Urology, Città Della Salute E Della Scienza Di Torino, University of Turin, Turin, Italy
| | - Katerina Rysankova
- Department of Urology and Surgical Studies, Faculty of Medicine, University Hospital Ostrava, Ostrava University, Ostrava, Czech Republic
| | | | | | | | - Gaelle Fiard
- Department of Urology, Grenoble Alpes University Hospital, Université Grenoble Alpes, CNRS, Grenoble INP, TIMC, Grenoble, France
| | - Rawad Abou Zahr
- Department of Urology, La Croix du Sud Hospital, Quint Fonsegrives, France
| | | | - Olivier Windisch
- Department of Urology, Hôpitaux Universitaires de Genève, Geneva, Switzerland
| | - Quentin Novello
- Department of Urology, Hôpitaux Universitaires de Genève, Geneva, Switzerland
| | - Daniel Benamran
- Department of Urology, Hôpitaux Universitaires de Genève, Geneva, Switzerland
| | - Gina Delavar
- Departement of Urology, Hôpital Cochin, Paris, France
| | - Julien Anract
- Departement of Urology, Hôpital Cochin, Paris, France
| | | | - Adam Halinski
- Department of Urology, Private Medical Center, Klinika Wisniowa", Zielona Góra, Poland
| | - Charles Dariane
- Department of Urology, Hôpital Européen Georges-Pompidou, Université de Paris, Paris, France
| | | | | | - Thierry Roumeguère
- Department of Urology, Jules Bordet Institute-Erasme Hospital, Hôpital Universitaire de Bruxelles, Université Libre de Bruxelles, Jules Bordet Institute, HUB, Rue Meylemeersch 90, 1070, Brussels, Belgium
| | - Alexandre Peltier
- Department of Urology, Jules Bordet Institute-Erasme Hospital, Hôpital Universitaire de Bruxelles, Université Libre de Bruxelles, Jules Bordet Institute, HUB, Rue Meylemeersch 90, 1070, Brussels, Belgium
| |
Collapse
|
16
|
External validation of the CholeS conversion from laparoscopic to open cholecystectomy (CLOC) risk score in Aotearoa New Zealand: a validation study. ANZ J Surg 2024; 94:1108-1113. [PMID: 38525949 DOI: 10.1111/ans.18921] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2023] [Revised: 01/29/2024] [Accepted: 02/19/2024] [Indexed: 03/26/2024]
Abstract
BACKGROUND Conversion of laparoscopic cholecystectomy to open is uncommon, but is associated with longer hospital stay and recovery. Prognosticating conversion may aid service planning and provision. We therefore aimed to assess the external validity of the largest risk score for operative conversion. METHODS CHOLENZ was a multicentre, prospective, national cohort study of cholecystectomy for benign biliary disease conducted by STRATA, a trainee-led collaborative network. Data were collected from patients undergoing cholecystectomy in New Zealand hospitals between 1 August and 30 October 2021 with 30-day follow-up. The Conversion from Laparoscopic to Open Cholecystectomy (CLOC) score from the CholeS study was assessed for external validity by interrogating its accuracy and calibration in the CHOLENZ dataset. RESULTS Of 1162 cholecystectomies started laparoscopically, 20 (1.7%) were converted to open in the CHOLENZ dataset. The CLOC score predicted 2.9% (IQR 1.3%-8.1%) would be converted. Area under the curve was 0.65 (95% 0.51-0.79) and calibration was acceptable with a Hosmer-Lemeshow p value of 0.45; with evidence of tendency to overestimate with interrogation of calibration across a continuous risk profile (intercept 1.27, slope 0.4). Sensitivity analysis with imputed data improved accuracy. Recalibration with the addition of body mass index, and preoperative bilirubin also improved accuracy to 0.86 (95% CI 0.78-0.95). CONCLUSIONS The CLOC score in its original form is not generalisable to the Aotearoa New Zealand setting and is therefore not suitable for clinical use in our local setting.
Collapse
|
17
|
Harrison-Brown M, Scholes C, Ebrahimi M, Bell C, Kirwan G. Applying models of care for total hip and knee arthroplasty: External validation of a published predictive model to identify extended stay risk prior to lower-limb arthroplasty. Clin Rehabil 2024; 38:700-712. [PMID: 38377957 DOI: 10.1177/02692155241233348] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/22/2024]
Abstract
OBJECTIVE This study aimed to externally validate a reported model for identifying patients requiring extended stay following lower limb arthroplasty in a new setting. DESIGN External validation of a previously reported prognostic model, using retrospective data. SETTING Medium-sized hospital orthopaedic department, Australia. PARTICIPANTS Electronic medical records were accessed for data collection between Sep-2019 and Feb-2020 and retrospective data extracted from 200 randomly selected total hip or knee arthroplasty patients. INTERVENTION Participants received total hip or knee replacement between 2-Feb-16 and 4-Apr-19. This study was a non-interventional retrospective study. MAIN MEASURES Model validation was assessed with discrimination, calibration on both original and adjusted forms of the candidate model. Decision curve analysis was conducted on the outputs of the adjusted model to determine net benefit at a predetermined decision threshold (0.5). RESULTS The original model performed poorly, grossly overestimating length of stay with mean calibration of -3.6 (95% confidence interval -3.9 to -3.2) and calibration slope of 0.52. Performance improved following adjustment of the model intercept and model coefficients (mean calibration 0.48, 95% confidence interval 0.16 to 0.80 and slope of 1.0), but remained poorly calibrated at low and medium risk threshold and net benefit was modest (three additional patients per hundred identified as at-risk) at the a-priori risk threshold. CONCLUSIONS External validation demonstrated poor performance when applied to a new patient population and would provide limited benefit for our institution. Implementation of predictive models for arthroplasty should include practical assessment of discrimination, calibration and net benefit at a clinically acceptable threshold.
Collapse
Affiliation(s)
| | | | | | - Christopher Bell
- Department of Orthopaedics, QEII Jubilee Hospital, Brisbane, Australia
| | - Garry Kirwan
- Department of Physiotherapy, QEII Jubilee Hospital, Brisbane, Australia
- School of Health Sciences and Social Work, Griffith University, Brisbane, Australia
| |
Collapse
|
18
|
Rezende KBDC, Bornia RG, Rolnik DL, Amim J, Ladeira LP, Teixeira VM, da Cunha AJL. Performance of the first-trimester Fetal Medicine Foundation competing risks model for preeclampsia prediction: an external validation study in Brazil. AJOG GLOBAL REPORTS 2024; 4:100346. [PMID: 38694483 PMCID: PMC11061323 DOI: 10.1016/j.xagr.2024.100346] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/04/2024] Open
Abstract
BACKGROUND The current version of the Fetal Medicine Foundation competing risks model for preeclampsia prediction has not been previously validated in Brazil. OBJECTIVE This study aimed (1) to validate the Fetal Medicine Foundation combined algorithm for the prediction of preterm preeclampsia in the Brazilian population and (2) to describe the accuracy and calibration of the Fetal Medicine Foundation algorithm when considering the prophylactic use of aspirin by clinical criteria. STUDY DESIGN This was a cohort study, including consecutive singleton pregnancies undergoing preeclampsia screening at 11 to 14 weeks of gestation, examining maternal characteristics, medical history, and biophysical markers between October 2010 and December 2018 in a university hospital in Brazil. Risks were calculated using the 2018 version of the algorithm available on the Fetal Medicine Foundation website, and cases were classified as low or high risk using a cutoff of 1/100 to evaluate predictive performance. Expected and observed cases with preeclampsia according to the Fetal Medicine Foundation-estimated risk range (≥1 in 10; 1 in 11 to 1 in 50; 1 in 51 to 1 in 100; 1 in 101 to 1 in 150; and <1 in 150) were compared. After identifying high-risk pregnant women who used aspirin, the treatment effect of 62% reduction in preterm preeclampsia identified in the Combined Multimarker Screening and Randomized Patient Treatment with Aspirin for Evidence-Based Preeclampsia Prevention trial was used to evaluate the predictive performance adjusted for the effect of aspirin. The number of potentially unpreventable cases in the group without aspirin use was estimated. RESULTS Among 2749 pregnancies, preterm preeclampsia occurred in 84 (3.1%). With a risk cutoff of 1/100, the screen-positive rate was 25.8%. The detection rate was 71.4%, with a false positive rate of 24.4%. The area under the curve was 0.818 (95% confidence interval, 0.773-0.863). In the risk range ≥1/10, there is an agreement between the number of expected cases and the number of observed cases, and in the other ranges, the predicted risk was lower than the observed rates. Accounting for the effect of aspirin resulted in an increase in detection rate and positive predictive values and a slight decrease in the false positive rate. With 27 cases of preterm preeclampsia in the high-risk group without aspirin use, we estimated that 16 of these cases of preterm preeclampsia would have been avoided if this group had received prophylaxis. CONCLUSION In a high-prevalence setting, the Fetal Medicine Foundation algorithm can identify women who are more likely to develop preterm preeclampsia. Not accounting for the effect of aspirin underestimates the screening performance.
Collapse
Affiliation(s)
- Karina Bilda de Castro Rezende
- Clinical Medicine Postgraduate Program, Faculty of Medicine, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil (Drs Rezende and da Cunha)
- Maternity School of the Federal University of Rio de Janeiro, Rio de Janeiro, Brazil (Drs Rezende, Bornia, Amim, and Ladeira and XX Teixeira)
- Multidisciplinary Laboratory of Epidemiology and Health – LAMPES, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil (Dr Rezende and da Cunha)
| | - Rita G. Bornia
- Maternity School of the Federal University of Rio de Janeiro, Rio de Janeiro, Brazil (Drs Rezende, Bornia, Amim, and Ladeira and XX Teixeira)
- Professional Master Perinatal Health, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil (Drs Bornia and Amim)
| | - Daniel L. Rolnik
- Department of Obstetrics and Gynaecology, Monash University, Melbourne, Australia (Drs Rolnik)
| | - Joffre Amim
- Maternity School of the Federal University of Rio de Janeiro, Rio de Janeiro, Brazil (Drs Rezende, Bornia, Amim, and Ladeira and XX Teixeira)
- Professional Master Perinatal Health, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil (Drs Bornia and Amim)
- Faculty of Medicine, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil (Dr Amim, XX Teixeira, and Dr da Cunha)
| | - Luiza P. Ladeira
- Maternity School of the Federal University of Rio de Janeiro, Rio de Janeiro, Brazil (Drs Rezende, Bornia, Amim, and Ladeira and XX Teixeira)
- Federal University of the State of Rio de Janeiro, Rio de Janeiro, Brazil (Dr Ladeira)
| | - Valentina M.G. Teixeira
- Maternity School of the Federal University of Rio de Janeiro, Rio de Janeiro, Brazil (Drs Rezende, Bornia, Amim, and Ladeira and XX Teixeira)
- Faculty of Medicine, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil (Dr Amim, XX Teixeira, and Dr da Cunha)
| | - Antonio Jose L.A. da Cunha
- Clinical Medicine Postgraduate Program, Faculty of Medicine, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil (Drs Rezende and da Cunha)
- Multidisciplinary Laboratory of Epidemiology and Health – LAMPES, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil (Dr Rezende and da Cunha)
- Faculty of Medicine, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil (Dr Amim, XX Teixeira, and Dr da Cunha)
| |
Collapse
|
19
|
Nemeth B, Smeets MJ, Cannegieter SC, van Smeden M. Tutorial: dos and don'ts in clinical prediction research for venous thromboembolism. Res Pract Thromb Haemost 2024; 8:102480. [PMID: 39099799 PMCID: PMC11295571 DOI: 10.1016/j.rpth.2024.102480] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2024] [Revised: 05/27/2024] [Accepted: 06/11/2024] [Indexed: 08/06/2024] Open
Abstract
Clinical prediction modeling has become an increasingly popular domain of venous thromboembolism research in recent years. Prediction models can help healthcare providers make decisions regarding starting or withholding therapeutic interventions, or referrals for further diagnostic workup, and can form a basis for risk stratification in clinical trials. The aim of the current guide is to assist in the practical application of complicated methodological requirements for well-performed prediction research by presenting key dos and don'ts while expanding the understanding of predictive research in general for (clinical) researchers who are not specifically trained in the topic; throughout we will use prognostic venous thromboembolism scores as an exemplar.
Collapse
Affiliation(s)
- Banne Nemeth
- Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, the Netherlands
| | - Mark J.R. Smeets
- Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, the Netherlands
| | - Suzanne C. Cannegieter
- Department of Clinical Epidemiology, Leiden University Medical Center, Leiden, the Netherlands
- Department of Thrombosis and Hemostasis, Leiden University Medical Center, Leiden, the Netherlands
| | - Maarten van Smeden
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
| |
Collapse
|
20
|
Wang S, Meng F, Chen P, Lv Y, Wu M, Tang H, Bao H, Wu X, Shao Y, Wang J, Dai J, Xu L, Wang X, Yin R. Cell-free DNA assay for malignancy classification of high-risk lung nodules. J Thorac Cardiovasc Surg 2024:S0022-5223(24)00370-2. [PMID: 38670484 DOI: 10.1016/j.jtcvs.2024.04.026] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/08/2023] [Revised: 03/18/2024] [Accepted: 04/15/2024] [Indexed: 04/28/2024]
Abstract
OBJECTIVE Although low-dose computed tomography has been proven effective to reduce lung cancer-specific mortality, a considerable proportion of surgically resected high-risk lung nodules were still confirmed pathologically benign. There is an unmet need of a novel method for malignancy classification in lung nodules. METHODS We recruited 307 patients with high-risk lung nodules who underwent curative surgery, and 247 and 60 cases were pathologically confirmed malignant and benign lung lesions, respectively. Plasma samples from each patient were collected before surgery and performed low-depth (5×) whole-genome sequencing. We extracted cell-free DNA characteristics and determined radiomic features. We built models to classify the malignancy using our data and further validated models with 2 independent lung nodule cohorts. RESULTS Our models using one type of profile were able to distinguish lung cancer and benign lung nodules at an area under the curve metrics of 0.69 to 0.91 in the study cohort. Integrating all the 5 base models using cell-free DNA profiles, the cell-free DNA-based ensemble model achieved an area under the curve of 0.95 (95% CI, 0.92-0.97) in the study cohort and 0.98 (95% CI, 0.96-1.00) in the validation cohort. At a specificity of 95.0%, the sensitivity reached 80.0% in the study cohort. With the same threshold, the specificity and sensitivity had similar performances in both validation cohorts. Furthermore, the performance of area under the curve reached 0.97 in both the study and validation cohorts when considering the radiomic profile. CONCLUSIONS The cell-free DNA profiles-based method is an efficient noninvasive tool to distinguish malignancies and high-risk but pathologically benign lung nodules.
Collapse
Affiliation(s)
- Siwei Wang
- Department of Thoracic Surgery, Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, Nanjing, Jiangsu, China; Clinical Research Institute of Traditional Chinese Medicine, Jiangsu Province Hospital of Chinese Medicine, The Affiliated Hospital of Nanjing University of Chinese Medicine, Nanjing, Jiangsu, China
| | - Fanchen Meng
- Department of Thoracic Surgery, Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, Nanjing, Jiangsu, China
| | - Peng Chen
- Department of Thoracic Surgery, Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, Nanjing, Jiangsu, China
| | - Yang Lv
- Department of Information Center, The Affiliated Jiangning Hospital of Nanjing Medical University, Nanjing, Jiangsu, China
| | - Min Wu
- Geneseeq Research Institute, Nanjing Geneseeq Technology Inc, Nanjing, Jiangsu, China
| | - Haimeng Tang
- Geneseeq Research Institute, Nanjing Geneseeq Technology Inc, Nanjing, Jiangsu, China
| | - Hua Bao
- Geneseeq Research Institute, Nanjing Geneseeq Technology Inc, Nanjing, Jiangsu, China
| | - Xue Wu
- Geneseeq Research Institute, Nanjing Geneseeq Technology Inc, Nanjing, Jiangsu, China
| | - Yang Shao
- Geneseeq Research Institute, Nanjing Geneseeq Technology Inc, Nanjing, Jiangsu, China; School of Public Health, Nanjing Medical University, Nanjing, Jiangsu, China
| | - Jie Wang
- Department of Science and Technology, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, Nanjing, Jiangsu, China; Biobank of Lung Cancer, Jiangsu Biobank of Clinical Resources, Nanjing, Jiangsu, China
| | - Juncheng Dai
- School of Public Health, Nanjing Medical University, Nanjing, Jiangsu, China; Collaborative Innovation Center for Cancer Personalized Medicine, Nanjing Medical University, Nanjing, Jiangsu, China
| | - Lin Xu
- Department of Thoracic Surgery, Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, Nanjing, Jiangsu, China; Collaborative Innovation Center for Cancer Personalized Medicine, Nanjing Medical University, Nanjing, Jiangsu, China
| | - Xiaoxiao Wang
- Clinical Research Institute of Traditional Chinese Medicine, Jiangsu Province Hospital of Chinese Medicine, The Affiliated Hospital of Nanjing University of Chinese Medicine, Nanjing, Jiangsu, China.
| | - Rong Yin
- Department of Thoracic Surgery, Jiangsu Key Laboratory of Molecular and Translational Cancer Research, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, Nanjing, Jiangsu, China; Department of Science and Technology, Nanjing Medical University Affiliated Cancer Hospital & Jiangsu Cancer Hospital & Jiangsu Institute of Cancer Research, Nanjing, Jiangsu, China; Biobank of Lung Cancer, Jiangsu Biobank of Clinical Resources, Nanjing, Jiangsu, China; Collaborative Innovation Center for Cancer Personalized Medicine, Nanjing Medical University, Nanjing, Jiangsu, China.
| |
Collapse
|
21
|
Collins GS, Moons KGM, Dhiman P, Riley RD, Beam AL, Van Calster B, Ghassemi M, Liu X, Reitsma JB, van Smeden M, Boulesteix AL, Camaradou JC, Celi LA, Denaxas S, Denniston AK, Glocker B, Golub RM, Harvey H, Heinze G, Hoffman MM, Kengne AP, Lam E, Lee N, Loder EW, Maier-Hein L, Mateen BA, McCradden MD, Oakden-Rayner L, Ordish J, Parnell R, Rose S, Singh K, Wynants L, Logullo P. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ 2024; 385:e078378. [PMID: 38626948 PMCID: PMC11019967 DOI: 10.1136/bmj-2023-078378] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 01/17/2024] [Indexed: 04/19/2024]
Affiliation(s)
- Gary S Collins
- Centre for Statistics in Medicine, UK EQUATOR Centre, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
| | - Karel G M Moons
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Paula Dhiman
- Centre for Statistics in Medicine, UK EQUATOR Centre, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
| | - Richard D Riley
- Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK
| | - Andrew L Beam
- Department of Epidemiology, Harvard T H Chan School of Public Health, Boston, MA, USA
| | - Ben Van Calster
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- Department of Biomedical Data Science, Leiden University Medical Centre, Leiden, Netherlands
| | - Marzyeh Ghassemi
- Department of Electrical Engineering and Computer Science, Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Xiaoxuan Liu
- Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
- University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
| | - Johannes B Reitsma
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Maarten van Smeden
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Anne-Laure Boulesteix
- Institute for Medical Information Processing, Biometry and Epidemiology, Faculty of Medicine, Ludwig-Maximilians-University of Munich and Munich Centre of Machine Learning, Germany
| | - Jennifer Catherine Camaradou
- Patient representative, Health Data Research UK patient and public involvement and engagement group
- Patient representative, University of East Anglia, Faculty of Health Sciences, Norwich Research Park, Norwich, UK
| | - Leo Anthony Celi
- Beth Israel Deaconess Medical Center, Boston, MA, USA
- Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, MA, USA
- Department of Biostatistics, Harvard T H Chan School of Public Health, Boston, MA, USA
| | - Spiros Denaxas
- Institute of Health Informatics, University College London, London, UK
- British Heart Foundation Data Science Centre, London, UK
| | - Alastair K Denniston
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK
- Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
| | - Ben Glocker
- Department of Computing, Imperial College London, London, UK
| | - Robert M Golub
- Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | | | - Georg Heinze
- Section for Clinical Biometrics, Centre for Medical Data Science, Medical University of Vienna, Vienna, Austria
| | - Michael M Hoffman
- Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada
- Department of Medical Biophysics, University of Toronto, Toronto, ON, Canada
- Department of Computer Science, University of Toronto, Toronto, ON, Canada
- Vector Institute for Artificial Intelligence, Toronto, ON, Canada
| | | | - Emily Lam
- Patient representative, Health Data Research UK patient and public involvement and engagement group
| | - Naomi Lee
- National Institute for Health and Care Excellence, London, UK
| | - Elizabeth W Loder
- The BMJ, London, UK
- Department of Neurology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Lena Maier-Hein
- Department of Intelligent Medical Systems, German Cancer Research Centre, Heidelberg, Germany
| | - Bilal A Mateen
- Institute of Health Informatics, University College London, London, UK
- Wellcome Trust, London, UK
- Alan Turing Institute, London, UK
| | - Melissa D McCradden
- Department of Bioethics, Hospital for Sick Children Toronto, ON, Canada
- Genetics and Genome Biology, SickKids Research Institute, Toronto, ON, Canada
| | - Lauren Oakden-Rayner
- Australian Institute for Machine Learning, University of Adelaide, Adelaide, SA, Australia
| | - Johan Ordish
- Medicines and Healthcare products Regulatory Agency, London, UK
| | - Richard Parnell
- Patient representative, Health Data Research UK patient and public involvement and engagement group
| | - Sherri Rose
- Department of Health Policy and Center for Health Policy, Stanford University, Stanford, CA, USA
| | - Karandeep Singh
- Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Maastricht, Netherlands
| | - Laure Wynants
- Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Maastricht, Netherlands
| | - Patricia Logullo
- Centre for Statistics in Medicine, UK EQUATOR Centre, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
| |
Collapse
|
22
|
Arakaki D, Iwata M, Terasawa T. External validation and update of the International Medical Prevention Registry on Venous Thromboembolism bleeding risk score for predicting bleeding in acutely ill hospitalized medical patients: a retrospective single-center cohort study in Japan. Thromb J 2024; 22:31. [PMID: 38549086 PMCID: PMC10976666 DOI: 10.1186/s12959-024-00603-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/18/2023] [Accepted: 03/21/2024] [Indexed: 04/10/2024] Open
Abstract
BACKGROUND The International Medical Prevention Registry for Venous Thromboembolism (IMPROVE) Bleeding Risk Score is the recommended risk assessment model (RAM) for predicting bleeding risk in acutely ill medical inpatients in Western countries. However, few studies have assessed its predictive performance in local Asian settings. METHODS We retrospectively identified acutely ill adolescents and adults (aged ≥ 15 years) who were admitted to our general internal medicine department between July 5, 2016 and July 5, 2021, and extracted data from their electronic medical records. The outcome of interest was the cumulative incidence of major and nonmajor but clinically relevant bleeding 14 days after admission. For the two-risk-group model, we estimated sensitivity, specificity, and positive and negative predictive values (PPV and NPV, respectively). For the 11-risk-group model, we estimated C statistic, expected and observed event ratio (E/O), calibration-in-the-large (CITL), and calibration slope. In addition, we recalibrated the intercept using local data to update the RAM. RESULTS Among the 3,876 included patients, 998 (26%) were aged ≥ 85 years, while 656 (17%) were hospitalized in the intensive care unit. The median length of hospital stay was 14 days. Clinically relevant bleeding occurred in 58 patients (1.5%), 49 (1.3%) of whom experienced major bleeding. Sensitivity, specificity, NPV, and PPV were 26.1% (95% confidence interval [CI]: 15.8-40.0%), 84.8% (83.6-85.9%), 98.7% (98.2-99.0%), and 2.5% (1.5-4.3%) for any bleeding and 30.9% (95% CI: 18.8-46.3%), 84.9% (83.7-86.0%), 99.0% (98.5-99.3%), and 2.5% (1.5-4.3%) for major bleeding, respectively. The C statistic, E/O, CITL, and calibration slope were 0.64 (95% CI: 0.58-0.71), 1.69 (1.45-2.05), - 0.55 (- 0.81 to - 0.29), and 0.58 (0.29-0.87) for any bleeding and 0.67 (95% CI: 0.60-0.74), 0.76 (0.61-0.87), 0.29 (0.00-0.58), and 0.42 (0.19-0.64) for major bleeding, respectively. Updating the model substantially corrected the poor calibration observed. CONCLUSIONS In our Japanese cohort, the IMPROVE bleeding RAM retained the reported moderate discriminative performance. Model recalibration substantially improved the poor calibration obtained using the original RAM. Before its introduction into clinical practice, the updated RAM needs further validation studies and an optimized threshold.
Collapse
Affiliation(s)
- Daichi Arakaki
- Department of Emergency Medicine and General Internal Medicine, Fujita Health University School of Medicine, 1-98 Dengakugakubo, Kutsukakecho, Toyoake, Achi, 470-1192, Toyoake, Aichi, Japan
- Department of Emergency and Critical Care, Nagoya University Hospital, Nagoya, Aichi, Japan
| | - Mitsunaga Iwata
- Department of Emergency Medicine and General Internal Medicine, Fujita Health University School of Medicine, 1-98 Dengakugakubo, Kutsukakecho, Toyoake, Achi, 470-1192, Toyoake, Aichi, Japan
| | - Teruhiko Terasawa
- Department of Emergency Medicine and General Internal Medicine, Fujita Health University School of Medicine, 1-98 Dengakugakubo, Kutsukakecho, Toyoake, Achi, 470-1192, Toyoake, Aichi, Japan.
| |
Collapse
|
23
|
Archer L, Relton SD, Akbari A, Best K, Bucknall M, Conroy S, Hattle M, Hollinghurst J, Humphrey S, Lyons RA, Richards S, Walters K, West R, van der Windt D, Riley RD, Clegg A. Development and external validation of the eFalls tool: a multivariable prediction model for the risk of ED attendance or hospitalisation with a fall or fracture in older adults. Age Ageing 2024; 53:afae057. [PMID: 38520142 PMCID: PMC10960070 DOI: 10.1093/ageing/afae057] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2023] [Indexed: 03/25/2024] Open
Abstract
BACKGROUND Falls are common in older adults and can devastate personal independence through injury such as fracture and fear of future falls. Methods to identify people for falls prevention interventions are currently limited, with high risks of bias in published prediction models. We have developed and externally validated the eFalls prediction model using routinely collected primary care electronic health records (EHR) to predict risk of emergency department attendance/hospitalisation with fall or fracture within 1 year. METHODS Data comprised two independent, retrospective cohorts of adults aged ≥65 years: the population of Wales, from the Secure Anonymised Information Linkage Databank (model development); the population of Bradford and Airedale, England, from Connected Bradford (external validation). Predictors included electronic frailty index components, supplemented with variables informed by literature reviews and clinical expertise. Fall/fracture risk was modelled using multivariable logistic regression with a Least Absolute Shrinkage and Selection Operator penalty. Predictive performance was assessed through calibration, discrimination and clinical utility. Apparent, internal-external cross-validation and external validation performance were assessed across general practices and in clinically relevant subgroups. RESULTS The model's discrimination performance (c-statistic) was 0.72 (95% confidence interval, CI: 0.68 to 0.76) on internal-external cross-validation and 0.82 (95% CI: 0.80 to 0.83) on external validation. Calibration was variable across practices, with some over-prediction in the validation population (calibration-in-the-large, -0.87; 95% CI: -0.96 to -0.78). Clinical utility on external validation was improved after recalibration. CONCLUSION The eFalls prediction model shows good performance and could support proactive stratification for falls prevention services if appropriately embedded into primary care EHR systems.
Collapse
Affiliation(s)
- Lucinda Archer
- Institute for Applied Health Research, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, University of Birmingham, Birmingham, UK
| | - Samuel D Relton
- Leeds Institute of Health Sciences, University of Leeds, Leeds, UK
| | - Ashley Akbari
- Population Data Science, Swansea University Medical School, Swansea University, Swansea, UK
| | - Kate Best
- Academic Unit for Ageing and Stroke Research, University of Leeds, Bradford Teaching Hospitals NHS Foundation Trust, Bradford, UK
| | | | - Simon Conroy
- Institute of Cardiovascular Science, University College London, London, UK
| | - Miriam Hattle
- Institute for Applied Health Research, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, University of Birmingham, Birmingham, UK
| | - Joe Hollinghurst
- Population Data Science, Swansea University Medical School, Swansea University, Swansea, UK
| | - Sara Humphrey
- Bradford District and Craven Health and Care Partnership, Bradford, UK
| | - Ronan A Lyons
- Population Data Science, Swansea University Medical School, Swansea University, Swansea, UK
| | - Suzanne Richards
- Leeds Institute of Health Sciences, University of Leeds, Leeds, UK
| | - Kate Walters
- Primary Care and Population Health, University College London, London, UK
| | - Robert West
- Leeds Institute of Health Sciences, University of Leeds, Leeds, UK
| | | | - Richard D Riley
- Institute for Applied Health Research, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, University of Birmingham, Birmingham, UK
| | - Andrew Clegg
- Academic Unit for Ageing and Stroke Research, University of Leeds, Bradford Teaching Hospitals NHS Foundation Trust, Bradford, UK
| | | |
Collapse
|
24
|
Barreñada L, Ledger A, Dhiman P, Collins G, Wynants L, Verbakel JY, Timmerman D, Valentin L, Van Calster B. ADNEX risk prediction model for diagnosis of ovarian cancer: systematic review and meta-analysis of external validation studies. BMJ MEDICINE 2024; 3:e000817. [PMID: 38375077 PMCID: PMC10875560 DOI: 10.1136/bmjmed-2023-000817] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/17/2023] [Accepted: 01/25/2024] [Indexed: 02/21/2024]
Abstract
Objectives To conduct a systematic review of studies externally validating the ADNEX (Assessment of Different Neoplasias in the adnexa) model for diagnosis of ovarian cancer and to present a meta-analysis of its performance. Design Systematic review and meta-analysis of external validation studies. Data sources Medline, Embase, Web of Science, Scopus, and Europe PMC, from 15 October 2014 to 15 May 2023. Eligibility criteria for selecting studies All external validation studies of the performance of ADNEX, with any study design and any study population of patients with an adnexal mass. Two independent reviewers extracted the data. Disagreements were resolved by discussion. Reporting quality of the studies was scored with the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) reporting guideline, and methodological conduct and risk of bias with PROBAST (Prediction model Risk Of Bias Assessment Tool). Random effects meta-analysis of the area under the receiver operating characteristic curve (AUC), sensitivity and specificity at the 10% risk of malignancy threshold, and net benefit and relative utility at the 10% risk of malignancy threshold were performed. Results 47 studies (17 007 tumours) were included, with a median study sample size of 261 (range 24-4905). On average, 61% of TRIPOD items were reported. Handling of missing data, justification of sample size, and model calibration were rarely described. 91% of validations were at high risk of bias, mainly because of the unexplained exclusion of incomplete cases, small sample size, or no assessment of calibration. The summary AUC to distinguish benign from malignant tumours in patients who underwent surgery was 0.93 (95% confidence interval 0.92 to 0.94, 95% prediction interval 0.85 to 0.98) for ADNEX with the serum biomarker, cancer antigen 125 (CA125), as a predictor (9202 tumours, 43 centres, 18 countries, and 21 studies) and 0.93 (95% confidence interval 0.91 to 0.94, 95% prediction interval 0.85 to 0.98) for ADNEX without CA125 (6309 tumours, 31 centres, 13 countries, and 12 studies). The estimated probability that the model has use clinically in a new centre was 95% (with CA125) and 91% (without CA125). When restricting analysis to studies with a low risk of bias, summary AUC values were 0.93 (with CA125) and 0.91 (without CA125), and estimated probabilities that the model has use clinically were 89% (with CA125) and 87% (without CA125). Conclusions The results of the meta-analysis indicated that ADNEX performed well in distinguishing between benign and malignant tumours in populations from different countries and settings, regardless of whether the serum biomarker, CA125, was used as a predictor. A key limitation was that calibration was rarely assessed. Systematic review registration PROSPERO CRD42022373182.
Collapse
Affiliation(s)
- Lasai Barreñada
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
| | - Ashleigh Ledger
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
| | - Paula Dhiman
- Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford Centre for Statistics in Medicine, Oxford, UK
| | - Gary Collins
- Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford Centre for Statistics in Medicine, Oxford, UK
| | - Laure Wynants
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- Department of Epidemiology, Universiteit Maastricht Care and Public Health Research Institute, Maastricht, Netherlands
| | - Jan Y Verbakel
- Department of Public Health and Primary care, KU Leuven, Leuven, Belgium
- Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
- Leuven Unit for Health Technology Assessment Research (LUHTAR), KU Leuven, Leuven, Belgium
| | - Dirk Timmerman
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- Department of Obstetrics and Gynaecology, UZ Leuven campus Gasthuisberg Dienst gynaecologie en verloskunde, Leuven, Belgium
| | - Lil Valentin
- Department of Obstetrics and Gynaecology, Skåne University Hospital, Malmo, Sweden
- Department of Clinical Sciences Malmö, Lund University, Lund, Sweden
| | - Ben Van Calster
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- Leuven Unit for Health Technology Assessment Research (LUHTAR), KU Leuven, Leuven, Belgium
- Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, Netherlands
| |
Collapse
|
25
|
Gehringer CK, Martin GP, Hyrich KL, Verstappen SMM, Sexton J, Kristianslund EK, Provan SA, Kvien TK, Sergeant JC. Developing and externally validating multinomial prediction models for methotrexate treatment outcomes in patients with rheumatoid arthritis: results from an international collaboration. J Clin Epidemiol 2024; 166:111239. [PMID: 38072179 DOI: 10.1016/j.jclinepi.2023.111239] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2023] [Revised: 11/23/2023] [Accepted: 12/05/2023] [Indexed: 01/01/2024]
Abstract
OBJECTIVES In rheumatology, there is a clinical need to identify patients at high risk (>50%) of not responding to the first-line therapy methotrexate (MTX) due to lack of disease control or discontinuation due to adverse events (AEs). Despite this need, previous prediction models in this context are at high risk of bias and ignore AEs. Our objectives were to (i) develop a multinomial model for outcomes of low disease activity and discontinuing due to AEs 6 months after starting MTX, (ii) update prognosis 3-month following treatment initiation, and (iii) externally validate these models. STUDY DESIGN AND SETTING A multinomial model for low disease activity (submodel 1) and discontinuing due to AEs (submodel 2) was developed using data from the UK Rheumatoid Arthritis Medication Study, updated using landmarking analysis, internally validated using bootstrapping, and externally validated in the Norwegian Disease-Modifying Antirheumatic Drug register. Performance was assessed using calibration (calibration-slope and calibration-in-the-large), and discrimination (concordance-statistic and polytomous discriminatory index). RESULTS The internally validated model showed good calibration in the development setting with a calibration-slope of 1.01 (0.87, 1.14) (submodel 1) and 0.83 (0.30, 1.34) (submodel 2), and moderate discrimination with a c-statistic of 0.72 (0.69, 0.74) and 0.53 (0.48, 0.59), respectively. Predictive performance decreased after external validation (calibration-slope 0.78 (0.64, 0.93) (submodel 1) and 0.86 (0.34, 1.38) (submodel 2)), which may be due to differences in disease-specific characteristics and outcome prevalence. CONCLUSION We addressed previously identified methodological limitations of prediction models for outcomes of MTX therapy. The multinomial approach predicted outcomes of disease activity more accurately than AEs, which should be addressed in future work to aid implementation into clinical practice.
Collapse
Affiliation(s)
- Celina K Gehringer
- Division of Musculoskeletal and Dermatological Sciences, Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, University of Manchester, Manchester, UK; Centre for Biostatistics, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK.
| | - Glen P Martin
- Division of Informatics, Imaging and Data Sciences, Centre for Health Informatics, University of Manchester, Manchester, UK
| | - Kimme L Hyrich
- Division of Musculoskeletal and Dermatological Sciences, Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, University of Manchester, Manchester, UK; NIHR Manchester Biomedical Research Centre, Manchester University NHS Foundation Trust, Manchester Academic Health Science Centre, Manchester, UK
| | - Suzanne M M Verstappen
- Division of Musculoskeletal and Dermatological Sciences, Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, University of Manchester, Manchester, UK; NIHR Manchester Biomedical Research Centre, Manchester University NHS Foundation Trust, Manchester Academic Health Science Centre, Manchester, UK
| | - Joseph Sexton
- Center for Treatment of Rheumatic and Musculoskeletal Diseases (REMEDY), Diakonhjemmet Hospital, Oslo, Norway
| | - Eirik K Kristianslund
- Center for Treatment of Rheumatic and Musculoskeletal Diseases (REMEDY), Diakonhjemmet Hospital, Oslo, Norway
| | - Sella A Provan
- Center for Treatment of Rheumatic and Musculoskeletal Diseases (REMEDY), Diakonhjemmet Hospital, Oslo, Norway
| | - Tore K Kvien
- Center for Treatment of Rheumatic and Musculoskeletal Diseases (REMEDY), Diakonhjemmet Hospital, Oslo, Norway; Faculty of Medicine, University of Oslo, Oslo, Norway
| | - Jamie C Sergeant
- Division of Musculoskeletal and Dermatological Sciences, Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, University of Manchester, Manchester, UK; Centre for Biostatistics, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK
| |
Collapse
|
26
|
Riley RD, Snell KIE, Archer L, Ensor J, Debray TPA, van Calster B, van Smeden M, Collins GS. Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study. BMJ 2024; 384:e074821. [PMID: 38253388 DOI: 10.1136/bmj-2023-074821] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/24/2024]
Affiliation(s)
- Richard D Riley
- Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham B15 2TT, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK
| | - Kym I E Snell
- Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham B15 2TT, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK
| | - Lucinda Archer
- Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham B15 2TT, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK
| | - Joie Ensor
- Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham B15 2TT, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK
| | - Thomas P A Debray
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Ben van Calster
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, Netherlands
| | - Maarten van Smeden
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| |
Collapse
|
27
|
Riley RD, Archer L, Snell KIE, Ensor J, Dhiman P, Martin GP, Bonnett LJ, Collins GS. Evaluation of clinical prediction models (part 2): how to undertake an external validation study. BMJ 2024; 384:e074820. [PMID: 38224968 PMCID: PMC10788734 DOI: 10.1136/bmj-2023-074820] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 09/13/2023] [Indexed: 01/17/2024]
Affiliation(s)
- Richard D Riley
- Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham B15 2TT, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK
| | - Lucinda Archer
- Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham B15 2TT, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK
| | - Kym I E Snell
- Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham B15 2TT, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK
| | - Joie Ensor
- Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham B15 2TT, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK
| | - Paula Dhiman
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - Glen P Martin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| | - Laura J Bonnett
- Department of Biostatistics, University of Liverpool, Liverpool, UK
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| |
Collapse
|
28
|
Chekroud AM, Hawrilenko M, Loho H, Bondar J, Gueorguieva R, Hasan A, Kambeitz J, Corlett PR, Koutsouleris N, Krumholz HM, Krystal JH, Paulus M. Illusory generalizability of clinical prediction models. Science 2024; 383:164-167. [PMID: 38207039 DOI: 10.1126/science.adg8538] [Citation(s) in RCA: 20] [Impact Index Per Article: 20.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2023] [Accepted: 11/10/2023] [Indexed: 01/13/2024]
Abstract
It is widely hoped that statistical models can improve decision-making related to medical treatments. Because of the cost and scarcity of medical outcomes data, this hope is typically based on investigators observing a model's success in one or two datasets or clinical contexts. We scrutinized this optimism by examining how well a machine learning model performed across several independent clinical trials of antipsychotic medication for schizophrenia. Models predicted patient outcomes with high accuracy within the trial in which the model was developed but performed no better than chance when applied out-of-sample. Pooling data across trials to predict outcomes in the trial left out did not improve predictions. These results suggest that models predicting treatment outcomes in schizophrenia are highly context-dependent and may have limited generalizability.
Collapse
Affiliation(s)
- Adam M Chekroud
- Spring Health, New York City, NY 10010, USA
- Department of Psychiatry, Yale University School of Medicine, New Haven, CT 06520, USA
| | | | - Hieronimus Loho
- Department of Psychiatry, Yale University School of Medicine, New Haven, CT 06520, USA
| | | | | | - Alkomiet Hasan
- Department of Psychiatry, Psychotherapy and Psychosomatics, University Augsburg, 86159 Augsburg, Germany
| | - Joseph Kambeitz
- Department of Psychiatry and Psychotherapy, University of Cologne, Faculty of Medicine and University Hospital of Cologne, Cologne, Germany
| | - Philip R Corlett
- Department of Psychiatry, Yale University School of Medicine, New Haven, CT 06520, USA
| | - Nikolaos Koutsouleris
- Department of Psychiatry and Psychotherapy, Ludwig-Maximilians-University, Munich, Germany
| | - Harlan M Krumholz
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, CT 06520, USA
| | - John H Krystal
- Department of Psychiatry, Yale University School of Medicine, New Haven, CT 06520, USA
| | - Martin Paulus
- Laureate Institute for Brain Research, Tulsa, OK 74136, USA
| |
Collapse
|
29
|
Schulte PJ, Goldberg JD, Oster RA, Ambrosius WT, Bonner LB, Cabral H, Carter RE, Chen Y, Desai M, Li D, Lindsell CJ, Pomann GM, Slade E, Tosteson TD, Yu F, Spratt H. Peer review of clinical and translational research manuscripts: Perspectives from statistical collaborators. J Clin Transl Sci 2024; 8:e20. [PMID: 38384899 PMCID: PMC10879991 DOI: 10.1017/cts.2023.707] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2023] [Revised: 11/29/2023] [Accepted: 12/19/2023] [Indexed: 02/23/2024] Open
Abstract
Research articles in the clinical and translational science literature commonly use quantitative data to inform evaluation of interventions, learn about the etiology of disease, or develop methods for diagnostic testing or risk prediction of future events. The peer review process must evaluate the methodology used therein, including use of quantitative statistical methods. In this manuscript, we provide guidance for peer reviewers tasked with assessing quantitative methodology, intended to complement guidelines and recommendations that exist for manuscript authors. We describe components of clinical and translational science research manuscripts that require assessment including study design and hypothesis evaluation, sampling and data acquisition, interventions (for studies that include an intervention), measurement of data, statistical analysis methods, presentation of the study results, and interpretation of the study results. For each component, we describe what reviewers should look for and assess; how reviewers should provide helpful comments for fixable errors or omissions; and how reviewers should communicate uncorrectable and irreparable errors. We then discuss the critical concepts of transparency and acceptance/revision guidelines when communicating with responsible journal editors.
Collapse
Affiliation(s)
- Phillip J. Schulte
- Division of Clinical Trials and Biostatistics, Department of Quantitative Health Sciences, Mayo Clinic, Rochester, MN, USA
| | - Judith D. Goldberg
- Division of Biostatistics, Department of Population Health, New York University Grossman School of Medicine, New York, NY, USA
| | - Robert A. Oster
- Department of Medicine, Division of Preventive Medicine, University of Alabama at Birmingham, Birmingham, AL, USA
| | - Walter T. Ambrosius
- Department of Biostatistics and Data Science, Wake Forest University School of Medicine, Winston-Salem, NC, USA
| | - Lauren Balmert Bonner
- Department of Preventive Medicine, Northwestern University Feinberg School of Medicine, Chicago, IL, USA
| | - Howard Cabral
- Department of Biostatistics, Boston University School of Public Health, Boston, MA, USA
| | - Rickey E. Carter
- Department of Quantitative Health Sciences, Mayo Clinic, Jacksonville, FL, USA
| | - Ye Chen
- Biostatistics, Epidemiology and Research Design (BERD), Tufts Clinical and Translational Science Institute (CTSI), Boston, MA, USA
| | - Manisha Desai
- Quantitative Sciences Unit, Departments of Medicine, Biomedical Data Science, and Epidemiology and Population Health, Stanford University, Stanford, CA, USA
| | - Dongmei Li
- Department of Clinical and Translational Research, Obstetrics and Gynecology and Public Health Sciences, University of Rochester Medical Center, Rochester, NY, USA
| | | | - Gina-Maria Pomann
- Department of Biostatistics and Bioinformatics, Duke University, Durham, NC, USA
| | - Emily Slade
- Department of Biostatistics, University of Kentucky, Lexington, KY, USA
| | - Tor D. Tosteson
- Department of Biomedical Data Science, Geisel School of Medicine, Dartmouth College, Hanover, NH, USA
| | - Fang Yu
- Department of Biostatistics, University of Nebraska Medical Center, Omaha, NE, USA
| | - Heidi Spratt
- Department of Biostatistics and Data Science, School of Public and Population Health, University of Texas Medical Branch, Galveston, TX, USA
| |
Collapse
|
30
|
Deisenhofer AK, Barkham M, Beierl ET, Schwartz B, Aafjes-van Doorn K, Beevers CG, Berwian IM, Blackwell SE, Bockting CL, Brakemeier EL, Brown G, Buckman JEJ, Castonguay LG, Cusack CE, Dalgleish T, de Jong K, Delgadillo J, DeRubeis RJ, Driessen E, Ehrenreich-May J, Fisher AJ, Fried EI, Fritz J, Furukawa TA, Gillan CM, Gómez Penedo JM, Hitchcock PF, Hofmann SG, Hollon SD, Jacobson NC, Karlin DR, Lee CT, Levinson CA, Lorenzo-Luaces L, McDanal R, Moggia D, Ng MY, Norris LA, Patel V, Piccirillo ML, Pilling S, Rubel JA, Salazar-de-Pablo G, Schleider JL, Schnurr PP, Schueller SM, Siegle GJ, Uher R, Watkins E, Webb CA, Wiltsey Stirman S, Wynants L, Youn SJ, Zilcha-Mano S, Lutz W, Cohen ZD. Implementing precision methods in personalizing psychological therapies: Barriers and possible ways forward. Behav Res Ther 2024; 172:104443. [PMID: 38086157 DOI: 10.1016/j.brat.2023.104443] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2023] [Revised: 11/21/2023] [Accepted: 11/27/2023] [Indexed: 12/26/2023]
Affiliation(s)
| | | | | | | | | | | | | | | | - Claudi L Bockting
- AmsterdamUMC, Department of Psychiatry, Research Program Amsterdam Public Health and Centre for Urban Mental Health, University of Amsterdam, the Netherlands
| | | | | | | | | | | | | | - Kim de Jong
- Leiden University, Institute of Psychology, USA
| | | | | | | | | | | | | | - Jessica Fritz
- University of Cambridge, UK; Philipps University of Marburg, Germany
| | | | - Claire M Gillan
- School of Psychology, Trinity College Institute for Neuroscience, And Global Brain Health Institute, Trinity College Dublin, USA
| | | | | | | | | | | | | | | | | | | | | | | | - Mei Yi Ng
- Florida International University, USA
| | | | | | | | | | | | | | - Jessica L Schleider
- Stony Brook University and Feinberg School of Medicine Northwestern University, USA
| | - Paula P Schnurr
- National Center for PTSD and Geisel School of Medicine at Dartmouth, USA
| | | | | | | | | | | | | | | | - Soo Jeong Youn
- Reliant Medical Group, OptumCare and Harvard Medical School, USA
| | | | | | - Zachary D Cohen
- University of California, Los Angeles and University of Arizona, USA.
| |
Collapse
|
31
|
Pieszko K, Hiczkiewicz J, Łojewska K, Uziębło-Życzkowska B, Krzesiński P, Gawałko M, Budnik M, Starzyk K, Wożakowska-Kapłon B, Daniłowicz-Szymanowicz L, Kaufmann D, Wójcik M, Błaszczyk R, Mizia-Stec K, Wybraniec M, Kosmalska K, Fijałkowski M, Szymańska A, Dłużniewski M, Kucio M, Haberka M, Kupczyńska K, Michalski B, Tomaszuk-Kazberuk A, Wilk-Śledziewska K, Wachnicka-Truty R, Koziński M, Kwieciński J, Wolny R, Kowalik E, Kolasa I, Jurek A, Budzianowski J, Burchardt P, Kapłon-Cieślicka A, Slomka PJ. Artificial intelligence in detecting left atrial appendage thrombus by transthoracic echocardiography and clinical features: the Left Atrial Thrombus on Transoesophageal Echocardiography (LATTEE) registry. Eur Heart J 2024; 45:32-41. [PMID: 37453044 PMCID: PMC10757867 DOI: 10.1093/eurheartj/ehad431] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/14/2022] [Revised: 05/03/2023] [Accepted: 06/22/2023] [Indexed: 07/18/2023] Open
Abstract
AIMS Transoesophageal echocardiography (TOE) is often performed before catheter ablation or cardioversion to rule out the presence of left atrial appendage thrombus (LAT) in patients on chronic oral anticoagulation (OAC), despite associated discomfort. A machine learning model [LAT-artificial intelligence (AI)] was developed to predict the presence of LAT based on clinical and transthoracic echocardiography (TTE) features. METHODS AND RESULTS Data from a 13-site prospective registry of patients who underwent TOE before cardioversion or catheter ablation were used. LAT-AI was trained to predict LAT using data from 12 sites (n = 2827) and tested externally in patients on chronic OAC from two sites (n = 1284). Areas under the receiver operating characteristic curve (AUC) of LAT-AI were compared with that of left ventricular ejection fraction (LVEF) and CHA2DS2-VASc score. A decision threshold allowing for a 99% negative predictive value was defined in the development cohort. A protocol where TOE in patients on chronic OAC is performed depending on the LAT-AI score was validated in the external cohort. In the external testing cohort, LAT was found in 5.5% of patients. LAT-AI achieved an AUC of 0.85 [95% confidence interval (CI): 0.82-0.89], outperforming LVEF (0.81, 95% CI 0.76-0.86, P < .0001) and CHA2DS2-VASc score (0.69, 95% CI: 0.63-0.7, P < .0001) in the entire external cohort. Based on the proposed protocol, 40% of patients on chronic OAC from the external cohort would safely avoid TOE. CONCLUSION LAT-AI allows accurate prediction of LAT. A LAT-AI-based protocol could be used to guide the decision to perform TOE despite chronic OAC.
Collapse
Affiliation(s)
- Konrad Pieszko
- ‘Club 30’, Polish Cardiac Society, Poland
- Department of Interventional Cardiology and Cardiac Surgery, University of Zielona Gora, Collegium Medicum, Zielona Gora, Poland
- WSSP ZOZ Nowa Sol, Nowa Sol, Poland
| | - Jarosław Hiczkiewicz
- Department of Interventional Cardiology and Cardiac Surgery, University of Zielona Gora, Collegium Medicum, Zielona Gora, Poland
- WSSP ZOZ Nowa Sol, Nowa Sol, Poland
| | | | - Beata Uziębło-Życzkowska
- ‘Club 30’, Polish Cardiac Society, Poland
- Department of Cardiology and Internal Diseases, Military Institute of Medicine, Warsaw, Poland
| | - Paweł Krzesiński
- ‘Club 30’, Polish Cardiac Society, Poland
- Department of Cardiology and Internal Diseases, Military Institute of Medicine, Warsaw, Poland
| | - Monika Gawałko
- ‘Club 30’, Polish Cardiac Society, Poland
- First Department of Cardiology, Medical University of Warsaw, Warsaw, Poland
- Department of Cardiology, Maastricht University Medical Centre and Cardiovascular Research Institute Maastricht, Maastricht, The Netherlands
- Institute of Pharmacology, West German Heart and Vascular Centre, University Duisburg-Essen, Essen, Germany
| | - Monika Budnik
- ‘Club 30’, Polish Cardiac Society, Poland
- First Department of Cardiology, Medical University of Warsaw, Warsaw, Poland
| | - Katarzyna Starzyk
- 1st Clinic of Cardiology and Electrotherapy, Swietokrzyskie Cardiology Centre, Kielce, Poland
| | - Beata Wożakowska-Kapłon
- 1st Clinic of Cardiology and Electrotherapy, Swietokrzyskie Cardiology Centre, Kielce, Poland
| | | | - Damian Kaufmann
- Department of Cardiology and Electrotherapy, Medical University of Gdansk, Gdansk, Poland
| | - Maciej Wójcik
- Department of Cardiology, Medical University of Lublin, Lublin, Poland
| | - Robert Błaszczyk
- Department of Cardiology, Medical University of Lublin, Lublin, Poland
| | - Katarzyna Mizia-Stec
- 1st Department of Cardiology, School of Medicine in Katowice, Medical University of Silesia, Katowice, Poland
| | - Maciej Wybraniec
- 1st Department of Cardiology, School of Medicine in Katowice, Medical University of Silesia, Katowice, Poland
| | | | | | - Anna Szymańska
- Department of Heart Diseases, Postgraduate Medical School, Warsaw, Poland
| | | | - Michał Kucio
- Department of Cardiology, School of Health Sciences, Medical University of Silesia, Katowice, Poland
| | - Maciej Haberka
- Department of Cardiology, School of Health Sciences, Medical University of Silesia, Katowice, Poland
| | | | - Błażej Michalski
- Department of Cardiology, Medical University of Lodz, Lodz, Poland
| | | | | | - Renata Wachnicka-Truty
- Department of Cardiology and Internal Medicine, Medical University of Gdansk, Gdynia, Poland
| | - Marek Koziński
- ‘Club 30’, Polish Cardiac Society, Poland
- Department of Cardiology and Internal Medicine, Medical University of Gdansk, Gdynia, Poland
| | - Jacek Kwieciński
- Department of Interventional Cardiology and Angiology, Institute of Cardiology, Warsaw, Poland
| | - Rafał Wolny
- Department of Interventional Cardiology and Angiology, Institute of Cardiology, Warsaw, Poland
| | - Ewa Kowalik
- Department of Congenital Heart Diseases, National Institute of Cardiology, Warsaw, Poland
| | - Iga Kolasa
- Department of Interventional Cardiology and Cardiac Surgery, University of Zielona Gora, Collegium Medicum, Zielona Gora, Poland
| | - Agnieszka Jurek
- ‘Club 30’, Polish Cardiac Society, Poland
- Department of Cardiology and Internal Diseases, Military Institute of Medicine, Warsaw, Poland
| | - Jan Budzianowski
- ‘Club 30’, Polish Cardiac Society, Poland
- Department of Interventional Cardiology and Cardiac Surgery, University of Zielona Gora, Collegium Medicum, Zielona Gora, Poland
- WSSP ZOZ Nowa Sol, Nowa Sol, Poland
| | - Paweł Burchardt
- ‘Club 30’, Polish Cardiac Society, Poland
- Department of Biology and Lipid Disorders, Poznan University of Medical Sciences, Poznan, Poland
| | - Agnieszka Kapłon-Cieślicka
- ‘Club 30’, Polish Cardiac Society, Poland
- First Department of Cardiology, Medical University of Warsaw, Warsaw, Poland
| | - Piotr J Slomka
- Department of Medicine (Division of Artificial Intelligence in Medicine), Cedars-Sinai Medical Center, 8700 Beverly Blvd, Suite Metro 203, 90048, Los Angeles, CA, USA
| |
Collapse
|
32
|
Hryciw BN, Rodic S, Selim S, Wang C, Lepage MF, Nguyen LH, Goyal V, van Walraven C. Derivation and External Validation of the Ottawa Bloodstream Infection Model for Acutely Ill Adults. J Gen Intern Med 2024; 39:103-112. [PMID: 37723368 PMCID: PMC10817882 DOI: 10.1007/s11606-023-08407-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/12/2023] [Accepted: 08/30/2023] [Indexed: 09/20/2023]
Abstract
BACKGROUND Knowing the probability that patients have a bloodstream infection (BSI) could influence the ordering of blood cultures and interpretation of their preliminary results. Many previous BSI probability models have limited applicability and accuracy. This study used currently recommended modeling techniques and a large sample to derive and validate the Ottawa BSI Model. METHODS At a tertiary care teaching hospital, we retrieved a random sample of 4180 adults having blood cultures in our emergency department or during the initial 48 h of the encounter. Variable selection was based on clinical experience and a systematic review of previous model performance. Model performance was measured in a temporal external validation group of 4680 patients. RESULTS A total of 327 derivation patients had a BSI (8.0%). BSI risk increased with increased number of culture sets (2 sets: adjusted odds ratio [aOR] 1.52 [1.10-2.11]; 3 sets: 1.99 [0.86-4.58]); with indwelling catheter (aOR 2.07 [1.34-3.20); with increasing temperature, heart rate, and neutrophil-lymphocyte ratio; and with decreasing systolic blood pressure, platelet count, urea-creatinine ratio, and estimated glomerular filtration rate. In the temporal external validation group, model discrimination was good (c-statistic 0.71 [0.69-0.74]) and calibration was very good (integrated calibration index .016 [.010-.024]). Exclusion of validation patients with acute SARS-CoV-2 infection improved discrimination slightly (c-statistic 0.73 [0.69-0.76]). CONCLUSIONS The Ottawa BSI Model uses commonly available data to return an expected BSI probability for acutely ill patients. However, it cannot exclude BSI and its complexity requires computational assistance to use.
Collapse
Affiliation(s)
- Brett N Hryciw
- Department of Medicine, University of Ottawa, Ottawa, Canada
| | - Stefan Rodic
- Department of Medicine, University of Ottawa, Ottawa, Canada
| | - Shehab Selim
- Department of Medicine, University of Ottawa, Ottawa, Canada
| | - Chuqi Wang
- Department of Medicine, University of Ottawa, Ottawa, Canada
| | | | | | - Vineet Goyal
- Department of Medicine, University of Ottawa, Ottawa, Canada
| | - Carl van Walraven
- Department of Medicine, University of Ottawa, Ottawa, Canada.
- Department of Epidemiology & Community Medicine, University of Ottawa, Ottawa Hospital Research Institute, ICES, Ottawa, Canada.
| |
Collapse
|
33
|
Rice ML, Barreto EF, Rule AD, Martin CE, Truong HL, Mara KC, Kashani KB, Thompson CA, Witzig TE, Barreto JN. Development and validation of a model to predict acute kidney injury following high-dose methotrexate in patients with lymphoma. Pharmacotherapy 2024; 44:4-12. [PMID: 37926860 DOI: 10.1002/phar.2889] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Revised: 09/26/2023] [Accepted: 09/27/2023] [Indexed: 11/07/2023]
Abstract
STUDY OBJECTIVE To develop and validate a model for predicting acute kidney injury (AKI) after high-dose methotrexate (HDMTX) exposure. DESIGN Retrospective analysis. SETTING Multisite integrated health system throughout Minnesota and Wisconsin. PATIENTS Adult patients with lymphoma who received HDMTX as a 4-h infusion. MEASUREMENTS AND MAIN RESULTS LASSO methodology was used to identify factors available at the outset of therapy that predicted incident AKI within 7 days following HDMTX. The model was then validated in an independent cohort. The incidence of AKI within 7 days following HDMTX was 21.6% (95% confidence interval (CI) 18.4%-24.8%) in the derivation cohort (435 unique patients who received a total of 1642 doses of HDMTX) and 15.6% (95% CI 5.3%-24.8%) in the validation cohort (55 unique patients who received a total of 247 doses of HDMTX). Factors significantly associated with AKI after HDMTX in the multivariable model included age ≥ 55 years, male sex, and lower HDMTX dose number. Other factors that were not found to be significantly associated with AKI on multivariable analysis, but were included in the final model, were body surface area, Charlson Comorbidity Index, and estimated glomerular filtration rate. The c-statistic of the model was 0.72 (95% CI 0.69-0.75) in the derivation cohort and 0.72 (95% CI 0.60-0.84) in the validation cohort. CONCLUSION This model utilizing identified sociodemographic and clinical factors is predictive of AKI following HDMTX administration in adult patients with lymphoma.
Collapse
Affiliation(s)
- Mikhaila L Rice
- Department of Pharmacy, Mayo Clinic, Rochester, Minnesota, USA
- Department of Pharmacy, Cleveland Clinic, Cleveland, Ohio, USA
| | - Erin F Barreto
- Department of Pharmacy, Mayo Clinic, Rochester, Minnesota, USA
| | - Andrew D Rule
- Division of Nephrology and Hypertension, Department of Internal Medicine, Mayo Clinic, Rochester, Minnesota, USA
| | | | - Huong L Truong
- Department of Pharmacy, Mayo Clinic, Rochester, Minnesota, USA
| | - Kristin C Mara
- Division of Biomedical Statistics and Informatics, Department of Health Sciences Research, Mayo Clinic, Rochester, Minnesota, USA
| | - Kianoush B Kashani
- Division of Nephrology and Hypertension, Department of Internal Medicine, Mayo Clinic, Rochester, Minnesota, USA
| | - Carrie A Thompson
- Division of Hematology, Department of Internal Medicine, Mayo Clinic, Rochester, Minnesota, USA
| | - Thomas E Witzig
- Division of Hematology, Department of Internal Medicine, Mayo Clinic, Rochester, Minnesota, USA
| | - Jason N Barreto
- Department of Pharmacy, Mayo Clinic, Rochester, Minnesota, USA
| |
Collapse
|
34
|
Riley RD, Pate A, Dhiman P, Archer L, Martin GP, Collins GS. Clinical prediction models and the multiverse of madness. BMC Med 2023; 21:502. [PMID: 38110939 PMCID: PMC10729337 DOI: 10.1186/s12916-023-03212-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Accepted: 12/05/2023] [Indexed: 12/20/2023] Open
Abstract
BACKGROUND Each year, thousands of clinical prediction models are developed to make predictions (e.g. estimated risk) to inform individual diagnosis and prognosis in healthcare. However, most are not reliable for use in clinical practice. MAIN BODY We discuss how the creation of a prediction model (e.g. using regression or machine learning methods) is dependent on the sample and size of data used to develop it-were a different sample of the same size used from the same overarching population, the developed model could be very different even when the same model development methods are used. In other words, for each model created, there exists a multiverse of other potential models for that sample size and, crucially, an individual's predicted value (e.g. estimated risk) may vary greatly across this multiverse. The more an individual's prediction varies across the multiverse, the greater the instability. We show how small development datasets lead to more different models in the multiverse, often with vastly unstable individual predictions, and explain how this can be exposed by using bootstrapping and presenting instability plots. We recommend healthcare researchers seek to use large model development datasets to reduce instability concerns. This is especially important to ensure reliability across subgroups and improve model fairness in practice. CONCLUSIONS Instability is concerning as an individual's predicted value is used to guide their counselling, resource prioritisation, and clinical decision making. If different samples lead to different models with very different predictions for the same individual, then this should cast doubt into using a particular model for that individual. Therefore, visualising, quantifying and reporting the instability in individual-level predictions is essential when proposing a new model.
Collapse
Affiliation(s)
- Richard D Riley
- College of Medical and Dental Sciences, Institute of Applied Health Research, University of Birmingham, Birmingham, B15 2TT, UK.
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK.
| | - Alexander Pate
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK
| | - Paula Dhiman
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK
| | - Lucinda Archer
- College of Medical and Dental Sciences, Institute of Applied Health Research, University of Birmingham, Birmingham, B15 2TT, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK
| | - Glen P Martin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK
| |
Collapse
|
35
|
van Royen FS, Asselbergs FW, Alfonso F, Vardas P, van Smeden M. Five critical quality criteria for artificial intelligence-based prediction models. Eur Heart J 2023; 44:4831-4834. [PMID: 37897346 DOI: 10.1093/eurheartj/ehad727] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/30/2023] Open
Abstract
To raise the quality of clinical artificial intelligence (AI) prediction modelling studies in the cardiovascular health domain and thereby improve their impact and relevancy, the editors for digital health, innovation, and quality standards of the European Heart Journal propose five minimal quality criteria for AI-based prediction model development and validation studies: complete reporting, carefully defined intended use of the model, rigorous validation, large enough sample size, and openness of code and software.
Collapse
Affiliation(s)
- Florien S van Royen
- Department of General Practice & Nursing Science, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Folkert W Asselbergs
- Department of Cardiology, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands
- Health Data Research UK and Institute of Health Informatics, University College London, London, UK
| | - Fernando Alfonso
- Department of Cardiology, Hospital Universitario de la Princesa, Universidad Autónoma de Madrid, IIS-IP. CIVER-CV, Madrid, Spain
| | - Panos Vardas
- Biomedical Research Foundation Academy of Athens (BRFAA) and Hygeia Hospitals Group, Athens, Greece
| | - Maarten van Smeden
- Department of Epidemiology & Health Economics, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Universiteitsweg 100, 3584 CG Utrecht, Netherlands
- Department of Data Science & Biostatistics, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Universiteitsweg 100, 3584 CG Utrecht, The Netherlands
| |
Collapse
|
36
|
Riley RD, Collins GS. Stability of clinical prediction models developed using statistical or machine learning methods. Biom J 2023; 65:e2200302. [PMID: 37466257 PMCID: PMC10952221 DOI: 10.1002/bimj.202200302] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2022] [Revised: 04/26/2023] [Accepted: 05/02/2023] [Indexed: 07/20/2023]
Abstract
Clinical prediction models estimate an individual's risk of a particular health outcome. A developed model is a consequence of the development dataset and model-building strategy, including the sample size, number of predictors, and analysis method (e.g., regression or machine learning). We raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model-building steps (those used to develop the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and deriving (i) a prediction instability plot of bootstrap model versus original model predictions; (ii) the mean absolute prediction error (mean absolute difference between individuals' original and bootstrap model predictions), and (iii) calibration, classification, and decision curve instability plots of bootstrap models applied in the original sample. A case study illustrates how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), while informing a model's critical appraisal (risk of bias rating), fairness, and further validation requirements.
Collapse
Affiliation(s)
- Richard D. Riley
- Institute of Applied Health ResearchCollege of Medical and Dental SciencesUniversity of BirminghamBirminghamUK
| | - Gary S. Collins
- Centre for Statistics in MedicineNuffield Department of OrthopaedicsRheumatology and Musculoskeletal SciencesUniversity of OxfordOxfordUK
| |
Collapse
|
37
|
van de Loo B, Heymans MW, Medlock S, Boyé NDA, van der Cammen TJM, Hartholt KA, Emmelot-Vonk MH, Mattace-Raso FUS, Abu-Hanna A, van der Velde N, van Schoor NM. Validation of the ADFICE_IT Models for Predicting Falls and Recurrent Falls in Geriatric Outpatients. J Am Med Dir Assoc 2023; 24:1996-2001. [PMID: 37268014 DOI: 10.1016/j.jamda.2023.04.021] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/25/2022] [Revised: 04/14/2023] [Accepted: 04/17/2023] [Indexed: 06/04/2023]
Abstract
OBJECTIVES Before being used in clinical practice, a prediction model should be tested in patients whose data were not used in model development. Previously, we developed the ADFICE_IT models for predicting any fall and recurrent falls, referred as Any_fall and Recur_fall. In this study, we externally validated the models and compared their clinical value to a practical screening strategy where patients are screened for falls history alone. DESIGN Retrospective, combined analysis of 2 prospective cohorts. SETTING AND PARTICIPANTS Data were included of 1125 patients (aged ≥65 years) who visited the geriatrics department or the emergency department. METHODS We evaluated the models' discrimination using the C-statistic. Models were updated using logistic regression if calibration intercept or slope values deviated significantly from their ideal values. Decision curve analysis was applied to compare the models' clinical value (ie, net benefit) against that of falls history for different decision thresholds. RESULTS During the 1-year follow-up, 428 participants (42.7%) endured 1 or more falls, and 224 participants (23.1%) endured a recurrent fall (≥2 falls). C-statistic values were 0.66 (95% CI 0.63-0.69) and 0.69 (95% CI 0.65-0.72) for the Any_fall and Recur_fall models, respectively. Any_fall overestimated the fall risk and we therefore updated only its intercept whereas Recur_fall showed good calibration and required no update. Compared with falls history, Any_fall and Recur_fall showed greater net benefit for decision thresholds of 35% to 60% and 15% to 45%, respectively. CONCLUSIONS AND IMPLICATIONS The models performed similarly in this data set of geriatric outpatients as in the development sample. This suggests that fall-risk assessment tools that were developed in community-dwelling older adults may perform well in geriatric outpatients. We found that in geriatric outpatients the models have greater clinical value across a wide range of decision thresholds compared with screening for falls history alone.
Collapse
Affiliation(s)
- Bob van de Loo
- Epidemiology and Data Science, Amsterdam UMC location Vrije Universiteit Amsterdam, Amsterdam, the Netherlands; Internal Medicine, Section of Geriatric Medicine, Amsterdam UMC location University of Amsterdam, Amsterdam, the Netherlands; Amsterdam Public Health research institute, Amsterdam, the Netherlands.
| | - Martijn W Heymans
- Epidemiology and Data Science, Amsterdam UMC location Vrije Universiteit Amsterdam, Amsterdam, the Netherlands; Amsterdam Public Health research institute, Amsterdam, the Netherlands
| | - Stephanie Medlock
- Department of Medical Informatics, Amsterdam UMC location University of Amsterdam, Amsterdam, the Netherlands; Amsterdam Public Health research institute, Amsterdam, the Netherlands
| | - Nicole D A Boyé
- Department of General Surgery, Curaçao Medical Center, Willemstad, Curaçao; Section of Geriatric Medicine, Department of Internal Medicine, Erasmus MC, University Medical Center, Rotterdam, the Netherlands
| | - Tischa J M van der Cammen
- Section of Geriatric Medicine, Department of Internal Medicine, Erasmus MC, University Medical Center, Rotterdam, the Netherlands; Department of Human-Centred Design, Faculty of Industrial Design Engineering, Delft University of Technology, Delft, the Netherlands
| | - Klaas A Hartholt
- Section of Geriatric Medicine, Department of Internal Medicine, Erasmus MC, University Medical Center, Rotterdam, the Netherlands; Department of Surgery-Traumatology, Reinier de Graaf Gasthuis, Delft, the Netherlands
| | - Marielle H Emmelot-Vonk
- Department of Geriatric Medicine, University Medical Centre Utrecht, Utrecht University, Utrecht, the Netherlands
| | - Francesco U S Mattace-Raso
- Section of Geriatric Medicine, Department of Internal Medicine, Erasmus MC, University Medical Center, Rotterdam, the Netherlands
| | - Ameen Abu-Hanna
- Department of Medical Informatics, Amsterdam UMC location University of Amsterdam, Amsterdam, the Netherlands; Amsterdam Public Health research institute, Amsterdam, the Netherlands
| | - Nathalie van der Velde
- Internal Medicine, Section of Geriatric Medicine, Amsterdam UMC location University of Amsterdam, Amsterdam, the Netherlands; Amsterdam Public Health research institute, Amsterdam, the Netherlands; Section of Geriatric Medicine, Department of Internal Medicine, Erasmus MC, University Medical Center, Rotterdam, the Netherlands
| | - Natasja M van Schoor
- Epidemiology and Data Science, Amsterdam UMC location Vrije Universiteit Amsterdam, Amsterdam, the Netherlands; Amsterdam Public Health research institute, Amsterdam, the Netherlands
| |
Collapse
|
38
|
Vigdal ØN, Storheim K, Killingmo RM, Rysstad T, Pripp AH, van der Gaag W, Chiarotto A, Koes B, Grotle M. External validation and updating of prognostic prediction models for nonrecovery among older adults seeking primary care for back pain. Pain 2023; 164:2759-2768. [PMID: 37490100 DOI: 10.1097/j.pain.0000000000002974] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2022] [Accepted: 03/23/2023] [Indexed: 07/26/2023]
Abstract
ABSTRACT Prognostic prediction models for 3 different definitions of nonrecovery were developed in the Back Complaints in the Elders study in the Netherlands. The models' performance was good (optimism-adjusted area under receiver operating characteristics [AUC] curve ≥0.77, R2 ≥0.3). This study aimed to assess the external validity of the 3 prognostic prediction models in the Norwegian Back Complaints in the Elders study. We conducted a prospective cohort study, including 452 patients aged ≥55 years, seeking primary care for a new episode of back pain. Nonrecovery was defined for 2 outcomes, combining 6- and 12-month follow-up data: Persistent back pain (≥3/10 on numeric rating scale) and persistent disability (≥4/24 on Roland-Morris Disability Questionnaire). We could not assess the third model (self-reported nonrecovery) because of substantial missing data (>50%). The models consisted of biopsychosocial prognostic factors. First, we assessed Nagelkerke R2 , discrimination (AUC) and calibration (calibration-in-the-large [CITL], slope, and calibration plot). Step 2 was to recalibrate the models based on CITL and slope. Step 3 was to reestimate the model coefficients and assess if this improved performance. The back pain model demonstrated acceptable discrimination (AUC 0.74, 95% confidence interval: 0.69-0.79), and R2 was 0.23. The disability model demonstrated excellent discrimination (AUC 0.81, 95% confidence interval: 0.76-0.85), and R2 was 0.35. Both models had poor calibration (CITL <0, slope <1). Recalibration yielded acceptable calibration for both models, according to the calibration plots. Step 3 did not improve performance substantially. The recalibrated models may need further external validation, and the models' clinical impact should be assessed.
Collapse
Affiliation(s)
- Ørjan Nesse Vigdal
- Department of Rehabilitation Science and Health Technology, Faculty of Health Science, OsloMet-Oslo Metropolitan University, Oslo, Norway
| | - Kjersti Storheim
- Department of Rehabilitation Science and Health Technology, Faculty of Health Science, OsloMet-Oslo Metropolitan University, Oslo, Norway
- Research and Communication Unit for Musculoskeletal Health (FORMI), Division of Clinical Neuroscience, Oslo University Hospital, Oslo, Norway
| | - Rikke Munk Killingmo
- Department of Rehabilitation Science and Health Technology, Faculty of Health Science, OsloMet-Oslo Metropolitan University, Oslo, Norway
| | - Tarjei Rysstad
- Department of Rehabilitation Science and Health Technology, Faculty of Health Science, OsloMet-Oslo Metropolitan University, Oslo, Norway
| | - Are Hugo Pripp
- Department of Rehabilitation Science and Health Technology, Faculty of Health Science, OsloMet-Oslo Metropolitan University, Oslo, Norway
| | - Wendelien van der Gaag
- Department of General Practice, Erasmus MC, University Medical Center, Rotterdam, the Netherlands
| | - Alessandro Chiarotto
- Department of General Practice, Erasmus MC, University Medical Center, Rotterdam, the Netherlands
| | - Bart Koes
- Department of General Practice, Erasmus MC, University Medical Center, Rotterdam, the Netherlands
- Center for Muscle and Health, University of Southern Denmark, Odense, Denmark
| | - Margreth Grotle
- Department of Rehabilitation Science and Health Technology, Faculty of Health Science, OsloMet-Oslo Metropolitan University, Oslo, Norway
- Research and Communication Unit for Musculoskeletal Health (FORMI), Division of Clinical Neuroscience, Oslo University Hospital, Oslo, Norway
| |
Collapse
|
39
|
Fernández-Cisnal A, Lopez-Ayala P, Valero E, Koechlin L, Catarralá A, Boeddinghaus J, Noceda J, Nestelberger T, Miró Ò, Julio N, Mueller C, Sanchis J. Derivation and external validation of machine-learning models for risk stratification in chest pain with normal troponin. EUROPEAN HEART JOURNAL. ACUTE CARDIOVASCULAR CARE 2023; 12:743-752. [PMID: 37531633 DOI: 10.1093/ehjacc/zuad089] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/27/2023] [Revised: 07/18/2023] [Accepted: 07/26/2023] [Indexed: 08/04/2023]
Abstract
AIMS Risk stratification of patients with chest pain and a high-sensitivity cardiac troponin T (hs-cTnT) concentration METHODS AND RESULTS Four machine-learning-based models and one logistic regression (LR) model were trained on 4075 patients (single-centre Spanish cohort) and externally validated on 3609 patients (international prospective Advantageous Predictors of Acute Coronary syndromes Evaluation cohort). Models were compared with GRACE and HEART scores and a single undetectable hs-cTnT-based strategy (u-cTn; hs-cTnT < 5 ng/L and time from symptoms onset >180 min). Probability thresholds for safe discharge were derived in the derivation cohort. The endpoint occurred in 105 (2.6%) patients in the training set and 98 (2.7%) in the external validation set. Gradient boosting full (GBf) showed the best discrimination (area under the curve = 0.808). Calibration was good for the reduced neural network and LR models. Gradient boosting full identified the highest proportion of patients for safe discharge (36.7 vs. 23.4 vs. 27.2%; GBf vs. LR vs. u-cTn, respectively) with similar safety (missed endpoint per 1000 patients: 2.2 vs. 3.5 vs. 3.1, respectively). All derived models were superior to the HEART and GRACE scores (P < 0.001). CONCLUSION Machine-learning and LR prediction models were superior to the HEART, GRACE, and u-cTn for risk stratification of patients with chest pain and a baseline hs-cTnT CLINICAL TRIAL REGISTRATION ClinicalTrials.gov number, NCT00470587, https://clinicaltrials.gov/ct2/show/NCT00470587.
Collapse
Affiliation(s)
- Agustín Fernández-Cisnal
- Cardiology Department, Hospital Clínico Universitario de València, Instituto de Investigación Sanitaria (INCLIVA), University of València, Centro de Investigación Biomédica en Red Enfermedades Cardiovaculares (CIBERCV), València, Spain
| | - Pedro Lopez-Ayala
- Cardiovascular Research Institute Basel (CRIB) and Department of Cardiology, University Heart Center Basel, University Hospital Basel, University of Basel, Basel, Switzerland
| | - Ernesto Valero
- Cardiology Department, Hospital Clínico Universitario de València, Instituto de Investigación Sanitaria (INCLIVA), University of València, Centro de Investigación Biomédica en Red Enfermedades Cardiovaculares (CIBERCV), València, Spain
| | - Luca Koechlin
- Cardiovascular Research Institute Basel (CRIB) and Department of Cardiology, University Heart Center Basel, University Hospital Basel, University of Basel, Basel, Switzerland
| | - Arturo Catarralá
- Clinical Biochemistry Department, Hospital Clínico Universitario de València, Instituto de Investigación Sanitaria (INCLIVA), València 46010, Spain
| | - Jasper Boeddinghaus
- Cardiovascular Research Institute Basel (CRIB) and Department of Cardiology, University Heart Center Basel, University Hospital Basel, University of Basel, Basel, Switzerland
| | - José Noceda
- Emergency Department, Hospital Clínico Universitario de València, Instituto de Investigación Sanitaria (INCLIVA), València 46010, Spain
| | - Thomas Nestelberger
- Cardiovascular Research Institute Basel (CRIB) and Department of Cardiology, University Heart Center Basel, University Hospital Basel, University of Basel, Basel, Switzerland
| | - Òscar Miró
- Emergency Department, Hospital Clinic, Barcelona, Catalonia, Spain
| | - Núñez Julio
- Cardiology Department, Hospital Clínico Universitario de València, Instituto de Investigación Sanitaria (INCLIVA), University of València, Centro de Investigación Biomédica en Red Enfermedades Cardiovaculares (CIBERCV), València, Spain
| | - Christian Mueller
- Cardiovascular Research Institute Basel (CRIB) and Department of Cardiology, University Heart Center Basel, University Hospital Basel, University of Basel, Basel, Switzerland
| | - Juan Sanchis
- Cardiology Department, Hospital Clínico Universitario de València, Instituto de Investigación Sanitaria (INCLIVA), University of València, Centro de Investigación Biomédica en Red Enfermedades Cardiovaculares (CIBERCV), València, Spain
| |
Collapse
|
40
|
Archer L, Snell KIE, Stynes S, Axén I, Dunn KM, Foster NE, Wynne-Jones G, van der Windt DA, Hill JC. Development and External Validation of Individualized Prediction Models for Pain Intensity Outcomes in Patients With Neck Pain, Low Back Pain, or Both in Primary Care Settings. Phys Ther 2023; 103:pzad128. [PMID: 37756617 PMCID: PMC10682973 DOI: 10.1093/ptj/pzad128] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/11/2022] [Revised: 05/05/2023] [Accepted: 07/14/2023] [Indexed: 09/29/2023]
Abstract
OBJECTIVE The purpose of this study was to develop and externally validate multivariable prediction models for future pain intensity outcomes to inform targeted interventions for patients with neck or low back pain in primary care settings. METHODS Model development data were obtained from a group of 679 adults with neck or low back pain who consulted a participating United Kingdom general practice. Predictors included self-report items regarding pain severity and impact from the STarT MSK Tool. Pain intensity at 2 and 6 months was modeled separately for continuous and dichotomized outcomes using linear and logistic regression, respectively. External validation of all models was conducted in a separate group of 586 patients recruited from a similar population with patients' predictor information collected both at point of consultation and 2 to 4 weeks later using self-report questionnaires. Calibration and discrimination of the models were assessed separately using STarT MSK Tool data from both time points to assess differences in predictive performance. RESULTS Pain intensity and patients reporting their condition would last a long time contributed most to predictions of future pain intensity conditional on other variables. On external validation, models were reasonably well calibrated on average when using tool measurements taken 2 to 4 weeks after consultation (calibration slope = 0.848 [95% CI = 0.767 to 0.928] for 2-month pain intensity score), but performance was poor using point-of-consultation tool data (calibration slope for 2-month pain intensity score of 0.650 [95% CI = 0.549 to 0.750]). CONCLUSION Model predictive accuracy was good when predictors were measured 2 to 4 weeks after primary care consultation, but poor when measured at the point of consultation. Future research will explore whether additional, nonmodifiable predictors improve point-of-consultation predictive performance. IMPACT External validation demonstrated that these individualized prediction models were not sufficiently accurate to recommend their use in clinical practice. Further research is required to improve performance through inclusion of additional nonmodifiable risk factors.
Collapse
Affiliation(s)
- Lucinda Archer
- School of Medicine, Keele University, Keele, Staffordshire, UK
- Institute for Applied Health Research, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, UK
| | - Kym I E Snell
- School of Medicine, Keele University, Keele, Staffordshire, UK
- Institute for Applied Health Research, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, UK
| | - Siobhán Stynes
- School of Medicine, Keele University, Keele, Staffordshire, UK
- Midlands Partnership Foundation NHS Trust, North Staffordshire Musculoskeletal Interface Service, Haywood Hospital, Staffordshire, UK
| | - Iben Axén
- Unit of Intervention and Implementation Research for Worker Health, Institute of Environmental Medicine, Karolinska Institutet, Nobels väg 13, Stockholm, Sweden
| | - Kate M Dunn
- School of Medicine, Keele University, Keele, Staffordshire, UK
| | - Nadine E Foster
- School of Medicine, Keele University, Keele, Staffordshire, UK
- Surgical Treatment and Rehabilitation Service (STARS) Education and Research Alliance, The University of Queensland and Metro North Hospital and Health Service, Queensland, Australia
| | | | | | - Jonathan C Hill
- School of Medicine, Keele University, Keele, Staffordshire, UK
| |
Collapse
|
41
|
Byrne JF, Mongan D, Murphy J, Healy C, Fӧcking M, Cannon M, Cotter DR. Prognostic models predicting transition to psychotic disorder using blood-based biomarkers: a systematic review and critical appraisal. Transl Psychiatry 2023; 13:333. [PMID: 37898606 PMCID: PMC10613280 DOI: 10.1038/s41398-023-02623-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/16/2022] [Revised: 09/15/2023] [Accepted: 10/06/2023] [Indexed: 10/30/2023] Open
Abstract
Accumulating evidence suggests individuals with psychotic disorder show abnormalities in metabolic and inflammatory processes. Recently, several studies have employed blood-based predictors in models predicting transition to psychotic disorder in risk-enriched populations. A systematic review of the performance and methodology of prognostic models using blood-based biomarkers in the prediction of psychotic disorder from risk-enriched populations is warranted. Databases (PubMed, EMBASE and PsycINFO) were searched for eligible texts from 1998 to 15/05/2023, which detailed model development or validation studies. The checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) was used to guide data extraction from eligible texts and the Prediction Model Risk of Bias Assessment Tool (PROBAST) was used to assess the risk of bias and applicability of the studies. A narrative synthesis of the included studies was performed. Seventeen eligible studies were identified: 16 eligible model development studies and one eligible model validation study. A wide range of biomarkers were assessed, including nucleic acids, proteins, metabolites, and lipids. The range of C-index (area under the curve) estimates reported for the models was 0.67-1.00. No studies assessed model calibration. According to PROBAST criteria, all studies were at high risk of bias in the analysis domain. While a wide range of potentially predictive biomarkers were identified in the included studies, most studies did not account for overfitting in model performance estimates, no studies assessed calibration, and all models were at high risk of bias according to PROBAST criteria. External validation of the models is needed to provide more accurate estimates of their performance. Future studies which follow the latest available methodological and reporting guidelines and adopt strategies to accommodate required sample sizes for model development or validation will clarify the value of including blood-based biomarkers in models predicting psychosis.
Collapse
Affiliation(s)
- Jonah F Byrne
- Department of Psychiatry, Royal College of Surgeons in Ireland, Dublin, Ireland.
- SFI FutureNeuro Research Centre, Royal College of Surgeons in Ireland, Dublin, Ireland.
| | - David Mongan
- Department of Psychiatry, Royal College of Surgeons in Ireland, Dublin, Ireland
- Centre for Public Health, Queen's University Belfast, Belfast, United Kingdom
| | - Jennifer Murphy
- Department of Psychiatry, Royal College of Surgeons in Ireland, Dublin, Ireland
| | - Colm Healy
- Department of Psychiatry, Royal College of Surgeons in Ireland, Dublin, Ireland
| | - Melanie Fӧcking
- Department of Psychiatry, Royal College of Surgeons in Ireland, Dublin, Ireland
| | - Mary Cannon
- Department of Psychiatry, Royal College of Surgeons in Ireland, Dublin, Ireland
- SFI FutureNeuro Research Centre, Royal College of Surgeons in Ireland, Dublin, Ireland
| | - David R Cotter
- Department of Psychiatry, Royal College of Surgeons in Ireland, Dublin, Ireland
- SFI FutureNeuro Research Centre, Royal College of Surgeons in Ireland, Dublin, Ireland
| |
Collapse
|
42
|
Notarnicola I, Ivziku D, Tartaglini D, Filomeno L, Gualandi R, Ricci S, Lommi M, Porcelli B, Raffaele B, Montini G, Ferramosca FMP, Di Maria E, De Benedictis A, Baysal E, Latina R, Rocco G, Stievano A. Self-Perceived Clinical Competence of Nurses in Different Working Experiences: A Cross-Sectional Study. Healthcare (Basel) 2023; 11:2808. [PMID: 37957953 PMCID: PMC10648505 DOI: 10.3390/healthcare11212808] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2023] [Revised: 10/19/2023] [Accepted: 10/21/2023] [Indexed: 11/15/2023] Open
Abstract
BACKGROUND Competence is an essential concept for measuring nurses' performance in terms of effectiveness and quality. To this end, our analysis highlighted the process of acquiring competencies, their self-evaluation into clinical practice, and how their proficiency levels change throughout the nursing career. In detail, this research explored nurses' perceived level of competence and the factors that influence it in different contexts. METHODS A cross-sectional survey using a structured questionnaire to assess the nursing participants' perception of their competencies in different clinical settings was accomplished. RESULTS A descriptive and bivariate analysis was performed on 431 nurses. Most respondents assessed their level of competence to be higher than their roles required. The Kruskal-Wallis test confirmed that nursing experience was a relevant factor influencing nursing competencies. CONCLUSIONS We suggest improving the competence of practicing nurses, using experience as a measurable effect of their development.
Collapse
Affiliation(s)
- Ippolito Notarnicola
- Centre of Excellence for Nursing Scholarship, OPI of Rome, 00136 Rome, Italy; (G.R.); (A.S.)
- Department of Nursing, Catholic University “Our Lady of Good Counsel”, 1000 Tirana, Albania
| | - Dhurata Ivziku
- Department of Health Profession, Foundation Policlinic Universitario Campus Bio-Medico, 00128 Rome, Italy; (D.I.); (D.T.); (R.G.); (A.D.B.)
| | - Daniela Tartaglini
- Department of Health Profession, Foundation Policlinic Universitario Campus Bio-Medico, 00128 Rome, Italy; (D.I.); (D.T.); (R.G.); (A.D.B.)
| | - Lucia Filomeno
- Department of Biomedicine and Prevetion, University Tor Vergata, 00133 Rome, Italy; (L.F.); (F.M.P.F.)
| | - Raffaella Gualandi
- Department of Health Profession, Foundation Policlinic Universitario Campus Bio-Medico, 00128 Rome, Italy; (D.I.); (D.T.); (R.G.); (A.D.B.)
| | - Simona Ricci
- UOC Care to the Person, Local Health Authority Roma 2, 00159 Rome, Italy; (S.R.); (M.L.); (B.P.); (B.R.); (G.M.); (E.D.M.)
| | - Marzia Lommi
- UOC Care to the Person, Local Health Authority Roma 2, 00159 Rome, Italy; (S.R.); (M.L.); (B.P.); (B.R.); (G.M.); (E.D.M.)
| | - Barbara Porcelli
- UOC Care to the Person, Local Health Authority Roma 2, 00159 Rome, Italy; (S.R.); (M.L.); (B.P.); (B.R.); (G.M.); (E.D.M.)
| | - Barbara Raffaele
- UOC Care to the Person, Local Health Authority Roma 2, 00159 Rome, Italy; (S.R.); (M.L.); (B.P.); (B.R.); (G.M.); (E.D.M.)
| | - Graziella Montini
- UOC Care to the Person, Local Health Authority Roma 2, 00159 Rome, Italy; (S.R.); (M.L.); (B.P.); (B.R.); (G.M.); (E.D.M.)
| | | | - Erica Di Maria
- UOC Care to the Person, Local Health Authority Roma 2, 00159 Rome, Italy; (S.R.); (M.L.); (B.P.); (B.R.); (G.M.); (E.D.M.)
| | - Anna De Benedictis
- Department of Health Profession, Foundation Policlinic Universitario Campus Bio-Medico, 00128 Rome, Italy; (D.I.); (D.T.); (R.G.); (A.D.B.)
| | - Ebru Baysal
- Department of Fundamentals Nursing, Faculty of Health Sciences, Manisa Celal Bayar University, 45030 Manisa, Turkey;
| | - Roberto Latina
- Department of Health Promotion, Mother and Child Care, Internal Medicine and Medical Specialties, University of Palermo, 90128 Palermo, Italy;
| | - Gennaro Rocco
- Centre of Excellence for Nursing Scholarship, OPI of Rome, 00136 Rome, Italy; (G.R.); (A.S.)
- Department of Nursing, Catholic University “Our Lady of Good Counsel”, 1000 Tirana, Albania
| | - Alessandro Stievano
- Centre of Excellence for Nursing Scholarship, OPI of Rome, 00136 Rome, Italy; (G.R.); (A.S.)
- Department of Nursing, Catholic University “Our Lady of Good Counsel”, 1000 Tirana, Albania
- Department of Clinical and Experimental Medicine, University of Messina, 98100 Messina, Italy
| |
Collapse
|
43
|
Chen TLW, Buddhiraju A, Seo HH, Subih MA, Tuchinda P, Kwon YM. Internal and External Validation of the Generalizability of Machine Learning Algorithms in Predicting Non-home Discharge Disposition Following Primary Total Knee Joint Arthroplasty. J Arthroplasty 2023; 38:1973-1981. [PMID: 36764409 DOI: 10.1016/j.arth.2023.01.065] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/24/2022] [Revised: 01/24/2023] [Accepted: 01/31/2023] [Indexed: 02/12/2023] Open
Abstract
BACKGROUND Nonhome discharge disposition following primary total knee arthroplasty (TKA) is associated with a higher rate of complications and constitutes a socioeconomic burden on the health care system. While existing algorithms predicting nonhome discharge disposition varied in degrees of mathematical complexity and prediction power, their capacity to generalize predictions beyond the development dataset remains limited. Therefore, this study aimed to establish the machine learning model generalizability by performing internal and external validations using nation-scale and institutional cohorts, respectively. METHODS Four machine learning models were trained using the national cohort. Recursive feature elimination and hyper-parameter tuning were applied. Internal validation was achieved through five-fold cross-validation during model training. The trained models' performance was externally validated using the institutional cohort and assessed by discrimination, calibration, and clinical utility. RESULTS The national (424,354 patients) and institutional (10,196 patients) cohorts had non-home discharge rates of 19.4 and 36.4%, respectively. The areas under the receiver operating curve of the model predictions were 0.83 to 0.84 during internal validation and increased to 0.88 to 0.89 during external validation. Artificial neural network and histogram-based gradient boosting elicited the best performance with a mean area under the receiver operating curve of 0.89, calibration slope of 1.39, and Brier score of 0.14, which indicated that the two models were robust in distinguishing non-home discharge and well-calibrated with accurate predictions of the probabilities. The low inter-dataset similarity indicated reliable external validation. Length of stay, age, body mass index, and sex were the strongest predictors of discharge destination after primary TKA. CONCLUSION The machine learning models demonstrated excellent predictive performance during both internal and external validations, supporting their generalizability across different patient cohorts and potential applicability in the clinical workflow.
Collapse
Affiliation(s)
- Tony Lin-Wei Chen
- Bioengineering Laboratory, Department of Orthopaedic Surgery, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
| | - Anirudh Buddhiraju
- Bioengineering Laboratory, Department of Orthopaedic Surgery, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
| | - Henry Hojoon Seo
- Bioengineering Laboratory, Department of Orthopaedic Surgery, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
| | - Murad Abdullah Subih
- Bioengineering Laboratory, Department of Orthopaedic Surgery, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
| | - Pete Tuchinda
- Bioengineering Laboratory, Department of Orthopaedic Surgery, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
| | - Young-Min Kwon
- Bioengineering Laboratory, Department of Orthopaedic Surgery, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
| |
Collapse
|
44
|
Shulman MA, Wallace S, Gilbert A, Reilly JR, Kasza J, Myles PS. Predicting Death or Disability after Surgery in the Older Adult. Anesthesiology 2023; 139:420-431. [PMID: 37432667 DOI: 10.1097/aln.0000000000004683] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/12/2023]
Abstract
BACKGROUND Older patients are vulnerable to developing new or worsening disability after surgery. Despite this, patient or surgical characteristics predisposing to postoperative disability are poorly defined. The aim of the study was to develop and validate a model, subsequently transformed to point-score form, to predict 6-month death or disability in older patients after surgery. METHODS The authors built a prospective, single-center registry to develop and validate the prediction model. The registry included patients 70 yr of age or older undergoing elective and nonelective, cardiac and noncardiac surgery between May 25, 2017, and February 11, 2021, and combined clinical data from the electronic medical record, hospital administrative data (International Classification of Diseases, Tenth Revision, Australian Modification codes) and World Health Organization (Geneva, Switzerland) Disability Assessment Schedule data collected directly from the patients. Death or disability was defined as being dead or having a World Health Organization Disability Assessment Schedule score 16% or greater. Included patients were randomly divided into model development (70%) and internal validation (30%) cohorts. Once constructed, the logistic regression and point-score models were assessed using the internal validation cohort and an external validation cohort comprising data from a separate randomized trial. RESULTS Of 2,176 patients who completed the World Health Organization Disability Assessment Schedule immediately before surgery, 927 (43%) patients were disabled, and 413 (19%) had significant disability. By 6 months after surgery, 1,640 patients (75%) had data available for the primary outcome analysis. Of these patients, 195 (12%) patients had died, and 691 (42%) were dead or disabled. The developed point-score model included the preoperative World Health Organization Disability Assessment Schedule score, patient age, dementia, and chronic kidney disease. The point score model retained good discrimination in the internal (area under the curve, 0.74; 95% CI, 0.69 to 0.79) and external (area under the curve, 0.77; 95% CI, 0.74 to 0.80) validation data sets. CONCLUSIONS The authors developed and validated a point score model to predict death or disability in older patients after surgery. EDITOR’S PERSPECTIVE
Collapse
Affiliation(s)
- Mark A Shulman
- Department of Anaesthesiology and Perioperative Medicine, Alfred Hospital and Monash University, Melbourne, Australia
| | - Sophie Wallace
- Department of Anaesthesiology and Perioperative Medicine, Alfred Hospital and Monash University, Melbourne, Australia
| | - Annie Gilbert
- Data Governance and Security, Alfred Hospital, Melbourne, Australia
| | - Jennifer R Reilly
- Department of Anaesthesiology and Perioperative Medicine, Alfred Hospital and Monash University, Melbourne, Australia
| | - Jessica Kasza
- School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
| | - Paul S Myles
- Department of Anaesthesiology and Perioperative Medicine, Alfred Hospital and Monash University, Melbourne, Australia
| |
Collapse
|
45
|
Bao Q, Song L, Ma L, Wang M, Hou Z, Lin J, Li C. Prediction of postoperative hypokalemia in patients with oral cancer undergoing en bloc cancer resection: a retrospective cohort study. BMC Oral Health 2023; 23:663. [PMID: 37710182 PMCID: PMC10500799 DOI: 10.1186/s12903-023-03371-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2022] [Accepted: 08/30/2023] [Indexed: 09/16/2023] Open
Abstract
BACKGROUND The factors associated with postoperative hypokalemia in patients with oral cancer remain unclear. We determined the preoperative factors associated with postoperative hypokalemia in patients with oral cancer following en bloc cancer resection and established a nomogram for postoperative hypokalemia prediction. METHODS Data from 381 patients with oral cancer who underwent en bloc cancer resection were retrospectively analyzed. Univariate and multivariate analyses were performed to identify the risk factors for postoperative hypokalemia. We used receiver operating characteristic (ROC) curves to quantify the factors' effectiveness. A nomogram was created to show each predictor's relative weight and the likelihood of postoperative hypokalemia development. The multinomial regression model's effectiveness was also evaluated. RESULTS Preoperative factors, including sex, preoperative serum potassium level, and preoperative platelet-to-lymphocyte ratio (PLR), were significantly associated with postoperative hypokalemia. Based on the ROC curve, the preoperative serum potassium and PLR cut-off levels were 3.98 mmol/L and 117, respectively. Further multivariate analysis indicated that female sex, preoperative serum potassium level < 3.98 mmol/L, and preoperative PLR ≥ 117 were independently associated with postoperative hypokalemia. We constructed a predictive nomogram with all these factors for the risk of postoperative hypokalemia with good discrimination and internal validation. CONCLUSIONS The predictive nomogram for postoperative hypokalemia risk constructed with these factors had good discrimination and internal validation. The developed nomogram will add value to these independent risk factors that can be identified at admission in order to predict postoperative hypokalemia.
Collapse
Affiliation(s)
- Qilin Bao
- Nutrition Department, West China Hospital of Stomatology, Sichuan University, State Key Laboratory of Oral Diseases, National Clinical Research Center for Oral Diseases, No. 14, Section Three, Ren Min Nan Road, Chengdu, Sichuan, China
| | - Lei Song
- Nutrition Department, West China Hospital of Stomatology, Sichuan University, State Key Laboratory of Oral Diseases, National Clinical Research Center for Oral Diseases, No. 14, Section Three, Ren Min Nan Road, Chengdu, Sichuan, China
| | - Liyuan Ma
- Medical Record Room, West China Hospital of Stomatology, Sichuan University, State Key Laboratory of Oral Diseases, National Clinical Research Center for Oral Diseases, No. 14, Section Three, Ren Min Nan Road, Chengdu, Sichuan, China
| | - Meng Wang
- Medical Record Room, West China Hospital of Stomatology, Sichuan University, State Key Laboratory of Oral Diseases, National Clinical Research Center for Oral Diseases, No. 14, Section Three, Ren Min Nan Road, Chengdu, Sichuan, China
| | - Zhaohuan Hou
- Department of Head and Neck Oncology, West China Hospital of Stomatology, Sichuan University, State Key Laboratory of Oral Diseases, National Clinical Research Center for Oral Diseases, No. 14, Section Three, Ren Min Nan Road, Chengdu, Sichuan, China
| | - Jie Lin
- Department of Oral Anesthesia, West China Hospital of Stomatology, Sichuan University, State Key Laboratory of Oral Diseases, National Clinical Research Center for Oral Diseases, No. 14, Section Three, Ren Min Nan Road, Chengdu, Sichuan, China
| | - Chunjie Li
- Department of Head and Neck Oncology, West China Hospital of Stomatology, Sichuan University, State Key Laboratory of Oral Diseases, National Clinical Research Center for Oral Diseases, No. 14, Section Three, Ren Min Nan Road, Chengdu, Sichuan, China.
| |
Collapse
|
46
|
Fuller GW, Hasan M, Hodkinson P, McAlpine D, Goodacre S, Bath PA, Sbaffi L, Omer Y, Wallis L, Marincowitz C. Training and testing of a gradient boosted machine learning model to predict adverse outcome in patients presenting to emergency departments with suspected covid-19 infection in a middle-income setting. PLOS DIGITAL HEALTH 2023; 2:e0000309. [PMID: 37729117 PMCID: PMC10511129 DOI: 10.1371/journal.pdig.0000309] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/04/2023] [Accepted: 06/27/2023] [Indexed: 09/22/2023]
Abstract
COVID-19 infection rates remain high in South Africa. Clinical prediction models may be helpful for rapid triage, and supporting clinical decision making, for patients with suspected COVID-19 infection. The Western Cape, South Africa, has integrated electronic health care data facilitating large-scale linked routine datasets. The aim of this study was to develop a machine learning model to predict adverse outcome in patients presenting with suspected COVID-19 suitable for use in a middle-income setting. A retrospective cohort study was conducted using linked, routine data, from patients presenting with suspected COVID-19 infection to public-sector emergency departments (EDs) in the Western Cape, South Africa between 27th August 2020 and 31st October 2021. The primary outcome was death or critical care admission at 30 days. An XGBoost machine learning model was trained and internally tested using split-sample validation. External validation was performed in 3 test cohorts: Western Cape patients presenting during the Omicron COVID-19 wave, a UK cohort during the ancestral COVID-19 wave, and a Sudanese cohort during ancestral and Eta waves. A total of 282,051 cases were included in a complete case training dataset. The prevalence of 30-day adverse outcome was 4.0%. The most important features for predicting adverse outcome were the requirement for supplemental oxygen, peripheral oxygen saturations, level of consciousness and age. Internal validation using split-sample test data revealed excellent discrimination (C-statistic 0.91, 95% CI 0.90 to 0.91) and calibration (CITL of 1.05). The model achieved C-statistics of 0.84 (95% CI 0.84 to 0.85), 0.72 (95% CI 0.71 to 0.73), and 0.62, (95% CI 0.59 to 0.65) in the Omicron, UK, and Sudanese test cohorts. Results were materially unchanged in sensitivity analyses examining missing data. An XGBoost machine learning model achieved good discrimination and calibration in prediction of adverse outcome in patients presenting with suspected COVID19 to Western Cape EDs. Performance was reduced in temporal and geographical external validation.
Collapse
Affiliation(s)
- Gordon Ward Fuller
- Centre for Urgent and Emergency Care Research (CURE), Health Services Research School of Health and Related Research, University of Sheffield, Sheffield, United Kingdom
| | - Madina Hasan
- Centre for Urgent and Emergency Care Research (CURE), Health Services Research School of Health and Related Research, University of Sheffield, Sheffield, United Kingdom
| | - Peter Hodkinson
- Division of Emergency Medicine, University of Cape Town, Cape Town, South Africa
| | - David McAlpine
- Division of Emergency Medicine, University of Cape Town, Cape Town, South Africa
| | - Steve Goodacre
- Centre for Urgent and Emergency Care Research (CURE), Health Services Research School of Health and Related Research, University of Sheffield, Sheffield, United Kingdom
| | - Peter A. Bath
- Centre for Urgent and Emergency Care Research (CURE), Health Services Research School of Health and Related Research, University of Sheffield, Sheffield, United Kingdom
- Information School, University of Sheffield, Sheffield, United Kingdom
| | - Laura Sbaffi
- Information School, University of Sheffield, Sheffield, United Kingdom
| | - Yasein Omer
- Division of Emergency Medicine, University of Cape Town, Cape Town, South Africa
| | - Lee Wallis
- Division of Emergency Medicine, University of Cape Town, Cape Town, South Africa
| | - Carl Marincowitz
- Centre for Urgent and Emergency Care Research (CURE), Health Services Research School of Health and Related Research, University of Sheffield, Sheffield, United Kingdom
| |
Collapse
|
47
|
Gao S, Albu E, Tuand K, Cossey V, Rademakers F, Van Calster B, Wynants L. Systematic review finds risk of bias and applicability concerns for models predicting central line-associated bloodstream infection. J Clin Epidemiol 2023; 161:127-139. [PMID: 37536503 DOI: 10.1016/j.jclinepi.2023.07.019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2023] [Revised: 07/25/2023] [Accepted: 07/26/2023] [Indexed: 08/05/2023]
Abstract
OBJECTIVES To systematically review the risk of bias and applicability of published prediction models for risk of central line-associated bloodstream infection (CLA-BSI) in hospitalized patients. STUDY DESIGN AND SETTING Systematic review of literature in PubMed, Embase, Web of Science Core Collection, and Scopus up to July 10, 2023. Two authors independently appraised risk models using CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) and assessed their risk of bias and applicability using Prediction model Risk Of Bias ASsessment Tool (PROBAST). RESULTS Sixteen studies were included, describing 37 models. When studies presented multiple algorithms, we focused on the model that was selected as the best by the study authors. Eventually we appraised 19 models, among which 15 were regression models and four machine learning models. All models were at a high risk of bias, primarily due to inappropriate proxy outcomes, predictors that are unavailable at prediction time in clinical practice, inadequate sample size, negligence of missing data, lack of model validation, and absence of calibration assessment. 18 out of 19 models had a high concern for applicability, one model had unclear concern for applicability due to incomplete reporting. CONCLUSION We did not identify a prediction model of potential clinical use. There is a pressing need to develop an applicable model for CLA-BSI.
Collapse
Affiliation(s)
- Shan Gao
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
| | - Elena Albu
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
| | - Krizia Tuand
- 2Bergen - Learning Centre Désiré Collen, KU Leuven Libraries, KU Leuven, Leuven, Belgium
| | - Veerle Cossey
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium; Department of Infection Control and Prevention, University Hospitals Leuven, Leuven, Belgium
| | | | - Ben Van Calster
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium; Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands; EPI-Center, KU Leuven, Leuven, Belgium.
| | - Laure Wynants
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium; EPI-Center, KU Leuven, Leuven, Belgium; Care & Public Health Research Institute, Maastricht University, Maastricht, The Netherlands
| |
Collapse
|
48
|
Goubar A, Martin F, Sackley C, Foster N, Ayis S, Gregson C, Cameron I, Walsh N, Sheehan K. Development and Validation of Multivariable Prediction Models for In-Hospital Death, 30-Day Death, and Change in Residence After Hip Fracture Surgery and the "Stratify-Hip" Algorithm. J Gerontol A Biol Sci Med Sci 2023; 78:1659-1668. [PMID: 36754375 PMCID: PMC10460557 DOI: 10.1093/gerona/glad053] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2022] [Indexed: 02/10/2023] Open
Abstract
BACKGROUND To develop and validate the stratify-hip algorithm (multivariable prediction models to predict those at low, medium, and high risk across in-hospital death, 30-day death, and residence change after hip fracture). METHODS Multivariable Fine-Gray and logistic regression of audit data linked to hospital records for older adults surgically treated for hip fracture in England/Wales 2011-14 (development n = 170 411) and 2015-16 (external validation, n = 90 102). Outcomes included time to in-hospital death, death at 30 days, and time to residence change. Predictors included age, sex, pre-fracture mobility, dementia, and pre-fracture residence (not for residence change). Model assumptions, performance, and sensitivity to missingness were assessed. Models were incorporated into the stratify-hip algorithm assigning patients to overall low (low risk across outcomes), medium (low death risk, medium/high risk of residence change), or high (high risk of in-hospital death, high/medium risk of 30-day death) risk. RESULTS For complete-case analysis, 6 780 of 141 158 patients (4.8%) died in-hospital, 8 693 of 149 258 patients (5.8%) died by 30 days, and 4 461 of 119 420 patients (3.7%) had residence change. Models demonstrated acceptable calibration (observed:expected ratio 0.90, 0.99, and 0.94), and discrimination (area under curve 73.1, 71.1, and 71.5; Brier score 5.7, 5.3, and 5.6) for in-hospital death, 30-day death, and residence change, respectively. Overall, 31%, 28%, and 41% of patients were assigned to overall low, medium, and high risk. External validation and missing data analyses elicited similar findings. The algorithm is available at https://stratifyhip.co.uk. CONCLUSIONS The current study developed and validated the stratify-hip algorithm as a new tool to risk stratify patients after hip fracture.
Collapse
Affiliation(s)
- Aicha Goubar
- School of Life Course and Population Sciences, Faculty of Life Science and Medicine, King’s College London, London, UK
| | - Finbarr C Martin
- School of Life Course and Population Sciences, Faculty of Life Science and Medicine, King’s College London, London, UK
| | - Catherine Sackley
- School of Life Course and Population Sciences, Faculty of Life Science and Medicine, King’s College London, London, UK
| | - Nadine E Foster
- Surgical Treatment and Rehabilitation Service (STARS) Education and Research Alliance, The University of Queensland and Metro North Health, Brisbane, Queensland, Australia
- Primary Care Centre Versus Arthritis, School of Medicine, Keele University, Keele, UK
| | - Salma Ayis
- School of Life Course and Population Sciences, Faculty of Life Science and Medicine, King’s College London, London, UK
| | - Celia L Gregson
- Musculoskeletal Research Unit, Translation Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
| | - Ian D Cameron
- John Walsh Centre for Rehabilitation Research, Northern Sydney Local Health District and University of Sydney, Ryde, New South Wales, Australia
| | - Nicola E Walsh
- Centre for Health and Clinical Research, University of the West of England Bristol, Bristol, UK
| | - Katie J Sheehan
- School of Life Course and Population Sciences, Faculty of Life Science and Medicine, King’s College London, London, UK
| |
Collapse
|
49
|
Dhiman P, Ma J, Qi C, Bullock G, Sergeant JC, Riley RD, Collins GS. Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review. BMC Med Res Methodol 2023; 23:188. [PMID: 37598153 PMCID: PMC10439652 DOI: 10.1186/s12874-023-02008-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2023] [Accepted: 08/04/2023] [Indexed: 08/21/2023] Open
Abstract
BACKGROUND Having an appropriate sample size is important when developing a clinical prediction model. We aimed to review how sample size is considered in studies developing a prediction model for a binary outcome. METHODS We searched PubMed for studies published between 01/07/2020 and 30/07/2020 and reviewed the sample size calculations used to develop the prediction models. Using the available information, we calculated the minimum sample size that would be needed to estimate overall risk and minimise overfitting in each study and summarised the difference between the calculated and used sample size. RESULTS A total of 119 studies were included, of which nine studies provided sample size justification (8%). The recommended minimum sample size could be calculated for 94 studies: 73% (95% CI: 63-82%) used sample sizes lower than required to estimate overall risk and minimise overfitting including 26% studies that used sample sizes lower than required to estimate overall risk only. A similar number of studies did not meet the ≥ 10EPV criteria (75%, 95% CI: 66-84%). The median deficit of the number of events used to develop a model was 75 [IQR: 234 lower to 7 higher]) which reduced to 63 if the total available data (before any data splitting) was used [IQR:225 lower to 7 higher]. Studies that met the minimum required sample size had a median c-statistic of 0.84 (IQR:0.80 to 0.9) and studies where the minimum sample size was not met had a median c-statistic of 0.83 (IQR: 0.75 to 0.9). Studies that met the ≥ 10 EPP criteria had a median c-statistic of 0.80 (IQR: 0.73 to 0.84). CONCLUSIONS Prediction models are often developed with no sample size calculation, as a consequence many are too small to precisely estimate the overall risk. We encourage researchers to justify, perform and report sample size calculations when developing a prediction model.
Collapse
Affiliation(s)
- Paula Dhiman
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK.
| | - Jie Ma
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK
| | - Cathy Qi
- Population Data Science, Faculty of Medicine, Health and Life Science, Swansea University Medical School, Swansea University, Singleton Park, Swansea, SA2 8PP, UK
| | - Garrett Bullock
- Department of Orthopaedic Surgery, Wake Forest School of Medicine, Winston-Salem, NC, USA
- Centre for Sport, Exercise and Osteoarthritis Research Versus Arthritis, University of Oxford, Oxford, UK
| | - Jamie C Sergeant
- Centre for Biostatistics, University of Manchester, Manchester Academic Health Science Centre, Manchester, M13 9PL, UK
- Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, University of Manchester, Manchester Academic Health Science Centre, Manchester, M13 9PT, UK
| | - Richard D Riley
- Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, B15 2TT, Birmingham, UK
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK
| |
Collapse
|
50
|
Cohen ZD, Schueller SM. Expanding, improving, and understanding behaviour research and therapy through digital mental health. Behav Res Ther 2023; 167:104358. [PMID: 37418857 DOI: 10.1016/j.brat.2023.104358] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/09/2023]
Affiliation(s)
- Zachary D Cohen
- Department of Psychiatry and Biobehavioral Sciences, University of California, Los Angeles, USA.
| | - Stephen M Schueller
- Department of Psychological Science, University of California, Irvine, USA; Department of Informatics, University of California, Irvine, USA
| |
Collapse
|