1
|
Jahan TA, Lapin NA, O'Connell MT, Jo C, Ma Y, Tareen NG, Copley LA. Accelerated Severity of Illness Score Enhances Prediction of Complicated Acute Hematogenous Osteomyelitis in Children. Pediatr Infect Dis J 2024:00006454-990000000-01001. [PMID: 39259854 DOI: 10.1097/inf.0000000000004535] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 09/13/2024]
Abstract
BACKGROUND Severity of illness determination for children with acute hematogenous osteomyelitis should be accomplished during the earliest stages of evaluation to guide treatment and establish prognosis. This study objectively defines an outcome of complicated osteomyelitis and explores an illness severity-based model with an improved ability to predict this outcome as soon and accurately as possible, comparing it to existing models. METHODS Children with Staphylococcus aureus acute hematogenous osteomyelitis (n = 438) were retrospectively studied to identify adverse events and predictors of severity. The outcome of complicated osteomyelitis was ultimately defined as the occurrence of any major or at least 3 minor adverse events, which occurred in 52 children. Twenty-four clinical and laboratory predictors were evaluated through univariate and stacked multivariable regression analyses of chronologically distinct groups of variables. Receiver operating characteristic curve analyses were conducted to compare models. RESULTS Accelerated Severity of Illness Score included: triage tachycardia [odds ratio: 10.2 (95% confidence interval: 3.48-32.3], triage tachypnea [6.0 (2.4-15.2)], C-reactive proteininitial ≥17.2 mg/dL [4.5 (1.8-11.8)], white blood cell count band percentageinitial >3.8% [4.6 (2.0-11.0)], hemoglobininitial ≤10.4 g/dL [6.0 (2.6-14.7)], methicillin-resistant S. aureus [3.0 (1.2-8.5)], septic arthritis [4.5 (1.8-12.3)] and platelet nadir [7.2 (2.7-20.4)]. The receiver operating characteristic curve of Accelerated Severity of Illness Score [area under the curve = 0.96 (0.941-0.980)] were superior to those of Modified Severity of Illness Score = 0.903 (0.859-0.947), Acute Score for Complications of Osteomyelitis Risk Evaluation = 0.878 (0.830-0.926) and Chronic Score for Complications of Osteomyelitis Risk Evaluation = 0.858 (0.811-0.904). Successive receiver operating characteristic curve analyses established an exponentially increasing risk of complicated osteomyelitis for children with mild (0/285 or 0%), moderate (4/63 or 6.3%), severe (15/50 or 30.0%) and hyper-severe (33/40 or 82.5%) acute hematogenous osteomyelitis (P<0.0001). CONCLUSIONS This study improves upon previous severity of illness models by identifying early predictors of a rigorously defined outcome of complicated osteomyelitis.
Collapse
Affiliation(s)
- Tahmina A Jahan
- From the Department of Pediatric Infectious Disease, Seattle Children's Hospital, University of Washington, Seattle, Washington
| | - Norman A Lapin
- Department of Orthopaedic Surgery, Texas Scottish Rite Hospital for Children, Center for Pediatric Bone Biology and Translational Research; Dallas, Texas
| | - Michael T O'Connell
- Department of Orthopaedic Surgery, Texas Scottish Rite Hospital for Children, Center for Pediatric Bone Biology and Translational Research; Dallas, Texas
| | - Chanhee Jo
- Department of Clinical Orthopaedic Research, Scottish Rite for Children
| | - Yuhan Ma
- Department of Clinical Orthopaedic Research, Scottish Rite for Children
| | - Naureen G Tareen
- Department of Pediatric Orthopaedic Surgery, Children's Medical Center-Dallas, Dallas, Texas
| | - Lawson A Copley
- Department of Orthopaedic Surgery, Texas Scottish Rite Hospital for Children, Center for Pediatric Bone Biology and Translational Research; Dallas, Texas
- Departments of Orthopaedic Surgery and Pediatrics, University of Texas Southwestern Medical Center
- Department of Pediatric Orthopaedic Surgery, Children's Health System of Texas, Dallas, Texas
| |
Collapse
|
2
|
Choi BM, Kim E, Kim DH, Kim KM, Bang JY, Noh GJ, Lee EK. Dosing Strategy of Ramosetron to Prevent Postoperative Nausea and Vomiting and Development of Prediction Models Using Data Obtained From Randomized Controlled Trials: A Comparative Study. Clin Ther 2024; 46:604-611. [PMID: 38897838 DOI: 10.1016/j.clinthera.2024.05.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2024] [Revised: 04/23/2024] [Accepted: 05/06/2024] [Indexed: 06/21/2024]
Abstract
PURPOSE The study aimed to compare the postoperative nausea and vomiting (PONV) preventive effect of repeated administration of ramosetron with the standard treatment group and compare models to predict the incidence of PONV using machine-learning techniques. METHODS A total of 261 patients scheduled for breast surgery were analyzed to evaluate the effectiveness of repeated intravenous administration of ramosetron. All patients were administered 0.3 mg ramosetron just before the end of surgery. For the repeated dose of ramosetron group, an additional dose of 0.3 mg was administered at 4, 22, and 46 hours after the end of the surgery. Postoperative nausea, vomiting, and retching were evaluated using the Rhodes Index of Nausea, Vomiting, and Retching at 6, 24, and 48 hours postoperatively. Previously published randomized controlled data were combined with the data of this study to create a new dataset of 1390 patients, and machine-learning-based PONV prediction models (classification tree, random forest, extreme gradient boosting, and neural network) was constructed and compared with the Apfel model. FINDINGS Fifty patients (38.5%) and 60 patients (45.8%) reported nausea, vomiting, or retching 48 hours postoperatively in the standard and repeated-dose groups, respectively (P = 0.317, χ2 test). Median sensitivity, specificity, and accuracy of the Apfel model analyzed using the training set were 0.815, 0.344, and 0.495, respectively. IMPLICATIONS The repeated administration of ramosetron did not reduce the incidence of PONV. The Apfel model had high sensitivity, however, its specificity and accuracy were lower than that in machine-learning-based models.
Collapse
Affiliation(s)
- Byung-Moon Choi
- Department of Anesthesiology and Pain Medicine, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea.
| | - Eunha Kim
- Department of Anesthesiology and Pain Medicine, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea
| | - Dong Ho Kim
- Department of Anesthesiology and Pain Medicine, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea
| | - Kyung Mi Kim
- Department of Anesthesiology and Pain Medicine, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea
| | - Ji-Yeon Bang
- Department of Anesthesiology and Pain Medicine, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea
| | - Gyu-Jeong Noh
- Department of Anesthesiology and Pain Medicine, Asan Medical Center, University of Ulsan College of Medicine, Seoul, Korea; Department of Clinical Pharmacology and Therapeutics, Asan Medical Centre, University of Ulsan College of Medicine, Seoul, Korea
| | - Eun-Kyung Lee
- Department of Statistics, Ewha Womans University, Seoul, Korea.
| |
Collapse
|
3
|
Gehringer CK, Martin GP, Van Calster B, Hyrich KL, Verstappen SMM, Sergeant JC. How to develop, validate, and update clinical prediction models using multinomial logistic regression. J Clin Epidemiol 2024; 174:111481. [PMID: 39067542 DOI: 10.1016/j.jclinepi.2024.111481] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2023] [Revised: 03/14/2024] [Accepted: 07/19/2024] [Indexed: 07/30/2024]
Abstract
OBJECTIVES Multicategory prediction models (MPMs) can be used in health care when the primary outcome of interest has more than two categories. The application of MPMs is scarce, possibly due to added methodological complexities compared to binary outcome models. We provide a guide of how to develop, validate, and update clinical prediction models based on multinomial logistic regression. STUDY DESIGN AND SETTING We present guidance and recommendations based on recent methodological literature, illustrated by a previously developed and validated MPM for treatment outcomes in rheumatoid arthritis. Prediction models using multinomial logistic regression can be developed for nominal outcomes, but also for ordinal outcomes. This article is intended to supplement existing general guidance on prediction model research. RESULTS This guide is split into three parts: 1) outcome definition and variable selection, 2) model development, and 3) model evaluation (including performance assessment, internal and external validation, and model recalibration). We outline how to evaluate and interpret the predictive performance of MPMs. R code is provided. CONCLUSION We recommend the application of MPMs in clinical settings where the prediction of a multicategory outcome is of interest. Future methodological research could focus on MPM-specific considerations for variable selection and sample size criteria for external validation.
Collapse
Affiliation(s)
- Celina K Gehringer
- Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, Division of Musculoskeletal and Dermatological Sciences, University of Manchester, Manchester, UK; Centre for Biostatistics, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK.
| | - Glen P Martin
- Division of Informatics, Imaging and Data Sciences, Centre for Health Informatics, University of Manchester, Manchester, UK
| | - Ben Van Calster
- Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, The Netherlands; Department of Development & Regeneration, KU Leuven, Leuven, Belgium
| | - Kimme L Hyrich
- Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, Division of Musculoskeletal and Dermatological Sciences, University of Manchester, Manchester, UK; NIHR Manchester Biomedical Research Centre, Manchester University NHS Foundation Trust, Manchester Academic Health Science Centre, Manchester, UK
| | - Suzanne M M Verstappen
- Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, Division of Musculoskeletal and Dermatological Sciences, University of Manchester, Manchester, UK; NIHR Manchester Biomedical Research Centre, Manchester University NHS Foundation Trust, Manchester Academic Health Science Centre, Manchester, UK
| | - Jamie C Sergeant
- Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, Division of Musculoskeletal and Dermatological Sciences, University of Manchester, Manchester, UK; Centre for Biostatistics, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK
| |
Collapse
|
4
|
Pavlou M, Omar RZ, Ambler G. Penalized Regression Methods With Modified Cross-Validation and Bootstrap Tuning Produce Better Prediction Models. Biom J 2024; 66:e202300245. [PMID: 38922968 DOI: 10.1002/bimj.202300245] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2023] [Revised: 04/22/2024] [Accepted: 05/06/2024] [Indexed: 06/28/2024]
Abstract
Risk prediction models fitted using maximum likelihood estimation (MLE) are often overfitted resulting in predictions that are too extreme and a calibration slope (CS) less than 1. Penalized methods, such as Ridge and Lasso, have been suggested as a solution to this problem as they tend to shrink regression coefficients toward zero, resulting in predictions closer to the average. The amount of shrinkage is regulated by a tuning parameter,λ , $\lambda ,$ commonly selected via cross-validation ("standard tuning"). Though penalized methods have been found to improve calibration on average, they often over-shrink and exhibit large variability in the selected λ $\lambda $ and hence the CS. This is a problem, particularly for small sample sizes, but also when using sample sizes recommended to control overfitting. We consider whether these problems are partly due to selecting λ $\lambda $ using cross-validation with "training" datasets of reduced size compared to the original development sample, resulting in an over-estimation of λ $\lambda $ and, hence, excessive shrinkage. We propose a modified cross-validation tuning method ("modified tuning"), which estimates λ $\lambda $ from a pseudo-development dataset obtained via bootstrapping from the original dataset, albeit of larger size, such that the resulting cross-validation training datasets are of the same size as the original dataset. Modified tuning can be easily implemented in standard software and is closely related to bootstrap selection of the tuning parameter ("bootstrap tuning"). We evaluated modified and bootstrap tuning for Ridge and Lasso in simulated and real data using recommended sample sizes, and sizes slightly lower and higher. They substantially improved the selection of λ $\lambda $ , resulting in improved CS compared to the standard tuning method. They also improved predictions compared to MLE.
Collapse
|
5
|
Gul H, Di Matteo A, Anioke I, Shuweidhi F, Mankia K, Ponchel F, Emery P. Predicting Flare in Patients With Rheumatoid Arthritis in Biologic Induced Remission, on Tapering, and on Stable Therapy. ACR Open Rheumatol 2024; 6:294-303. [PMID: 38411023 PMCID: PMC11089437 DOI: 10.1002/acr2.11656] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2023] [Revised: 01/02/2024] [Accepted: 01/11/2024] [Indexed: 02/28/2024] Open
Abstract
OBJECTIVE The tapering of biologic disease-modifying antirheumatic drug (b-DMARD) therapy for patients with rheumatoid arthritis (RA) in stable remission is frequently undertaken, but specific guidance on how to successfully taper is lacking. The objective of this study is to identify predictors of flare in patients in stable b-DMARD-induced clinical remission, who did or did not follow structured b-DMARD tapering. METHODS Patients with RA receiving b-DMARD treatment who had achieved sustained remission according to a Disease Activity Score in 28 joints using the C-reactive protein level (DAS28-CRP) <2.6 for ≥6 months were offered tapering. Clinical, ultrasound (US) (total power Doppler [PD]/grayscale abnormalities), CD4+ T cell subsets, and patient-reported outcomes (PROs) were collected at inclusion. The primary endpoint was the occurrence of flare (loss of DAS28-CRP remission) over 12 months. Logistic regression analyses identified predictors of flare. Dichotomization into high/low-risk groups was based on 80% specificity using the area under the receiving operator curve (AUROC). RESULTS Of 63 patients choosing tapering, 23 (37%) flared compared with 12 of 60 (20%) on stable treatment (P = 0.043). All patients who flared regained remission upon reinstating treatment. In the tapering group, flare was associated with lower regulatory T cell (Treg) (P < 0.0001) and higher CRP levels (P < 0.0001), erythrocyte sedimentation rate (P < 0.035), and inflammation-related cells (IRCs) (P = 0.054); stepwise modeling selected Tregs (odds ratio [OR] = 0.350, P = 0.004), IRCs (OR = 1.871, P = 0.007), and CRP level (OR = 1.577, P = 0.004) with 81.7% accuracy and AUROC = 0.890. In the continued therapy group, modeling retained the tender joint count, total PD, and visual analog scale pain score, with 82.1% accuracy and AUROC = 0.899. Most patients in the study were considered low risk of flare (80 of 123 patients [65%]). Only 5 of 37 (13.5%) of the low-risk patients who tapered flared, which was notable compared with the continued therapy group (20% flare). CONCLUSION Flare on tapering b-DMARDs was predicted by lower Tregs and elevated inflammation biomarkers (IRCs/CRP level); flare on continued b-DMARDs was associated with raised pain parameters and US inflammation. Knowledge of these biomarkers should improve outcomes by targeted selection for tapering, and by increased monitoring of those on continued therapy predicted to flare.
Collapse
Affiliation(s)
| | - Andrea Di Matteo
- University of Leeds and National Institute for Health and Care Research Leeds Biomedical Research Centre, Leeds Teaching Hospitals NHS TrustLeedsUK
| | | | | | - Kulveer Mankia
- University of Leeds and National Institute for Health and Care Research Leeds Biomedical Research Centre, Leeds Teaching Hospitals NHS TrustLeedsUK
| | | | - Paul Emery
- University of Leeds and National Institute for Health and Care Research Leeds Biomedical Research Centre, Leeds Teaching Hospitals NHS TrustLeedsUK
| |
Collapse
|
6
|
Chu HO, Buchan E, Smith D, Goldberg Oppenheimer P. Development and application of an optimised Bayesian shrinkage prior for spectroscopic biomedical diagnostics. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 245:108014. [PMID: 38246097 DOI: 10.1016/j.cmpb.2024.108014] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/08/2023] [Revised: 01/06/2024] [Accepted: 01/08/2024] [Indexed: 01/23/2024]
Abstract
BACKGROUND AND OBJECTIVE Classification of vibrational spectra is often challenging for biological substances containing similar molecular bonds, interfering with spectral outputs. To address this, various approaches are widely studied. However, whilst providing powerful estimations, these techniques are computationally extensive and frequently overfit the data. Shrinkage priors, which favour models with relatively few predictor variables, are often applied in Bayesian penalisation techniques to avoid overfitting. METHODS Using the logit-normal continuous analogue of the spike-and-slab (LN-CASS) as the shrinkage prior and modelling, we have established classification for accurate analysis, with the established system found to be faster than conventional least absolute shrinkage and selection operator, horseshoe or spike-and-slab. These were examined versus coefficient data based on a linear regression model and vibrational spectra produced via density functional theory calculations. Then applied to Raman spectra from saliva to classify the sample sex. RESULTS Subsequently applied to the acquired spectra from saliva, the evaluated models exhibited high accuracy (AUC>90 %) even when number of parameters was higher than the number of observations. Analyses of spectra for all Bayesian models yielded high-classification accuracy upon cross-validation. Further, for saliva sensing, LN-CASS was found to be the only classifier with 100 %-accuracy in predicting the output based on a leave-one-out cross validation. CONCLUSIONS With potential applications in aiding diagnosis from small spectroscopic datasets and are compatible with a range of spectroscopic data formats. As seen with the classification of IR and Raman spectra. These results are highly promising for emerging developments of spectroscopic platforms for biomedical diagnostic sensing systems.
Collapse
Affiliation(s)
- Hin On Chu
- School of Chemical Engineering, University of Birmingham, Birmingham B15 2TT, UK
| | - Emma Buchan
- School of Chemical Engineering, University of Birmingham, Birmingham B15 2TT, UK
| | - David Smith
- School of Mathematics, Watson Building, University of Birmingham, Birmingham B15 2TT, UK
| | - Pola Goldberg Oppenheimer
- School of Chemical Engineering, University of Birmingham, Birmingham B15 2TT, UK; Healthcare Technologies Institute, Institute of Translational Medicine, Mindelsohn Way, Birmingham B15 2TH, UK.
| |
Collapse
|
7
|
Gehringer CK, Martin GP, Hyrich KL, Verstappen SMM, Sexton J, Kristianslund EK, Provan SA, Kvien TK, Sergeant JC. Developing and externally validating multinomial prediction models for methotrexate treatment outcomes in patients with rheumatoid arthritis: results from an international collaboration. J Clin Epidemiol 2024; 166:111239. [PMID: 38072179 DOI: 10.1016/j.jclinepi.2023.111239] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2023] [Revised: 11/23/2023] [Accepted: 12/05/2023] [Indexed: 01/01/2024]
Abstract
OBJECTIVES In rheumatology, there is a clinical need to identify patients at high risk (>50%) of not responding to the first-line therapy methotrexate (MTX) due to lack of disease control or discontinuation due to adverse events (AEs). Despite this need, previous prediction models in this context are at high risk of bias and ignore AEs. Our objectives were to (i) develop a multinomial model for outcomes of low disease activity and discontinuing due to AEs 6 months after starting MTX, (ii) update prognosis 3-month following treatment initiation, and (iii) externally validate these models. STUDY DESIGN AND SETTING A multinomial model for low disease activity (submodel 1) and discontinuing due to AEs (submodel 2) was developed using data from the UK Rheumatoid Arthritis Medication Study, updated using landmarking analysis, internally validated using bootstrapping, and externally validated in the Norwegian Disease-Modifying Antirheumatic Drug register. Performance was assessed using calibration (calibration-slope and calibration-in-the-large), and discrimination (concordance-statistic and polytomous discriminatory index). RESULTS The internally validated model showed good calibration in the development setting with a calibration-slope of 1.01 (0.87, 1.14) (submodel 1) and 0.83 (0.30, 1.34) (submodel 2), and moderate discrimination with a c-statistic of 0.72 (0.69, 0.74) and 0.53 (0.48, 0.59), respectively. Predictive performance decreased after external validation (calibration-slope 0.78 (0.64, 0.93) (submodel 1) and 0.86 (0.34, 1.38) (submodel 2)), which may be due to differences in disease-specific characteristics and outcome prevalence. CONCLUSION We addressed previously identified methodological limitations of prediction models for outcomes of MTX therapy. The multinomial approach predicted outcomes of disease activity more accurately than AEs, which should be addressed in future work to aid implementation into clinical practice.
Collapse
Affiliation(s)
- Celina K Gehringer
- Division of Musculoskeletal and Dermatological Sciences, Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, University of Manchester, Manchester, UK; Centre for Biostatistics, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK.
| | - Glen P Martin
- Division of Informatics, Imaging and Data Sciences, Centre for Health Informatics, University of Manchester, Manchester, UK
| | - Kimme L Hyrich
- Division of Musculoskeletal and Dermatological Sciences, Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, University of Manchester, Manchester, UK; NIHR Manchester Biomedical Research Centre, Manchester University NHS Foundation Trust, Manchester Academic Health Science Centre, Manchester, UK
| | - Suzanne M M Verstappen
- Division of Musculoskeletal and Dermatological Sciences, Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, University of Manchester, Manchester, UK; NIHR Manchester Biomedical Research Centre, Manchester University NHS Foundation Trust, Manchester Academic Health Science Centre, Manchester, UK
| | - Joseph Sexton
- Center for Treatment of Rheumatic and Musculoskeletal Diseases (REMEDY), Diakonhjemmet Hospital, Oslo, Norway
| | - Eirik K Kristianslund
- Center for Treatment of Rheumatic and Musculoskeletal Diseases (REMEDY), Diakonhjemmet Hospital, Oslo, Norway
| | - Sella A Provan
- Center for Treatment of Rheumatic and Musculoskeletal Diseases (REMEDY), Diakonhjemmet Hospital, Oslo, Norway
| | - Tore K Kvien
- Center for Treatment of Rheumatic and Musculoskeletal Diseases (REMEDY), Diakonhjemmet Hospital, Oslo, Norway; Faculty of Medicine, University of Oslo, Oslo, Norway
| | - Jamie C Sergeant
- Division of Musculoskeletal and Dermatological Sciences, Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, University of Manchester, Manchester, UK; Centre for Biostatistics, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK
| |
Collapse
|
8
|
Collins GS, Dhiman P, Ma J, Schlussel MM, Archer L, Van Calster B, Harrell FE, Martin GP, Moons KGM, van Smeden M, Sperrin M, Bullock GS, Riley RD. Evaluation of clinical prediction models (part 1): from development to external validation. BMJ 2024; 384:e074819. [PMID: 38191193 PMCID: PMC10772854 DOI: 10.1136/bmj-2023-074819] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 09/04/2023] [Indexed: 01/10/2024]
Affiliation(s)
- Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
| | - Paula Dhiman
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
| | - Jie Ma
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
| | - Michael M Schlussel
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
| | - Lucinda Archer
- Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, UK
| | - Ben Van Calster
- KU Leuven, Department of Development and Regeneration, Leuven, Belgium
- Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, Netherlands
- EPI-Centre, KU Leuven, Belgium
| | - Frank E Harrell
- Department of Biostatistics, Vanderbilt University, Nashville, TN, USA
| | - Glen P Martin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| | - Karel G M Moons
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Maarten van Smeden
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
| | - Matthew Sperrin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| | - Garrett S Bullock
- Department of Orthopaedic Surgery, Wake Forest School of Medicine, Winston-Salem, NC, USA
- Centre for Sport, Exercise and Osteoarthritis Research Versus Arthritis, University of Oxford, Oxford, UK
| | - Richard D Riley
- Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, UK
| |
Collapse
|
9
|
Seers T, Reynard C, Martin GP, Body R. Development and Internal Validation of a Multivariable Prediction Model to Predict Repeat Attendances in the Pediatric Emergency Department: A Retrospective Cohort Study. Pediatr Emerg Care 2024; 40:16-21. [PMID: 37195679 DOI: 10.1097/pec.0000000000002975] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 05/18/2023]
Abstract
OBJECTIVE Unplanned reattendances to the pediatric emergency department (PED) occur commonly in clinical practice. Multiple factors influence the decision to return to care, and understanding risk factors may allow for better design of clinical services. We developed a clinical prediction model to predict return to the PED within 72 hours from the index visit. METHODS We retrospectively reviewed all attendances to the PED of Royal Manchester Children's Hospital between 2009 and 2019. Attendances were excluded if they were admitted to hospital, aged older than 16 years or died in the PED. Variables were collected from Electronic Health Records reflecting triage codes. Data were split temporally into a training (80%) set for model development and a test (20%) set for internal validation. We developed the prediction model using LASSO penalized logistic regression. RESULTS A total of 308,573 attendances were included in the study. There were 14,276 (4.63%) returns within 72 hours of index visit. The final model had an area under the receiver operating characteristic curve of 0.64 (95% confidence interval, 0.63-0.65) on temporal validation. The calibration of the model was good, although with some evidence of miscalibration at the high extremes of the risk distribution. After-visit diagnoses codes reflecting a nonspecific problem ("unwell child") were more common in children who went on to reattend. CONCLUSIONS We developed and internally validated a clinical prediction model for unplanned reattendance to the PED using routinely collected clinical data, including markers of socioeconomic deprivation. This model allows for easy identification of children at the greatest risk of return to PED.
Collapse
Affiliation(s)
- Tim Seers
- From the Emergency Department, Manchester Royal Infirmary, Manchester University NHS Foundation Trust, Manchester, United Kingdom
| | | | | | | |
Collapse
|
10
|
Riley RD, Pate A, Dhiman P, Archer L, Martin GP, Collins GS. Clinical prediction models and the multiverse of madness. BMC Med 2023; 21:502. [PMID: 38110939 PMCID: PMC10729337 DOI: 10.1186/s12916-023-03212-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Accepted: 12/05/2023] [Indexed: 12/20/2023] Open
Abstract
BACKGROUND Each year, thousands of clinical prediction models are developed to make predictions (e.g. estimated risk) to inform individual diagnosis and prognosis in healthcare. However, most are not reliable for use in clinical practice. MAIN BODY We discuss how the creation of a prediction model (e.g. using regression or machine learning methods) is dependent on the sample and size of data used to develop it-were a different sample of the same size used from the same overarching population, the developed model could be very different even when the same model development methods are used. In other words, for each model created, there exists a multiverse of other potential models for that sample size and, crucially, an individual's predicted value (e.g. estimated risk) may vary greatly across this multiverse. The more an individual's prediction varies across the multiverse, the greater the instability. We show how small development datasets lead to more different models in the multiverse, often with vastly unstable individual predictions, and explain how this can be exposed by using bootstrapping and presenting instability plots. We recommend healthcare researchers seek to use large model development datasets to reduce instability concerns. This is especially important to ensure reliability across subgroups and improve model fairness in practice. CONCLUSIONS Instability is concerning as an individual's predicted value is used to guide their counselling, resource prioritisation, and clinical decision making. If different samples lead to different models with very different predictions for the same individual, then this should cast doubt into using a particular model for that individual. Therefore, visualising, quantifying and reporting the instability in individual-level predictions is essential when proposing a new model.
Collapse
Affiliation(s)
- Richard D Riley
- College of Medical and Dental Sciences, Institute of Applied Health Research, University of Birmingham, Birmingham, B15 2TT, UK.
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK.
| | - Alexander Pate
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK
| | - Paula Dhiman
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK
| | - Lucinda Archer
- College of Medical and Dental Sciences, Institute of Applied Health Research, University of Birmingham, Birmingham, B15 2TT, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK
| | - Glen P Martin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, OX3 7LD, UK
| |
Collapse
|
11
|
Riley RD, Collins GS. Stability of clinical prediction models developed using statistical or machine learning methods. Biom J 2023; 65:e2200302. [PMID: 37466257 PMCID: PMC10952221 DOI: 10.1002/bimj.202200302] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2022] [Revised: 04/26/2023] [Accepted: 05/02/2023] [Indexed: 07/20/2023]
Abstract
Clinical prediction models estimate an individual's risk of a particular health outcome. A developed model is a consequence of the development dataset and model-building strategy, including the sample size, number of predictors, and analysis method (e.g., regression or machine learning). We raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model-building steps (those used to develop the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and deriving (i) a prediction instability plot of bootstrap model versus original model predictions; (ii) the mean absolute prediction error (mean absolute difference between individuals' original and bootstrap model predictions), and (iii) calibration, classification, and decision curve instability plots of bootstrap models applied in the original sample. A case study illustrates how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), while informing a model's critical appraisal (risk of bias rating), fairness, and further validation requirements.
Collapse
Affiliation(s)
- Richard D. Riley
- Institute of Applied Health ResearchCollege of Medical and Dental SciencesUniversity of BirminghamBirminghamUK
| | - Gary S. Collins
- Centre for Statistics in MedicineNuffield Department of OrthopaedicsRheumatology and Musculoskeletal SciencesUniversity of OxfordOxfordUK
| |
Collapse
|
12
|
Ruijter BEW, Bik CA, Schofield I, Niessen SJM. External validation of a United Kingdom primary-care Cushing's prediction tool in a population of referred Dutch dogs. J Vet Intern Med 2023; 37:2052-2063. [PMID: 37665189 PMCID: PMC10658492 DOI: 10.1111/jvim.16848] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2023] [Accepted: 08/23/2023] [Indexed: 09/05/2023] Open
Abstract
BACKGROUND A prediction tool was developed and internally validated to aid the diagnosis of Cushing's syndrome in dogs attending UK primary-care practices. External validation is an important part of model validation to assess model performance when used in different populations. OBJECTIVES To assess the original prediction model's transportability, applicability, and diagnostic performance in a secondary-care practice in the Netherlands. ANIMALS Two hundred thirty client-owned dogs. METHODS Retrospective observational study. Medical records of dogs under investigation of Cushing's syndrome between 2011 and 2020 were reviewed. Dogs diagnosed with Cushing's syndrome by the attending internists and fulfilling ALIVE criteria were defined as cases, others as non-cases. All dogs were scored using the aforementioned prediction tool. Dog characteristics and predictor-outcome effects in development and validation data sets were compared to assess model transportability. Calibration and discrimination were examined to assess model performance. RESULTS Eighty of 230 dogs were defined as cases. Significant differences in dog characteristics were found between UK primary-care and Dutch secondary-care populations. Not all predictors from the original model were confirmed to be significant predictors in the validation sample. The model systematically overestimated the probability of having Cushing's syndrome (a = -1.10, P < .001). Calibration slope was 1.35 and discrimination proved excellent (area under the receiver operating curve = 0.83). CONCLUSIONS AND CLINICAL IMPORTANCE The prediction model had moderate transportability, excellent discriminatory ability, and overall overestimated probability of having Cushing's syndrome. This study confirms its utility, though emphasizes that ongoing validation efforts of disease prediction tools are a worthwhile effort.
Collapse
Affiliation(s)
| | - Céline Anne Bik
- MCD‐AniCura – Internal Medicine, Isolatorweg 45Amsterdam 1014ASThe Netherlands
| | - Imogen Schofield
- Royal Veterinary College, Hawkshead LaneHatfield AL9 7TAUnited Kingdom
| | - Stijn Johannes Maria Niessen
- Royal Veterinary College – Veterinary Clinical Sciences, North MimmsHertsUnited Kingdom
- Veterinary Specialist ConsultationsHilversumThe Netherlands
| |
Collapse
|
13
|
Reyes-Santias F, García-García C, Aibar-Guzmán B, García-Campos A, Cordova-Arevalo O, Mendoza-Pintos M, Cinza-Sanjurjo S, Portela-Romero M, Mazón-Ramos P, Gonzalez-Juanatey JR. Cost Analysis of Magnetic Resonance Imaging and Computed Tomography in Cardiology: A Case Study of a University Hospital Complex in the Euro Region. Healthcare (Basel) 2023; 11:2084. [PMID: 37510526 PMCID: PMC10379578 DOI: 10.3390/healthcare11142084] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2023] [Revised: 07/12/2023] [Accepted: 07/17/2023] [Indexed: 07/30/2023] Open
Abstract
INTRODUCTION In recent years, several hospitals have incorporated MRI equipment managed directly by their cardiology departments. The aim of our work is to determine the total cost per test of both CT and MRI in the setting of a Cardiology Department of a tertiary hospital. MATERIALS AND METHODS The process followed for estimating the costs of CT and MRI tests consists of three phases: (1) Identification of the phases of the testing process; (2) Identification of the resources consumed in carrying out the tests; (3) Quantification and assessment of inputs. RESULTS MRI involves higher personnel (EUR 66.03 vs. EUR 49.17) and equipment (EUR 89.98 vs. EUR 33.73) costs, while CT consumes higher expenditures in consumables (EUR 93.28 vs. EUR 22.95) and overheads (EUR 1.64 vs. EUR 1.55). The total cost of performing each test is higher in MRI (EUR 180.60 vs. EUR 177.73). CONCLUSIONS We can conclude that the unit cost of each CT and MRI performed in that unit are EUR 177.73 and EUR 180.60, respectively, attributable to consumables in the case of CT and to amortization of equipment and staff time in the case of MRI.
Collapse
Affiliation(s)
- Francisco Reyes-Santias
- Servicio de Cardiología, Complejo Hospitalario Universitario de Santiago de Compostela, Choupana s/n, 15706 Santiago de Compostela, Spain
- Instituto de Investigación Sanitaria de Santiago de Compostela (IDIS), Choupana s/n, 15706 Santiago de Compostela, Spain
- Centro de Investigación Biomédica en Red-Enfermedades Cardiovasculares (CIBERCV), Av. Monforte de Lemos, 3-5. Pabellón 11. Planta 0, 28029 Madrid, Spain
- Department of Business, University of Vigo, 36310 Vigo, Spain
| | - Carlos García-García
- Department of Pharmacology, Pharmacy and Pharmaceutical Technology, R+D Pharma Group (GI-1645), Faculty of Pharmacy, Health Research Institute of Santiago de Compostela (IDIS), University of Santiago de Compostela, 15782 Santiago de Compostela, Spain
| | - Beatriz Aibar-Guzmán
- Departamento de Economía Financiera y Contabilidad, Facultad de Ciencias Económicas y Empresariales, Universidad de Santiago de Compostela, Av. Burgo, s/n, 15782 Santiago Compostela, Spain
| | - Ana García-Campos
- Servicio de Cardiología, Complejo Hospitalario Universitario de Santiago de Compostela, Choupana s/n, 15706 Santiago de Compostela, Spain
- Instituto de Investigación Sanitaria de Santiago de Compostela (IDIS), Choupana s/n, 15706 Santiago de Compostela, Spain
- Centro de Investigación Biomédica en Red-Enfermedades Cardiovasculares (CIBERCV), Av. Monforte de Lemos, 3-5. Pabellón 11. Planta 0, 28029 Madrid, Spain
| | | | | | - Sergio Cinza-Sanjurjo
- Instituto de Investigación Sanitaria de Santiago de Compostela (IDIS), Choupana s/n, 15706 Santiago de Compostela, Spain
- Centro de Investigación Biomédica en Red-Enfermedades Cardiovasculares (CIBERCV), Av. Monforte de Lemos, 3-5. Pabellón 11. Planta 0, 28029 Madrid, Spain
- CS Milladoiro, Área Sanitaria Integrada Santiago de Compostela, 15895 Travesía do Porto, Spain
| | - Manuel Portela-Romero
- Instituto de Investigación Sanitaria de Santiago de Compostela (IDIS), Choupana s/n, 15706 Santiago de Compostela, Spain
- Centro de Investigación Biomédica en Red-Enfermedades Cardiovasculares (CIBERCV), Av. Monforte de Lemos, 3-5. Pabellón 11. Planta 0, 28029 Madrid, Spain
- CS Concepción Arenal, Área Sanitaria Integrada Santiago de Compostela, Rúa de Santiago León de Caracas, 12, 15701 Santiago de Compostela, Spain
| | - Pilar Mazón-Ramos
- Servicio de Cardiología, Complejo Hospitalario Universitario de Santiago de Compostela, Choupana s/n, 15706 Santiago de Compostela, Spain
- Instituto de Investigación Sanitaria de Santiago de Compostela (IDIS), Choupana s/n, 15706 Santiago de Compostela, Spain
- Centro de Investigación Biomédica en Red-Enfermedades Cardiovasculares (CIBERCV), Av. Monforte de Lemos, 3-5. Pabellón 11. Planta 0, 28029 Madrid, Spain
| | - Jose Ramon Gonzalez-Juanatey
- Servicio de Cardiología, Complejo Hospitalario Universitario de Santiago de Compostela, Choupana s/n, 15706 Santiago de Compostela, Spain
- Instituto de Investigación Sanitaria de Santiago de Compostela (IDIS), Choupana s/n, 15706 Santiago de Compostela, Spain
- Centro de Investigación Biomédica en Red-Enfermedades Cardiovasculares (CIBERCV), Av. Monforte de Lemos, 3-5. Pabellón 11. Planta 0, 28029 Madrid, Spain
| |
Collapse
|
14
|
Sperrin M, Riley RD, Collins GS, Martin GP. Targeted validation: validating clinical prediction models in their intended population and setting. Diagn Progn Res 2022; 6:24. [PMID: 36550534 PMCID: PMC9773429 DOI: 10.1186/s41512-022-00136-8] [Citation(s) in RCA: 14] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/24/2022] [Accepted: 11/14/2022] [Indexed: 12/24/2022] Open
Abstract
Clinical prediction models must be appropriately validated before they can be used. While validation studies are sometimes carefully designed to match an intended population/setting of the model, it is common for validation studies to take place with arbitrary datasets, chosen for convenience rather than relevance. We call estimating how well a model performs within the intended population/setting "targeted validation". Use of this term sharpens the focus on the intended use of a model, which may increase the applicability of developed models, avoid misleading conclusions, and reduce research waste. It also exposes that external validation may not be required when the intended population for the model matches the population used to develop the model; here, a robust internal validation may be sufficient, especially if the development dataset was large.
Collapse
Affiliation(s)
- Matthew Sperrin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK.
| | - Richard D Riley
- Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - Glen P Martin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK
| |
Collapse
|