1
|
Arakaki D, Iwata M, Terasawa T. External validation and update of the International Medical Prevention Registry on Venous Thromboembolism bleeding risk score for predicting bleeding in acutely ill hospitalized medical patients: a retrospective single-center cohort study in Japan. Thromb J 2024; 22:31. [PMID: 38549086 PMCID: PMC10976666 DOI: 10.1186/s12959-024-00603-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/18/2023] [Accepted: 03/21/2024] [Indexed: 04/10/2024] Open
Abstract
BACKGROUND The International Medical Prevention Registry for Venous Thromboembolism (IMPROVE) Bleeding Risk Score is the recommended risk assessment model (RAM) for predicting bleeding risk in acutely ill medical inpatients in Western countries. However, few studies have assessed its predictive performance in local Asian settings. METHODS We retrospectively identified acutely ill adolescents and adults (aged ≥ 15 years) who were admitted to our general internal medicine department between July 5, 2016 and July 5, 2021, and extracted data from their electronic medical records. The outcome of interest was the cumulative incidence of major and nonmajor but clinically relevant bleeding 14 days after admission. For the two-risk-group model, we estimated sensitivity, specificity, and positive and negative predictive values (PPV and NPV, respectively). For the 11-risk-group model, we estimated C statistic, expected and observed event ratio (E/O), calibration-in-the-large (CITL), and calibration slope. In addition, we recalibrated the intercept using local data to update the RAM. RESULTS Among the 3,876 included patients, 998 (26%) were aged ≥ 85 years, while 656 (17%) were hospitalized in the intensive care unit. The median length of hospital stay was 14 days. Clinically relevant bleeding occurred in 58 patients (1.5%), 49 (1.3%) of whom experienced major bleeding. Sensitivity, specificity, NPV, and PPV were 26.1% (95% confidence interval [CI]: 15.8-40.0%), 84.8% (83.6-85.9%), 98.7% (98.2-99.0%), and 2.5% (1.5-4.3%) for any bleeding and 30.9% (95% CI: 18.8-46.3%), 84.9% (83.7-86.0%), 99.0% (98.5-99.3%), and 2.5% (1.5-4.3%) for major bleeding, respectively. The C statistic, E/O, CITL, and calibration slope were 0.64 (95% CI: 0.58-0.71), 1.69 (1.45-2.05), - 0.55 (- 0.81 to - 0.29), and 0.58 (0.29-0.87) for any bleeding and 0.67 (95% CI: 0.60-0.74), 0.76 (0.61-0.87), 0.29 (0.00-0.58), and 0.42 (0.19-0.64) for major bleeding, respectively. Updating the model substantially corrected the poor calibration observed. CONCLUSIONS In our Japanese cohort, the IMPROVE bleeding RAM retained the reported moderate discriminative performance. Model recalibration substantially improved the poor calibration obtained using the original RAM. Before its introduction into clinical practice, the updated RAM needs further validation studies and an optimized threshold.
Collapse
Affiliation(s)
- Daichi Arakaki
- Department of Emergency Medicine and General Internal Medicine, Fujita Health University School of Medicine, 1-98 Dengakugakubo, Kutsukakecho, Toyoake, Achi, 470-1192, Toyoake, Aichi, Japan
- Department of Emergency and Critical Care, Nagoya University Hospital, Nagoya, Aichi, Japan
| | - Mitsunaga Iwata
- Department of Emergency Medicine and General Internal Medicine, Fujita Health University School of Medicine, 1-98 Dengakugakubo, Kutsukakecho, Toyoake, Achi, 470-1192, Toyoake, Aichi, Japan
| | - Teruhiko Terasawa
- Department of Emergency Medicine and General Internal Medicine, Fujita Health University School of Medicine, 1-98 Dengakugakubo, Kutsukakecho, Toyoake, Achi, 470-1192, Toyoake, Aichi, Japan.
| |
Collapse
|
2
|
Dhana A, Gupta RK, Hamada Y, Kengne AP, Kerkhoff AD, Yoon C, Cattamanchi A, Reeve BWP, Theron G, Ndlangalavu G, Wood R, Drain PK, Calderwood CJ, Noursadeghi M, Boyles T, Meintjes G, Maartens G, Barr DA. Clinical utility of WHO-recommended screening tools and development and validation of novel clinical prediction models for pulmonary tuberculosis screening among outpatients living with HIV: an individual participant data meta-analysis. Eur Respir Rev 2023; 32:230021. [PMID: 37286216 PMCID: PMC10245131 DOI: 10.1183/16000617.0021-2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2023] [Accepted: 03/15/2023] [Indexed: 06/09/2023] Open
Abstract
BACKGROUND The World Health Organization (WHO) recommends that outpatient people living with HIV (PLHIV) undergo tuberculosis screening with the WHO four-symptom screen (W4SS) or C-reactive protein (CRP) (5 mg·L-1 cut-off) followed by confirmatory testing if screen positive. We conducted an individual participant data meta-analysis to determine the performance of WHO-recommended screening tools and two newly developed clinical prediction models (CPMs). METHODS Following a systematic review, we identified studies that recruited adult outpatient PLHIV irrespective of tuberculosis signs and symptoms or with a positive W4SS, evaluated CRP and collected sputum for culture. We used logistic regression to develop an extended CPM (which included CRP and other predictors) and a CRP-only CPM. We used internal-external cross-validation to evaluate performance. RESULTS We pooled data from eight cohorts (n=4315 participants). The extended CPM had excellent discrimination (C-statistic 0.81); the CRP-only CPM had similar discrimination. The C-statistics for WHO-recommended tools were lower. Both CPMs had equivalent or higher net benefit compared with the WHO-recommended tools. Compared with both CPMs, CRP (5 mg·L-1 cut-off) had equivalent net benefit across a clinically useful range of threshold probabilities, while the W4SS had a lower net benefit. The W4SS would capture 91% of tuberculosis cases and require confirmatory testing for 78% of participants. CRP (5 mg·L-1 cut-off), the extended CPM (4.2% threshold) and the CRP-only CPM (3.6% threshold) would capture similar percentages of cases but reduce confirmatory tests required by 24, 27 and 36%, respectively. CONCLUSIONS CRP sets the standard for tuberculosis screening among outpatient PLHIV. The choice between using CRP at 5 mg·L-1 cut-off or in a CPM depends on available resources.
Collapse
Affiliation(s)
- Ashar Dhana
- Department of Medicine, University of Cape Town, Cape Town, South Africa
| | - Rishi K Gupta
- Institute for Global Health, University College London, London, UK
| | - Yohhei Hamada
- Institute for Global Health, University College London, London, UK
- Centre for International Cooperation and Global TB Information, The Research Institute of Tuberculosis, Japan Anti-Tuberculosis Association, Tokyo, Japan
| | - Andre P Kengne
- Non-communicable Diseases Research Unit, South African Medical Research Council, Cape Town, South Africa
| | - Andrew D Kerkhoff
- Division of HIV, Infectious Diseases and Global Medicine, Zuckerberg San Francisco General Hospital and Trauma Center, University of California San Francisco, San Francisco, CA, USA
| | - Christina Yoon
- Department of Medicine, Division of Pulmonary and Critical Care Medicine and Center for Tuberculosis, University of California San Francisco, San Francisco, CA, USA
| | - Adithya Cattamanchi
- Department of Medicine, Division of Pulmonary and Critical Care Medicine and Center for Tuberculosis, University of California San Francisco, San Francisco, CA, USA
| | - Byron W P Reeve
- DSI-NRF Centre of Excellence for Biomedical Tuberculosis Research, South African Medical Research Council Centre for Tuberculosis Research, Division of Molecular Biology and Human Genetics, Faculty of Medicine and Health Sciences, Stellenbosch University, Cape Town, South Africa
| | - Grant Theron
- DSI-NRF Centre of Excellence for Biomedical Tuberculosis Research, South African Medical Research Council Centre for Tuberculosis Research, Division of Molecular Biology and Human Genetics, Faculty of Medicine and Health Sciences, Stellenbosch University, Cape Town, South Africa
| | - Gcobisa Ndlangalavu
- DSI-NRF Centre of Excellence for Biomedical Tuberculosis Research, South African Medical Research Council Centre for Tuberculosis Research, Division of Molecular Biology and Human Genetics, Faculty of Medicine and Health Sciences, Stellenbosch University, Cape Town, South Africa
| | - Robin Wood
- Institute of Infectious Disease and Molecular Medicine (IDM), Faculty of Health Sciences, University of Cape Town, Cape Town, South Africa
| | - Paul K Drain
- Departments of Global Health, Medicine, and Epidemiology, University of Washington, Seattle, WA, USA
| | - Claire J Calderwood
- Institute for Global Health, University College London, London, UK
- Clinical Research Department, Faculty of Infectious and Tropical Diseases, London School of Hygiene and Tropical Medicine, London, UK
- The Research Unit Zimbabwe, Biomedical Research and Training Institute, Harare, Zimbabwe
| | - Mahdad Noursadeghi
- Division of Infection and Immunity, University College London, London, UK
| | - Tom Boyles
- Helen Joseph Hospital, Johannesburg, South Africa
| | - Graeme Meintjes
- Department of Medicine, University of Cape Town, Cape Town, South Africa
- Wellcome Centre for Infectious Diseases Research in Africa (CIDRI-Africa), Institute of Infectious Disease and Molecular Medicine, University of Cape Town, Cape Town, South Africa
| | - Gary Maartens
- Department of Medicine, University of Cape Town, Cape Town, South Africa
- Wellcome Centre for Infectious Diseases Research in Africa (CIDRI-Africa), Institute of Infectious Disease and Molecular Medicine, University of Cape Town, Cape Town, South Africa
| | - David A Barr
- Institute of Infection and Global Health, University of Liverpool, Liverpool, UK
| |
Collapse
|
3
|
Mohanannair Geethadevi G, Quinn TJ, George J, Anstey KJ, Bell JS, Sarwar MR, Cross AJ. Multi-domain prognostic models used in middle-aged adults without known cognitive impairment for predicting subsequent dementia. Cochrane Database Syst Rev 2023; 6:CD014885. [PMID: 37265424 PMCID: PMC10239281 DOI: 10.1002/14651858.cd014885.pub2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 06/03/2023]
Abstract
BACKGROUND Dementia, a global health priority, has no current cure. Around 50 million people worldwide currently live with dementia, and this number is expected to treble by 2050. Some health conditions and lifestyle behaviours can increase or decrease the risk of dementia and are known as 'predictors'. Prognostic models combine such predictors to measure the risk of future dementia. Models that can accurately predict future dementia would help clinicians select high-risk adults in middle age and implement targeted risk reduction. OBJECTIVES Our primary objective was to identify multi-domain prognostic models used in middle-aged adults (aged 45 to 65 years) for predicting dementia or cognitive impairment. Eligible multi-domain prognostic models involved two or more of the modifiable dementia predictors identified in a 2020 Lancet Commission report and a 2019 World Health Organization (WHO) report (less education, hearing loss, traumatic brain injury, hypertension, excessive alcohol intake, obesity, smoking, depression, social isolation, physical inactivity, diabetes mellitus, air pollution, poor diet, and cognitive inactivity). Our secondary objectives were to summarise the prognostic models, to appraise their predictive accuracy (discrimination and calibration) as reported in the development and validation studies, and to identify the implications of using dementia prognostic models for the management of people at a higher risk for future dementia. SEARCH METHODS We searched MEDLINE, Embase, PsycINFO, CINAHL, and ISI Web of Science Core Collection from inception until 6 June 2022. We performed forwards and backwards citation tracking of included studies using the Web of Science platform. SELECTION CRITERIA: We included development and validation studies of multi-domain prognostic models. The minimum eligible follow-up was five years. Our primary outcome was an incident clinical diagnosis of dementia based on validated diagnostic criteria, and our secondary outcome was dementia or cognitive impairment determined by any other method. DATA COLLECTION AND ANALYSIS Two review authors independently screened the references, extracted data using a template based on the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS), and assessed risk of bias and applicability of included studies using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). We synthesised the C-statistics of models that had been externally validated in at least three comparable studies. MAIN RESULTS: We identified 20 eligible studies; eight were development studies and 12 were validation studies. There were 14 unique prognostic models: seven models with validation studies and seven models with development-only studies. The models included a median of nine predictors (range 6 to 34); the median number of modifiable predictors was five (range 2 to 11). The most common modifiable predictors in externally validated models were diabetes, hypertension, smoking, physical activity, and obesity. In development-only models, the most common modifiable predictors were obesity, diabetes, hypertension, and smoking. No models included hearing loss or air pollution as predictors. Nineteen studies had a high risk of bias according to the PROBAST assessment, mainly because of inappropriate analysis methods, particularly lack of reported calibration measures. Applicability concerns were low for 12 studies, as their population, predictors, and outcomes were consistent with those of interest for this review. Applicability concerns were high for nine studies, as they lacked baseline cognitive screening or excluded an age group within the range of 45 to 65 years. Only one model, Cardiovascular Risk Factors, Ageing, and Dementia (CAIDE), had been externally validated in multiple studies, allowing for meta-analysis. The CAIDE model included eight predictors (four modifiable predictors): age, education, sex, systolic blood pressure, body mass index (BMI), total cholesterol, physical activity and APOEƐ4 status. Overall, our confidence in the prediction accuracy of CAIDE was very low; our main reasons for downgrading the certainty of the evidence were high risk of bias across all the studies, high concern of applicability, non-overlapping confidence intervals (CIs), and a high degree of heterogeneity. The summary C-statistic was 0.71 (95% CI 0.66 to 0.76; 3 studies; very low-certainty evidence) for the incident clinical diagnosis of dementia, and 0.67 (95% CI 0.61 to 0.73; 3 studies; very low-certainty evidence) for dementia or cognitive impairment based on cognitive scores. Meta-analysis of calibration measures was not possible, as few studies provided these data. AUTHORS' CONCLUSIONS We identified 14 unique multi-domain prognostic models used in middle-aged adults for predicting subsequent dementia. Diabetes, hypertension, obesity, and smoking were the most common modifiable risk factors used as predictors in the models. We performed meta-analyses of C-statistics for one model (CAIDE), but the summary values were unreliable. Owing to lack of data, we were unable to meta-analyse the calibration measures of CAIDE. This review highlights the need for further robust external validations of multi-domain prognostic models for predicting future risk of dementia in middle-aged adults.
Collapse
Affiliation(s)
| | - Terry J Quinn
- Institute of Cardiovascular and Medical Sciences, University of Glasgow, Glasgow, UK
| | - Johnson George
- Centre for Medicine Use and Safety, Faculty of Pharmacy and Pharmaceutical Sciences, Monash University, Melbourne, Australia
- Faculty of Medicine, Nursing and Health Sciences, School of Public Health and Preventive Medicine, Melbourne, Australia
| | - Kaarin J Anstey
- School of Psychology, The University of New South Wales, Sydney, Australia
- Ageing Futures Institute, The University of New South Wales, Sydney, Australia
| | - J Simon Bell
- Centre for Medicine Use and Safety, Faculty of Pharmacy and Pharmaceutical Sciences, Monash University, Melbourne, Australia
| | - Muhammad Rehan Sarwar
- Centre for Medicine Use and Safety, Faculty of Pharmacy and Pharmaceutical Sciences, Monash University, Melbourne, Australia
| | - Amanda J Cross
- Centre for Medicine Use and Safety, Faculty of Pharmacy and Pharmaceutical Sciences, Monash University, Melbourne, Australia
| |
Collapse
|
4
|
Debray TPA, Collins GS, Riley RD, Snell KIE, Van Calster B, Reitsma JB, Moons KGM. Transparent reporting of multivariable prediction models developed or validated using clustered data (TRIPOD-Cluster): explanation and elaboration. BMJ 2023; 380:e071058. [PMID: 36750236 PMCID: PMC9903176 DOI: 10.1136/bmj-2022-071058] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 10/07/2022] [Indexed: 02/09/2023]
Affiliation(s)
- Thomas P A Debray
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, Botnar Research Centre, University of Oxford, Oxford, UK
- National Institute for Health and Care Research Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford, UK
| | - Richard D Riley
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK
| | - Kym I E Snell
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK
| | - Ben Van Calster
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- EPI-centre, KU Leuven, Leuven, Belgium
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, Netherlands
| | - Johannes B Reitsma
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
| | - Karel G M Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
| |
Collapse
|
5
|
Neligan A, Adan G, Nevitt SJ, Pullen A, Sander JW, Bonnett L, Marson AG. Prognosis of adults and children following a first unprovoked seizure. Cochrane Database Syst Rev 2023; 1:CD013847. [PMID: 36688481 PMCID: PMC9869434 DOI: 10.1002/14651858.cd013847.pub2] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/24/2023]
Abstract
BACKGROUND Epilepsy is clinically defined as two or more unprovoked epileptic seizures more than 24 hours apart. Given that, a diagnosis of epilepsy can be associated with significant morbidity and mortality, it is imperative that clinicians (and people with seizures and their relatives) have access to accurate and reliable prognostic estimates, to guide clinical practice on the risks of developing further unprovoked seizures (and by definition, a diagnosis of epilepsy) following single unprovoked epileptic seizure. OBJECTIVES 1. To provide an accurate estimate of the proportion of individuals going on to have further unprovoked seizures at subsequent time points following a single unprovoked epileptic seizure (or cluster of epileptic seizures within a 24-hour period, or a first episode of status epilepticus), of any seizure type (overall prognosis). 2. To evaluate the mortality rate following a first unprovoked epileptic seizure. SEARCH METHODS We searched the following databases on 19 September 2019 and again on 30 March 2021, with no language restrictions. The Cochrane Register of Studies (CRS Web), MEDLINE Ovid (1946 to March 29, 2021), SCOPUS (1823 onwards), ClinicalTrials.gov, the World Health Organization (WHO) International Clinical Trials Registry Platform (ICTRP). CRS Web includes randomized or quasi-randomized, controlled trials from PubMed, Embase, ClinicalTrials.gov, the World Health Organization International Clinical Trials Registry Platform (ICTRP), the Cochrane Central Register of Controlled Trials (CENTRAL), and the Specialized Registers of Cochrane Review Groups including Epilepsy. In MEDLINE (Ovid) the coverage end date always lags a few days behind the search date. SELECTION CRITERIA We included studies, both retrospective and prospective, of all age groups (except those in the neonatal period (< 1 month of age)), of people with a single unprovoked seizure, followed up for a minimum of six months, with no upper limit of follow-up, with the study end point being seizure recurrence, death, or loss to follow-up. To be included, studies must have included at least 30 participants. We excluded studies that involved people with seizures that occur as a result of an acute precipitant or provoking factor, or in close temporal proximity to an acute neurological insult, since these are not considered epileptic in aetiology (acute symptomatic seizures). We also excluded people with situational seizures, such as febrile convulsions. DATA COLLECTION AND ANALYSIS Two review authors conducted the initial screening of titles and abstracts identified through the electronic searches, and removed non-relevant articles. We obtained the full-text articles of all remaining potentially relevant studies, or those whose relevance could not be determined from the abstract alone and two authors independently assessed for eligibility. All disagreements were resolved through discussion with no need to defer to a third review author. We extracted data from included studies using a data extraction form based on the checklist for critical appraisal and data extraction for systematicreviews of prediction modelling studies (CHARMS). Two review authors then appraised the included studies, using a standardised approach based on the quality in prognostic studies (QUIPS) tool, which was adapted for overall prognosis (seizure recurrence). We conducted a meta-analysis using Review Manager 2014, with a random-effects generic inverse variance meta-analysis model, which accounted for any between-study heterogeneity in the prognostic effect. We then summarised the meta-analysis by the pooled estimate (the average prognostic factor effect), its 95% confidence interval (CI), the estimates of I² and Tau² (heterogeneity), and a 95% prediction interval for the prognostic effect in a single population at three various time points, 6 months, 12 months and 24 months. Subgroup analysis was performed according to the ages of the cohorts included; studies involving all ages, studies that recruited adult only and those that were purely paediatric. MAIN RESULTS Fifty-eight studies (involving 54 cohorts), with a total of 12,160 participants (median 147, range 31 to 1443), met the inclusion criteria for the review. Of the 58 studies, 26 studies were paediatric studies, 16 were adult and the remaining 16 studies were a combination of paediatric and adult populations. Most included studies had a cohort study design with two case-control studies and one nested case-control study. Thirty-two studies (29 cohorts) reported a prospective longitudinal design whilst 15 studies had a retrospective design whilst the remaining studies were randomised controlled trials. Nine of the studies included presented mortality data following a first unprovoked seizure. For a mortality study to be included, a proportional mortality ratio (PMR) or a standardised mortality ratio (SMR) had to be given at a specific time point following a first unprovoked seizure. To be included in the meta-analysis a study had to present clear seizure recurrence data at 6 months, 12 months or 24 months. Forty-six studies were included in the meta-analysis, of which 23 were paediatric, 13 were adult, and 10 were a combination of paediatric and adult populations. A meta-analysis was performed at three time points; six months, one year and two years for all ages combined, paediatric and adult studies, respectively. We found an estimated overall seizure recurrence of all included studies at six months of 27% (95% CI 24% to 31%), 36% (95% CI 33% to 40%) at one year and 43% (95% CI 37% to 44%) at two years, with slightly lower estimates for adult subgroup analysis and slightly higher estimates for paediatric subgroup analysis. It was not possible to provide a summary estimate of the risk of seizure recurrence beyond these time points as most of the included studies were of short follow-up and too few studies presented recurrence rates at a single time point beyond two years. The evidence presented was found to be of moderate certainty. AUTHORS' CONCLUSIONS Despite the limitations of the data (moderate-certainty of evidence), mainly relating to clinical and methodological heterogeneity we have provided summary estimates for the likely risk of seizure recurrence at six months, one year and two years for both children and adults. This provides information that is likely to be useful for the clinician counselling patients (or their parents) on the probable risk of further seizures in the short-term whilst acknowledging the paucity of long-term recurrence data, particularly beyond 10 years.
Collapse
Affiliation(s)
- Aidan Neligan
- Homerton University Hospital, NHS Foundation Trust, London, UK
- Department of Clinical and Experimental Epilepsy, UCL Queen Square Institute of Neurology, London, UK
| | - Guleed Adan
- Department of Pharmacology and Therapeutics, Institute of Systems, Molecular and Integrative Biology, University of Liverpool, Liverpool, UK
- The Walton Centre NHS Foundation Trust, Liverpool, UK
| | - Sarah J Nevitt
- Department of Health Data Science, University of Liverpool, Liverpool, UK
| | | | - Josemir W Sander
- Department of Clinical and Experimental Epilepsy, UCL Queen Square Institute of Neurology, London, UK
- National Hospital for Neurology and Neurosurgery, London, UK
| | - Laura Bonnett
- Department of Health Data Science, University of Liverpool, Liverpool, UK
| | - Anthony G Marson
- Department of Pharmacology and Therapeutics, Institute of Systems, Molecular and Integrative Biology, University of Liverpool, Liverpool, UK
- The Walton Centre NHS Foundation Trust, Liverpool, UK
- Liverpool Health Partners, Liverpool, UK
| |
Collapse
|
6
|
Karageorgou M, Hughes DM, Myint AS, Pritchard DM, Bonnett LJ. Clinical prediction models assessing response to radiotherapy for rectal cancer: protocol for a systematic review. Diagn Progn Res 2022; 6:19. [PMID: 36199114 PMCID: PMC9535989 DOI: 10.1186/s41512-022-00132-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/31/2022] [Accepted: 08/02/2022] [Indexed: 11/17/2022] Open
Abstract
BACKGROUND Rectal cancer has a high prevalence. The standard of care for management of localised disease involves major surgery and/or chemoradiotherapy, but these modalities are sometimes associated with mortality and morbidity. The notion of 'watch and wait' has therefore emerged and offers an organ-sparing approach to patients after administering a less invasive initial treatment, such as X-ray brachytherapy (Papillon technique). It is thus important to evaluate how likely patients are to respond to such therapies, to develop patient-tailored treatment pathways. We propose a systematic review to identify published clinical prediction models of the response of rectal cancer to treatment that includes radiotherapy and here present our protocol. METHODS Included studies will develop multivariable clinical prediction models which assess response to treatment and overall survival of adult patients who have been diagnosed with any stage of rectal cancer and have received radiotherapy treatment with curative intent. Cohort studies and randomised controlled trials will be included. The primary outcome will be the occurrence of salvage surgery at 1 year after treatment. Secondary outcomes include salavage surgery at at any reported time point, the predictive accuracy of models, the quality of the developed models and the feasibility of using the model in clinical practice. Ovid MEDLINE, PubMed, Cochrane Library, EMBASE and CINAHL will be searched from inception to 24 February 2022. Keywords and phrases related to rectal cancer, radiotherapy and prediction models will be used. Studies will be selected once the deduplication, title, abstract and full-text screening process have been completed by two independent reviewers. The PRISMA-P checklist will be followed. A third reviewer will resolve any disagreement. The data extraction form will be pilot-tested using a representative 5% sample of the studies reviewed. The CHARMS checklist will be implemented. Risk of bias in each study will be assessed using the PROBAST tool. A narrative synthesis will be performed and if sufficient data are identified, meta-analysis will be undertaken as described in Debray et al. DISCUSSION: This systematic review will identify factors that predict response to the treatment protocol. Any gaps for potential development of new clinical prediction models will be highlighted. TRIAL REGISTRATION CRD42022277704.
Collapse
Affiliation(s)
- Margarita Karageorgou
- Department of Molecular and Clinical Cancer Medicine, Institute of Systems, Molecular and Integrative Biology, University of Liverpool, Liverpool, UK
| | - David M Hughes
- Department of Health Data Science, Institute of Population Health, University of Liverpool, Waterhouse Building, Block B, Brownlow Street, Liverpool, L69 3GF, UK
| | - Arthur Sun Myint
- Department of Molecular and Clinical Cancer Medicine, Institute of Systems, Molecular and Integrative Biology, University of Liverpool, Liverpool, UK
- Papillon Suite, Clatterbridge Cancer Centre, Liverpool, UK
| | - D Mark Pritchard
- Department of Molecular and Clinical Cancer Medicine, Institute of Systems, Molecular and Integrative Biology, University of Liverpool, Liverpool, UK
| | - Laura J Bonnett
- Department of Health Data Science, Institute of Population Health, University of Liverpool, Waterhouse Building, Block B, Brownlow Street, Liverpool, L69 3GF, UK.
| |
Collapse
|
7
|
Chang WJ, Naylor J, Natarajan P, Liu V, Adie S. Evaluating methodological quality of prognostic prediction models on patient reported outcome measurements after total hip replacement and total knee replacement surgery: a systematic review protocol. Syst Rev 2022; 11:165. [PMID: 35948989 PMCID: PMC9364604 DOI: 10.1186/s13643-022-02039-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/13/2021] [Accepted: 07/28/2022] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND Prediction models for poor patient-reported surgical outcomes after total hip replacement (THR) and total knee replacement (TKR) may provide a method for improving appropriate surgical care for hip and knee osteoarthritis. There are concerns about methodological issues and the risk of bias of studies producing prediction models. A critical evaluation of the methodological quality of prediction modelling studies in THR and TKR is needed to ensure their clinical usefulness. This systematic review aims to (1) evaluate and report the quality of risk stratification and prediction modelling studies that predict patient-reported outcomes after THR and TKR; (2) identify areas of methodological deficit and provide recommendations for future research; and (3) synthesise the evidence on prediction models associated with post-operative patient-reported outcomes after THR and TKR surgeries. METHODS MEDLINE, EMBASE, and CINAHL electronic databases will be searched to identify relevant studies. Title and abstract and full-text screening will be performed by two independent reviewers. We will include (1) prediction model development studies without external validation; (2) prediction model development studies with external validation of independent data; (3) external model validation studies; and (4) studies updating a previously developed prediction model. Data extraction spreadsheets will be developed based on the CHARMS checklist and TRIPOD statement and piloted on two relevant studies. Study quality and risk of bias will be assessed using the PROBAST tool. Prediction models will be summarised qualitatively. Meta-analyses on the predictive performance of included models will be conducted if appropriate. A narrative review will be used to synthesis the evidence if there are insufficient data to perform meta-analyses. DISCUSSION This systematic review will evaluate the methodological quality and usefulness of prediction models for poor outcomes after THR or TKR. This information is essential to provide evidence-based healthcare for end-stage hip and knee osteoarthritis. Findings of this review will contribute to the identification of key areas for improvement in conducting prognostic research in this field and facilitate the progress in evidence-based tailored treatments for hip and knee osteoarthritis. SYSTEMATIC REVIEW REGISTRATION PROSPERO registration number CRD42021271828.
Collapse
Affiliation(s)
- Wei-Ju Chang
- Centre for Pain IMPACT, Neuroscience Research Australia (NeuRA), 139 Barker St, Randwick, NSW, 2031, Australia. .,School of Health Sciences, College of Health, Medicine and Wellbeing, The University of Newcastle, Callaghan, NSW, 2038, Australia.
| | - Justine Naylor
- School of Clinical Medicine, UNSW Medicine & Health, South West Clinical Campuses, Discipline of Surgery, Faculty of Medicine and Health, UNSW, Sydney, NSW, Australia.,Whitlam Orthopaedic Research Centre, Ingham Institute for Applied Medical Research, 1 Campbell St, Liverpool, NSW, 2170, Australia
| | - Pragadesh Natarajan
- St George and Sutherland Clinical School, University of New South Wales, Clinical Sciences (WRPitney) Building, Short Street, St George Hospital, Kogarah, NSW, 2217, Australia
| | - Victor Liu
- St George and Sutherland Clinical School, University of New South Wales, Clinical Sciences (WRPitney) Building, Short Street, St George Hospital, Kogarah, NSW, 2217, Australia
| | - Sam Adie
- St George and Sutherland Clinical School, University of New South Wales, Clinical Sciences (WRPitney) Building, Short Street, St George Hospital, Kogarah, NSW, 2217, Australia.,St. George and Sutherland Centre for Clinical Orthopaedic Research (SCORe), Suite 201, Level 2 131 Princes Highway, Kogarah, NSW, 2217, Australia.,School of Clinical Medicine, UNSW Medicine & Health, UNSW, New South Wales, Sydney, Australia
| |
Collapse
|
8
|
Allotey J, Snell KI, Smuk M, Hooper R, Chan CL, Ahmed A, Chappell LC, von Dadelszen P, Dodds J, Green M, Kenny L, Khalil A, Khan KS, Mol BW, Myers J, Poston L, Thilaganathan B, Staff AC, Smith GC, Ganzevoort W, Laivuori H, Odibo AO, Ramírez JA, Kingdom J, Daskalakis G, Farrar D, Baschat AA, Seed PT, Prefumo F, da Silva Costa F, Groen H, Audibert F, Masse J, Skråstad RB, Salvesen KÅ, Haavaldsen C, Nagata C, Rumbold AR, Heinonen S, Askie LM, Smits LJ, Vinter CA, Magnus PM, Eero K, Villa PM, Jenum AK, Andersen LB, Norman JE, Ohkuchi A, Eskild A, Bhattacharya S, McAuliffe FM, Galindo A, Herraiz I, Carbillon L, Klipstein-Grobusch K, Yeo S, Teede HJ, Browne JL, Moons KG, Riley RD, Thangaratinam S. Validation and development of models using clinical, biochemical and ultrasound markers for predicting pre-eclampsia: an individual participant data meta-analysis. Health Technol Assess 2021; 24:1-252. [PMID: 33336645 DOI: 10.3310/hta24720] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/12/2022] Open
Abstract
BACKGROUND Pre-eclampsia is a leading cause of maternal and perinatal mortality and morbidity. Early identification of women at risk is needed to plan management. OBJECTIVES To assess the performance of existing pre-eclampsia prediction models and to develop and validate models for pre-eclampsia using individual participant data meta-analysis. We also estimated the prognostic value of individual markers. DESIGN This was an individual participant data meta-analysis of cohort studies. SETTING Source data from secondary and tertiary care. PREDICTORS We identified predictors from systematic reviews, and prioritised for importance in an international survey. PRIMARY OUTCOMES Early-onset (delivery at < 34 weeks' gestation), late-onset (delivery at ≥ 34 weeks' gestation) and any-onset pre-eclampsia. ANALYSIS We externally validated existing prediction models in UK cohorts and reported their performance in terms of discrimination and calibration. We developed and validated 12 new models based on clinical characteristics, clinical characteristics and biochemical markers, and clinical characteristics and ultrasound markers in the first and second trimesters. We summarised the data set-specific performance of each model using a random-effects meta-analysis. Discrimination was considered promising for C-statistics of ≥ 0.7, and calibration was considered good if the slope was near 1 and calibration-in-the-large was near 0. Heterogeneity was quantified using I 2 and τ2. A decision curve analysis was undertaken to determine the clinical utility (net benefit) of the models. We reported the unadjusted prognostic value of individual predictors for pre-eclampsia as odds ratios with 95% confidence and prediction intervals. RESULTS The International Prediction of Pregnancy Complications network comprised 78 studies (3,570,993 singleton pregnancies) identified from systematic reviews of tests to predict pre-eclampsia. Twenty-four of the 131 published prediction models could be validated in 11 UK cohorts. Summary C-statistics were between 0.6 and 0.7 for most models, and calibration was generally poor owing to large between-study heterogeneity, suggesting model overfitting. The clinical utility of the models varied between showing net harm to showing minimal or no net benefit. The average discrimination for IPPIC models ranged between 0.68 and 0.83. This was highest for the second-trimester clinical characteristics and biochemical markers model to predict early-onset pre-eclampsia, and lowest for the first-trimester clinical characteristics models to predict any pre-eclampsia. Calibration performance was heterogeneous across studies. Net benefit was observed for International Prediction of Pregnancy Complications first and second-trimester clinical characteristics and clinical characteristics and biochemical markers models predicting any pre-eclampsia, when validated in singleton nulliparous women managed in the UK NHS. History of hypertension, parity, smoking, mode of conception, placental growth factor and uterine artery pulsatility index had the strongest unadjusted associations with pre-eclampsia. LIMITATIONS Variations in study population characteristics, type of predictors reported, too few events in some validation cohorts and the type of measurements contributed to heterogeneity in performance of the International Prediction of Pregnancy Complications models. Some published models were not validated because model predictors were unavailable in the individual participant data. CONCLUSION For models that could be validated, predictive performance was generally poor across data sets. Although the International Prediction of Pregnancy Complications models show good predictive performance on average, and in the singleton nulliparous population, heterogeneity in calibration performance is likely across settings. FUTURE WORK Recalibration of model parameters within populations may improve calibration performance. Additional strong predictors need to be identified to improve model performance and consistency. Validation, including examination of calibration heterogeneity, is required for the models we could not validate. STUDY REGISTRATION This study is registered as PROSPERO CRD42015029349. FUNDING This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 24, No. 72. See the NIHR Journals Library website for further project information.
Collapse
|
9
|
Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets. J Clin Epidemiol 2021; 137:83-91. [PMID: 33836256 DOI: 10.1016/j.jclinepi.2021.03.025] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2020] [Revised: 03/05/2021] [Accepted: 03/29/2021] [Indexed: 12/23/2022]
Abstract
OBJECTIVE To illustrate how to evaluate the need of complex strategies for developing generalizable prediction models in large clustered datasets. STUDY DESIGN AND SETTING We developed eight Cox regression models to estimate the risk of heart failure using a large population-level dataset. These models differed in the number of predictors, the functional form of the predictor effects (non-linear effects and interaction) and the estimation method (maximum likelihood and penalization). Internal-external cross-validation was used to evaluate the models' generalizability across the included general practices. RESULTS Among 871,687 individuals from 225 general practices, 43,987 (5.5%) developed heart failure during a median follow-up time of 5.8 years. For discrimination, the simplest prediction model yielded a good concordance statistic, which was not much improved by adopting complex strategies. Between-practice heterogeneity in discrimination was similar in all models. For calibration, the simplest model performed satisfactorily. Although accounting for non-linear effects and interaction slightly improved the calibration slope, it also led to more heterogeneity in the observed/expected ratio. Similar results were found in a second case study involving patients with stroke. CONCLUSION In large clustered datasets, prediction model studies may adopt internal-external cross-validation to evaluate the generalizability of competing models, and to identify promising modelling strategies.
Collapse
|
10
|
Meid AD, Gonzalez-Gonzalez AI, Dinh TS, Blom J, van den Akker M, Elders P, Thiem U, Küllenberg de Gaudry D, Swart KMA, Rudolf H, Bosch-Lenders D, Trampisch HJ, Meerpohl JJ, Gerlach FM, Flaig B, Kom G, Snell KIE, Perera R, Haefeli WE, Glasziou P, Muth C. Predicting hospital admissions from individual patient data (IPD): an applied example to explore key elements driving external validity. BMJ Open 2021; 11:e045572. [PMID: 34348947 PMCID: PMC8340284 DOI: 10.1136/bmjopen-2020-045572] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/18/2022] Open
Abstract
OBJECTIVE To explore factors that potentially impact external validation performance while developing and validating a prognostic model for hospital admissions (HAs) in complex older general practice patients. STUDY DESIGN AND SETTING Using individual participant data from four cluster-randomised trials conducted in the Netherlands and Germany, we used logistic regression to develop a prognostic model to predict all-cause HAs within a 6-month follow-up period. A stratified intercept was used to account for heterogeneity in baseline risk between the studies. The model was validated both internally and by using internal-external cross-validation (IECV). RESULTS Prior HAs, physical components of the health-related quality of life comorbidity index, and medication-related variables were used in the final model. While achieving moderate discriminatory performance, internal bootstrap validation revealed a pronounced risk of overfitting. The results of the IECV, in which calibration was highly variable even after accounting for between-study heterogeneity, agreed with this finding. Heterogeneity was equally reflected in differing baseline risk, predictor effects and absolute risk predictions. CONCLUSIONS Predictor effect heterogeneity and differing baseline risk can explain the limited external performance of HA prediction models. With such drivers known, model adjustments in external validation settings (eg, intercept recalibration, complete updating) can be applied more purposefully. TRIAL REGISTRATION NUMBER PROSPERO id: CRD42018088129.
Collapse
Affiliation(s)
- Andreas Daniel Meid
- Department of Clinical Pharmacology & Pharmacoepidemiology, Heidelberg University, Heidelberg, Baden-Württemberg, Germany
| | - Ana Isabel Gonzalez-Gonzalez
- Institute of General Practice, Goethe University, Frankfurt am Main, Hessen, Germany
- Red de Investigación en Servicios de Salud en Enfermedades Crónicas (REDISSEC), Madrid, Spain
| | - Truc Sophia Dinh
- Institute of General Practice, Goethe University, Frankfurt am Main, Hessen, Germany
| | - Jeanet Blom
- Department of Public Health and Primary Care, Leiden University Medical Center, Leiden, The Netherlands
| | - Marjan van den Akker
- Institute of General Practice, Goethe University, Frankfurt am Main, Hessen, Germany
- School of CAPHRI, Department of Family Medicine, Maastricht University, Maastricht, The Netherlands
| | - Petra Elders
- Department of General Practice and Elderly Care Medicine, Amsterdam UMC, Vrije Universiteit, Amstedarm Public Health Research Institute, Amsterdam, The Netherlands
| | - Ulrich Thiem
- Chair of Geriatrics and Gerontology, University Clinic Eppendorf, Hamburg, Germany
| | - Daniela Küllenberg de Gaudry
- Institute for Evidence in Medicine (for Cochrane Germany Foundation), Medical Center-University of Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany
| | - Karin M A Swart
- Department of General Practice and Elderly Care Medicine, Amsterdam UMC, Vrije Universiteit, Amstedarm Public Health Research Institute, Amsterdam, The Netherlands
| | - Henrik Rudolf
- Department of Medical Informatics, Biometry and Epidemiology, Ruhr University Bochum, Bochum, Nordrhein-Westfalen, Germany
| | - Donna Bosch-Lenders
- School of CAPHRI, Department of Family Medicine, Maastricht University, Maastricht, The Netherlands
| | - Hans J Trampisch
- Department of Medical Informatics, Biometry and Epidemiology, Ruhr University Bochum, Bochum, Nordrhein-Westfalen, Germany
| | - Joerg J Meerpohl
- Institute for Evidence in Medicine (for Cochrane Germany Foundation), Medical Center-University of Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany
| | - Ferdinand M Gerlach
- Institute of General Practice, Goethe University, Frankfurt am Main, Hessen, Germany
| | - Benno Flaig
- Institute of General Practice, Goethe University, Frankfurt am Main, Hessen, Germany
| | | | - Kym I E Snell
- Centre for Prognosis Research, School of Primary Care Research, Community and Social Care, Keele University, Keele, UK
| | - Rafael Perera
- Nuffield Department of Primary Care, University of Oxford, Oxford, UK
| | - Walter Emil Haefeli
- Department of Clinical Pharmacology & Pharmacoepidemiology, Heidelberg University, Heidelberg, Baden-Württemberg, Germany
| | - Paul Glasziou
- Centre for Research in Evidence-Based Practice, Bond University, Robina, Queensland, Australia
| | - Christiane Muth
- Institute of General Practice, Goethe University, Frankfurt am Main, Hessen, Germany
- Department of General Practice and Family Medicine, Medical Faculty OWL, University of Bielefeld, Bielefeld, Germany
| |
Collapse
|
11
|
Mohanannair Geethadevi G, Quinn TJ, George J, Anstey K, Bell JS, Cross AJ. Multi-domain prognostic models used in middle aged adults without known cognitive impairment for predicting subsequent dementia. Hippokratia 2021. [DOI: 10.1002/14651858.cd014885] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022]
Affiliation(s)
| | - Terry J Quinn
- Institute of Cardiovascular and Medical Sciences; University of Glasgow; Glasgow UK
| | - Johnson George
- Centre for Medicine Use and Safety, Faculty of Pharmacy and Pharmaceutical Sciences; Monash University; Parkville Australia
| | - Kaarin Anstey
- Centre for Mental Health Research; The Australian National University; Canberra Australia
| | - J Simon Bell
- Centre for Medicine Use and Safety, Faculty of Pharmacy and Pharmaceutical Sciences; Monash University; Parkville Australia
| | - Amanda J Cross
- Centre for Medicine Use and Safety, Faculty of Pharmacy and Pharmaceutical Sciences; Monash University; Parkville Australia
| |
Collapse
|
12
|
de Jong VMT, Moons KGM, Eijkemans MJC, Riley RD, Debray TPA. Developing more generalizable prediction models from pooled studies and large clustered data sets. Stat Med 2021; 40:3533-3559. [PMID: 33948970 PMCID: PMC8252590 DOI: 10.1002/sim.8981] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2019] [Revised: 02/16/2021] [Accepted: 03/22/2021] [Indexed: 12/14/2022]
Abstract
Prediction models often yield inaccurate predictions for new individuals. Large data sets from pooled studies or electronic healthcare records may alleviate this with an increased sample size and variability in sample characteristics. However, existing strategies for prediction model development generally do not account for heterogeneity in predictor‐outcome associations between different settings and populations. This limits the generalizability of developed models (even from large, combined, clustered data sets) and necessitates local revisions. We aim to develop methodology for producing prediction models that require less tailoring to different settings and populations. We adopt internal‐external cross‐validation to assess and reduce heterogeneity in models' predictive performance during the development. We propose a predictor selection algorithm that optimizes the (weighted) average performance while minimizing its variability across the hold‐out clusters (or studies). Predictors are added iteratively until the estimated generalizability is optimized. We illustrate this by developing a model for predicting the risk of atrial fibrillation and updating an existing one for diagnosing deep vein thrombosis, using individual participant data from 20 cohorts (N = 10 873) and 11 diagnostic studies (N = 10 014), respectively. Meta‐analysis of calibration and discrimination performance in each hold‐out cluster shows that trade‐offs between average and heterogeneity of performance occurred. Our methodology enables the assessment of heterogeneity of prediction model performance during model development in multiple or clustered data sets, thereby informing researchers on predictor selection to improve the generalizability to different settings and populations, and reduce the need for model tailoring. Our methodology has been implemented in the R package metamisc.
Collapse
Affiliation(s)
- Valentijn M T de Jong
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.,Cochrane Netherlands, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Karel G M Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.,Cochrane Netherlands, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Marinus J C Eijkemans
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Richard D Riley
- Centre for Prognosis Research, School of Medicine, Keele University, Staffordshire, UK
| | - Thomas P A Debray
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.,Cochrane Netherlands, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
13
|
Ensor J, Snell KIE, Debray TPA, Lambert PC, Look MP, Mamas MA, Moons KGM, Riley RD. Individual participant data meta-analysis for external validation, recalibration, and updating of a flexible parametric prognostic model. Stat Med 2021; 40:3066-3084. [PMID: 33768582 DOI: 10.1002/sim.8959] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2019] [Revised: 03/04/2021] [Accepted: 03/05/2021] [Indexed: 12/14/2022]
Abstract
Individual participant data (IPD) from multiple sources allows external validation of a prognostic model across multiple populations. Often this reveals poor calibration, potentially causing poor predictive performance in some populations. However, rather than discarding the model outright, it may be possible to modify the model to improve performance using recalibration techniques. We use IPD meta-analysis to identify the simplest method to achieve good model performance. We examine four options for recalibrating an existing time-to-event model across multiple populations: (i) shifting the baseline hazard by a constant, (ii) re-estimating the shape of the baseline hazard, (iii) adjusting the prognostic index as a whole, and (iv) adjusting individual predictor effects. For each strategy, IPD meta-analysis examines (heterogeneity in) model performance across populations. Additionally, the probability of achieving good performance in a new population can be calculated allowing ranking of recalibration methods. In an applied example, IPD meta-analysis reveals that the existing model had poor calibration in some populations, and large heterogeneity across populations. However, re-estimation of the intercept substantially improved the expected calibration in new populations, and reduced between-population heterogeneity. Comparing recalibration strategies showed that re-estimating both the magnitude and shape of the baseline hazard gave the highest predicted probability of good performance in a new population. In conclusion, IPD meta-analysis allows a prognostic model to be externally validated in multiple settings, and enables recalibration strategies to be compared and ranked to decide on the least aggressive recalibration strategy to achieve acceptable external model performance without discarding existing model information.
Collapse
Affiliation(s)
- Joie Ensor
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK
| | - Kym I E Snell
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK
| | - Thomas P A Debray
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands.,Cochrane Netherlands, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Paul C Lambert
- Biostatistics Research Group, Department of Health Sciences, University of Leicester, Centre for Medicine, Leicester, UK.,Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
| | - Maxime P Look
- Department of Medical Oncology, Erasmus MC Cancer Institute, Erasmus University Medical Center, Rotterdam, The Netherlands
| | - Mamas A Mamas
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK.,Department of Cardiology, Royal Stoke University Hospital, Stoke-on-Trent, UK
| | - Karel G M Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands.,Cochrane Netherlands, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Richard D Riley
- Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK
| |
Collapse
|
14
|
Identifying Adolescents at Risk for Depression: A Prediction Score Performance in Cohorts Based in 3 Different Continents. J Am Acad Child Adolesc Psychiatry 2021; 60:262-273. [PMID: 31953186 PMCID: PMC8215370 DOI: 10.1016/j.jaac.2019.12.004] [Citation(s) in RCA: 39] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/11/2019] [Revised: 12/19/2019] [Accepted: 01/08/2020] [Indexed: 12/13/2022]
Abstract
OBJECTIVE Prediction models have become frequent in the medical literature, but most published studies are conducted in a single setting. Heterogeneity between development and validation samples has been posited as a major obstacle for the generalization of models. We aimed to develop a multivariable prognostic model using sociodemographic variables easily obtainable from adolescents at age 15 to predict a depressive disorder diagnosis at age 18 and to evaluate its generalizability in 2 samples from diverse socioeconomic and cultural settings. METHOD Data from the 1993 Pelotas Birth Cohort were used to develop the prediction model, and its generalizability was evaluated in 2 representative cohort studies: the Environmental Risk (E-Risk) Longitudinal Twin Study and the Dunedin Multidisciplinary Health and Development Study. RESULTS At age 15, 2,192 adolescents with no evidence of current or previous depression were included (44.6% male). The apparent C-statistic of the models derived in Pelotas ranged from 0.76 to 0.79, and the model obtained from a penalized logistic regression was selected for subsequent external evaluation. Major discrepancies between the samples were identified, impacting the external prognostic performance of the model (Dunedin and E-Risk C-statistics of 0.63 and 0.59, respectively). The implementation of recommended strategies to account for this heterogeneity among samples improved the model's calibration in both samples. CONCLUSION An adolescent depression risk score comprising easily obtainable predictors was developed with good prognostic performance in a Brazilian sample. Heterogeneity among settings was not trivial, but strategies to deal with sample diversity were identified as pivotal for providing better risk stratification across samples. Future efforts should focus on developing better methodological approaches for incorporating heterogeneity in prognostic research.
Collapse
|
15
|
Adan G, Neligan A, Nevitt SJ, Pullen A, Sander JW, Marson AG. Prognostic factors predicting an unprovoked seizure recurrence in children and adults following a first unprovoked seizure. Hippokratia 2021. [DOI: 10.1002/14651858.cd013848] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Affiliation(s)
- Guleed Adan
- Department of Molecular and Clinical Pharmacology; Institute of Translational Medicine, University of Liverpool; Liverpool UK
- The Walton Centre NHS Foundation Trust; Liverpool UK
| | - Aidan Neligan
- Homerton University Hospital, NHS Foundation Trust; London UK
- Department of Clinical and Experimental Epilepsy; UCL Queen Square Institute of Neurology; London UK
| | - Sarah J Nevitt
- Department of Health Data Science; University of Liverpool; Liverpool UK
| | | | - Josemir W Sander
- Department of Clinical and Experimental Epilepsy; UCL Queen Square Institute of Neurology; London UK
- National Hospital for Neurology and Neurosurgery; London UK
| | - Anthony G Marson
- Department of Molecular and Clinical Pharmacology; Institute of Translational Medicine, University of Liverpool; Liverpool UK
- The Walton Centre NHS Foundation Trust; Liverpool UK
- Liverpool Health Partners; Liverpool UK
| |
Collapse
|
16
|
Neligan A, Adan G, Nevitt SJ, Pullen A, Sander JW, Marson AG. Prognosis of adults and children following a first unprovoked seizure. Hippokratia 2021. [DOI: 10.1002/14651858.cd013847] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Affiliation(s)
- Aidan Neligan
- Homerton University Hospital, NHS Foundation Trust; London UK
- Department of Clinical and Experimental Epilepsy; UCL Queen Square Institute of Neurology; London UK
| | - Guleed Adan
- Department of Molecular and Clinical Pharmacology; Institute of Translational Medicine, University of Liverpool; Liverpool UK
- The Walton Centre NHS Foundation Trust; Liverpool UK
| | - Sarah J Nevitt
- Department of Health Data Science; University of Liverpool; Liverpool UK
| | | | - Josemir W Sander
- Department of Clinical and Experimental Epilepsy; UCL Queen Square Institute of Neurology; London UK
- National Hospital for Neurology and Neurosurgery; London UK
| | - Anthony G Marson
- Department of Molecular and Clinical Pharmacology; Institute of Translational Medicine, University of Liverpool; Liverpool UK
- The Walton Centre NHS Foundation Trust; Liverpool UK
- Liverpool Health Partners; Liverpool UK
| |
Collapse
|
17
|
Martin GP, Sperrin M, Sotgiu G. Performance of prediction models for COVID-19: the Caudine Forks of the external validation. Eur Respir J 2020; 56:13993003.03728-2020. [PMID: 33060155 PMCID: PMC7562696 DOI: 10.1183/13993003.03728-2020] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2020] [Accepted: 10/06/2020] [Indexed: 02/06/2023]
Abstract
Healthcare systems worldwide have observed significant changes to meet demands due to the coronavirus disease 2019 (COVID-19) pandemic. The uncertainty surrounding optimal treatment, the rapid public health urgency and clinical emergencies have caused a chaotic disruption of the cases and their related contacts at inpatient and outpatient settings. Developing more tailored healthcare plans based on the currently available scientific evidence, could help improve clinical efficacy, treatment outcomes, prognosis, and health efficiency. Existing evidence suggests that none of the COVID-19 prediction models can be supported for clinical use. Here we discuss “what next” in COVID-19 prediction.https://bit.ly/2SMtoLV
Collapse
Affiliation(s)
- Glen P Martin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| | - Matthew Sperrin
- Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| | - Giovanni Sotgiu
- Clinical Epidemiology and Medical Statistics Unit, Dept of Medical, Surgical and Experimental Sciences, University of Sassari, Sassari, Italy
| |
Collapse
|
18
|
Sufriyana H, Husnayain A, Chen YL, Kuo CY, Singh O, Yeh TY, Wu YW, Su ECY. Comparison of Multivariable Logistic Regression and Other Machine Learning Algorithms for Prognostic Prediction Studies in Pregnancy Care: Systematic Review and Meta-Analysis. JMIR Med Inform 2020; 8:e16503. [PMID: 33200995 PMCID: PMC7708089 DOI: 10.2196/16503] [Citation(s) in RCA: 38] [Impact Index Per Article: 9.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/07/2019] [Revised: 06/22/2020] [Accepted: 10/24/2020] [Indexed: 02/06/2023] Open
Abstract
Background Predictions in pregnancy care are complex because of interactions among multiple factors. Hence, pregnancy outcomes are not easily predicted by a single predictor using only one algorithm or modeling method. Objective This study aims to review and compare the predictive performances between logistic regression (LR) and other machine learning algorithms for developing or validating a multivariable prognostic prediction model for pregnancy care to inform clinicians’ decision making. Methods Research articles from MEDLINE, Scopus, Web of Science, and Google Scholar were reviewed following several guidelines for a prognostic prediction study, including a risk of bias (ROB) assessment. We report the results based on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. Studies were primarily framed as PICOTS (population, index, comparator, outcomes, timing, and setting): Population: men or women in procreative management, pregnant women, and fetuses or newborns; Index: multivariable prognostic prediction models using non-LR algorithms for risk classification to inform clinicians’ decision making; Comparator: the models applying an LR; Outcomes: pregnancy-related outcomes of procreation or pregnancy outcomes for pregnant women and fetuses or newborns; Timing: pre-, inter-, and peripregnancy periods (predictors), at the pregnancy, delivery, and either puerperal or neonatal period (outcome), and either short- or long-term prognoses (time interval); and Setting: primary care or hospital. The results were synthesized by reporting study characteristics and ROBs and by random effects modeling of the difference of the logit area under the receiver operating characteristic curve of each non-LR model compared with the LR model for the same pregnancy outcomes. We also reported between-study heterogeneity by using τ2 and I2. Results Of the 2093 records, we included 142 studies for the systematic review and 62 studies for a meta-analysis. Most prediction models used LR (92/142, 64.8%) and artificial neural networks (20/142, 14.1%) among non-LR algorithms. Only 16.9% (24/142) of studies had a low ROB. A total of 2 non-LR algorithms from low ROB studies significantly outperformed LR. The first algorithm was a random forest for preterm delivery (logit AUROC 2.51, 95% CI 1.49-3.53; I2=86%; τ2=0.77) and pre-eclampsia (logit AUROC 1.2, 95% CI 0.72-1.67; I2=75%; τ2=0.09). The second algorithm was gradient boosting for cesarean section (logit AUROC 2.26, 95% CI 1.39-3.13; I2=75%; τ2=0.43) and gestational diabetes (logit AUROC 1.03, 95% CI 0.69-1.37; I2=83%; τ2=0.07). Conclusions Prediction models with the best performances across studies were not necessarily those that used LR but also used random forest and gradient boosting that also performed well. We recommend a reanalysis of existing LR models for several pregnancy outcomes by comparing them with those algorithms that apply standard guidelines. Trial Registration PROSPERO (International Prospective Register of Systematic Reviews) CRD42019136106; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=136106
Collapse
Affiliation(s)
- Herdiantri Sufriyana
- Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei, Taiwan.,Department of Medical Physiology, College of Medicine, University of Nahdlatul Ulama Surabaya, Surabaya, Indonesia
| | - Atina Husnayain
- Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei, Taiwan.,Department of Biostatistics, Epidemiology, and Population Health, Faculty of Medicine, Public Health and Nursing, Universitas Gadjah Mada, Yogyakarta, Indonesia
| | - Ya-Lin Chen
- Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei, Taiwan.,School of Pharmacy, College of Pharmacy, Taipei Medical University, Taipei, Taiwan
| | - Chao-Yang Kuo
- Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei, Taiwan
| | - Onkar Singh
- Bioinformatics Program, Taiwan International Graduate Program, Institute of Information Science, Academia Sinica, Taipei, Taiwan.,Institute of Biomedical Informatics, National Yang-Ming University, Taipei, Taiwan
| | - Tso-Yang Yeh
- School of Dentistry, College of Oral Medicine, Taipei Medical University, Taipei, Taiwan
| | - Yu-Wei Wu
- Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei, Taiwan.,Clinical Big Data Research Center, Taipei Medical University Hospital, Taipei, Taiwan
| | - Emily Chia-Yu Su
- Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei, Taiwan.,Clinical Big Data Research Center, Taipei Medical University Hospital, Taipei, Taiwan
| |
Collapse
|
19
|
Snell KIE, Allotey J, Smuk M, Hooper R, Chan C, Ahmed A, Chappell LC, Von Dadelszen P, Green M, Kenny L, Khalil A, Khan KS, Mol BW, Myers J, Poston L, Thilaganathan B, Staff AC, Smith GCS, Ganzevoort W, Laivuori H, Odibo AO, Arenas Ramírez J, Kingdom J, Daskalakis G, Farrar D, Baschat AA, Seed PT, Prefumo F, da Silva Costa F, Groen H, Audibert F, Masse J, Skråstad RB, Salvesen KÅ, Haavaldsen C, Nagata C, Rumbold AR, Heinonen S, Askie LM, Smits LJM, Vinter CA, Magnus P, Eero K, Villa PM, Jenum AK, Andersen LB, Norman JE, Ohkuchi A, Eskild A, Bhattacharya S, McAuliffe FM, Galindo A, Herraiz I, Carbillon L, Klipstein-Grobusch K, Yeo SA, Browne JL, Moons KGM, Riley RD, Thangaratinam S. External validation of prognostic models predicting pre-eclampsia: individual participant data meta-analysis. BMC Med 2020; 18:302. [PMID: 33131506 PMCID: PMC7604970 DOI: 10.1186/s12916-020-01766-9] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/19/2020] [Accepted: 08/26/2020] [Indexed: 01/05/2023] Open
Abstract
BACKGROUND Pre-eclampsia is a leading cause of maternal and perinatal mortality and morbidity. Early identification of women at risk during pregnancy is required to plan management. Although there are many published prediction models for pre-eclampsia, few have been validated in external data. Our objective was to externally validate published prediction models for pre-eclampsia using individual participant data (IPD) from UK studies, to evaluate whether any of the models can accurately predict the condition when used within the UK healthcare setting. METHODS IPD from 11 UK cohort studies (217,415 pregnant women) within the International Prediction of Pregnancy Complications (IPPIC) pre-eclampsia network contributed to external validation of published prediction models, identified by systematic review. Cohorts that measured all predictor variables in at least one of the identified models and reported pre-eclampsia as an outcome were included for validation. We reported the model predictive performance as discrimination (C-statistic), calibration (calibration plots, calibration slope, calibration-in-the-large), and net benefit. Performance measures were estimated separately in each available study and then, where possible, combined across studies in a random-effects meta-analysis. RESULTS Of 131 published models, 67 provided the full model equation and 24 could be validated in 11 UK cohorts. Most of the models showed modest discrimination with summary C-statistics between 0.6 and 0.7. The calibration of the predicted compared to observed risk was generally poor for most models with observed calibration slopes less than 1, indicating that predictions were generally too extreme, although confidence intervals were wide. There was large between-study heterogeneity in each model's calibration-in-the-large, suggesting poor calibration of the predicted overall risk across populations. In a subset of models, the net benefit of using the models to inform clinical decisions appeared small and limited to probability thresholds between 5 and 7%. CONCLUSIONS The evaluated models had modest predictive performance, with key limitations such as poor calibration (likely due to overfitting in the original development datasets), substantial heterogeneity, and small net benefit across settings. The evidence to support the use of these prediction models for pre-eclampsia in clinical decision-making is limited. Any models that we could not validate should be examined in terms of their predictive performance, net benefit, and heterogeneity across multiple UK settings before consideration for use in practice. TRIAL REGISTRATION PROSPERO ID: CRD42015029349 .
Collapse
Affiliation(s)
- Kym I E Snell
- Centre for Prognosis Research, School of Primary, Community and Social Care, Keele University, Keele, UK.
| | - John Allotey
- Barts Research Centre for Women's Health (BARC), Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
- Pragmatic Clinical Trials Unit, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
| | - Melanie Smuk
- Pragmatic Clinical Trials Unit, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
| | - Richard Hooper
- Pragmatic Clinical Trials Unit, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
| | - Claire Chan
- Pragmatic Clinical Trials Unit, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
| | - Asif Ahmed
- MirZyme Therapeutics, Innovation Birmingham Campus, Birmingham, UK
| | - Lucy C Chappell
- Department of Women and Children's Health, School of Life Course Sciences, King's College London, London, UK
| | - Peter Von Dadelszen
- Department of Women and Children's Health, School of Life Course Sciences, King's College London, London, UK
| | - Marcus Green
- Action on Pre-eclampsia (APEC) Charity, Worcestershire, UK
| | - Louise Kenny
- Faculty Health & Life Sciences, University of Liverpool, Liverpool, UK
| | - Asma Khalil
- Fetal Medicine Unit, St George's University Hospitals NHS Foundation Trust and Molecular and Clinical Sciences Research Institute, St George's University of London, London, UK
| | - Khalid S Khan
- Barts Research Centre for Women's Health (BARC), Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
- Pragmatic Clinical Trials Unit, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
| | - Ben W Mol
- Department of Obstetrics and Gynaecology, Monash University, Monash Medical Centre, Clayton, Victoria, Australia
| | - Jenny Myers
- Maternal and Fetal Health Research Centre, Manchester Academic Health Science Centre, University of Manchester, Central Manchester NHS Trust, Manchester, UK
| | - Lucilla Poston
- Department of Women and Children's Health, School of Life Course Sciences, King's College London, London, UK
| | - Basky Thilaganathan
- Fetal Medicine Unit, St George's University Hospitals NHS Foundation Trust and Molecular and Clinical Sciences Research Institute, St George's University of London, London, UK
| | - Anne C Staff
- Division of Obstetrics and Gynaecology, Oslo University Hospital, and Faculty of Medicine, University of Oslo, Oslo, Norway
| | - Gordon C S Smith
- Department of Obstetrics and Gynaecology, NIHR Biomedical Research Centre, Cambridge University, Cambridge, UK
| | - Wessel Ganzevoort
- Department of Obstetrics, Amsterdam UMC University of Amsterdam, Amsterdam, The Netherlands
| | - Hannele Laivuori
- Department of Medical and Clinical Genetics, University of Helsinki and Helsinki University Hospital, Helsinki, Finland
- Institute for Molecular Medicine Finland, Helsinki Institute of Life Science, University of Helsinki, Helsinki, Finland
- Department of Obstetrics and Gynecology, Faculty of Medicine and Health Technology, Tampere University Hospital and Tampere University, Tampere, Finland
| | | | - Javier Arenas Ramírez
- Department of Obstetrics and Gynaecology, University Hospital de Cabueñes, Gijón, Spain
| | - John Kingdom
- Maternal-Fetal Medicine Division, Department OBGYN, Mount Sinai Hospital, University of Toronto, Toronto, Canada
| | - George Daskalakis
- Department of Obstetrics and Gynecology, National and Kapodistrian University of Athens, Alexandra Hospital, Athens, Greece
| | - Diane Farrar
- Bradford Institute for Health Research, Bradford Teaching Hospitals, Bradford, UK
| | - Ahmet A Baschat
- Johns Hopkins Center for Fetal Therapy, Department of Gynecology & Obstetrics, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Paul T Seed
- Department of Women and Children's Health, School of Life Course Sciences, King's College London, London, UK
| | - Federico Prefumo
- Department of Obstetrics and Gynaecology, University of Brescia, Brescia, Italy
| | - Fabricio da Silva Costa
- Department of Gynecology and Obstetrics, Ribeirão Preto Medical School, University of São Paulo, Ribeirão Preto, São Paulo, Brazil
| | - Henk Groen
- Department of Epidemiology, University of Groningen, University Medical Center Groningen, Groningen, The Netherlands
| | - Francois Audibert
- Department of Obstetrics and Gynecology, CHU Ste Justine, Université de Montréal, Montreal, Canada
| | - Jacques Masse
- Department of Molecular Biology, Medical Biochemistry and Pathology, Laval University, Quebec City, Canada
| | - Ragnhild B Skråstad
- Department of Clinical and Molecular Medicine, Faculty of Medicine and Health Sciences, Norwegian University of Science and Technology - NTNU, Trondheim, Norway
- Department of Clinical Pharmacology, St. Olav University Hospital, Trondheim, Norway
| | - Kjell Å Salvesen
- Department of Obstetrics and Gynecology, Trondheim University Hospital, Trondheim, Norway
- Department of Laboratory Medicine, Children's and Women's Health, Norwegian University of Science and Technology, Trondheim, Norway
| | - Camilla Haavaldsen
- Department of Obstetrics and Gynaecology, Akershus University Hospital, Lørenskog, Norway
| | - Chie Nagata
- Department of Education for Clinical Research, National Center for Child Health and Development, Tokyo, Japan
| | - Alice R Rumbold
- South Australian Health and Medical Research Institute and Robinson Research Institute, The University of Adelaide, Adelaide, Australia
| | - Seppo Heinonen
- Department of Obstetrics and Gynaecology, University of Helsinki and Helsinki University Hospital, Helsinki, Finland
| | - Lisa M Askie
- NHMRC Clinical Trials Centre, University of Sydney, Sydney, Australia
| | - Luc J M Smits
- Care and Public Health Research Institute, Maastricht University Medical Centre, Maastricht, The Netherlands
| | - Christina A Vinter
- Department of Gynecology and Obstetrics, Odense University Hospital, University of Southern Denmark, Odense, Denmark
| | - Per Magnus
- Centre for Fertility and Health, Norwegian Institute of Public Health, Oslo, Norway
| | - Kajantie Eero
- National Institute for Health and Welfare, Helsinki, Finland
- Children's Hospital, University of Helsinki and Helsinki University Hospital, Helsinki, Finland
| | - Pia M Villa
- Department of Obstetrics and Gynaecology, University of Helsinki and Helsinki University Hospital, Helsinki, Finland
| | - Anne K Jenum
- General Practice Research Unit (AFE), Department of General Practice, Institute of Health and Society, Faculty of Medicine, University of Oslo, Oslo, Norway
| | - Louise B Andersen
- Institute for Clinical Research, University of Southern Denmark, Odense, Denmark
- Department of Obstetrics and Gynecology, Odense University Hospital, Odense, Denmark
| | - Jane E Norman
- MRC Centre for Reproductive Health, University of Edinburgh, Edinburgh, UK
| | - Akihide Ohkuchi
- Department of Obstetrics and Gynecology, Jichi Medical University School of Medicine, Shimotsuke-shi, Tochigi, Japan
| | - Anne Eskild
- Department of Obstetrics and Gynaecology, Akershus University Hospital, Lørenskog, Norway
- Institute of Clinical Medicine, University of Oslo, Oslo, Norway
| | - Sohinee Bhattacharya
- Obstetrics & Gynaecology, Institute of Applied Health Sciences, School of Medicine, Medical Sciences and Nutrition, University of Aberdeen, Aberdeen, UK
| | - Fionnuala M McAuliffe
- UCD Perinatal Research Centre, School of Medicine, University College Dublin, National Maternity Hospital, Dublin, Ireland
| | - Alberto Galindo
- Fetal Medicine Unit, Maternal and Child Health and Development Network (SAMID), Department of Obstetrics and Gynaecology, Hospital Universitario, Instituto de Investigación Hospital, Universidad Complutense de Madrid, Madrid, Spain
| | - Ignacio Herraiz
- Fetal Medicine Unit, Maternal and Child Health and Development Network (SAMID), Department of Obstetrics and Gynaecology, Hospital Universitario, Instituto de Investigación Hospital, Universidad Complutense de Madrid, Madrid, Spain
| | - Lionel Carbillon
- Department of Obstetrics and Gynecology, Assistance Publique-Hôpitaux de Paris Université Paris, Paris, France
| | - Kerstin Klipstein-Grobusch
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Seon Ae Yeo
- University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Joyce L Browne
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Karel G M Moons
- Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, The Netherlands
- Cochrane Netherlands, Utrecht, The Netherlands
| | - Richard D Riley
- Centre for Prognosis Research, School of Primary, Community and Social Care, Keele University, Keele, UK
| | - Shakila Thangaratinam
- Institute of Metabolism and Systems Research, WHO Collaborating Centre for Women's Health, University of Birmingham, Birmingham, UK
| |
Collapse
|
20
|
A prognostic model predicted deterioration in health-related quality of life in older patients with multimorbidity and polypharmacy. J Clin Epidemiol 2020; 130:1-12. [PMID: 33065164 DOI: 10.1016/j.jclinepi.2020.10.006] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/16/2020] [Revised: 08/12/2020] [Accepted: 10/07/2020] [Indexed: 01/07/2023]
Abstract
OBJECTIVES To develop and validate a prognostic model to predict deterioration in health-related quality of life (dHRQoL) in older general practice patients with at least one chronic condition and one chronic prescription. STUDY DESIGN AND SETTING We used individual participant data from five cluster-randomized trials conducted in the Netherlands and Germany to predict dHRQoL, defined as a decrease in EQ-5D-3 L index score of ≥5% after 6-month follow-up in logistic regression models with stratified intercepts to account for between-study heterogeneity. The model was validated internally and by using internal-external cross-validation (IECV). RESULTS In 3,582 patients with complete data, of whom 1,046 (29.2%) showed deterioration in HRQoL, and 12/87 variables were selected that were related to single (chronic) conditions, inappropriate medication, medication underuse, functional status, well-being, and HRQoL. Bootstrap internal validation showed a C-statistic of 0.71 (0.69 to 0.72) and a calibration slope of 0.88 (0.78 to 0.98). In the IECV loop, the model provided a pooled C-statistic of 0.68 (0.65 to 0.70) and calibration-in-the-large of 0 (-0.13 to 0.13). HRQoL/functionality had the strongest prognostic value. CONCLUSION The model performed well in terms of discrimination, calibration, and generalizability and might help clinicians identify older patients at high risk of dHRQoL. REGISTRATION PROSPERO ID: CRD42018088129.
Collapse
|
21
|
Takada T, van Doorn S, Parpia S, de Wit K, Anderson DR, Stevens SM, Woller SC, Ten Cate-Hoek AJ, Elf JL, Kraaijenhagen RA, Schutgens REG, Wells PS, Kearon C, Moons KGM, Geersing GJ. Diagnosing deep vein thrombosis in cancer patients with suspected symptoms: An individual participant data meta-analysis. J Thromb Haemost 2020; 18:2245-2252. [PMID: 32433797 DOI: 10.1111/jth.14900] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2020] [Revised: 04/23/2020] [Accepted: 05/11/2020] [Indexed: 12/28/2022]
Abstract
BACKGROUND A previous individual participant data (IPD) meta-analysis showed that the Wells rule and D-dimer testing cannot exclude suspected deep vein thrombosis (DVT) in cancer patients. OBJECTIVES To explore reasons for this reduced diagnostic accuracy and to optimize the diagnostic pathway for cancer patients suspected of DVT. PATIENTS AND METHODS Using IPD from 13 studies in patients with suspected DVT, DVT prevalence and the predictive value of the Wells rule items and D-dimer testing were compared between patients with and without cancer. Next, we developed a prediction model with five variables selected from all available diagnostic predictors. RESULTS Among the 10 002 suspected DVT patients, there were 834 patients with cancer. The median prevalence of DVT in these patients with cancer was 37.5% (interquartile range [IQR], 30.8-45.5), whereas it was 15.1% (IQR, 11.5-16.7) in patients without cancer. Diagnostic performance of individual Wells rule items and D-dimer testing was similar across patients with and without cancer, except "immobility" and "history of DVT." The newly developed rule showed a pooled c-statistic 0.80 (95% confidence interval [CI], 0.75-0.83) and good calibration. However, using this model, still only 4.3% (95% CI, 3.0-5.7) of the suspected patients with cancer could be identified with a predicted DVT posttest probability of <2%. CONCLUSIONS Likely because of the high prevalence of DVT, clinical models followed by D-dimer testing fail to rule out DVT efficiently in cancer patients suspected of DVT. Direct referral for compression ultrasonography appears to be the preferred approach for diagnosis of suspected DVT in cancer patients.
Collapse
Affiliation(s)
- Toshihiko Takada
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Sander van Doorn
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Sameer Parpia
- Department of Oncology, McMaster University, Hamilton, Canada
| | - Kerstin de Wit
- Department of Medicine, McMaster University, Hamilton, Canada
| | - David R Anderson
- Division of Haematology, Department of Medicine, Dalhousie University, Halifax, Canada
| | - Scott M Stevens
- Department of Medicine, Intermountain Medical Center, Murray, UT, USA
- Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, USA
| | - Scott C Woller
- Department of Medicine, Intermountain Medical Center, Murray, UT, USA
- Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, USA
| | - Arina J Ten Cate-Hoek
- Department of Internal Medicine, Maastricht University Medical Center, Maastricht, The Netherlands
| | - Johan L Elf
- Vascular Center, Skane University Hospital, Malmö, Sweden
| | | | - Roger E G Schutgens
- Van Creveld Clinic, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Phil S Wells
- Department of Medicine, Ottawa Hospital, University of Ottawa, Ottawa, Canada
| | - Clive Kearon
- Department of Medicine, McMaster University, Hamilton, Canada
| | - Karel G M Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Geert-Jan Geersing
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
22
|
Eriksson LSE, Epstein E, Testa AC, Fischerova D, Valentin L, Sladkevicius P, Franchi D, Frühauf F, Fruscio R, Haak LA, Opolskiene G, Mascilini F, Alcazar JL, Van Holsbeke C, Chiappa V, Bourne T, Lindqvist PG, Van Calster B, Timmerman D, Verbakel JY, Van den Bosch T, Wynants L. Ultrasound-based risk model for preoperative prediction of lymph-node metastases in women with endometrial cancer: model-development study. ULTRASOUND IN OBSTETRICS & GYNECOLOGY : THE OFFICIAL JOURNAL OF THE INTERNATIONAL SOCIETY OF ULTRASOUND IN OBSTETRICS AND GYNECOLOGY 2020; 56:443-452. [PMID: 31840873 DOI: 10.1002/uog.21950] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/25/2019] [Revised: 12/06/2019] [Accepted: 12/07/2019] [Indexed: 06/10/2023]
Abstract
OBJECTIVE To develop a preoperative risk model, using endometrial biopsy results and clinical and ultrasound variables, to predict the individual risk of lymph-node metastases in women with endometrial cancer. METHODS A mixed-effects logistic regression model for prediction of lymph-node metastases was developed in 1501 prospectively included women with endometrial cancer undergoing transvaginal ultrasound examination before surgery, from 16 European centers. Missing data, including missing lymph-node status, were imputed. Discrimination, calibration and clinical utility of the model were evaluated using leave-center-out cross validation. The predictive performance of the model was compared with that of risk classification from endometrial biopsy alone (high-risk defined as endometrioid cancer Grade 3/non-endometrioid cancer) or combined endometrial biopsy and ultrasound (high-risk defined as endometrioid cancer Grade 3/non-endometrioid cancer/deep myometrial invasion/cervical stromal invasion/extrauterine spread). RESULTS Lymphadenectomy was performed in 691 women, of whom 127 had lymph-node metastases. The model for prediction of lymph-node metastases included the predictors age, duration of abnormal bleeding, endometrial biopsy result, tumor extension and tumor size according to ultrasound and undefined tumor with an unmeasurable endometrium. The model's area under the curve was 0.73 (95% CI, 0.68-0.78), the calibration slope was 1.06 (95% CI, 0.79-1.34) and the calibration intercept was 0.06 (95% CI, -0.15 to 0.27). Using a risk threshold for lymph-node metastases of 5% compared with 20%, the model had, respectively, a sensitivity of 98% vs 48% and specificity of 11% vs 80%. The model had higher sensitivity and specificity than did classification as high-risk, according to endometrial biopsy alone (50% vs 35% and 80% vs 77%, respectively) or combined endometrial biopsy and ultrasound (80% vs 75% and 53% vs 52%, respectively). The model's clinical utility was higher than that of endometrial biopsy alone or combined endometrial biopsy and ultrasound at any given risk threshold. CONCLUSIONS Based on endometrial biopsy results and clinical and ultrasound characteristics, the individual risk of lymph-node metastases in women with endometrial cancer can be estimated reliably before surgery. The model is superior to risk classification by endometrial biopsy alone or in combination with ultrasound. Copyright © 2019 ISUOG. Published by John Wiley & Sons Ltd.
Collapse
Affiliation(s)
- L S E Eriksson
- Department of Pelvic Cancer, Karolinska University Hospital, Stockholm, Sweden
- Department of Women's and Children's Health, Karolinska Institutet, Stockholm, Sweden
| | - E Epstein
- Department of Clinical Science and Education, Karolinska Institutet, Stockholm, Sweden
- Department of Obstetrics and Gynecology, Sodersjukhuset, Stockholm, Sweden
| | - A C Testa
- Department of Gynecological Oncology, Catholic University of the Sacred Heart, Rome, Italy
| | - D Fischerova
- Department of Obstetrics and Gynecology, First Faculty of Medicine, Charles University, Prague, Czech Republic
| | - L Valentin
- Department of Obstetrics and Gynecology, Skåne University Hospital, Lund University, Malmö, Sweden
| | - P Sladkevicius
- Department of Obstetrics and Gynecology, Skåne University Hospital, Lund University, Malmö, Sweden
| | - D Franchi
- Department of Gynecological Oncology, European Institute of Oncology, Milan, Italy
| | - F Frühauf
- Department of Obstetrics and Gynecology, First Faculty of Medicine, Charles University, Prague, Czech Republic
| | - R Fruscio
- Clinic of Obstetrics and Gynecology, University of Milan Bicocca, San Gerardo Hospital, Monza, Italy
| | - L A Haak
- Institute for the Care of Mother and Child, Prague, Czech Republic
- Third Faculty of Medicine, Charles University, Prague, Czech Republic
| | - G Opolskiene
- Center of Obstetrics and Gynecology, Vilnius University Hospital Santaros Klinikos, Vilnius, Lithuania
| | - F Mascilini
- Department of Woman and Child Health and Public Health, Fondazione Policlinico Universitario Agostino Gemelli, IRCSS, Rome, Italy
| | - J L Alcazar
- Department of Obstetrics and Gynecology, Clinica Universidad de Navarra, Pamplona, Spain
| | - C Van Holsbeke
- Department of Obstetrics and Gynecology, Ziekenhuis Oost-Limburg, Genk, Belgium
| | - V Chiappa
- Department of Obstetrics and Gynecology, National Cancer Institute, Milan, Italy
| | - T Bourne
- Department of Obstetrics and Gynecology, Queen Charlotte's and Chelsea Hospital, Imperial College London, London, UK
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
| | - P G Lindqvist
- Department of Clinical Science and Education, Karolinska Institutet, Stockholm, Sweden
- Department of Obstetrics and Gynecology, Sodersjukhuset, Stockholm, Sweden
| | - B Van Calster
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
| | - D Timmerman
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- Department of Obstetrics and Gynecology, University Hospital Leuven, Leuven, Belgium
| | - J Y Verbakel
- Department of Public Health and Primary Care, KU Leuven, Leuven, Belgium
- Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
| | - T Van den Bosch
- Department of Obstetrics and Gynecology, University Hospital Leuven, Leuven, Belgium
| | - L Wynants
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
- Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Maastricht, The Netherlands
| |
Collapse
|
23
|
Establishment and evaluation of a multicenter collaborative prediction model construction framework supporting model generalization and continuous improvement: A pilot study. Int J Med Inform 2020; 141:104173. [PMID: 32531725 DOI: 10.1016/j.ijmedinf.2020.104173] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2020] [Revised: 04/10/2020] [Accepted: 05/09/2020] [Indexed: 12/13/2022]
Abstract
BACKGROUND AND OBJECTIVE In recent years, an increasing number of clinical prediction models have been developed to serve clinical care. Establishing a data-driven prediction model based on large-scale electronic health record (EHR) data can provide a more empirical basis for clinical decision making. However, research on model generalization and continuous improvement is insufficiently focused, which also hinders the application and evaluation of prediction models in real clinical environments. Therefore, this study proposes a multicenter collaborative prediction model construction framework to build a prediction model with greater generalizability and continuous improvement capabilities while preserving patient data security and privacy. MATERIALS AND METHODS Based on a multicenter collaborative research network, such as the Observational Health Data Sciences and Informatics (OHDSI), a multicenter collaborative prediction model construction framework is proposed. Based on the idea of multi-source transfer learning, in each source hospital, a base classifier was trained according to the model research setting. Then, in the target hospital with missing calibration data, a prediction model was established through weighted integration of base classifiers from source hospitals based on the smoothness assumption. Moreover, a passive-aggressive online learning algorithm was used for continuous improvement of the prediction model, which can help to maintain a high predictive performance to provide reliable clinical decision-making abilities. To evaluate the proposed prediction model construction framework, a prototype system for colorectal cancer prognosis prediction was developed. To evaluate the performance of models, 70,906 patients were screened, including 70,090 from 5 US hospital-specific datasets and 816 from a Chinese hospital-specific dataset. The area under the receiver operating characteristic curve (AUC) and the estimated calibration index (ECI) were used to evaluate the discrimination and calibration of models. RESULTS Regarding the colorectal cancer prognosis prediction in our prototype system, compared with the reference models, our model achieved a better performance in model calibration (ECI = 9.294 [9.146, 9.441]) and a similar ability in model discrimination (AUC = 0.783 [0.780, 0.786]). Furthermore, the online learning process provided in this study can continuously improve the performance of the prediction model when patient data with specified labels arrive (the AUC value increased from 0.709 to 0.715 and the ECI value decreased from 13.013 to 9.634 after 650 patient instances with specified labels from the Chinese hospital arrived), enabling the prediction model to maintain a good predictive performance during clinical application. CONCLUSIONS This study proposes and evaluates a multicenter collaborative prediction model construction framework that can support the construction of prediction models with better generalizability and continuous improvement capabilities without the need to aggregate multicenter patient-level data.
Collapse
|
24
|
Phillips B, Morgan JE, Haeusler GM, Riley RD. Individual participant data validation of the PICNICC prediction model for febrile neutropenia. Arch Dis Child 2020; 105:439-445. [PMID: 31690548 PMCID: PMC7212933 DOI: 10.1136/archdischild-2019-317308] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/26/2019] [Revised: 09/20/2019] [Accepted: 10/18/2019] [Indexed: 12/23/2022]
Abstract
BACKGROUND Risk-stratified approaches to managing cancer therapies and their consequent complications rely on accurate predictions to work effectively. The risk-stratified management of fever with neutropenia is one such very common area of management in paediatric practice. Such rules are frequently produced and promoted without adequate confirmation of their accuracy. METHODS An individual participant data meta-analytic validation of the 'Predicting Infectious ComplicatioNs In Children with Cancer' (PICNICC) prediction model for microbiologically documented infection in paediatric fever with neutropenia was undertaken. Pooled estimates were produced using random-effects meta-analysis of the area under the curve-receiver operating characteristic curve (AUC-ROC), calibration slope and ratios of expected versus observed cases (E/O). RESULTS The PICNICC model was poorly predictive of microbiologically documented infection (MDI) in these validation cohorts. The pooled AUC-ROC was 0.59, 95% CI 0.41 to 0.78, tau2=0, compared with derivation value of 0.72, 95% CI 0.71 to 0.76. There was poor discrimination (pooled slope estimate 0.03, 95% CI -0.19 to 0.26) and calibration in the large (pooled E/O ratio 1.48, 95% CI 0.87 to 2.1). Three different simple recalibration approaches failed to improve performance meaningfully. CONCLUSION This meta-analysis shows the PICNICC model should not be used at admission to predict MDI. Further work should focus on validating alternative prediction models. Validation across multiple cohorts from diverse locations is essential before widespread clinical adoption of such rules to avoid overtreating or undertreating children with fever with neutropenia.
Collapse
Affiliation(s)
- Bob Phillips
- Centre for Reviews and Dissemination, University of York, York, UK .,Leeds Children's Hospital, Leeds, UK
| | - Jessica Elizabeth Morgan
- Centre for Reviews and Dissemination, University of York, York, UK,Leeds Children's Hospital, Leeds, UK
| | - Gabrielle M Haeusler
- Infectious Diseases and Infection Control, Peter MacCallum Cancer Centre, Melbourne, Victoria, Australia
| | - Richard D Riley
- Research Institute for Primary Care and Health Sciences, Keele University, Keele, UK
| | | |
Collapse
|
25
|
Jiang W, Wang J, Shen X, Lu W, Wang Y, Li W, Gao Z, Xu J, Li X, Liu R, Zheng M, Chang B, Li J, Yang J, Chang B. Establishment and Validation of a Risk Prediction Model for Early Diabetic Kidney Disease Based on a Systematic Review and Meta-Analysis of 20 Cohorts. Diabetes Care 2020; 43:925-933. [PMID: 32198286 DOI: 10.2337/dc19-1897] [Citation(s) in RCA: 86] [Impact Index Per Article: 21.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/24/2019] [Accepted: 12/27/2019] [Indexed: 02/03/2023]
Abstract
BACKGROUND Identifying patients at high risk of diabetic kidney disease (DKD) helps improve clinical outcome. PURPOSE To establish a model for predicting DKD. DATA SOURCES The derivation cohort was from a meta-analysis. The validation cohort was from a Chinese cohort. STUDY SELECTION Cohort studies that reported risk factors of DKD with their corresponding risk ratios (RRs) in patients with type 2 diabetes were selected. All patients had estimated glomerular filtration rate (eGFR) ≥60 mL/min/1.73 m2 and urinary albumin-to-creatinine ratio (UACR) <30 mg/g at baseline. DATA EXTRACTION Risk factors and their corresponding RRs were extracted. Only risk factors with statistical significance were included in our DKD risk prediction model. DATA SYNTHESIS Twenty cohorts including 41,271 patients with type 2 diabetes were included in our meta-analysis. Age, BMI, smoking, diabetic retinopathy, hemoglobin A1c, systolic blood pressure, HDL cholesterol, triglycerides, UACR, and eGFR were statistically significant. All these risk factors were included in the model except eGFR because of the significant heterogeneity among studies. All risk factors were scored according to their weightings, and the highest score was 37.0. The model was validated in an external cohort with a median follow-up of 2.9 years. A cutoff value of 16 was selected with a sensitivity of 0.847 and a specificity of 0.677. LIMITATIONS There was huge heterogeneity among studies involving eGFR. More evidence is needed to power it as a risk factor of DKD. CONCLUSIONS The DKD risk prediction model consisting of nine risk factors established in this study is a simple tool for detecting patients at high risk of DKD.
Collapse
Affiliation(s)
- Wenhui Jiang
- NHC Key Laboratory of Hormones and Development (Tianjin Medical University), Tianjin Key Laboratory of Metabolic Diseases, Tianjin Medical University Chu Hsien-I Memorial Hospital and Tianjin Institute of Endocrinology, Tianjin, China
| | - Jingyu Wang
- NHC Key Laboratory of Hormones and Development (Tianjin Medical University), Tianjin Key Laboratory of Metabolic Diseases, Tianjin Medical University Chu Hsien-I Memorial Hospital and Tianjin Institute of Endocrinology, Tianjin, China
| | - Xiaofang Shen
- NHC Key Laboratory of Hormones and Development (Tianjin Medical University), Tianjin Key Laboratory of Metabolic Diseases, Tianjin Medical University Chu Hsien-I Memorial Hospital and Tianjin Institute of Endocrinology, Tianjin, China
| | - Wenli Lu
- Department of Epidemiology and Health Statistics, School of Public Health, Tianjin Medical University, Tianjin, China
| | - Yuan Wang
- Department of Epidemiology and Health Statistics, School of Public Health, Tianjin Medical University, Tianjin, China
| | - Wen Li
- Department of Epidemiology and Health Statistics, School of Public Health, Tianjin Medical University, Tianjin, China
| | - Zhongai Gao
- NHC Key Laboratory of Hormones and Development (Tianjin Medical University), Tianjin Key Laboratory of Metabolic Diseases, Tianjin Medical University Chu Hsien-I Memorial Hospital and Tianjin Institute of Endocrinology, Tianjin, China
| | - Jie Xu
- NHC Key Laboratory of Hormones and Development (Tianjin Medical University), Tianjin Key Laboratory of Metabolic Diseases, Tianjin Medical University Chu Hsien-I Memorial Hospital and Tianjin Institute of Endocrinology, Tianjin, China
| | - Xiaochen Li
- NHC Key Laboratory of Hormones and Development (Tianjin Medical University), Tianjin Key Laboratory of Metabolic Diseases, Tianjin Medical University Chu Hsien-I Memorial Hospital and Tianjin Institute of Endocrinology, Tianjin, China
| | - Ran Liu
- NHC Key Laboratory of Hormones and Development (Tianjin Medical University), Tianjin Key Laboratory of Metabolic Diseases, Tianjin Medical University Chu Hsien-I Memorial Hospital and Tianjin Institute of Endocrinology, Tianjin, China
| | - Miaoyan Zheng
- NHC Key Laboratory of Hormones and Development (Tianjin Medical University), Tianjin Key Laboratory of Metabolic Diseases, Tianjin Medical University Chu Hsien-I Memorial Hospital and Tianjin Institute of Endocrinology, Tianjin, China
| | - Bai Chang
- NHC Key Laboratory of Hormones and Development (Tianjin Medical University), Tianjin Key Laboratory of Metabolic Diseases, Tianjin Medical University Chu Hsien-I Memorial Hospital and Tianjin Institute of Endocrinology, Tianjin, China
| | - Jing Li
- NHC Key Laboratory of Hormones and Development (Tianjin Medical University), Tianjin Key Laboratory of Metabolic Diseases, Tianjin Medical University Chu Hsien-I Memorial Hospital and Tianjin Institute of Endocrinology, Tianjin, China
| | - Juhong Yang
- NHC Key Laboratory of Hormones and Development (Tianjin Medical University), Tianjin Key Laboratory of Metabolic Diseases, Tianjin Medical University Chu Hsien-I Memorial Hospital and Tianjin Institute of Endocrinology, Tianjin, China
| | - Baocheng Chang
- NHC Key Laboratory of Hormones and Development (Tianjin Medical University), Tianjin Key Laboratory of Metabolic Diseases, Tianjin Medical University Chu Hsien-I Memorial Hospital and Tianjin Institute of Endocrinology, Tianjin, China
| |
Collapse
|
26
|
Comparison of Centor and McIsaac scores in primary care: a meta-analysis over multiple thresholds. Br J Gen Pract 2020; 70:e245-e254. [PMID: 32152041 PMCID: PMC7065683 DOI: 10.3399/bjgp20x708833] [Citation(s) in RCA: 20] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/04/2019] [Accepted: 10/15/2019] [Indexed: 01/14/2023] Open
Abstract
Background Centor and McIsaac scores are both used to diagnose group A beta-haemolytic streptococcus (GABHS) infection, but have not been compared through meta-analysis. Aim To compare the performance of Centor and McIsaac scores at diagnosing patients with GABHS presenting to primary care with pharyngitis. Design and setting A meta-analysis of diagnostic test accuracy studies conducted in primary care was performed using a novel model that incorporates data at multiple thresholds. Method MEDLINE, EMBASE, and PsycINFO were searched for studies published between January 1980 and February 2019. Included studies were: cross-sectional; recruited patients with sore throats from primary care; used the Centor or McIsaac score; had GABHS infection as the target diagnosis; used throat swab culture as the reference standard; and reported 2 × 2 tables across multiple thresholds. Selection and data extraction were conducted by two independent reviewers. QUADAS-2 was used to assess study quality. Summary receiver operating characteristic (SROC) curves were synthesised. Calibration curves were used to assess the transferability of results into practice. Results Ten studies using the Centor score and eight using the McIsaac score were included. The prevalence of GABHS ranged between 4% and 44%. The areas under the SROC curves for McIsaac and Centor scores were 0.7052 and 0.6888, respectively. The P-value for the difference (0.0164) was 0.419, suggesting the SROC curves for the tests are equivalent. Both scores demonstrated poor calibration. Conclusion Both Centor and McIsaac scores provide only fair discrimination of those with and without GABHS, and appear broadly equivalent in performance. The poor calibration for a positive test result suggests other point-of-care tests are required to rule in GABHS; however, with both Centor and McIsaac scores, a score of ≤0 may be sufficient to rule out infection.
Collapse
|
27
|
Multicenter External Validation of the Liverpool Uveal Melanoma Prognosticator Online: An OOG Collaborative Study. Cancers (Basel) 2020; 12:cancers12020477. [PMID: 32085617 PMCID: PMC7072188 DOI: 10.3390/cancers12020477] [Citation(s) in RCA: 20] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2019] [Revised: 01/22/2020] [Accepted: 02/13/2020] [Indexed: 12/19/2022] Open
Abstract
Uveal melanoma (UM) is fatal in ~50% of patients as a result of disseminated disease. This study aims to externally validate the Liverpool Uveal Melanoma Prognosticator Online V3 (LUMPO3) to determine its reliability in predicting survival after treatment for choroidal melanoma when utilizing external data from other ocular oncology centers. Anonymized data of 1836 UM patients from seven international ocular oncology centers were analyzed with LUMPO3 to predict the 10-year survival for each patient in each external dataset. The analysts were masked to the patient outcomes. Model predictions were sent to an independent statistician to evaluate LUMPO3’s performance using discrimination and calibration methods. LUMPO3’s ability to discriminate between UM patients who died of metastatic UM and those who were still alive was fair-to-good, with C-statistics ranging from 0.64 to 0.85 at year 1. The pooled estimate for all external centers was 0.72 (95% confidence interval: 0.68 to 0.75). Agreement between observed and predicted survival probabilities was generally good given differences in case mix and survival rates between different centers. Despite the differences between the international cohorts of patients with primary UM, LUMPO3 is a valuable tool for predicting all-cause mortality in this disease when using data from external centers.
Collapse
|
28
|
Giardiello D, Steyerberg EW, Hauptmann M, Adank MA, Akdeniz D, Blomqvist C, Bojesen SE, Bolla MK, Brinkhuis M, Chang-Claude J, Czene K, Devilee P, Dunning AM, Easton DF, Eccles DM, Fasching PA, Figueroa J, Flyger H, García-Closas M, Haeberle L, Haiman CA, Hall P, Hamann U, Hopper JL, Jager A, Jakubowska A, Jung A, Keeman R, Kramer I, Lambrechts D, Le Marchand L, Lindblom A, Lubiński J, Manoochehri M, Mariani L, Nevanlinna H, Oldenburg HSA, Pelders S, Pharoah PDP, Shah M, Siesling S, Smit VTHBM, Southey MC, Tapper WJ, Tollenaar RAEM, van den Broek AJ, van Deurzen CHM, van Leeuwen FE, van Ongeval C, Van't Veer LJ, Wang Q, Wendt C, Westenend PJ, Hooning MJ, Schmidt MK. Prediction and clinical utility of a contralateral breast cancer risk model. Breast Cancer Res 2019; 21:144. [PMID: 31847907 PMCID: PMC6918633 DOI: 10.1186/s13058-019-1221-1] [Citation(s) in RCA: 25] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2019] [Accepted: 10/29/2019] [Indexed: 02/08/2023] Open
Abstract
BACKGROUND Breast cancer survivors are at risk for contralateral breast cancer (CBC), with the consequent burden of further treatment and potentially less favorable prognosis. We aimed to develop and validate a CBC risk prediction model and evaluate its applicability for clinical decision-making. METHODS We included data of 132,756 invasive non-metastatic breast cancer patients from 20 studies with 4682 CBC events and a median follow-up of 8.8 years. We developed a multivariable Fine and Gray prediction model (PredictCBC-1A) including patient, primary tumor, and treatment characteristics and BRCA1/2 germline mutation status, accounting for the competing risks of death and distant metastasis. We also developed a model without BRCA1/2 mutation status (PredictCBC-1B) since this information was available for only 6% of patients and is routinely unavailable in the general breast cancer population. Prediction performance was evaluated using calibration and discrimination, calculated by a time-dependent area under the curve (AUC) at 5 and 10 years after diagnosis of primary breast cancer, and an internal-external cross-validation procedure. Decision curve analysis was performed to evaluate the net benefit of the model to quantify clinical utility. RESULTS In the multivariable model, BRCA1/2 germline mutation status, family history, and systemic adjuvant treatment showed the strongest associations with CBC risk. The AUC of PredictCBC-1A was 0.63 (95% prediction interval (PI) at 5 years, 0.52-0.74; at 10 years, 0.53-0.72). Calibration-in-the-large was -0.13 (95% PI: -1.62-1.37), and the calibration slope was 0.90 (95% PI: 0.73-1.08). The AUC of Predict-1B at 10 years was 0.59 (95% PI: 0.52-0.66); calibration was slightly lower. Decision curve analysis for preventive contralateral mastectomy showed potential clinical utility of PredictCBC-1A between thresholds of 4-10% 10-year CBC risk for BRCA1/2 mutation carriers and non-carriers. CONCLUSIONS We developed a reasonably calibrated model to predict the risk of CBC in women of European-descent; however, prediction accuracy was moderate. Our model shows potential for improved risk counseling, but decision-making regarding contralateral preventive mastectomy, especially in the general breast cancer population where limited information of the mutation status in BRCA1/2 is available, remains challenging.
Collapse
Affiliation(s)
- Daniele Giardiello
- Division of Molecular Pathology, The Netherlands Cancer Institute - Antoni van Leeuwenhoek Hospital, Amsterdam, The Netherlands
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands
| | - Ewout W Steyerberg
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands
- Department of Public Health, Erasmus MC Cancer Institute, Rotterdam, The Netherlands
| | - Michael Hauptmann
- Institute of Biometry and Registry Research, Brandenburg Medical School, Neuruppin, Germany
- Department of Epidemiology and Biostatistics, The Netherlands Cancer Institute - Antoni van Leeuwenhoek Hospital, Amsterdam, The Netherlands
| | - Muriel A Adank
- The Netherlands Cancer Institute - Antoni van Leeuwenhoek hospital, Family Cancer Clinic, Amsterdam, The Netherlands
| | - Delal Akdeniz
- Department of Medical Oncology, Family Cancer Clinic, Erasmus MC Cancer Institute, Rotterdam, The Netherlands
| | - Carl Blomqvist
- Department of Oncology, Helsinki University Hospital, University of Helsinki, Helsinki, Finland
- Department of Oncology, Örebro University Hospital, Örebro, Sweden
| | - Stig E Bojesen
- Copenhagen General Population Study, Herlev and Gentofte Hospital, Copenhagen University Hospital, Herlev, Denmark
- Department of Clinical Biochemistry, Herlev and Gentofte Hospital, Copenhagen University Hospital, Herlev, Denmark
- Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | - Manjeet K Bolla
- Centre for Cancer Genetic Epidemiology, Department of Public Health and Primary Care, University of Cambridge, Cambridge, UK
| | - Mariël Brinkhuis
- East-Netherlands, Laboratory for Pathology, Hengelo, The Netherlands
| | - Jenny Chang-Claude
- Division of Cancer Epidemiology, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Cancer Epidemiology Group, University Cancer Center Hamburg (UCCH), University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - Kamila Czene
- Department of Medical Epidemiology and Biostatistics, Karolinska Institute, Stockholm, Sweden
| | - Peter Devilee
- Department of Pathology, Leiden University Medical Center, Leiden, The Netherlands
- Department of Human Genetics, Leiden University Medical Center, Leiden, The Netherlands
| | - Alison M Dunning
- Centre for Cancer Genetic Epidemiology, Department of Oncology, University of Cambridge, Cambridge, UK
| | - Douglas F Easton
- Centre for Cancer Genetic Epidemiology, Department of Public Health and Primary Care, University of Cambridge, Cambridge, UK
- Centre for Cancer Genetic Epidemiology, Department of Oncology, University of Cambridge, Cambridge, UK
| | - Diana M Eccles
- Cancer Sciences Academic Unit, Faculty of Medicine, University of Southampton, Southampton, UK
| | - Peter A Fasching
- Department of Medicine Division of Hematology and Oncology, University of California at Los Angeles, David Geffen School of Medicine, Los Angeles, CA, USA
- Department of Gynecology and Obstetrics, Comprehensive Cancer Center ER-EMN, University Hospital Erlangen, Friedrich-Alexander-University Erlangen-Nuremberg, Erlangen, Germany
| | - Jonine Figueroa
- Usher Institute of Population Health Sciences and Informatics, The University of Edinburgh Medical School, Edinburgh, UK
- Cancer Research UK Edinburgh Centre, Edinburgh, UK
- Department of Health and Human Services, Division of Cancer Epidemiology and Genetics, National Cancer Institute, National Institutes of Health, Bethesda, MD, USA
| | - Henrik Flyger
- Department of Breast Surgery, Herlev and Gentofte Hospital, Copenhagen University Hospital, Herlev, Denmark
| | - Montserrat García-Closas
- Department of Health and Human Services, Division of Cancer Epidemiology and Genetics, National Cancer Institute, National Institutes of Health, Bethesda, MD, USA
- Division of Genetics and Epidemiology, Institute of Cancer Research, London, UK
| | - Lothar Haeberle
- Department of Gynecology and Obstetrics, Comprehensive Cancer Center ER-EMN, University Hospital Erlangen, Friedrich-Alexander-University Erlangen-Nuremberg, Erlangen, Germany
| | - Christopher A Haiman
- Department of Preventive Medicine, Keck School of Medicine, University of Southern California, Los Angeles, CA, USA
| | - Per Hall
- Department of Medical Epidemiology and Biostatistics, Karolinska Institute, Stockholm, Sweden
- Department of Oncology, Södersjukhuset, Stockholm, Sweden
| | - Ute Hamann
- Molecular Genetics of Breast Cancer, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - John L Hopper
- Centre for Epidemiology and Biostatistics, Melbourne School of Population and Global Health, The University of Melbourne, Melbourne, Victoria, Australia
| | - Agnes Jager
- Department of Medical Oncology, Erasmus MC Cancer Institute, Rotterdam, The Netherlands
| | - Anna Jakubowska
- Department of Genetics and Pathology, Pomeranian Medical University, Szczecin, Poland
- Independent Laboratory of Molecular Biology and Genetic Diagnostics, Pomeranian Medical University, Szczecin, Poland
| | - Audrey Jung
- Division of Cancer Epidemiology, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Renske Keeman
- Division of Molecular Pathology, The Netherlands Cancer Institute - Antoni van Leeuwenhoek Hospital, Amsterdam, The Netherlands
| | - Iris Kramer
- Division of Molecular Pathology, The Netherlands Cancer Institute - Antoni van Leeuwenhoek Hospital, Amsterdam, The Netherlands
| | - Diether Lambrechts
- VIB Center for Cancer Biology, VIB, Leuven, Belgium
- Laboratory for Translational Genetics, Department of Human Genetics, University of Leuven, Leuven, Belgium
| | - Loic Le Marchand
- University of Hawaii Cancer Center, Epidemiology Program, Honolulu, HI, USA
| | - Annika Lindblom
- Department of Molecular Medicine and Surgery, Karolinska Institutet, Stockholm, Sweden
- Department of Clinical Genetics, Karolinska University Hospital, Stockholm, Sweden
| | - Jan Lubiński
- Department of Genetics and Pathology, Pomeranian Medical University, Szczecin, Poland
| | - Mehdi Manoochehri
- Molecular Genetics of Breast Cancer, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Luigi Mariani
- Unit of Clinical Epidemiology and Trial Organization, Fondazione IRCCS Istituto Nazionale dei Tumori, Milan, Italy
| | - Heli Nevanlinna
- Department of Obstetrics and Gynecology, Helsinki University Hospital, University of Helsinki, Helsinki, Finland
| | - Hester S A Oldenburg
- Department of Surgical Oncology, The Netherlands Cancer Institute - Antoni van Leeuwenhoek Hospital, Amsterdam, The Netherlands
| | - Saskia Pelders
- Department of Medical Oncology, Family Cancer Clinic, Erasmus MC Cancer Institute, Rotterdam, The Netherlands
| | - Paul D P Pharoah
- Centre for Cancer Genetic Epidemiology, Department of Public Health and Primary Care, University of Cambridge, Cambridge, UK
- Centre for Cancer Genetic Epidemiology, Department of Oncology, University of Cambridge, Cambridge, UK
| | - Mitul Shah
- Centre for Cancer Genetic Epidemiology, Department of Oncology, University of Cambridge, Cambridge, UK
| | - Sabine Siesling
- Department of Research, Netherlands Comprehensive Cancer Organisation, Utrecht, The Netherlands
| | - Vincent T H B M Smit
- Department of Pathology, Leiden University Medical Center, Leiden, The Netherlands
| | - Melissa C Southey
- Precision Medicine, School of Clinical Sciences at Monash Health, Monash University, Clayton, Victoria, Australia
- Department of Clinical Pathology, The University of Melbourne, Melbourne, Victoria, Australia
| | | | - Rob A E M Tollenaar
- Department of Surgery, Leiden University Medical Center, Leiden, The Netherlands
| | - Alexandra J van den Broek
- Division of Molecular Pathology, The Netherlands Cancer Institute - Antoni van Leeuwenhoek Hospital, Amsterdam, The Netherlands
| | | | - Flora E van Leeuwen
- Division of Psychosocial Research and Epidemiology, The Netherlands Cancer Institute - Antoni van Leeuwenhoek Hospital, Plesmanlaan 121, 1066, CX, Amsterdam, The Netherlands
| | - Chantal van Ongeval
- Leuven Multidisciplinary Breast Center, Department of Oncology, Leuven Cancer Institute, University Hospitals Leuven, Leuven, Belgium
| | - Laura J Van't Veer
- Division of Molecular Pathology, The Netherlands Cancer Institute - Antoni van Leeuwenhoek Hospital, Amsterdam, The Netherlands
| | - Qin Wang
- Centre for Cancer Genetic Epidemiology, Department of Public Health and Primary Care, University of Cambridge, Cambridge, UK
| | - Camilla Wendt
- Department of Clinical Science and Education, Södersjukhuset, Karolinska Institutet, Stockholm, Sweden
| | | | - Maartje J Hooning
- Department of Medical Oncology, Family Cancer Clinic, Erasmus MC Cancer Institute, Rotterdam, The Netherlands
| | - Marjanka K Schmidt
- Division of Molecular Pathology, The Netherlands Cancer Institute - Antoni van Leeuwenhoek Hospital, Amsterdam, The Netherlands.
- Division of Psychosocial Research and Epidemiology, The Netherlands Cancer Institute - Antoni van Leeuwenhoek Hospital, Plesmanlaan 121, 1066, CX, Amsterdam, The Netherlands.
| |
Collapse
|
29
|
Stevens RJ, Poppe KK. Validation of clinical prediction models: what does the "calibration slope" really measure? J Clin Epidemiol 2019; 118:93-99. [PMID: 31605731 DOI: 10.1016/j.jclinepi.2019.09.016] [Citation(s) in RCA: 29] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/23/2019] [Revised: 08/22/2019] [Accepted: 09/19/2019] [Indexed: 10/25/2022]
Abstract
BACKGROUND AND OBJECTIVES Definitions of calibration, an aspect of model validation, have evolved over time. We examine use and interpretation of the statistic currently referred to as the calibration slope. METHODS The history of the term "calibration slope", and usage in papers published in 2016 and 2017, were reviewed. The behaviour of the slope in illustrative hypothetical examples and in two examples in the clinical literature was demonstrated. RESULTS The paper in which the statistic was proposed described it as a measure of "spread" and did not use the term "calibration". In illustrative examples, slope of 1 can be associated with good or bad calibration, and this holds true across different definitions of calibration. In data extracted from a previous study, the slope was correlated with discrimination, not overall calibration. Many authors of recent papers interpret the slope as a measure of calibration; a minority interpret it as a measure of discrimination or do not explicitly categorise it as either. Seventeen of thirty-three papers used the slope as the sole measure of calibration. CONCLUSION Misunderstanding about this statistic has led to many papers in which it is the sole measure of calibration, which should be discouraged.
Collapse
Affiliation(s)
- Richard J Stevens
- Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK.
| | - Katrina K Poppe
- Faculty of Medical and Health Sciences, University of Auckland, Auckland, New Zealand
| |
Collapse
|
30
|
Sprügel MI, Kuramatsu JB, Volbers B, Gerner ST, Sembill JA, Madžar D, Bobinger T, Kölbl K, Hoelter P, Lücking H, Dörfler A, Schwab S, Huttner HB. Perihemorrhagic edema. Neurology 2019; 93:e1159-e1170. [DOI: 10.1212/wnl.0000000000008129] [Citation(s) in RCA: 22] [Impact Index Per Article: 4.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/27/2018] [Accepted: 04/24/2019] [Indexed: 01/24/2023] Open
Abstract
ObjectiveTo determine the influence of intracerebral hemorrhage (ICH) location and volume and hematoma surface on perihemorrhagic edema evolution.MethodsPatients with ICH of the prospective Universitätsklinikum Erlangen Cohort of Patients With Spontaneous Intracerebral Hemorrhage (UKER-ICH) cohort study (NCT03183167) between 2010 and 2013 were analyzed. Hematoma and edema volume during hospital stay were volumetrically assessed, and time course of edema evolution and peak edema correlated to hematoma volume, location, and surface to verify the strength of the parameters on edema evolution.ResultsOverall, 300 patients with supratentorial ICH were analyzed. Peak edema showed a high correlation with hematoma surface (R2 = 0.864, p < 0.001) rather than with hematoma volumes, regardless of hematoma location. Smaller hematomas with a higher ratio of hematoma surface to volume showed exponentially higher relative edema (R2 = 0.755, p < 0.001). Multivariable logistic regression analysis revealed a cutoff ICH volume of 30 mL, beyond which an increase of total mass lesion volume (combined volume of hematoma and edema) was not associated with worse functional outcome. Specifically, peak edema was associated with worse functional outcome in ICH <30 mL (odds ratio [OR] 2.63, 95% confidence interval [CI] 1.68–4.12, p < 0.001), contrary to ICH ≥30 mL (OR 1.20, 95% CI 0.88–1.63, p = 0.247). There were no significant differences between patients with lobar and those with deep ICH after adjustment for hematoma volumes.ConclusionsPeak perihemorrhagic edema, although influencing mortality, is not associated with worse functional outcomes in ICH volumes >30 mL. Although hematoma volume correlates with peak edema extent, hematoma surface is the major parameter for edema evolution. The effect of edema on functional outcome is therefore more pronounced in smaller and irregularly shaped hematomas, and these patients may particularly benefit from edema-modifying therapies.
Collapse
|
31
|
Debray TP, de Jong VM, Moons KG, Riley RD. Evidence synthesis in prognosis research. Diagn Progn Res 2019; 3:13. [PMID: 31338426 PMCID: PMC6621956 DOI: 10.1186/s41512-019-0059-4] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/23/2018] [Accepted: 04/16/2019] [Indexed: 12/11/2022] Open
Abstract
Over the past few years, evidence synthesis has become essential to investigate and improve the generalizability of medical research findings. This strategy often involves a meta-analysis to formally summarize quantities of interest, such as relative treatment effect estimates. The use of meta-analysis methods is, however, less straightforward in prognosis research because substantial variation exists in research objectives, analysis methods and the level of reported evidence. We present a gentle overview of statistical methods that can be used to summarize data of prognostic factor and prognostic model studies. We discuss how aggregate data, individual participant data, or a combination thereof can be combined through meta-analysis methods. Recent examples are provided throughout to illustrate the various methods.
Collapse
Affiliation(s)
- Thomas P.A. Debray
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Universiteitsweg 100, Utrecht, 3584 CG The Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Universiteitsweg 100, Utrecht, 3584 CG The Netherlands
| | - Valentijn M.T. de Jong
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Universiteitsweg 100, Utrecht, 3584 CG The Netherlands
| | - Karel G.M. Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Universiteitsweg 100, Utrecht, 3584 CG The Netherlands
- Cochrane Netherlands, University Medical Center Utrecht, Universiteitsweg 100, Utrecht, 3584 CG The Netherlands
| | - Richard D. Riley
- Research Institute for Primary Care & Health Sciences, Keele University, Staffordshire, ST5 5BG UK
| |
Collapse
|
32
|
Damen JA, Pajouheshnia R, Heus P, Moons KGM, Reitsma JB, Scholten RJPM, Hooft L, Debray TPA. Performance of the Framingham risk models and pooled cohort equations for predicting 10-year risk of cardiovascular disease: a systematic review and meta-analysis. BMC Med 2019; 17:109. [PMID: 31189462 PMCID: PMC6563379 DOI: 10.1186/s12916-019-1340-7] [Citation(s) in RCA: 112] [Impact Index Per Article: 22.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/05/2019] [Accepted: 05/07/2019] [Indexed: 12/11/2022] Open
Abstract
BACKGROUND The Framingham risk models and pooled cohort equations (PCE) are widely used and advocated in guidelines for predicting 10-year risk of developing coronary heart disease (CHD) and cardiovascular disease (CVD) in the general population. Over the past few decades, these models have been extensively validated within different populations, which provided mounting evidence that local tailoring is often necessary to obtain accurate predictions. The objective is to systematically review and summarize the predictive performance of three widely advocated cardiovascular risk prediction models (Framingham Wilson 1998, Framingham ATP III 2002 and PCE 2013) in men and women separately, to assess the generalizability of performance across different subgroups and geographical regions, and to determine sources of heterogeneity in the findings across studies. METHODS A search was performed in October 2017 to identify studies investigating the predictive performance of the aforementioned models. Studies were included if they externally validated one or more of the original models in the general population for the same outcome as the original model. We assessed risk of bias for each validation and extracted data on population characteristics and model performance. Performance estimates (observed versus expected (OE) ratio and c-statistic) were summarized using a random effects models and sources of heterogeneity were explored with meta-regression. RESULTS The search identified 1585 studies, of which 38 were included, describing a total of 112 external validations. Results indicate that, on average, all models overestimate the 10-year risk of CHD and CVD (pooled OE ratio ranged from 0.58 (95% CI 0.43-0.73; Wilson men) to 0.79 (95% CI 0.60-0.97; ATP III women)). Overestimation was most pronounced for high-risk individuals and European populations. Further, discriminative performance was better in women for all models. There was considerable heterogeneity in the c-statistic between studies, likely due to differences in population characteristics. CONCLUSIONS The Framingham Wilson, ATP III and PCE discriminate comparably well but all overestimate the risk of developing CVD, especially in higher risk populations. Because the extent of miscalibration substantially varied across settings, we highly recommend that researchers further explore reasons for overprediction and that the models be updated for specific populations.
Collapse
Affiliation(s)
- Johanna A Damen
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands. .,Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, P.O. Box 85500, Str. 6.131, 3508, GA, Utrecht, The Netherlands.
| | - Romin Pajouheshnia
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, P.O. Box 85500, Str. 6.131, 3508, GA, Utrecht, The Netherlands
| | - Pauline Heus
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.,Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, P.O. Box 85500, Str. 6.131, 3508, GA, Utrecht, The Netherlands
| | - Karel G M Moons
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.,Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, P.O. Box 85500, Str. 6.131, 3508, GA, Utrecht, The Netherlands
| | - Johannes B Reitsma
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.,Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, P.O. Box 85500, Str. 6.131, 3508, GA, Utrecht, The Netherlands
| | - Rob J P M Scholten
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.,Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, P.O. Box 85500, Str. 6.131, 3508, GA, Utrecht, The Netherlands
| | - Lotty Hooft
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.,Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, P.O. Box 85500, Str. 6.131, 3508, GA, Utrecht, The Netherlands
| | - Thomas P A Debray
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.,Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, P.O. Box 85500, Str. 6.131, 3508, GA, Utrecht, The Netherlands
| |
Collapse
|
33
|
Archer R, Hock E, Hamilton J, Stevens J, Essat M, Poku E, Clowes M, Pandor A, Stevenson M. Assessing prognosis and prediction of treatment response in early rheumatoid arthritis: systematic reviews. Health Technol Assess 2019; 22:1-294. [PMID: 30501821 DOI: 10.3310/hta22660] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/20/2022] Open
Abstract
BACKGROUND Rheumatoid arthritis (RA) is a chronic, debilitating disease associated with reduced quality of life and substantial costs. It is unclear which tests and assessment tools allow the best assessment of prognosis in people with early RA and whether or not variables predict the response of patients to different drug treatments. OBJECTIVE To systematically review evidence on the use of selected tests and assessment tools in patients with early RA (1) in the evaluation of a prognosis (review 1) and (2) as predictive markers of treatment response (review 2). DATA SOURCES Electronic databases (e.g. MEDLINE, EMBASE, The Cochrane Library, Web of Science Conference Proceedings; searched to September 2016), registers, key websites, hand-searching of reference lists of included studies and key systematic reviews and contact with experts. STUDY SELECTION Review 1 - primary studies on the development, external validation and impact of clinical prediction models for selected outcomes in adult early RA patients. Review 2 - primary studies on the interaction between selected baseline covariates and treatment (conventional and biological disease-modifying antirheumatic drugs) on salient outcomes in adult early RA patients. RESULTS Review 1 - 22 model development studies and one combined model development/external validation study reporting 39 clinical prediction models were included. Five external validation studies evaluating eight clinical prediction models for radiographic joint damage were also included. c-statistics from internal validation ranged from 0.63 to 0.87 for radiographic progression (different definitions, six studies) and 0.78 to 0.82 for the Health Assessment Questionnaire (HAQ). Predictive performance in external validations varied considerably. Three models [(1) Active controlled Study of Patients receiving Infliximab for the treatment of Rheumatoid arthritis of Early onset (ASPIRE) C-reactive protein (ASPIRE CRP), (2) ASPIRE erythrocyte sedimentation rate (ASPIRE ESR) and (3) Behandelings Strategie (BeSt)] were externally validated using the same outcome definition in more than one population. Results of the random-effects meta-analysis suggested substantial uncertainty in the expected predictive performance of models in a new sample of patients. Review 2 - 12 studies were identified. Covariates examined included anti-citrullinated protein/peptide anti-body (ACPA) status, smoking status, erosions, rheumatoid factor status, C-reactive protein level, erythrocyte sedimentation rate, swollen joint count (SJC), body mass index and vascularity of synovium on power Doppler ultrasound (PDUS). Outcomes examined included erosions/radiographic progression, disease activity, physical function and Disease Activity Score-28 remission. There was statistical evidence to suggest that ACPA status, SJC and PDUS status at baseline may be treatment effect modifiers, but not necessarily that they are prognostic of response for all treatments. Most of the results were subject to considerable uncertainty and were not statistically significant. LIMITATIONS The meta-analysis in review 1 was limited by the availability of only a small number of external validation studies. Studies rarely investigated the interaction between predictors and treatment. SUGGESTED RESEARCH PRIORITIES Collaborative research (including the use of individual participant data) is needed to further develop and externally validate the clinical prediction models. The clinical prediction models should be validated with respect to individual treatments. Future assessments of treatment by covariate interactions should follow good statistical practice. CONCLUSIONS Review 1 - uncertainty remains over the optimal prediction model(s) for use in clinical practice. Review 2 - in general, there was insufficient evidence that the effect of treatment depended on baseline characteristics. STUDY REGISTRATION This study is registered as PROSPERO CRD42016042402. FUNDING The National Institute for Health Research Health Technology Assessment programme.
Collapse
Affiliation(s)
- Rachel Archer
- School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK
| | - Emma Hock
- School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK
| | - Jean Hamilton
- School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK
| | - John Stevens
- School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK
| | - Munira Essat
- School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK
| | - Edith Poku
- School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK
| | - Mark Clowes
- School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK
| | - Abdullah Pandor
- School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK
| | - Matt Stevenson
- School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK
| |
Collapse
|
34
|
Damen JAAG, Debray TPA, Pajouheshnia R, Reitsma JB, Scholten RJPM, Moons KGM, Hooft L. Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study. BMJ Open 2019; 9:e026160. [PMID: 30940759 PMCID: PMC6500242 DOI: 10.1136/bmjopen-2018-026160] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/22/2018] [Revised: 11/05/2018] [Accepted: 02/04/2019] [Indexed: 12/15/2022] Open
Abstract
OBJECTIVES To empirically assess the relation between study characteristics and prognostic model performance in external validation studies of multivariable prognostic models. DESIGN Meta-epidemiological study. DATA SOURCES AND STUDY SELECTION On 16 October 2018, we searched electronic databases for systematic reviews of prognostic models. Reviews from non-overlapping clinical fields were selected if they reported common performance measures (either the concordance (c)-statistic or the ratio of observed over expected number of events (OE ratio)) from 10 or more validations of the same prognostic model. DATA EXTRACTION AND ANALYSES Study design features, population characteristics, methods of predictor and outcome assessment, and the aforementioned performance measures were extracted from the included external validation studies. Random effects meta-regression was used to quantify the association between the study characteristics and model performance. RESULTS We included 10 systematic reviews, describing a total of 224 external validations, of which 221 reported c-statistics and 124 OE ratios. Associations between study characteristics and model performance were heterogeneous across systematic reviews. C-statistics were most associated with variation in population characteristics, outcome definitions and measurement and predictor substitution. For example, validations with eligibility criteria comparable to the development study were associated with higher c-statistics compared with narrower criteria (difference in logit c-statistic 0.21(95% CI 0.07 to 0.35), similar to an increase from 0.70 to 0.74). Using a case-control design was associated with higher OE ratios, compared with using data from a cohort (difference in log OE ratio 0.97(95% CI 0.38 to 1.55), similar to an increase in OE ratio from 1.00 to 2.63). CONCLUSIONS Variation in performance of prognostic models across studies is mainly associated with variation in case-mix, study designs, outcome definitions and measurement methods and predictor substitution. Researchers developing and validating prognostic models should realise the potential influence of these study characteristics on the predictive performance of prognostic models.
Collapse
Affiliation(s)
- Johanna A A G Damen
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Thomas P A Debray
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Romin Pajouheshnia
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Johannes B Reitsma
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Rob J P M Scholten
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Karel G M Moons
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| | - Lotty Hooft
- Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
35
|
Shinohara K, Tanaka S, Imai H, Noma H, Maruo K, Cipriani A, Yamawaki S, Furukawa TA. Development and validation of a prediction model for the probability of responding to placebo in antidepressant trials: a pooled analysis of individual patient data. EVIDENCE-BASED MENTAL HEALTH 2019; 22:10-16. [PMID: 30665989 PMCID: PMC10270413 DOI: 10.1136/ebmental-2018-300073] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/25/2018] [Revised: 12/12/2018] [Accepted: 12/21/2018] [Indexed: 12/11/2022]
Abstract
BACKGROUND Identifying potential placebo responders among apparent drug responders is critical to dissect drug-specific and nonspecific effects in depression. OBJECTIVE This project aimed to develop and test a prediction model for the probability of responding to placebo in antidepressant trials. Such a model will allow us to estimate the probability of placebo response among drug responders in antidepressants trials. METHODS We identified all placebo-controlled, double-blind randomised controlled trials (RCTs) of second generation antidepressants for major depressive disorder conducted in Japan and requested their individual patient data (IPD) to pharmaceutical companies. We obtained IPD (n=1493) from four phase II/III RCTs comparing mirtazapine, escitalopram, duloxetine, paroxetine and placebo. Out of 1493 participants in the four clinical trials, 440 participants allocated to placebo were included in the analyses. Our primary outcome was response, defined as 50% or greater reduction on Hamilton Rating Scale for Depression at study endpoint. We used multivariable logistic regression to develop a prediction model. All available candidate of predictor variables were tested through a backward variable selection and covariates were selected for the prediction model. The performance of the model was assessed by using Hosmer-Lemeshow test for calibration and the area under the ROC curve for discrimination. FINDINGS Placebo response rates differed between 31% and 59% (grand average: 43%) among four trials. Four variables were selected from all candidate variables and included in the final model: age at onset, age at baseline, bodily symptoms, and study-level difference. The final model performed satisfactorily in terms of calibration (Hosmer-Lemeshow p=0.92) and discrimination (the area under the ROC curve (AUC): 0.70). CONCLUSIONS Our model is expected to help researchers discriminate individuals who are more likely to respond to placebo from those who are less likely so. CLINICAL IMPLICATIONS A larger sample and more precise individual participant information should be collected for better performance. Examination of external validity in independent datasets is warranted. TRIAL REGISTRATION NUMBER CRD42017055912.
Collapse
Affiliation(s)
- Kiyomi Shinohara
- Department of Health Promotion and Human Behavior and of Clinical Epidemiology, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan
| | - Shiro Tanaka
- Department of Clinical Biostatistics, Kyoto University Graduate School of Medicine, Kyoto, Japan
| | - Hissei Imai
- Department of Health Promotion and Human Behavior and of Clinical Epidemiology, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan
| | - Hisashi Noma
- Department of Data Science, The Institute of Statistical Mathematics, Tokyo, Japan
| | - Kazushi Maruo
- Department of Biostatistics, Faculty of Medicine, University of Tsukuba, Tsukuba, Japan
| | - Andrea Cipriani
- Department of Psychiatry, University of Oxford, Oxford, UK
- Oxford Health NHS Foundation Trust, Warneford Hospital, Oxford, UK
| | - Shigeto Yamawaki
- Academic-Industrial Cooperation Office, Hiroshima University, Hiroshima, Japan
| | - Toshi A Furukawa
- Department of Health Promotion and Human Behavior and of Clinical Epidemiology, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan
| |
Collapse
|
36
|
Moons KGM, Wolff RF, Riley RD, Whiting PF, Westwood M, Collins GS, Reitsma JB, Kleijnen J, Mallett S. PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration. Ann Intern Med 2019; 170:W1-W33. [PMID: 30596876 DOI: 10.7326/m18-1377] [Citation(s) in RCA: 652] [Impact Index Per Article: 130.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/22/2022] Open
Abstract
Prediction models in health care use predictors to estimate for an individual the probability that a condition or disease is already present (diagnostic model) or will occur in the future (prognostic model). Publications on prediction models have become more common in recent years, and competing prediction models frequently exist for the same outcome or target population. Health care providers, guideline developers, and policymakers are often unsure which model to use or recommend, and in which persons or settings. Hence, systematic reviews of these studies are increasingly demanded, required, and performed. A key part of a systematic review of prediction models is examination of risk of bias and applicability to the intended population and setting. To help reviewers with this process, the authors developed PROBAST (Prediction model Risk Of Bias ASsessment Tool) for studies developing, validating, or updating (for example, extending) prediction models, both diagnostic and prognostic. PROBAST was developed through a consensus process involving a group of experts in the field. It includes 20 signaling questions across 4 domains (participants, predictors, outcome, and analysis). This explanation and elaboration document describes the rationale for including each domain and signaling question and guides researchers, reviewers, readers, and guideline developers in how to use them to assess risk of bias and applicability concerns. All concepts are illustrated with published examples across different topics. The latest version of the PROBAST checklist, accompanying documents, and filled-in examples can be downloaded from www.probast.org.
Collapse
Affiliation(s)
- Karel G M Moons
- Julius Center for Health Sciences and Primary Care and Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands (K.G.M., J.B.R.)
| | - Robert F Wolff
- Kleijnen Systematic Reviews, York, United Kingdom (R.F.W., M.W.)
| | - Richard D Riley
- Centre for Prognosis Research, Research Institute for Primary Care and Health Sciences, Keele University, Keele, United Kingdom (R.D.R.)
| | - Penny F Whiting
- Bristol Medical School of the University of Bristol and National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West, University Hospitals Bristol National Health Service Foundation Trust, Bristol, United Kingdom (P.F.W.)
| | - Marie Westwood
- Kleijnen Systematic Reviews, York, United Kingdom (R.F.W., M.W.)
| | - Gary S Collins
- Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom (G.S.C.)
| | - Johannes B Reitsma
- Julius Center for Health Sciences and Primary Care and Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands (K.G.M., J.B.R.)
| | - Jos Kleijnen
- Kleijnen Systematic Reviews, York, United Kingdom, and School for Public Health and Primary Care, Maastricht University, Maastricht, the Netherlands (J.K.)
| | - Sue Mallett
- Institute of Applied Health Research, National Institute for Health Research Birmingham Biomedical Research Centre, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom (S.M.)
| |
Collapse
|
37
|
Bonnett LJ, Varese F, Smith CT, Flores A, Yung AR. Individualised prediction of psychosis in individuals meeting at-risk mental state (ARMS) criteria: protocol for a systematic review of clinical prediction models. Diagn Progn Res 2019; 3:21. [PMID: 31768418 PMCID: PMC6873538 DOI: 10.1186/s41512-019-0066-5] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/12/2019] [Accepted: 09/22/2019] [Indexed: 01/22/2023] Open
Abstract
BACKGROUND Psychotic disorders affect about 3% of the population worldwide and are associated with high personal, social and economic costs. They tend to have their first onset in adolescence. Increasing emphasis has been placed on early intervention to detect illness and minimise disability. In the late 1990s, criteria were developed to identify individuals at high risk for psychotic disorder. These are known as the at-risk mental state (ARMS) criteria. While ARMS individuals have a risk of psychosis much greater than the general population, most individuals meeting the ARMS criteria will not develop psychosis. Despite this, the National Institute for Health and Care Excellence recommends cognitive behavioural therapy (CBT) for all ARMS people.Clinical prediction models that combine multiple patient characteristics to predict individual outcome risk may facilitate identification of patients who would benefit from CBT and conversely those that would benefit from less costly and less intensive regular mental state monitoring. The study will systematically review the evidence on clinical prediction models aimed at making individualised predictions for the transition to psychosis. METHODS Database searches will be conducted on PsycINFO, Medline, EMBASE and CINAHL. Reference lists and subject experts will be utilised. No language restrictions will be placed on publications, but searches will be restricted to 1994 onwards, the initial year of the first prospective study using ARMS criteria. Studies of any design will be included if they examined, in ARMS patients, whether more than one factor in combination is associated with the risk of transition to psychosis. Study quality will be assessed using the prediction model risk of bias assessment tool (PROBAST). Clinical prediction models will be summarised qualitatively, and if tested in multiple validation studies, their predictive performance will be summarised using a random-effects meta-analysis model. DISCUSSION The results of the review will identify prediction models for the risk of transition to psychosis. These will be informative for clinicians currently treating ARMS patients and considering potential preventive interventions. The conclusions of the review will also inform the possible update and external validation of prediction models and clinical prediction rules to identify those at high or low risk of transition to psychosis. TRIAL REGISTRATION The review has been registered with PROSPERO (CRD42018108488).
Collapse
Affiliation(s)
- Laura J. Bonnett
- 0000 0004 1936 8470grid.10025.36Department of Biostatistics, University of Liverpool, Liverpool, UK
| | - Filippo Varese
- 0000000121662407grid.5379.8School of Health Sciences, Division of Psychology & Mental Health, University of Manchester, Manchester, UK
- 0000 0004 0430 6955grid.450837.dGreater Manchester Mental Health NHS Foundation Trust, Manchester, UK
| | - Catrin Tudur Smith
- 0000 0004 1936 8470grid.10025.36Department of Biostatistics, University of Liverpool, Liverpool, UK
| | - Allan Flores
- 0000 0004 1936 8470grid.10025.36Department of Biostatistics, University of Liverpool, Liverpool, UK
| | - Alison R. Yung
- 0000000121662407grid.5379.8School of Health Sciences, Division of Psychology & Mental Health, University of Manchester, Manchester, UK
- 0000 0004 0430 6955grid.450837.dGreater Manchester Mental Health NHS Foundation Trust, Manchester, UK
- 0000 0001 2179 088Xgrid.1008.9Centre for Youth Mental Health, University of Melbourne, Melbourne, Australia
| |
Collapse
|
38
|
Snell KIE, Ensor J, Debray TPA, Moons KGM, Riley RD. Meta-analysis of prediction model performance across multiple studies: Which scale helps ensure between-study normality for the C-statistic and calibration measures? Stat Methods Med Res 2018; 27:3505-3522. [PMID: 28480827 PMCID: PMC6193210 DOI: 10.1177/0962280217705678] [Citation(s) in RCA: 64] [Impact Index Per Article: 10.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/22/2023]
Abstract
If individual participant data are available from multiple studies or clusters, then a prediction model can be externally validated multiple times. This allows the model's discrimination and calibration performance to be examined across different settings. Random-effects meta-analysis can then be used to quantify overall (average) performance and heterogeneity in performance. This typically assumes a normal distribution of 'true' performance across studies. We conducted a simulation study to examine this normality assumption for various performance measures relating to a logistic regression prediction model. We simulated data across multiple studies with varying degrees of variability in baseline risk or predictor effects and then evaluated the shape of the between-study distribution in the C-statistic, calibration slope, calibration-in-the-large, and E/O statistic, and possible transformations thereof. We found that a normal between-study distribution was usually reasonable for the calibration slope and calibration-in-the-large; however, the distributions of the C-statistic and E/O were often skewed across studies, particularly in settings with large variability in the predictor effects. Normality was vastly improved when using the logit transformation for the C-statistic and the log transformation for E/O, and therefore we recommend these scales to be used for meta-analysis. An illustrated example is given using a random-effects meta-analysis of the performance of QRISK2 across 25 general practices.
Collapse
Affiliation(s)
- Kym IE Snell
- Research Institute for Primary Care and
Health Sciences, Keele University, Staffordshire, UK
| | - Joie Ensor
- Research Institute for Primary Care and
Health Sciences, Keele University, Staffordshire, UK
| | - Thomas PA Debray
- Julius Center for Health Sciences and
Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands
- Cochrane Netherlands, University Medical
Center Utrecht, Utrecht, The Netherlands
| | - Karel GM Moons
- Julius Center for Health Sciences and
Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands
- Cochrane Netherlands, University Medical
Center Utrecht, Utrecht, The Netherlands
| | - Richard D Riley
- Research Institute for Primary Care and
Health Sciences, Keele University, Staffordshire, UK
| |
Collapse
|
39
|
POPCORN: A web service for individual PrognOsis prediction based on multi-center clinical data CollabORatioN without patient-level data sharing. J Biomed Inform 2018; 86:1-14. [PMID: 30103028 DOI: 10.1016/j.jbi.2018.08.008] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2018] [Revised: 08/06/2018] [Accepted: 08/08/2018] [Indexed: 12/23/2022]
Abstract
BACKGROUND AND OBJECTIVE Clinical prognosis prediction plays an important role in clinical research and practice. The construction of prediction models based on electronic health record data has recently become a research focus. Due to the lack of external validation, prediction models based on single-center, hospital-specific datasets may not perform well with datasets from other medical institutions. Therefore, research investigating prognosis prediction model construction based on a collaborative analysis of multi-center electronic health record data could increase the number and coverage of patients used for model training, enrich patient prognostic features and ultimately improve the accuracy and generalization of prognosis prediction. MATERIALS AND METHODS A web service for individual prognosis prediction based on multi-center clinical data collaboration without patient-level data sharing (POPCORN) was proposed. POPCORN focuses on solving key issues in multi-center collaborative research based on electronic health record systems; these issues include the standardization of clinical data expression, the preservation of patient privacy during model training and the effect of case mix variance on the prediction model construction and application. POPCORN is based on a multivariable meta-analysis and a Bayesian framework and can construct suitable prediction models for multiple clinical scenarios that can effectively adapt to complex clinical application environments. RESULTS POPCORN was validated using a joint, multi-center collaborative research network between China and the United States with patients diagnosed with colorectal cancer. The performance of the models based on POPCORN was comparable to that of the standard prognosis prediction model; however, POPCORN did not expose raw patient data. The prediction models had similar AUC, but the BMA model had the lowest ECI across all prediction models, indicating that this model had better calibration performance than the other models, especially for patients in Chinese hospitals. CONCLUSIONS The POPCORN system can build prediction models that perform well in complex clinical application scenarios and can provide effective decision support for individual patient prognostic predictions.
Collapse
|
40
|
Westeneng HJ, Debray TPA, Visser AE, van Eijk RPA, Rooney JPK, Calvo A, Martin S, McDermott CJ, Thompson AG, Pinto S, Kobeleva X, Rosenbohm A, Stubendorff B, Sommer H, Middelkoop BM, Dekker AM, van Vugt JJFA, van Rheenen W, Vajda A, Heverin M, Kazoka M, Hollinger H, Gromicho M, Körner S, Ringer TM, Rödiger A, Gunkel A, Shaw CE, Bredenoord AL, van Es MA, Corcia P, Couratier P, Weber M, Grosskreutz J, Ludolph AC, Petri S, de Carvalho M, Van Damme P, Talbot K, Turner MR, Shaw PJ, Al-Chalabi A, Chiò A, Hardiman O, Moons KGM, Veldink JH, van den Berg LH. Prognosis for patients with amyotrophic lateral sclerosis: development and validation of a personalised prediction model. Lancet Neurol 2018; 17:423-433. [PMID: 29598923 DOI: 10.1016/s1474-4422(18)30089-9] [Citation(s) in RCA: 289] [Impact Index Per Article: 48.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/25/2017] [Revised: 02/06/2018] [Accepted: 02/21/2018] [Indexed: 12/23/2022]
Abstract
BACKGROUND Amyotrophic lateral sclerosis (ALS) is a relentlessly progressive, fatal motor neuron disease with a variable natural history. There are no accurate models that predict the disease course and outcomes, which complicates risk assessment and counselling for individual patients, stratification of patients for trials, and timing of interventions. We therefore aimed to develop and validate a model for predicting a composite survival endpoint for individual patients with ALS. METHODS We obtained data for patients from 14 specialised ALS centres (each one designated as a cohort) in Belgium, France, the Netherlands, Germany, Ireland, Italy, Portugal, Switzerland, and the UK. All patients were diagnosed in the centres after excluding other diagnoses and classified according to revised El Escorial criteria. We assessed 16 patient characteristics as potential predictors of a composite survival outcome (time between onset of symptoms and non-invasive ventilation for more than 23 h per day, tracheostomy, or death) and applied backward elimination with bootstrapping in the largest population-based dataset for predictor selection. Data were gathered on the day of diagnosis or as soon as possible thereafter. Predictors that were selected in more than 70% of the bootstrap resamples were used to develop a multivariable Royston-Parmar model for predicting the composite survival outcome in individual patients. We assessed the generalisability of the model by estimating heterogeneity of predictive accuracy across external populations (ie, populations not used to develop the model) using internal-external cross-validation, and quantified the discrimination using the concordance (c) statistic (area under the receiver operator characteristic curve) and calibration using a calibration slope. FINDINGS Data were collected between Jan 1, 1992, and Sept 22, 2016 (the largest data-set included data from 1936 patients). The median follow-up time was 97·5 months (IQR 52·9-168·5). Eight candidate predictors entered the prediction model: bulbar versus non-bulbar onset (univariable hazard ratio [HR] 1·71, 95% CI 1·63-1·79), age at onset (1·03, 1·03-1·03), definite versus probable or possible ALS (1·47, 1·39-1·55), diagnostic delay (0·52, 0·51-0·53), forced vital capacity (HR 0·99, 0·99-0·99), progression rate (6·33, 5·92-6·76), frontotemporal dementia (1·34, 1·20-1·50), and presence of a C9orf72 repeat expansion (1·45, 1·31-1·61), all p<0·0001. The c statistic for external predictive accuracy of the model was 0·78 (95% CI 0·77-0·80; 95% prediction interval [PI] 0·74-0·82) and the calibration slope was 1·01 (95% CI 0·95-1·07; 95% PI 0·83-1·18). The model was used to define five groups with distinct median predicted (SE) and observed (SE) times in months from symptom onset to the composite survival outcome: very short 17·7 (0·20), 16·5 (0·23); short 25·3 (0·06), 25·2 (0·35); intermediate 32·2 (0·09), 32·8 (0·46); long 43·7 (0·21), 44·6 (0·74); and very long 91·0 (1·84), 85·6 (1·96). INTERPRETATION We have developed an externally validated model to predict survival without tracheostomy and non-invasive ventilation for more than 23 h per day in European patients with ALS. This model could be applied to individualised patient management, counselling, and future trial design, but to maximise the benefit and prevent harm it is intended to be used by medical doctors only. FUNDING Netherlands ALS Foundation.
Collapse
Affiliation(s)
- Henk-Jan Westeneng
- Department of Neurology, Brain Centre Rudolf Magnus, University Medical Centre Utrecht, Utrecht, Netherlands
| | - Thomas P A Debray
- Department of Epidemiology, Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht, Netherlands; Cochrane Netherlands, University Medical Centre Utrecht, Utrecht, Netherlands
| | - Anne E Visser
- Department of Neurology, Brain Centre Rudolf Magnus, University Medical Centre Utrecht, Utrecht, Netherlands
| | - Ruben P A van Eijk
- Department of Neurology, Brain Centre Rudolf Magnus, University Medical Centre Utrecht, Utrecht, Netherlands
| | - James P K Rooney
- Academic Unit of Neurology, Trinity Biomedical Sciences Institute, Trinity College, Dublin, Ireland
| | - Andrea Calvo
- 'Rita Levi Montalcini' Department of Neuroscience, ALS Centre, University of Torino, Torino, Italy
| | - Sarah Martin
- Maurice Wohl Clinical Neuroscience Institute, and United Kingdom Dementia Research Institute at the Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
| | | | | | - Susana Pinto
- Institute of Physiology-Instituto de Medicina Molecular, Faculty of Medicine, University of Lisbon, Lisbon, Portugal
| | - Xenia Kobeleva
- Department of Neurology, Hannover Medical School, Hannover, Germany
| | | | | | - Helma Sommer
- Neuromuscular Diseases Centre/ALS Clinic, Kantonsspital St Gallen, St Gallen, Switzerland
| | - Bas M Middelkoop
- Department of Neurology, Brain Centre Rudolf Magnus, University Medical Centre Utrecht, Utrecht, Netherlands
| | - Annelot M Dekker
- Department of Neurology, Brain Centre Rudolf Magnus, University Medical Centre Utrecht, Utrecht, Netherlands
| | - Joke J F A van Vugt
- Department of Neurology, Brain Centre Rudolf Magnus, University Medical Centre Utrecht, Utrecht, Netherlands
| | - Wouter van Rheenen
- Department of Neurology, Brain Centre Rudolf Magnus, University Medical Centre Utrecht, Utrecht, Netherlands
| | - Alice Vajda
- Academic Unit of Neurology, Trinity Biomedical Sciences Institute, Trinity College, Dublin, Ireland
| | - Mark Heverin
- Academic Unit of Neurology, Trinity Biomedical Sciences Institute, Trinity College, Dublin, Ireland
| | - Mbombe Kazoka
- Sheffield Institute for Translational Neuroscience, University of Sheffield, Sheffield, UK
| | - Hannah Hollinger
- Sheffield Institute for Translational Neuroscience, University of Sheffield, Sheffield, UK
| | - Marta Gromicho
- Institute of Physiology-Instituto de Medicina Molecular, Faculty of Medicine, University of Lisbon, Lisbon, Portugal
| | - Sonja Körner
- Department of Neurology, Hannover Medical School, Hannover, Germany
| | - Thomas M Ringer
- Hans-Berger Department of Neurology, Jena University Hospital, Jena, Germany
| | - Annekathrin Rödiger
- Hans-Berger Department of Neurology, Jena University Hospital, Jena, Germany
| | - Anne Gunkel
- Hans-Berger Department of Neurology, Jena University Hospital, Jena, Germany
| | - Christopher E Shaw
- Maurice Wohl Clinical Neuroscience Institute, and United Kingdom Dementia Research Institute at the Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
| | - Annelien L Bredenoord
- Department of Epidemiology, Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht, Netherlands
| | - Michael A van Es
- Department of Neurology, Brain Centre Rudolf Magnus, University Medical Centre Utrecht, Utrecht, Netherlands
| | - Philippe Corcia
- Centre de compétence SLA-fédération Tours-Limoges, CHU de Tours, Tours, France
| | - Philippe Couratier
- Centre de compétence SLA-fédération Tours-Limoges, CHU de Limoges, Limoges, France
| | - Markus Weber
- Neuromuscular Diseases Centre/ALS Clinic, Kantonsspital St Gallen, St Gallen, Switzerland
| | - Julian Grosskreutz
- Hans-Berger Department of Neurology, Jena University Hospital, Jena, Germany
| | | | - Susanne Petri
- Department of Neurology, Hannover Medical School, Hannover, Germany
| | - Mamede de Carvalho
- Institute of Physiology-Instituto de Medicina Molecular, Faculty of Medicine, University of Lisbon, Lisbon, Portugal
| | - Philip Van Damme
- Department of Neurology, University Hospital Leuven, Leuven, Belgium; Department of Neurosciences, Katholieke Universiteit, Leuven (University of Leuven) and Centre for Brain and Disease Research, VIB, Leuven, Belgium
| | - Kevin Talbot
- Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
| | - Martin R Turner
- Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
| | - Pamela J Shaw
- Sheffield Institute for Translational Neuroscience, University of Sheffield, Sheffield, UK
| | - Ammar Al-Chalabi
- Maurice Wohl Clinical Neuroscience Institute, and United Kingdom Dementia Research Institute at the Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
| | - Adriano Chiò
- 'Rita Levi Montalcini' Department of Neuroscience, ALS Centre, University of Torino, Torino, Italy
| | - Orla Hardiman
- Academic Unit of Neurology, Trinity Biomedical Sciences Institute, Trinity College, Dublin, Ireland; Department of Neurology, Beaumont Hospital, Beaumont, Ireland
| | - Karel G M Moons
- Department of Epidemiology, Julius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht, Netherlands; Cochrane Netherlands, University Medical Centre Utrecht, Utrecht, Netherlands
| | - Jan H Veldink
- Department of Neurology, Brain Centre Rudolf Magnus, University Medical Centre Utrecht, Utrecht, Netherlands
| | - Leonard H van den Berg
- Department of Neurology, Brain Centre Rudolf Magnus, University Medical Centre Utrecht, Utrecht, Netherlands.
| |
Collapse
|
41
|
Wynants L, Riley RD, Timmerman D, Van Calster B. Random-effects meta-analysis of the clinical utility of tests and prediction models. Stat Med 2018; 37:2034-2052. [PMID: 29575170 DOI: 10.1002/sim.7653] [Citation(s) in RCA: 22] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2017] [Revised: 01/20/2018] [Accepted: 02/10/2018] [Indexed: 11/10/2022]
Abstract
The use of data from multiple studies or centers for the validation of a clinical test or a multivariable prediction model allows researchers to investigate the test's/model's performance in multiple settings and populations. Recently, meta-analytic techniques have been proposed to summarize discrimination and calibration across study populations. Here, we rather consider performance in terms of net benefit, which is a measure of clinical utility that weighs the benefits of true positive classifications against the harms of false positives. We posit that it is important to examine clinical utility across multiple settings of interest. This requires a suitable meta-analysis method, and we propose a Bayesian trivariate random-effects meta-analysis of sensitivity, specificity, and prevalence. Across a range of chosen harm-to-benefit ratios, this provides a summary measure of net benefit, a prediction interval, and an estimate of the probability that the test/model is clinically useful in a new setting. In addition, the prediction interval and probability of usefulness can be calculated conditional on the known prevalence in a new setting. The proposed methods are illustrated by 2 case studies: one on the meta-analysis of published studies on ear thermometry to diagnose fever in children and one on the validation of a multivariable clinical risk prediction model for the diagnosis of ovarian cancer in a multicenter dataset. Crucially, in both case studies the clinical utility of the test/model was heterogeneous across settings, limiting its usefulness in practice. This emphasizes that heterogeneity in clinical utility should be assessed before a test/model is routinely implemented.
Collapse
Affiliation(s)
- L Wynants
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
| | - R D Riley
- Research Institute for Primary Care and Health Sciences, Keele University, Keele, Staffordshire, ST5 5BG, UK
| | - D Timmerman
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium.,Department of Obstetrics and Gynecology, University Hospitals Leuven, Leuven, Belgium
| | - B Van Calster
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium
| |
Collapse
|
42
|
Riley RD, Jackson D, Salanti G, Burke DL, Price M, Kirkham J, White IR. Multivariate and network meta-analysis of multiple outcomes and multiple treatments: rationale, concepts, and examples. BMJ 2017; 358:j3932. [PMID: 28903924 PMCID: PMC5596393 DOI: 10.1136/bmj.j3932] [Citation(s) in RCA: 155] [Impact Index Per Article: 22.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
Abstract
Organisations such as the National Institute for Health and Care Excellence require the synthesis of evidence from existing studies to inform their decisions—for example, about the best available treatments with respect to multiple efficacy and safety outcomes. However, relevant studies may not provide direct evidence about all the treatments or outcomes of interest. Multivariate and network meta-analysis methods provide a framework to address this, using correlated or indirect evidence from such studies alongside any direct evidence. In this article, the authors describe the key concepts and assumptions of these methods, outline how correlated and indirect evidence arises, and illustrate the contribution of such evidence in real clinical examples involving multiple outcomes and multiple treatments
Collapse
Affiliation(s)
- Richard D Riley
- Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire, UK
| | | | - Georgia Salanti
- Institute of Social and Preventive Medicine, University of Bern, Switzerland
- University of Ioannina School of Medicine, Ioannina, Greece
| | - Danielle L Burke
- Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire, UK
| | - Malcolm Price
- Institute of Applied Health Research, University of Birmingham, UK
| | - Jamie Kirkham
- MRC North West Hub for Trials Methodology Research, Department of Biostatistics, University of Liverpool, Liverpool, UK
| | - Ian R White
- MRC Biostatistics Unit, Cambridge, UK
- MRC Clinical Trials Unit at UCL, London, UK
| |
Collapse
|
43
|
Willis BH, Riley RD. Measuring the statistical validity of summary meta-analysis and meta-regression results for use in clinical practice. Stat Med 2017. [PMID: 28620945 PMCID: PMC5575530 DOI: 10.1002/sim.7372] [Citation(s) in RCA: 81] [Impact Index Per Article: 11.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/30/2023]
Abstract
An important question for clinicians appraising a meta‐analysis is: are the findings likely to be valid in their own practice—does the reported effect accurately represent the effect that would occur in their own clinical population? To this end we advance the concept of statistical validity—where the parameter being estimated equals the corresponding parameter for a new independent study. Using a simple (‘leave‐one‐out’) cross‐validation technique, we demonstrate how we may test meta‐analysis estimates for statistical validity using a new validation statistic, Vn, and derive its distribution. We compare this with the usual approach of investigating heterogeneity in meta‐analyses and demonstrate the link between statistical validity and homogeneity. Using a simulation study, the properties of Vn and the Q statistic are compared for univariate random effects meta‐analysis and a tailored meta‐regression model, where information from the setting (included as model covariates) is used to calibrate the summary estimate to the setting of application. Their properties are found to be similar when there are 50 studies or more, but for fewer studies Vn has greater power but a higher type 1 error rate than Q. The power and type 1 error rate of Vn are also shown to depend on the within‐study variance, between‐study variance, study sample size, and the number of studies in the meta‐analysis. Finally, we apply Vn to two published meta‐analyses and conclude that it usefully augments standard methods when deciding upon the likely validity of summary meta‐analysis estimates in clinical practice. © 2017 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.
Collapse
Affiliation(s)
- Brian H Willis
- Institute of Applied Health Research, University of Birmingham, U.K
| | - Richard D Riley
- Research Institute for Primary Care and Health Sciences, Keele University, U.K
| |
Collapse
|
44
|
van Doorn S, Debray TPA, Kaasenbrood F, Hoes AW, Rutten FH, Moons KGM, Geersing GJ. Predictive performance of the CHA2DS2-VASc rule in atrial fibrillation: a systematic review and meta-analysis. J Thromb Haemost 2017; 15:1065-1077. [PMID: 28375552 DOI: 10.1111/jth.13690] [Citation(s) in RCA: 22] [Impact Index Per Article: 3.1] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2016] [Indexed: 11/29/2022]
Abstract
Essentials The widely recommended CHA2DS2-VASc shows conflicting results in contemporary validation studies. We performed a systematic review and meta-analysis of 19 studies validating CHA2DS2-VASc. There was high heterogeneity in stroke risks for different CHA2DS2-VASc scores. This was not explained by differences between setting of care, or by performing meta-regression. SUMMARY Background The CHA2DS2-VASc decision rule is widely recommended for estimating stroke risk in patients with atrial fibrillation (AF), although validation studies show ambiguous and conflicting results. Objectives To: (i) review existing studies validating CHA2DS2-VASc in AF patients who are not (yet) anticoagulated; (ii) meta-analyze estimates of stroke risk per score; and (iii) explore sources of heterogeneity across the validation studies. Methods We performed a systematic literature review and random effects meta-analysis of studies externally validating CHA2DS2-VASc in AF patients not receiving anticoagulants. To explore between-study heterogeneity in stroke risk, we stratified studies to the clinical setting in which patient enrollment started, and performed meta-regression. Results In total, 19 studies were evaluated, with over two million person-years of follow-up. In studies recruiting AF patients in hospitals, stroke risks for scores of 0, 1 and 2 were 0.4% (approximate 95% prediction interval [PI] 0.2-3.2%), 1.2% (95% PI 0.1-3.8%), and 2.2% (95% PI 0.03-7.8%), respectively. These were consistently higher than those in studies recruiting patients from the open general population, with risks of 0.2% (95% PI 0.0-0.9%), 0.7% (95% PI 0.3-1.2%) and 1.5% (95% PI 0.4-3.3%) for scores of 0, 1, and 2, respectively. Heterogeneity, as reflected by the wide PIs, could not be fully explained by meta-regression. Conclusions Studies validating CHA2DS2-VASc show high heterogeneity in predicted stroke risks for different scores.
Collapse
Affiliation(s)
- S van Doorn
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands
| | - T P A Debray
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands
| | - F Kaasenbrood
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands
| | - A W Hoes
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands
| | - F H Rutten
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands
| | - K G M Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands
| | - G J Geersing
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands
| |
Collapse
|
45
|
Allotey J, Snell KIE, Chan C, Hooper R, Dodds J, Rogozinska E, Khan KS, Poston L, Kenny L, Myers J, Thilaganathan B, Chappell L, Mol BW, Von Dadelszen P, Ahmed A, Green M, Poon L, Khalil A, Moons KGM, Riley RD, Thangaratinam S. External validation, update and development of prediction models for pre-eclampsia using an Individual Participant Data (IPD) meta-analysis: the International Prediction of Pregnancy Complication Network (IPPIC pre-eclampsia) protocol. Diagn Progn Res 2017; 1:16. [PMID: 31093545 PMCID: PMC6460674 DOI: 10.1186/s41512-017-0016-z] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/06/2017] [Accepted: 09/19/2017] [Indexed: 01/13/2023] Open
Abstract
BACKGROUND Pre-eclampsia, a condition with raised blood pressure and proteinuria is associated with an increased risk of maternal and offspring mortality and morbidity. Early identification of mothers at risk is needed to target management. METHODS/DESIGN We aim to systematically review the existing literature to identify prediction models for pre-eclampsia. We have established the International Prediction of Pregnancy Complication Network (IPPIC), made up of 72 researchers from 21 countries who have carried out relevant primary studies or have access to existing registry databases, and collectively possess data from more than two million patients. We will use the individual participant data (IPD) from these studies to externally validate these existing prediction models and summarise model performance across studies using random-effects meta-analysis for any, late (after 34 weeks) and early (before 34 weeks) onset pre-eclampsia. If none of the models perform well, we will recalibrate (update), or develop and validate new prediction models using the IPD. We will assess the differential accuracy of the models in various settings and subgroups according to the risk status. We will also validate or develop prediction models based on clinical characteristics only; clinical and biochemical markers; clinical and ultrasound parameters; and clinical, biochemical and ultrasound tests. DISCUSSION Numerous systematic reviews with aggregate data meta-analysis have evaluated various risk factors separately or in combination for predicting pre-eclampsia, but these are affected by many limitations. Our large-scale collaborative IPD approach encourages consensus towards well developed, and validated prognostic models, rather than a number of competing non-validated ones. The large sample size from our IPD will also allow development and validation of multivariable prediction model for the relatively rare outcome of early onset pre-eclampsia. TRIAL REGISTRATION The project was registered on Prospero on the 27 November 2015 with ID: CRD42015029349.
Collapse
Affiliation(s)
- John Allotey
- 0000 0001 2171 1133grid.4868.2Women’s Health Research Unit, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
- 0000 0001 2171 1133grid.4868.2Pragmatic Clinical Trials Unit, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
- 0000 0001 2171 1133grid.4868.2Multidisciplinary Evidence Synthesis Hub (MESH), Queen Mary University of London, London, UK
| | - Kym I. E. Snell
- 0000 0004 0415 6205grid.9757.cResearch Institute for Primary Care and Health Sciences, Keele University, Keele, UK
| | - Claire Chan
- 0000 0001 2171 1133grid.4868.2Pragmatic Clinical Trials Unit, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
| | - Richard Hooper
- 0000 0001 2171 1133grid.4868.2Pragmatic Clinical Trials Unit, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
| | - Julie Dodds
- 0000 0001 2171 1133grid.4868.2Women’s Health Research Unit, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
- 0000 0001 2171 1133grid.4868.2Pragmatic Clinical Trials Unit, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
- 0000 0001 2171 1133grid.4868.2Multidisciplinary Evidence Synthesis Hub (MESH), Queen Mary University of London, London, UK
| | - Ewelina Rogozinska
- 0000 0001 2171 1133grid.4868.2Women’s Health Research Unit, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
- 0000 0001 2171 1133grid.4868.2Pragmatic Clinical Trials Unit, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
- 0000 0001 2171 1133grid.4868.2Multidisciplinary Evidence Synthesis Hub (MESH), Queen Mary University of London, London, UK
| | - Khalid S. Khan
- 0000 0001 2171 1133grid.4868.2Women’s Health Research Unit, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
- 0000 0001 2171 1133grid.4868.2Pragmatic Clinical Trials Unit, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
- 0000 0001 2171 1133grid.4868.2Multidisciplinary Evidence Synthesis Hub (MESH), Queen Mary University of London, London, UK
| | - Lucilla Poston
- 0000 0001 2322 6764grid.13097.3cDivision of Women’s Health, Women’s Health Academic Centre, King’s College London, London, UK
| | - Louise Kenny
- 0000000123318773grid.7872.aIrish Centre for Fetal and Neonatal Translational Research [INFANT], University College Cork, Cork, Ireland
| | - Jenny Myers
- 0000000121662407grid.5379.8Maternal and Fetal Heath Research Centre, Manchester Academic Health Science Centre, University of Manchester, Central Manchester NHS Trust, Manchester, UK
| | - Basky Thilaganathan
- grid.264200.2Fetal Medicine Unit, St George’s University Hospitals NHS Foundation Trust and Molecular and Clinical Sciences Research Institute, St George’s University of London, London, UK
| | - Lucy Chappell
- 0000 0001 2322 6764grid.13097.3cDivision of Women’s Health, Women’s Health Academic Centre, King’s College London, London, UK
| | - Ben W. Mol
- 0000 0004 1936 7304grid.1010.0The Robinson Research Institute, School of Paediatrics and Reproductive Health, University of Adelaide, Adelaide, Australia
| | - Peter Von Dadelszen
- 0000 0001 2161 2573grid.4464.2Institute of Cardiovascular and Cell Sciences, St George’s, University of London, London, UK
| | - Asif Ahmed
- 0000 0004 0376 4727grid.7273.1Aston Medical School, Aston University, Birmingham, UK
| | - Marcus Green
- Action on Pre-eclampsia (APEC) Charity, Worcestershire, UK
| | - Liona Poon
- 0000 0004 0391 9020grid.46699.34Harris Birthright Research Centre for Fetal Medicine, King’s College Hospital, London, UK
- 0000 0004 1937 0482grid.10784.3aDepartment of Obstetrics and Gynaecology, The Chinese University of Hong Kong, Hong Kong, Hong Kong
| | - Asma Khalil
- grid.264200.2Fetal Medicine Unit, St George’s University Hospitals NHS Foundation Trust and Molecular and Clinical Sciences Research Institute, St George’s University of London, London, UK
| | - Karel G. M. Moons
- 0000000090126352grid.7692.aJulius Centre for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht, the Netherlands
| | - Richard D. Riley
- 0000 0004 0415 6205grid.9757.cResearch Institute for Primary Care and Health Sciences, Keele University, Keele, UK
| | - Shakila Thangaratinam
- 0000 0001 2171 1133grid.4868.2Women’s Health Research Unit, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
- 0000 0001 2171 1133grid.4868.2Pragmatic Clinical Trials Unit, Barts and the London School of Medicine and Dentistry, Queen Mary University of London, London, UK
- 0000 0001 2171 1133grid.4868.2Multidisciplinary Evidence Synthesis Hub (MESH), Queen Mary University of London, London, UK
| | | |
Collapse
|
46
|
Wynants L, Vergouwe Y, Van Huffel S, Timmerman D, Van Calster B. Does ignoring clustering in multicenter data influence the performance of prediction models? A simulation study. Stat Methods Med Res 2016; 27:1723-1736. [PMID: 27647815 DOI: 10.1177/0962280216668555] [Citation(s) in RCA: 22] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/20/2023]
Abstract
Clinical risk prediction models are increasingly being developed and validated on multicenter datasets. In this article, we present a comprehensive framework for the evaluation of the predictive performance of prediction models at the center level and the population level, considering population-averaged predictions, center-specific predictions, and predictions assuming an average random center effect. We demonstrated in a simulation study that calibration slopes do not only deviate from one because of over- or underfitting of patterns in the development dataset, but also as a result of the choice of the model (standard versus mixed effects logistic regression), the type of predictions (marginal versus conditional versus assuming an average random effect), and the level of model validation (center versus population). In particular, when data is heavily clustered (ICC 20%), center-specific predictions offer the best predictive performance at the population level and the center level. We recommend that models should reflect the data structure, while the level of model validation should reflect the research question.
Collapse
Affiliation(s)
- L Wynants
- 1 KU Leuven Department of Electrical Engineering (ESAT), STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics, Leuven, Belgium.,2 KU Leuven iMinds Department Medical Information Technologies, Leuven, Belgium
| | - Y Vergouwe
- 3 Center for Medical Decision Sciences, Department of Public Health, Erasmus Medical Center, Rotterdam, The Netherlands
| | - S Van Huffel
- 1 KU Leuven Department of Electrical Engineering (ESAT), STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics, Leuven, Belgium.,2 KU Leuven iMinds Department Medical Information Technologies, Leuven, Belgium
| | - D Timmerman
- 4 KU Leuven Department of Development and Regeneration, Leuven, Belgium
| | - B Van Calster
- 3 Center for Medical Decision Sciences, Department of Public Health, Erasmus Medical Center, Rotterdam, The Netherlands.,4 KU Leuven Department of Development and Regeneration, Leuven, Belgium
| |
Collapse
|
47
|
Riley RD, Ensor J, Snell KIE, Debray TPA, Altman DG, Moons KGM, Collins GS. External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ 2016; 353:i3140. [PMID: 27334381 PMCID: PMC4916924 DOI: 10.1136/bmj.i3140] [Citation(s) in RCA: 285] [Impact Index Per Article: 35.6] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 05/18/2016] [Indexed: 12/18/2022]
Affiliation(s)
- Richard D Riley
- Research Institute for Primary Care and Health Sciences, Keele University, Keele ST5 5BG, Staffordshire, UK
| | - Joie Ensor
- Research Institute for Primary Care and Health Sciences, Keele University, Keele ST5 5BG, Staffordshire, UK
| | - Kym I E Snell
- Institute of Applied Health Research, University of Birmingham, Edgbaston, Birmingham, UK
| | - Thomas P A Debray
- Julius Centre for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, Netherlands Cochrane Netherlands, University Medical Center Utrecht, Utrecht, Netherlands
| | - Doug G Altman
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - Karel G M Moons
- Julius Centre for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, Netherlands Cochrane Netherlands, University Medical Center Utrecht, Utrecht, Netherlands
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| |
Collapse
|
48
|
Damen JAAG, Hooft L, Schuit E, Debray TPA, Collins GS, Tzoulaki I, Lassale CM, Siontis GCM, Chiocchia V, Roberts C, Schlüssel MM, Gerry S, Black JA, Heus P, van der Schouw YT, Peelen LM, Moons KGM. Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ 2016; 353:i2416. [PMID: 27184143 PMCID: PMC4868251 DOI: 10.1136/bmj.i2416] [Citation(s) in RCA: 463] [Impact Index Per Article: 57.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 04/19/2016] [Indexed: 12/23/2022]
Abstract
OBJECTIVE To provide an overview of prediction models for risk of cardiovascular disease (CVD) in the general population. DESIGN Systematic review. DATA SOURCES Medline and Embase until June 2013. ELIGIBILITY CRITERIA FOR STUDY SELECTION Studies describing the development or external validation of a multivariable model for predicting CVD risk in the general population. RESULTS 9965 references were screened, of which 212 articles were included in the review, describing the development of 363 prediction models and 473 external validations. Most models were developed in Europe (n=167, 46%), predicted risk of fatal or non-fatal coronary heart disease (n=118, 33%) over a 10 year period (n=209, 58%). The most common predictors were smoking (n=325, 90%) and age (n=321, 88%), and most models were sex specific (n=250, 69%). Substantial heterogeneity in predictor and outcome definitions was observed between models, and important clinical and methodological information were often missing. The prediction horizon was not specified for 49 models (13%), and for 92 (25%) crucial information was missing to enable the model to be used for individual risk prediction. Only 132 developed models (36%) were externally validated and only 70 (19%) by independent investigators. Model performance was heterogeneous and measures such as discrimination and calibration were reported for only 65% and 58% of the external validations, respectively. CONCLUSIONS There is an excess of models predicting incident CVD in the general population. The usefulness of most of the models remains unclear owing to methodological shortcomings, incomplete presentation, and lack of external validation and model impact studies. Rather than developing yet another similar CVD risk prediction model, in this era of large datasets, future research should focus on externally validating and comparing head-to-head promising CVD risk models that already exist, on tailoring or even combining these models to local settings, and investigating whether these models can be extended by addition of new predictors.
Collapse
Affiliation(s)
- Johanna A A G Damen
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, Netherlands Cochrane Netherlands, University Medical Center Utrecht, PO Box 85500, Str 6.131, 3508 GA Utrecht, Netherlands
| | - Lotty Hooft
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, Netherlands Cochrane Netherlands, University Medical Center Utrecht, PO Box 85500, Str 6.131, 3508 GA Utrecht, Netherlands
| | - Ewoud Schuit
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, Netherlands Cochrane Netherlands, University Medical Center Utrecht, PO Box 85500, Str 6.131, 3508 GA Utrecht, Netherlands Stanford Prevention Research Center, Stanford University, Stanford, CA, USA
| | - Thomas P A Debray
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, Netherlands Cochrane Netherlands, University Medical Center Utrecht, PO Box 85500, Str 6.131, 3508 GA Utrecht, Netherlands
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - Ioanna Tzoulaki
- Department of Epidemiology and Biostatistics, School of Public Health, Imperial College London, London, UK
| | - Camille M Lassale
- Department of Epidemiology and Biostatistics, School of Public Health, Imperial College London, London, UK
| | - George C M Siontis
- Department of Cardiology, Bern University Hospital, 3010 Bern, Switzerland
| | - Virginia Chiocchia
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK Surgical Intervention Trials Unit, University of Oxford, Oxford, UK
| | - Corran Roberts
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - Michael Maia Schlüssel
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - Stephen Gerry
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - James A Black
- MRC Epidemiology Unit, University of Cambridge School of Clinical Medicine, Cambridge, UK
| | - Pauline Heus
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, Netherlands Cochrane Netherlands, University Medical Center Utrecht, PO Box 85500, Str 6.131, 3508 GA Utrecht, Netherlands
| | - Yvonne T van der Schouw
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, Netherlands
| | - Linda M Peelen
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, Netherlands
| | - Karel G M Moons
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, Netherlands Cochrane Netherlands, University Medical Center Utrecht, PO Box 85500, Str 6.131, 3508 GA Utrecht, Netherlands
| |
Collapse
|
49
|
Burke DL, Bujkiewicz S, Riley RD. Bayesian bivariate meta-analysis of correlated effects: Impact of the prior distributions on the between-study correlation, borrowing of strength, and joint inferences. Stat Methods Med Res 2016; 27:428-450. [PMID: 26988929 PMCID: PMC5810917 DOI: 10.1177/0962280216631361] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/21/2023]
Abstract
Multivariate random-effects meta-analysis allows the joint synthesis of correlated results from multiple studies, for example, for multiple outcomes or multiple treatment groups. In a Bayesian univariate meta-analysis of one endpoint, the importance of specifying a sensible prior distribution for the between-study variance is well understood. However, in multivariate meta-analysis, there is little guidance about the choice of prior distributions for the variances or, crucially, the between-study correlation, ρB; for the latter, researchers often use a Uniform(−1,1) distribution assuming it is vague. In this paper, an extensive simulation study and a real illustrative example is used to examine the impact of various (realistically) vague prior distributions for ρB and the between-study variances within a Bayesian bivariate random-effects meta-analysis of two correlated treatment effects. A range of diverse scenarios are considered, including complete and missing data, to examine the impact of the prior distributions on posterior results (for treatment effect and between-study correlation), amount of borrowing of strength, and joint predictive distributions of treatment effectiveness in new studies. Two key recommendations are identified to improve the robustness of multivariate meta-analysis results. First, the routine use of a Uniform(−1,1) prior distribution for ρB should be avoided, if possible, as it is not necessarily vague. Instead, researchers should identify a sensible prior distribution, for example, by restricting values to be positive or negative as indicated by prior knowledge. Second, it remains critical to use sensible (e.g. empirically based) prior distributions for the between-study variances, as an inappropriate choice can adversely impact the posterior distribution for ρB, which may then adversely affect inferences such as joint predictive probabilities. These recommendations are especially important with a small number of studies and missing data.
Collapse
Affiliation(s)
- Danielle L Burke
- 1 Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire, UK
| | - Sylwia Bujkiewicz
- 2 Biostatistics Group, Department of Health Sciences, University of Leicester, Leicester, UK
| | - Richard D Riley
- 1 Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire, UK
| |
Collapse
|
50
|
Bennette CS, Ramsey SD, McDermott CL, Carlson JJ, Basu A, Veenstra DL. Predicting Low Accrual in the National Cancer Institute's Cooperative Group Clinical Trials. J Natl Cancer Inst 2016; 108:djv324. [PMID: 26714555 PMCID: PMC5964887 DOI: 10.1093/jnci/djv324] [Citation(s) in RCA: 63] [Impact Index Per Article: 7.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/02/2015] [Revised: 07/27/2015] [Accepted: 10/08/2015] [Indexed: 11/13/2022] Open
Abstract
BACKGROUND The extent to which trial-level factors differentially influence accrual to trials has not been comprehensively studied. Our objective was to evaluate the empirical relationship and predictive properties of putative risk factors for low accrual in the National Cancer Institute's (NCI's) Cooperative Group Program, now the National Clinical Trials Network (NCTN). METHODS Data from 787 phase II/III adult NCTN-sponsored trials launched between 2000 and 2011 were used to develop a logistic regression model to predict low accrual, defined as trials that closed with or were accruing at less than 50% of target; 46 trials opened between 2012 and 2013 were used for prospective validation. Candidate predictors were identified from a literature review and expert interviews; final predictors were selected using stepwise regression. Model performance was evaluated by calibration and discrimination via the area under the curve (AUC). All statistical tests were two-sided. RESULTS Eighteen percent (n = 145) of NCTN-sponsored trials closed with low accrual or were accruing at less than 50% of target three years or more after initiation. A multivariable model of twelve trial-level risk factors had good calibration and discrimination for predicting trials with low accrual (AUC in trials launched 2000-2011 = 0.739, 95% confidence interval [CI] = 0.696 to 0.783]; 2012-2013: AUC = 0.732, 95% CI = 0.547 to 0.917). Results were robust to different definitions of low accrual and predictor selection strategies. CONCLUSIONS We identified multiple characteristics of NCTN-sponsored trials associated with low accrual, several of which have not been previously empirically described, and developed a prediction model that can provide a useful estimate of accrual risk based on these factors. Future work should assess the role of such prediction tools in trial design and prioritization decisions.
Collapse
Affiliation(s)
- Caroline S Bennette
- Pharmaceutical Outcomes Research and Policy Program, University of Washington, Seattle, WA (CSB, SDR, CLM, JJC, AB, DLV) and the Hutchinson Institute for Outcomes Research, Fred Hutchinson Cancer Research Center (SDR)
| | - Scott D Ramsey
- Pharmaceutical Outcomes Research and Policy Program, University of Washington, Seattle, WA (CSB, SDR, CLM, JJC, AB, DLV) and the Hutchinson Institute for Outcomes Research, Fred Hutchinson Cancer Research Center (SDR)
| | - Cara L McDermott
- Pharmaceutical Outcomes Research and Policy Program, University of Washington, Seattle, WA (CSB, SDR, CLM, JJC, AB, DLV) and the Hutchinson Institute for Outcomes Research, Fred Hutchinson Cancer Research Center (SDR)
| | - Josh J Carlson
- Pharmaceutical Outcomes Research and Policy Program, University of Washington, Seattle, WA (CSB, SDR, CLM, JJC, AB, DLV) and the Hutchinson Institute for Outcomes Research, Fred Hutchinson Cancer Research Center (SDR)
| | - Anirban Basu
- Pharmaceutical Outcomes Research and Policy Program, University of Washington, Seattle, WA (CSB, SDR, CLM, JJC, AB, DLV) and the Hutchinson Institute for Outcomes Research, Fred Hutchinson Cancer Research Center (SDR)
| | - David L Veenstra
- Pharmaceutical Outcomes Research and Policy Program, University of Washington, Seattle, WA (CSB, SDR, CLM, JJC, AB, DLV) and the Hutchinson Institute for Outcomes Research, Fred Hutchinson Cancer Research Center (SDR)
| |
Collapse
|