1
|
Yland JJ, Zad Z, Wang TR, Wesselink AK, Jiang T, Hatch EE, Paschalidis IC, Wise LA. Predictive models of miscarriage on the basis of data from a preconception cohort study. Fertil Steril 2024; 122:140-149. [PMID: 38604264 DOI: 10.1016/j.fertnstert.2024.04.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2023] [Revised: 03/30/2024] [Accepted: 04/01/2024] [Indexed: 04/13/2024]
Abstract
OBJECTIVE To use self-reported preconception data to derive models that predict the risk of miscarriage. DESIGN Prospective preconception cohort study. SETTING Not applicable. PATIENTS Study participants were female, aged 21-45 years, residents of the United States or Canada, and attempting spontaneous pregnancy at enrollment during 2013-2022. Participants were followed for up to 12 months of pregnancy attempts; those who conceived were followed through pregnancy and postpartum. We restricted analyses to participants who conceived during the study period. EXPOSURE On baseline and follow-up questionnaires completed every 8 weeks until pregnancy, we collected self-reported data on sociodemographic factors, reproductive history, lifestyle, anthropometrics, diet, medical history, and male partner characteristics. We included 160 potential predictor variables in our models. MAIN OUTCOME MEASURES The primary outcome was a miscarriage, defined as pregnancy loss before 20 weeks of gestation. We followed participants from their first positive pregnancy test until miscarriage or a censoring event (induced abortion, ectopic pregnancy, loss of follow-up, or 20 weeks of gestation), whichever occurred first. We fit both survival and static models using Cox proportional hazards models, logistic regression, support vector machines, gradient-boosted trees, and random forest algorithms. We evaluated model performance using the concordance index (survival models) and the weighted F1 score (static models). RESULTS Among the 8,720 participants who conceived, 20.4% reported miscarriage. In multivariable models, the strongest predictors of miscarriage were female age, history of miscarriage, and male partner age. The weighted F1 score ranged from 73%-89% for static models and the concordance index ranged from 53%-56% for survival models, indicating better discrimination for the static models compared with the survival models (i.e., the ability of the model to discriminate between individuals with and without miscarriage). No appreciable differences were observed across strata of miscarriage history or among models restricted to ≥8 weeks of gestation. CONCLUSION Our findings suggest that miscarriage is not easily predicted on the basis of preconception lifestyle characteristics and that advancing age and a history of miscarriage are the most important predictors of incident miscarriage.
Collapse
Affiliation(s)
- Jennifer J Yland
- Department of Epidemiology, Boston University School of Public Health, Boston, Massachusetts.
| | - Zahra Zad
- Hariri Institute for Computing and Computational Science & Engineering, Boston University, Boston, Massachusetts; Division of Systems Engineering, Department of Electrical and Computer Engineering, Boston University, Boston, Massachusetts
| | - Tanran R Wang
- Department of Epidemiology, Boston University School of Public Health, Boston, Massachusetts
| | - Amelia K Wesselink
- Department of Epidemiology, Boston University School of Public Health, Boston, Massachusetts
| | - Tammy Jiang
- Department of Epidemiology, Boston University School of Public Health, Boston, Massachusetts
| | - Elizabeth E Hatch
- Department of Epidemiology, Boston University School of Public Health, Boston, Massachusetts
| | - Ioannis Ch Paschalidis
- Hariri Institute for Computing and Computational Science & Engineering, Boston University, Boston, Massachusetts; Division of Systems Engineering, Department of Electrical and Computer Engineering, Boston University, Boston, Massachusetts; Department of Biomedical Engineering, Boston University, Boston, Massachusetts
| | - Lauren A Wise
- Department of Epidemiology, Boston University School of Public Health, Boston, Massachusetts
| |
Collapse
|
2
|
van Os HJA, Kanning JP, Wermer MJH, Chavannes NH, Numans ME, Ruigrok YM, van Zwet EW, Putter H, Steyerberg EW, Groenwold RHH. Developing Clinical Prediction Models Using Primary Care Electronic Health Record Data: The Impact of Data Preparation Choices on Model Performance. FRONTIERS IN EPIDEMIOLOGY 2022; 2:871630. [PMID: 38455328 PMCID: PMC10910909 DOI: 10.3389/fepid.2022.871630] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/08/2022] [Accepted: 04/11/2022] [Indexed: 03/09/2024]
Abstract
Objective To quantify prediction model performance in relation to data preparation choices when using electronic health records (EHR). Study Design and Setting Cox proportional hazards models were developed for predicting the first-ever main adverse cardiovascular events using Dutch primary care EHR data. The reference model was based on a 1-year run-in period, cardiovascular events were defined based on both EHR diagnosis and medication codes, and missing values were multiply imputed. We compared data preparation choices based on (i) length of the run-in period (2- or 3-year run-in); (ii) outcome definition (EHR diagnosis codes or medication codes only); and (iii) methods addressing missing values (mean imputation or complete case analysis) by making variations on the derivation set and testing their impact in a validation set. Results We included 89,491 patients in whom 6,736 first-ever main adverse cardiovascular events occurred during a median follow-up of 8 years. Outcome definition based only on diagnosis codes led to a systematic underestimation of risk (calibration curve intercept: 0.84; 95% CI: 0.83-0.84), while complete case analysis led to overestimation (calibration curve intercept: -0.52; 95% CI: -0.53 to -0.51). Differences in the length of the run-in period showed no relevant impact on calibration and discrimination. Conclusion Data preparation choices regarding outcome definition or methods to address missing values can have a substantial impact on the calibration of predictions, hampering reliable clinical decision support. This study further illustrates the urgency of transparent reporting of modeling choices in an EHR data setting.
Collapse
Affiliation(s)
- Hendrikus J. A. van Os
- Department of Neurology, Leiden University Medical Hospital, Leiden, Netherlands
- National eHealth Living Lab, Leiden University Medical Hospital, Leiden, Netherlands
- Department of Public Health & Primary Care, Leiden University Medical Hospital, Leiden, Netherlands
| | - Jos P. Kanning
- Department of Neurology, University Medical Center Utrecht, Utrecht, Netherlands
| | - Marieke J. H. Wermer
- Department of Neurology, Leiden University Medical Hospital, Leiden, Netherlands
| | - Niels H. Chavannes
- National eHealth Living Lab, Leiden University Medical Hospital, Leiden, Netherlands
- Department of Public Health & Primary Care, Leiden University Medical Hospital, Leiden, Netherlands
| | - Mattijs E. Numans
- Department of Public Health & Primary Care, Leiden University Medical Hospital, Leiden, Netherlands
| | - Ynte M. Ruigrok
- Department of Neurology, University Medical Center Utrecht, Utrecht, Netherlands
| | - Erik W. van Zwet
- Department of Biomedical Data Sciences, Leiden University Medical Hospital, Leiden, Netherlands
| | - Hein Putter
- Department of Biomedical Data Sciences, Leiden University Medical Hospital, Leiden, Netherlands
| | - Ewout W. Steyerberg
- Department of Biomedical Data Sciences, Leiden University Medical Hospital, Leiden, Netherlands
| | - Rolf H. H. Groenwold
- Department of Biomedical Data Sciences, Leiden University Medical Hospital, Leiden, Netherlands
- Department of Clinical Epidemiology, Leiden University Medical Hospital, Leiden, Netherlands
| |
Collapse
|
3
|
Atrial fibrillation: trends in prevalence and antithrombotic prescriptions in the community. Neth Heart J 2022; 30:459-465. [PMID: 35230637 PMCID: PMC9475006 DOI: 10.1007/s12471-022-01667-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/13/2022] [Indexed: 11/09/2022] Open
Abstract
Introduction In the past decade, the atrial fibrillation (AF) landscape, including the treatment modalities, has drastically changed. This raises the question how AF prevalence and choices in antithrombotic therapy prescription have developed in the community over time. Methods Routine care data from the Julius General Practitioners’ Network (JGPN) were used to calculate the yearly prevalence of AF and to quantify the percentage of all patients who were prescribed a platelet inhibitor, vitamin K antagonist (VKA), non-VKA oral anticoagulant (NOAC) or no antithrombotic medication. To explore whether certain patient characteristics are associated with selective prescription of oral anticoagulants (OAC), we applied logistic regression analyses. Results From 2008 through 2017, the JGPN database included 7459 unique AF patients. During this period, the prevalence of AF increased from 0.4% to 1.4%. The percentage of patients prescribed a VKA declined from 47% to 41%, whereas the percentage of patients prescribed a NOAC rose from 0% to 20%. In patients with new-onset AF, older age, heart failure, diabetes mellitus, vascular disease and dementia were independently associated with a higher likelihood of VKA rather than NOAC prescription. In 2017, 25% of all patients with AF and a CHA2DS2-VASc score ≥ 2 were not prescribed OAC therapy (i.e. 8% with platelet inhibitor monotherapy and 17% without any antithrombotic therapy). Conclusion Between 2008 and 2017, AF prevalence in the community more than tripled. Prescription patterns showed possible ‘channelling’ of VKAs over NOACs in frailer, elderly patients, whereas still about one in every four AF patients with a CHA2DS2-VASc score ≥ 2 was not prescribed any prophylactic OAC therapy. Supplementary Information The online version of this article (10.1007/s12471-022-01667-x) contains supplementary material, which is available to authorized users.
Collapse
|
4
|
Yland JJ, Wang T, Zad Z, Willis SK, Wang TR, Wesselink AK, Jiang T, Hatch EE, Wise LA, Paschalidis IC. Predictive models of pregnancy based on data from a preconception cohort study. Hum Reprod 2022; 37:565-576. [PMID: 35024824 PMCID: PMC8888990 DOI: 10.1093/humrep/deab280] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/25/2021] [Revised: 11/30/2021] [Indexed: 01/16/2023] Open
Abstract
STUDY QUESTION Can we derive adequate models to predict the probability of conception among couples actively trying to conceive? SUMMARY ANSWER Leveraging data collected from female participants in a North American preconception cohort study, we developed models to predict pregnancy with performance of ∼70% in the area under the receiver operating characteristic curve (AUC). WHAT IS KNOWN ALREADY Earlier work has focused primarily on identifying individual risk factors for infertility. Several predictive models have been developed in subfertile populations, with relatively low discrimination (AUC: 59-64%). STUDY DESIGN, SIZE, DURATION Study participants were female, aged 21-45 years, residents of the USA or Canada, not using fertility treatment, and actively trying to conceive at enrollment (2013-2019). Participants completed a baseline questionnaire at enrollment and follow-up questionnaires every 2 months for up to 12 months or until conception. We used data from 4133 participants with no more than one menstrual cycle of pregnancy attempt at study entry. PARTICIPANTS/MATERIALS, SETTING, METHODS On the baseline questionnaire, participants reported data on sociodemographic factors, lifestyle and behavioral factors, diet quality, medical history and selected male partner characteristics. A total of 163 predictors were considered in this study. We implemented regularized logistic regression, support vector machines, neural networks and gradient boosted decision trees to derive models predicting the probability of pregnancy: (i) within fewer than 12 menstrual cycles of pregnancy attempt time (Model I), and (ii) within 6 menstrual cycles of pregnancy attempt time (Model II). Cox models were used to predict the probability of pregnancy within each menstrual cycle for up to 12 cycles of follow-up (Model III). We assessed model performance using the AUC and the weighted-F1 score for Models I and II, and the concordance index for Model III. MAIN RESULTS AND THE ROLE OF CHANCE Model I and II AUCs were 70% and 66%, respectively, in parsimonious models, and the concordance index for Model III was 63%. The predictors that were positively associated with pregnancy in all models were: having previously breastfed an infant and using multivitamins or folic acid supplements. The predictors that were inversely associated with pregnancy in all models were: female age, female BMI and history of infertility. Among nulligravid women with no history of infertility, the most important predictors were: female age, female BMI, male BMI, use of a fertility app, attempt time at study entry and perceived stress. LIMITATIONS, REASONS FOR CAUTION Reliance on self-reported predictor data could have introduced misclassification, which would likely be non-differential with respect to the pregnancy outcome given the prospective design. In addition, we cannot be certain that all relevant predictor variables were considered. Finally, though we validated the models using split-sample replication techniques, we did not conduct an external validation study. WIDER IMPLICATIONS OF THE FINDINGS Given a wide range of predictor data, machine learning algorithms can be leveraged to analyze epidemiologic data and predict the probability of conception with discrimination that exceeds earlier work. STUDY FUNDING/COMPETING INTEREST(S) The research was partially supported by the U.S. National Science Foundation (under grants DMS-1664644, CNS-1645681 and IIS-1914792) and the National Institutes for Health (under grants R01 GM135930 and UL54 TR004130). In the last 3 years, L.A.W. has received in-kind donations for primary data collection in PRESTO from FertilityFriend.com, Kindara.com, Sandstone Diagnostics and Swiss Precision Diagnostics. L.A.W. also serves as a fibroid consultant to AbbVie, Inc. The other authors declare no competing interests. TRIAL REGISTRATION NUMBER N/A.
Collapse
Affiliation(s)
- Jennifer J Yland
- Department of Epidemiology, Boston University School of Public Health, Boston, MA, USA,Correspondence address. Department of Epidemiology, Boston University School of Public Health, 715 Albany Street, Boston, MA 02118, USA. E-mail:
| | - Taiyao Wang
- Center for Information and Systems Engineering, Boston University, Boston, MA, USA,Philips Research North America, Cambridge, MA, USA
| | - Zahra Zad
- Center for Information and Systems Engineering, Boston University, Boston, MA, USA,Division of Systems Engineering, Department of Electrical and Computer Engineering, Boston University, Boston, MA, USA
| | - Sydney K Willis
- Department of Epidemiology, Boston University School of Public Health, Boston, MA, USA
| | - Tanran R Wang
- Department of Epidemiology, Boston University School of Public Health, Boston, MA, USA
| | - Amelia K Wesselink
- Department of Epidemiology, Boston University School of Public Health, Boston, MA, USA
| | - Tammy Jiang
- Department of Epidemiology, Boston University School of Public Health, Boston, MA, USA
| | - Elizabeth E Hatch
- Department of Epidemiology, Boston University School of Public Health, Boston, MA, USA
| | - Lauren A Wise
- Department of Epidemiology, Boston University School of Public Health, Boston, MA, USA
| | - Ioannis Ch Paschalidis
- Center for Information and Systems Engineering, Boston University, Boston, MA, USA,Division of Systems Engineering, Department of Electrical and Computer Engineering, Boston University, Boston, MA, USA,Department of Biomedical Engineering, Boston University, Boston, MA, USA
| |
Collapse
|
5
|
Naccarelli GV, Ruzieh M, Wolbrette DL, Sendra-Ferrer M, van Harskamp J, Bentz B, Caputo G, McConkey N, Mills K, Wasemiller S, Plamenac J, Leslie D, Glasser FD, Abendroth TW. Oral Anticoagulation Use in High-Risk Patients Is Improved by Elimination of False-Positive and Inactive Atrial Fibrillation Cases. Am J Med 2021; 134:e366-e373. [PMID: 33359273 DOI: 10.1016/j.amjmed.2020.11.024] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/13/2020] [Revised: 11/04/2020] [Accepted: 11/10/2020] [Indexed: 11/28/2022]
Abstract
BACKGROUND Multiple registries have reported that >40% of high-risk atrial fibrillation patients are not taking oral anticoagulants. The purpose of our study was to determine the presence or absence of active atrial fibrillation and CHA2DS2-VASc (Congestive heart failure, Hypertension, Age ≥75 y, Diabetes mellitus, prior Stroke [or transient ischemic attack or thromboembolism], Vascular disease, Age 65-74 y, Sex category) risk factors to accurately identify high-risk atrial fibrillation (CHA2DS2-VASc ≥2) patients requiring oral anticoagulants and the magnitude of the anticoagulant treatment gap. METHODS We retrospectively adjudicated 6514 patients with atrial fibrillation documented by at least one of: billing diagnosis, electronic medical record encounter diagnosis, electronic medical record problem list, or electrocardiogram interpretation. RESULTS After review, 4555/6514 (69.9%) had active atrial fibrillation, while 1201 had no documented history of atrial fibrillation and 758 had a history of atrial fibrillation that was no longer active. After removing the 1201 patients without a confirmed atrial fibrillation diagnosis, oral anticoagulant use in high-risk patients increased to 71.1% (P < .0001 compared with 62.9% at baseline). Oral anticoagulant use increased to 79.7% when the 758 inactive atrial fibrillation patients were also eliminated from the analysis (P < .0001 compared with baseline). In the active high-risk atrial fibrillation group, there was no significant difference in the use of oral anticoagulants between men (80.7%) and women (78.8%) with a CHA2DS2-VASc ≥2, or in women with a CHA2DS2-VASc ≥3 (79.9%). CONCLUSIONS Current registries and health system health records with unadjudicated diagnoses over-report the number of high-risk atrial fibrillation patients not taking oral anticoagulants. Expert adjudication identifies a smaller treatment gap than previously described.
Collapse
Affiliation(s)
- Gerald V Naccarelli
- Penn State University Heart and Vascular Institute, Penn State University College of Medicine, Hershey, Pa.
| | | | - Deborah L Wolbrette
- Penn State University Heart and Vascular Institute, Penn State University College of Medicine, Hershey, Pa
| | - Mauricio Sendra-Ferrer
- Division of Cardiovascular Medicine, University of Missouri School of Medicine, Columbia
| | | | - Barbara Bentz
- Penn State University Heart and Vascular Institute, Penn State University College of Medicine, Hershey, Pa
| | - Gregory Caputo
- Division of Internal Medicine, Penn State University College of Medicine, Hershey, Pa
| | - Nathan McConkey
- Penn State University Heart and Vascular Institute, Penn State University College of Medicine, Hershey, Pa
| | - Kevin Mills
- Penn State University Heart and Vascular Institute, Penn State University College of Medicine, Hershey, Pa
| | | | - Jovan Plamenac
- Penn State University Heart and Vascular Institute, Penn State University College of Medicine, Hershey, Pa
| | - Douglas Leslie
- Department of Public Health Sciences, Penn State University College of Medicine, Hershey, Pa
| | - Frendy D Glasser
- Penn State University Center for Quality Innovation, Hershey, Pa
| | - Thomas W Abendroth
- Penn State University Center for Quality Innovation, Hershey, Pa; Penn State University College of Medicine, Penn State Health, The Milton S. Hershey Medical Center, Hershey, Pa
| |
Collapse
|
6
|
Ageron FX, Coats TJ, Darioli V, Roberts I. Validation of the BATT score for prehospital risk stratification of traumatic haemorrhagic death: usefulness for tranexamic acid treatment criteria. Scand J Trauma Resusc Emerg Med 2021; 29:6. [PMID: 33407716 PMCID: PMC7789642 DOI: 10.1186/s13049-020-00827-5] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2020] [Accepted: 12/15/2020] [Indexed: 12/23/2022] Open
Abstract
BACKGROUND Tranexamic acid reduces surgical blood loss and reduces deaths from bleeding in trauma patients. Tranexamic acid must be given urgently, preferably by paramedics at the scene of the injury or in the ambulance. We developed a simple score (Bleeding Audit Triage Trauma score) to predict death from bleeding. METHODS We conducted an external validation of the BATT score using data from the UK Trauma Audit Research Network (TARN) from 1st January 2017 to 31st December 2018. We evaluated the impact of tranexamic acid treatment thresholds in trauma patients. RESULTS We included 104,862 trauma patients with an injury severity score of 9 or above. Tranexamic acid was administered to 9915 (9%) patients. Of these 5185 (52%) received prehospital tranexamic acid. The BATT score had good accuracy (Brier score = 6%) and good discrimination (C-statistic 0.90; 95% CI 0.89-0.91). Calibration in the large showed no substantial difference between predicted and observed death due to bleeding (1.15% versus 1.16%, P = 0.81). Pre-hospital tranexamic acid treatment of trauma patients with a BATT score of 2 or more would avoid 210 bleeding deaths by treating 61,598 patients instead of avoiding 55 deaths by treating 9915 as currently. CONCLUSION The BATT score identifies trauma patient at risk of significant haemorrhage. A score of 2 or more would be an appropriate threshold for pre-hospital tranexamic acid treatment.
Collapse
Affiliation(s)
- Francois-Xavier Ageron
- Clinical Trials Unit, London School of Hygiene & Tropical Medicine, London, WC1E 7HT UK
- Department of Emergency Medicine, Lausanne University Hospital, University of Lausanne, 1011 Lausanne, Switzerland
| | | | - Vincent Darioli
- Department of Emergency Medicine, Lausanne University Hospital, University of Lausanne, 1011 Lausanne, Switzerland
| | - Ian Roberts
- Clinical Trials Unit, London School of Hygiene & Tropical Medicine, London, WC1E 7HT UK
| |
Collapse
|
7
|
Davidson J, Banerjee A, Muzambi R, Smeeth L, Warren-Gash C. Validity of Acute Cardiovascular Outcome Diagnoses Recorded in European Electronic Health Records: A Systematic Review. Clin Epidemiol 2020; 12:1095-1111. [PMID: 33116903 PMCID: PMC7569174 DOI: 10.2147/clep.s265619] [Citation(s) in RCA: 28] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2020] [Accepted: 08/06/2020] [Indexed: 12/18/2022] Open
Abstract
BACKGROUND Electronic health records are widely used in cardiovascular disease research. We appraised the validity of stroke, acute coronary syndrome and heart failure diagnoses in studies conducted using European electronic health records. METHODS Using a prespecified strategy, we systematically searched seven databases from dates of inception to April 2019. Two reviewers independently completed study selection, followed by partial parallel data extraction and risk of bias assessment. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value estimates were narratively synthesized and heterogeneity between sensitivity and PPV estimates were assessed using I2. RESULTS We identified 81 studies, of which 20 validated heart failure diagnoses, 31 validated acute coronary syndrome diagnoses with 29 specifically recording estimates for myocardial infarction, and 41 validated stroke diagnoses. Few studies reported specificity or negative predictive value estimates. Sensitivity was ≤66% in all but one heart failure study, ≥80% for 91% of myocardial infarction studies, and ≥70% for 73% of stroke studies. PPV was ≥80% in 74% of heart failure, 88% of myocardial infarction, and 70% of stroke studies. PPV by stroke subtype was variable, at ≥80% for 80% of ischaemic stroke but only 44% of haemorrhagic stroke. There was considerable heterogeneity (I2 >75%) between sensitivity and PPV estimates for all diagnoses. CONCLUSION Overall, European electronic health record stroke, acute coronary syndrome and heart failure diagnoses are accurate for use in research, although validity estimates for heart failure and individual stroke subtypes were lower. Where possible, researchers should validate data before use or carefully interpret the results of previous validation studies for their own study purposes.
Collapse
Affiliation(s)
- Jennifer Davidson
- Faculty of Epidemiology & Population Health, London School of Hygiene and Tropical Medicine, London, UK
| | - Amitava Banerjee
- Institute of Health Informatics, University College London, London, UK
| | - Rutendo Muzambi
- Faculty of Epidemiology & Population Health, London School of Hygiene and Tropical Medicine, London, UK
| | - Liam Smeeth
- Faculty of Epidemiology & Population Health, London School of Hygiene and Tropical Medicine, London, UK
| | - Charlotte Warren-Gash
- Faculty of Epidemiology & Population Health, London School of Hygiene and Tropical Medicine, London, UK
| |
Collapse
|
8
|
Marchevsky AM, Walts AE, Lissenberg-Witte BI, Thunnissen E. Pathologists should probably forget about kappa. Percent agreement, diagnostic specificity and related metrics provide more clinically applicable measures of interobserver variability. Ann Diagn Pathol 2020; 47:151561. [PMID: 32623312 DOI: 10.1016/j.anndiagpath.2020.151561] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2020] [Accepted: 06/19/2020] [Indexed: 02/09/2023]
Abstract
Kappa statistics have been widely used in the pathology literature to compare interobserver diagnostic variability (IOV) among different pathologists but there has been limited discussion about the clinical significance of kappa scores. Five representative and recent pathology papers were queried using clinically relevant specific questions to learn how IOV was evaluated and how the clinical applicability of results was interpreted. The papers supported our anecdotal impression that pathologists usually assess IOV using Cohen's or Fleiss' kappa statistics and interpret the results using some variation of the scale proposed by Landis and Koch. The papers did not cite or propose specific guidelines to comment on the clinical applicability of results. The solutions proposed to decrease IOV included the development of better diagnostic criteria and additional educational efforts, but the possibility that the entities themselves represented a continuum of morphologic findings rather than distinct diagnostic categories was not considered in any of the studies. A dataset from a previous study of IOV reported by Thunnissen et al. was recalculated to estimate percent agreement among 19 international lung pathologists for the diagnosis of 74 challenging lung neuroendocrine neoplasms. Kappa scores and diagnostic sensitivity, specificity, positive and negative predictive values were calculated using the majority consensus diagnosis for each case as the gold reference diagnosis for that case. Diagnostic specificity estimates among multiple pathologists were > 90%, although kappa scores were considerably more variable. We explain why kappa scores are of limited clinical applicability in pathology and propose the use of positive and negative percent agreement and diagnostic specificity against a gold reference diagnosis to evaluate IOV among two and multiple raters, respectively.
Collapse
Affiliation(s)
- Alberto M Marchevsky
- Department of Pathology & Laboratory Medicine, Cedars-Sinai Medical Center, Los Angeles, CA, United States of America.
| | - Ann E Walts
- Department of Pathology & Laboratory Medicine, Cedars-Sinai Medical Center, Los Angeles, CA, United States of America
| | | | - Erik Thunnissen
- Department of Pathology, UMC, Vrije Universiteit Amsterdam, the Netherlands
| |
Collapse
|
9
|
Cowley LE, Farewell DM, Maguire S, Kemp AM. Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature. Diagn Progn Res 2019; 3:16. [PMID: 31463368 PMCID: PMC6704664 DOI: 10.1186/s41512-019-0060-y] [Citation(s) in RCA: 120] [Impact Index Per Article: 24.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/13/2018] [Accepted: 05/12/2019] [Indexed: 12/20/2022] Open
Abstract
Clinical prediction rules (CPRs) that predict the absolute risk of a clinical condition or future outcome for individual patients are abundant in the medical literature; however, systematic reviews have demonstrated shortcomings in the methodological quality and reporting of prediction studies. To maximise the potential and clinical usefulness of CPRs, they must be rigorously developed and validated, and their impact on clinical practice and patient outcomes must be evaluated. This review aims to present a comprehensive overview of the stages involved in the development, validation and evaluation of CPRs, and to describe in detail the methodological standards required at each stage, illustrated with examples where appropriate. Important features of the study design, statistical analysis, modelling strategy, data collection, performance assessment, CPR presentation and reporting are discussed, in addition to other, often overlooked aspects such as the acceptability, cost-effectiveness and longer-term implementation of CPRs, and their comparison with clinical judgement. Although the development and evaluation of a robust, clinically useful CPR is anything but straightforward, adherence to the plethora of methodological standards, recommendations and frameworks at each stage will assist in the development of a rigorous CPR that has the potential to contribute usefully to clinical practice and decision-making and have a positive impact on patient care.
Collapse
Affiliation(s)
- Laura E. Cowley
- Division of Population Medicine, School of Medicine, Neuadd Meirionnydd, Heath Park, Cardiff University, Wales, CF14 4YS UK
| | - Daniel M. Farewell
- Division of Population Medicine, School of Medicine, Neuadd Meirionnydd, Heath Park, Cardiff University, Wales, CF14 4YS UK
| | - Sabine Maguire
- Division of Population Medicine, School of Medicine, Neuadd Meirionnydd, Heath Park, Cardiff University, Wales, CF14 4YS UK
| | - Alison M. Kemp
- Division of Population Medicine, School of Medicine, Neuadd Meirionnydd, Heath Park, Cardiff University, Wales, CF14 4YS UK
| |
Collapse
|