1
|
Visweswaran S, Luo Y, Peleg M. Fairness and inclusion methods for biomedical informatics research. J Biomed Inform 2024:104713. [PMID: 39187169 DOI: 10.1016/j.jbi.2024.104713] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/19/2024] [Revised: 08/21/2024] [Accepted: 08/22/2024] [Indexed: 08/28/2024]
Affiliation(s)
- Shyam Visweswaran
- Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA, USA
| | - Yuan Luo
- Department of Preventive Medicine, Northwestern University, Chicago, IL, USA
| | - Mor Peleg
- Department of Information Systems, University of Haifa, Haifa, Israel.
| |
Collapse
|
2
|
Wang HE, Weiner JP, Saria S, Lehmann H, Kharrazi H. Assessing racial bias in healthcare predictive models: Practical lessons from an empirical evaluation of 30-day hospital readmission models. J Biomed Inform 2024; 156:104683. [PMID: 38925281 DOI: 10.1016/j.jbi.2024.104683] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/13/2023] [Revised: 05/20/2024] [Accepted: 06/23/2024] [Indexed: 06/28/2024]
Abstract
OBJECTIVE Despite increased availability of methodologies to identify algorithmic bias, the operationalization of bias evaluation for healthcare predictive models is still limited. Therefore, this study proposes a process for bias evaluation through an empirical assessment of common hospital readmission models. The process includes selecting bias measures, interpretation, determining disparity impact and potential mitigations. METHODS This retrospective analysis evaluated racial bias of four common models predicting 30-day unplanned readmission (i.e., LACE Index, HOSPITAL Score, and the CMS readmission measure applied as is and retrained). The models were assessed using 2.4 million adult inpatient discharges in Maryland from 2016 to 2019. Fairness metrics that are model-agnostic, easy to compute, and interpretable were implemented and apprised to select the most appropriate bias measures. The impact of changing model's risk thresholds on these measures was further assessed to guide the selection of optimal thresholds to control and mitigate bias. RESULTS Four bias measures were selected for the predictive task: zero-one-loss difference, false negative rate (FNR) parity, false positive rate (FPR) parity, and generalized entropy index. Based on these measures, the HOSPITAL score and the retrained CMS measure demonstrated the lowest racial bias. White patients showed a higher FNR while Black patients resulted in a higher FPR and zero-one-loss. As the models' risk threshold changed, trade-offs between models' fairness and overall performance were observed, and the assessment showed all models' default thresholds were reasonable for balancing accuracy and bias. CONCLUSIONS This study proposes an Applied Framework to Assess Fairness of Predictive Models (AFAFPM) and demonstrates the process using 30-day hospital readmission model as the example. It suggests the feasibility of applying algorithmic bias assessment to determine optimized risk thresholds so that predictive models can be used more equitably and accurately. It is evident that a combination of qualitative and quantitative methods and a multidisciplinary team are necessary to identify, understand and respond to algorithm bias in real-world healthcare settings. Users should also apply multiple bias measures to ensure a more comprehensive, tailored, and balanced view. The results of bias measures, however, must be interpreted with caution and consider the larger operational, clinical, and policy context.
Collapse
Affiliation(s)
- H Echo Wang
- Department of Health Policy and Management, Johns Hopkins School of Public Health, Baltimore, MD, USA.
| | - Jonathan P Weiner
- Department of Health Policy and Management, Johns Hopkins School of Public Health, Baltimore, MD, USA; Center for Population Health Information Technology, Johns Hopkins School of Public Health, Baltimore, MD, USA.
| | - Suchi Saria
- Department of Computer Science and Statistics, Whiting School of Engineering, Johns Hopkins University, Baltimore, MD, USA.
| | - Harold Lehmann
- Biomedical Informatics and Data Science, Division of General Internal Medicine, Johns Hopkins School of Medicine, Baltimore, MD, USA.
| | - Hadi Kharrazi
- Department of Health Policy and Management, Johns Hopkins School of Public Health, Baltimore, MD, USA; Center for Population Health Information Technology, Johns Hopkins School of Public Health, Baltimore, MD, USA; Biomedical Informatics and Data Science, Division of General Internal Medicine, Johns Hopkins School of Medicine, Baltimore, MD, USA.
| |
Collapse
|
3
|
Jain SS, Elias P, Poterucha T, Randazzo M, Lopez Jimenez F, Khera R, Perez M, Ouyang D, Pirruccello J, Salerno M, Einstein AJ, Avram R, Tison GH, Nadkarni G, Natarajan V, Pierson E, Beecy A, Kumaraiah D, Haggerty C, Avari Silva JN, Maddox TM. Artificial Intelligence in Cardiovascular Care-Part 2: Applications: JACC Review Topic of the Week. J Am Coll Cardiol 2024; 83:2487-2496. [PMID: 38593945 DOI: 10.1016/j.jacc.2024.03.401] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/26/2024] [Accepted: 03/14/2024] [Indexed: 04/11/2024]
Abstract
Recent artificial intelligence (AI) advancements in cardiovascular care offer potential enhancements in effective diagnosis, treatment, and outcomes. More than 600 U.S. Food and Drug Administration-approved clinical AI algorithms now exist, with 10% focusing on cardiovascular applications, highlighting the growing opportunities for AI to augment care. This review discusses the latest advancements in the field of AI, with a particular focus on the utilization of multimodal inputs and the field of generative AI. Further discussions in this review involve an approach to understanding the larger context in which AI-augmented care may exist, and include a discussion of the need for rigorous evaluation, appropriate infrastructure for deployment, ethics and equity assessments, regulatory oversight, and viable business cases for deployment. Embracing this rapidly evolving technology while setting an appropriately high evaluation benchmark with careful and patient-centered implementation will be crucial for cardiology to leverage AI to enhance patient care and the provider experience.
Collapse
Affiliation(s)
- Sneha S Jain
- Division of Cardiology, Stanford University School of Medicine, Palo Alto, California, USA
| | - Pierre Elias
- Seymour, Paul and Gloria Milstein Division of Cardiology, Columbia University Irving Medical Center, New York, New York, USA; Department of Biomedical Informatics Columbia University Irving Medical Center, New York, New York, USA
| | - Timothy Poterucha
- Seymour, Paul and Gloria Milstein Division of Cardiology, Columbia University Irving Medical Center, New York, New York, USA
| | - Michael Randazzo
- Division of Cardiology, University of Chicago Medical Center, Chicago, Illinois, USA
| | | | - Rohan Khera
- Division of Cardiology, Yale School of Medicine, New Haven, Connecticut, USA
| | - Marco Perez
- Division of Cardiology, Stanford University School of Medicine, Palo Alto, California, USA
| | - David Ouyang
- Division of Cardiology, Cedars-Sinai Medical Center, Los Angeles, California, USA
| | - James Pirruccello
- Division of Cardiology, University of California San Francisco, San Francisco, California, USA
| | - Michael Salerno
- Division of Cardiology, Stanford University School of Medicine, Palo Alto, California, USA
| | - Andrew J Einstein
- Seymour, Paul and Gloria Milstein Division of Cardiology, Columbia University Irving Medical Center, New York, New York, USA
| | - Robert Avram
- Division of Cardiology, Montreal Heart Institute, Montreal, Quebec, Canada
| | - Geoffrey H Tison
- Division of Cardiology, University of California San Francisco, San Francisco, California, USA
| | - Girish Nadkarni
- Icahn School of Medicine at Mount Sinai, New York, New York, USA
| | | | - Emma Pierson
- Department of Computer Science, Cornell Tech, New York, New York, USA
| | - Ashley Beecy
- NewYork-Presbyterian Health System, New York, New York, USA; Division of Cardiology, Weill Cornell Medical College, New York, New York, USA
| | - Deepa Kumaraiah
- Seymour, Paul and Gloria Milstein Division of Cardiology, Columbia University Irving Medical Center, New York, New York, USA; NewYork-Presbyterian Health System, New York, New York, USA
| | - Chris Haggerty
- Department of Biomedical Informatics Columbia University Irving Medical Center, New York, New York, USA; NewYork-Presbyterian Health System, New York, New York, USA
| | - Jennifer N Avari Silva
- Division of Cardiology, Washington University School of Medicine, St Louis, Missouri, USA
| | - Thomas M Maddox
- Division of Cardiology, Washington University School of Medicine, St Louis, Missouri, USA.
| |
Collapse
|
4
|
Bear Don't Walk OJ, Paullada A, Everhart A, Casanova-Perez R, Cohen T, Veinot T. Opportunities for incorporating intersectionality into biomedical informatics. J Biomed Inform 2024; 154:104653. [PMID: 38734158 PMCID: PMC11146624 DOI: 10.1016/j.jbi.2024.104653] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/18/2023] [Revised: 04/06/2024] [Accepted: 05/08/2024] [Indexed: 05/13/2024]
Abstract
Many approaches in biomedical informatics (BMI) rely on the ability to define, gather, and manipulate biomedical data to support health through a cyclical research-practice lifecycle. Researchers within this field are often fortunate to work closely with healthcare and public health systems to influence data generation and capture and have access to a vast amount of biomedical data. Many informaticists also have the expertise to engage with stakeholders, develop new methods and applications, and influence policy. However, research and policy that explicitly seeks to address the systemic drivers of health would more effectively support health. Intersectionality is a theoretical framework that can facilitate such research. It holds that individual human experiences reflect larger socio-structural level systems of privilege and oppression, and cannot be truly understood if these systems are examined in isolation. Intersectionality explicitly accounts for the interrelated nature of systems of privilege and oppression, providing a lens through which to examine and challenge inequities. In this paper, we propose intersectionality as an intervention into how we conduct BMI research. We begin by discussing intersectionality's history and core principles as they apply to BMI. We then elaborate on the potential for intersectionality to stimulate BMI research. Specifically, we posit that our efforts in BMI to improve health should address intersectionality's five key considerations: (1) systems of privilege and oppression that shape health; (2) the interrelated nature of upstream health drivers; (3) the nuances of health outcomes within groups; (4) the problematic and power-laden nature of categories that we assign to people in research and in society; and (5) research to inform and support social change.
Collapse
Affiliation(s)
- Oliver J Bear Don't Walk
- Department of Biomedical Informatics and Medical Education, University of Washington, United States.
| | - Amandalynne Paullada
- Department of Biomedical Informatics and Medical Education, University of Washington, United States
| | - Avery Everhart
- Department of Geography, Faculty of Arts, University of British Columbia, Canada
| | - Reggie Casanova-Perez
- Department of Biomedical Informatics and Medical Education, University of Washington, United States
| | - Trevor Cohen
- Department of Biomedical Informatics and Medical Education, University of Washington, United States
| | - Tiffany Veinot
- School of Information and School of Public Health, University of Michigan, United States
| |
Collapse
|
5
|
Deardorff WJ, Diaz-Ramirez LG, Boscardin WJ, Smith AK, Lee SJ. Around the EQUATOR with Clin-STAR: Prediction modeling opportunities and challenges in aging research. J Am Geriatr Soc 2024; 72:1658-1668. [PMID: 38032070 PMCID: PMC11137550 DOI: 10.1111/jgs.18704] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/16/2023] [Revised: 10/16/2023] [Accepted: 10/30/2023] [Indexed: 12/01/2023]
Abstract
The 2015 Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement was published to improve reporting transparency for prediction modeling studies. The objective of this review is to highlight methodologic challenges that aging-focused researchers will encounter when designing and reporting studies involving prediction models for older adults and provide guidance for addressing these challenges. In following the 22-item TRIPOD checklist, researchers must consider the representativeness of cohorts used (e.g., whether older adults with frailty, cognitive impairment, and social isolation were included), strategies for incorporating common geriatric predictors (e.g., age, comorbidities, functional status, and frailty), methods for handling missing data and competing risk of death, and assessment of model performance heterogeneity across important subgroups (e.g., age, sex, race, and ethnicity). We provide guidance to help aging-focused researchers develop, validate, and report models that can inform and improve patient care, which we label "TRIPOD-65."
Collapse
Affiliation(s)
- W. James Deardorff
- Division of Geriatrics, University of California, San
Francisco, San Francisco, California
- San Francisco Veterans Affairs Medical Center, San
Francisco, California
| | - L. Grisell Diaz-Ramirez
- Division of Geriatrics, University of California, San
Francisco, San Francisco, California
- San Francisco Veterans Affairs Medical Center, San
Francisco, California
| | - W. John Boscardin
- Division of Geriatrics, University of California, San
Francisco, San Francisco, California
- San Francisco Veterans Affairs Medical Center, San
Francisco, California
- Department of Epidemiology and Biostatistics, University of
California, San Francisco, San Francisco, California
| | - Alexander K. Smith
- Division of Geriatrics, University of California, San
Francisco, San Francisco, California
- San Francisco Veterans Affairs Medical Center, San
Francisco, California
| | - Sei J. Lee
- Division of Geriatrics, University of California, San
Francisco, San Francisco, California
- San Francisco Veterans Affairs Medical Center, San
Francisco, California
| |
Collapse
|
6
|
Zhang K, Han Y, Gu F, Gu Z, Zhao J, Chen J, Chen B, Gao M, Hou Z, Yu X, Cai T, Gao Y, Hu R, Xie J, Liu T, Liu K. Association between serum chloride and in-hospital mortality in congestive heart failure with diabetes: Data from the MIMIC-IV database. J Diabetes Metab Disord 2024; 23:859-870. [PMID: 38932886 PMCID: PMC11196478 DOI: 10.1007/s40200-023-01362-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/09/2023] [Accepted: 11/27/2023] [Indexed: 06/28/2024]
Abstract
Background Congestive heart failure (CHF) demonstrates a heightened prevalence in individuals with diabetes mellitus within Intensive Care Units. The occurrence of abnormal chloride levels is frequently observed in critically ill patients, yet its clinical significance remains subject to debate. This study endeavors to explore the relationship between serum chloride levels and in-hospital mortality among patients affected by both congestive heart failure and diabetes. Methods A retrospective cohort study was conducted, utilizing data from the Medical Information Mart for Intensive Care-IV (MIMIC-IV) database, focusing on adult patients in the United States. The impact of serum chloride levels upon ICU admission on in-hospital mortality was analyzed using multivariable logistic regression models, generalized additive models and subgroup analysis. Results The study encompassed 7,063 patients with coexisting diabetes and congestive heart failure. The fully adjusted model revealed an inverse association between serum chloride levels and in-hospital mortality. As a tertile variable (Q3 vs Q1), the odds ratio (OR) was 0.73 with a 95% confidence interval (CI) of 0.54-0.98 (p = 0.039). As a continuous variable, per 1 mmol/L increment, the OR (95% CI) was 0.97 (0.96-0.99, p = 0.01). The relationship between serum chloride and in-hospital mortality demonstrated linearity (non-linear p = 0.958). Stratified analyses further validated the robustness of this correlation. Conclusions Serum chloride levels exhibited a negative association with in-hospital mortality in patients with both congestive heart failure and diabetes. Nevertheless, prospective, randomized, controlled studies are warranted to corroborate and validate the findings presented in this investigation.
Collapse
Affiliation(s)
- Kai Zhang
- Cardiovascular Surgery Department of the Second Hospital of Jilin University, No. 218, Ziqiang Street, Changchun, Jilin Province China
| | - Yu Han
- Department of Ophthalmology, First Hospital of Jilin University, Changchun, China
| | - Fangming Gu
- Bethune Second College of Clinical Medicine, Jilin University, Changchun, China
| | - Zhaoxuan Gu
- Bethune Second College of Clinical Medicine, Jilin University, Changchun, China
| | - JiaYu Zhao
- Bethune Second College of Clinical Medicine, Jilin University, Changchun, China
| | - Jianguo Chen
- Bethune First College of Clinical Medicine, Jilin University, Changchun, China
| | - Bowen Chen
- Bethune First College of Clinical Medicine, Jilin University, Changchun, China
| | - Min Gao
- Department of Cancer Center, The First Hospital of Jilin University, Changchun, China
| | - Zhengyan Hou
- Bethune Second College of Clinical Medicine, Jilin University, Changchun, China
| | - Xiaoqi Yu
- Bethune Second College of Clinical Medicine, Jilin University, Changchun, China
| | - Tianyi Cai
- Bethune Second College of Clinical Medicine, Jilin University, Changchun, China
| | - Yafang Gao
- Bethune Second College of Clinical Medicine, Jilin University, Changchun, China
| | - Rui Hu
- Bethune Third College of Clinical Medicine, Jilin University, Changchun, China
| | - Jinyu Xie
- Bethune Second College of Clinical Medicine, Jilin University, Changchun, China
| | - Tianzhou Liu
- Department of Gastrointestinal Surgery, The Second Hospital of Jilin University, Changchun, China
| | - Kexiang Liu
- Cardiovascular Surgery Department of the Second Hospital of Jilin University, No. 218, Ziqiang Street, Changchun, Jilin Province China
| |
Collapse
|
7
|
Li C, Lai D, Jiang X, Zhang K. FERI: A Multitask-based Fairness Achieving Algorithm with Applications to Fair Organ Transplantation. AMIA JOINT SUMMITS ON TRANSLATIONAL SCIENCE PROCEEDINGS. AMIA JOINT SUMMITS ON TRANSLATIONAL SCIENCE 2024; 2024:593-602. [PMID: 38827050 PMCID: PMC11141863] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Subscribe] [Scholar Register] [Indexed: 06/04/2024]
Abstract
Liver transplantation often faces fairness challenges across subgroups defined by sensitive attributes such as age group, gender, and race/ethnicity. Machine learning models for outcome prediction can introduce additional biases. Therefore, we introduce Fairness through the Equitable Rate of Improvement in Multitask Learning (FERI) algorithm for fair predictions of graft failure risk in liver transplant patients. FERI constrains subgroup loss by balancing learning rates and preventing subgroup dominance in the training process. Our results show that FERI maintained high predictive accuracy with AUROC and AUPRC comparable to baseline models. More importantly, FERI demonstrated an ability to improve fairness without sacrificing accuracy. Specifically, for the gender, FERI reduced the demographic parity disparity by 71.74%, and for the age group, it decreased the equalized odds disparity by 40.46%. Therefore, the FERI algorithm advanced fairness-aware predictive modeling in healthcare and provides an invaluable tool for equitable healthcare systems.
Collapse
Affiliation(s)
- Can Li
- Department of Biostatistics and Data Science, School of Public Health, The University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Dejian Lai
- Department of Biostatistics and Data Science, School of Public Health, The University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Xiaoqian Jiang
- Department of Health Data Science and Artificial Intelligence, McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Kai Zhang
- Department of Health Data Science and Artificial Intelligence, McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, USA
| |
Collapse
|
8
|
Wang HE, Weiner JP, Saria S, Kharrazi H. Evaluating Algorithmic Bias in 30-Day Hospital Readmission Models: Retrospective Analysis. J Med Internet Res 2024; 26:e47125. [PMID: 38422347 PMCID: PMC11066744 DOI: 10.2196/47125] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2023] [Revised: 12/28/2023] [Accepted: 02/27/2024] [Indexed: 03/02/2024] Open
Abstract
BACKGROUND The adoption of predictive algorithms in health care comes with the potential for algorithmic bias, which could exacerbate existing disparities. Fairness metrics have been proposed to measure algorithmic bias, but their application to real-world tasks is limited. OBJECTIVE This study aims to evaluate the algorithmic bias associated with the application of common 30-day hospital readmission models and assess the usefulness and interpretability of selected fairness metrics. METHODS We used 10.6 million adult inpatient discharges from Maryland and Florida from 2016 to 2019 in this retrospective study. Models predicting 30-day hospital readmissions were evaluated: LACE Index, modified HOSPITAL score, and modified Centers for Medicare & Medicaid Services (CMS) readmission measure, which were applied as-is (using existing coefficients) and retrained (recalibrated with 50% of the data). Predictive performances and bias measures were evaluated for all, between Black and White populations, and between low- and other-income groups. Bias measures included the parity of false negative rate (FNR), false positive rate (FPR), 0-1 loss, and generalized entropy index. Racial bias represented by FNR and FPR differences was stratified to explore shifts in algorithmic bias in different populations. RESULTS The retrained CMS model demonstrated the best predictive performance (area under the curve: 0.74 in Maryland and 0.68-0.70 in Florida), and the modified HOSPITAL score demonstrated the best calibration (Brier score: 0.16-0.19 in Maryland and 0.19-0.21 in Florida). Calibration was better in White (compared to Black) populations and other-income (compared to low-income) groups, and the area under the curve was higher or similar in the Black (compared to White) populations. The retrained CMS and modified HOSPITAL score had the lowest racial and income bias in Maryland. In Florida, both of these models overall had the lowest income bias and the modified HOSPITAL score showed the lowest racial bias. In both states, the White and higher-income populations showed a higher FNR, while the Black and low-income populations resulted in a higher FPR and a higher 0-1 loss. When stratified by hospital and population composition, these models demonstrated heterogeneous algorithmic bias in different contexts and populations. CONCLUSIONS Caution must be taken when interpreting fairness measures' face value. A higher FNR or FPR could potentially reflect missed opportunities or wasted resources, but these measures could also reflect health care use patterns and gaps in care. Simply relying on the statistical notions of bias could obscure or underplay the causes of health disparity. The imperfect health data, analytic frameworks, and the underlying health systems must be carefully considered. Fairness measures can serve as a useful routine assessment to detect disparate model performances but are insufficient to inform mechanisms or policy changes. However, such an assessment is an important first step toward data-driven improvement to address existing health disparities.
Collapse
Affiliation(s)
- H Echo Wang
- Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, United States
| | - Jonathan P Weiner
- Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, United States
- Johns Hopkins Center for Population Health Information Technology, Baltimore, MD, United States
| | - Suchi Saria
- Whiting School of Engineering, Johns Hopkins University, Baltimore, MD, United States
| | - Hadi Kharrazi
- Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, United States
- Johns Hopkins Center for Population Health Information Technology, Baltimore, MD, United States
| |
Collapse
|
9
|
Vaidya A, Chen RJ, Williamson DFK, Song AH, Jaume G, Yang Y, Hartvigsen T, Dyer EC, Lu MY, Lipkova J, Shaban M, Chen TY, Mahmood F. Demographic bias in misdiagnosis by computational pathology models. Nat Med 2024; 30:1174-1190. [PMID: 38641744 DOI: 10.1038/s41591-024-02885-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/03/2023] [Accepted: 02/23/2024] [Indexed: 04/21/2024]
Abstract
Despite increasing numbers of regulatory approvals, deep learning-based computational pathology systems often overlook the impact of demographic factors on performance, potentially leading to biases. This concern is all the more important as computational pathology has leveraged large public datasets that underrepresent certain demographic groups. Using publicly available data from The Cancer Genome Atlas and the EBRAINS brain tumor atlas, as well as internal patient data, we show that whole-slide image classification models display marked performance disparities across different demographic groups when used to subtype breast and lung carcinomas and to predict IDH1 mutations in gliomas. For example, when using common modeling approaches, we observed performance gaps (in area under the receiver operating characteristic curve) between white and Black patients of 3.0% for breast cancer subtyping, 10.9% for lung cancer subtyping and 16.0% for IDH1 mutation prediction in gliomas. We found that richer feature representations obtained from self-supervised vision foundation models reduce performance variations between groups. These representations provide improvements upon weaker models even when those weaker models are combined with state-of-the-art bias mitigation strategies and modeling choices. Nevertheless, self-supervised vision foundation models do not fully eliminate these discrepancies, highlighting the continuing need for bias mitigation efforts in computational pathology. Finally, we demonstrate that our results extend to other demographic factors beyond patient race. Given these findings, we encourage regulatory and policy agencies to integrate demographic-stratified evaluation into their assessment guidelines.
Collapse
Affiliation(s)
- Anurag Vaidya
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Health Sciences and Technology, Harvard-MIT, Cambridge, MA, USA
| | - Richard J Chen
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
| | - Drew F K Williamson
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology and Laboratory Medicine, Emory University School of Medicine, Atlanta, GA, USA
| | - Andrew H Song
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Guillaume Jaume
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Yuzhe Yang
- Electrical Engineering and Computer Science, MIT, Cambridge, MA, USA
| | - Thomas Hartvigsen
- School of Data Science, University of Virginia, Charlottesville, VA, USA
| | - Emma C Dyer
- T.H. Chan School of Public Health, Harvard University, Cambridge, MA, USA
| | - Ming Y Lu
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Electrical Engineering and Computer Science, MIT, Cambridge, MA, USA
| | - Jana Lipkova
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Muhammad Shaban
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Tiffany Y Chen
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Faisal Mahmood
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA.
- Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA, USA.
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA.
- Harvard Data Science Initiative, Harvard University, Cambridge, MA, USA.
| |
Collapse
|
10
|
Movva R, Koh PW, Pierson E. Using unlabeled data to enhance fairness of medical AI. Nat Med 2024; 30:944-945. [PMID: 38641743 DOI: 10.1038/s41591-024-02892-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/21/2024]
Affiliation(s)
- Rajiv Movva
- Department of Computer Science, Cornell Tech, New York, NY, USA
| | - Pang Wei Koh
- Department of Computer Science, University of Washington, Seattle, WA, USA
| | - Emma Pierson
- Department of Computer Science, Cornell Tech, New York, NY, USA.
| |
Collapse
|
11
|
Wang R, Kuo PC, Chen LC, Seastedt KP, Gichoya JW, Celi LA. Drop the shortcuts: image augmentation improves fairness and decreases AI detection of race and other demographics from medical images. EBioMedicine 2024; 102:105047. [PMID: 38471396 PMCID: PMC10945176 DOI: 10.1016/j.ebiom.2024.105047] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2023] [Revised: 02/15/2024] [Accepted: 02/21/2024] [Indexed: 03/14/2024] Open
Abstract
BACKGROUND It has been shown that AI models can learn race on medical images, leading to algorithmic bias. Our aim in this study was to enhance the fairness of medical image models by eliminating bias related to race, age, and sex. We hypothesise models may be learning demographics via shortcut learning and combat this using image augmentation. METHODS This study included 44,953 patients who identified as Asian, Black, or White (mean age, 60.68 years ±18.21; 23,499 women) for a total of 194,359 chest X-rays (CXRs) from MIMIC-CXR database. The included CheXpert images comprised 45,095 patients (mean age 63.10 years ±18.14; 20,437 women) for a total of 134,300 CXRs were used for external validation. We also collected 1195 3D brain magnetic resonance imaging (MRI) data from the ADNI database, which included 273 participants with an average age of 76.97 years ±14.22, and 142 females. DL models were trained on either non-augmented or augmented images and assessed using disparity metrics. The features learned by the models were analysed using task transfer experiments and model visualisation techniques. FINDINGS In the detection of radiological findings, training a model using augmented CXR images was shown to reduce disparities in error rate among racial groups (-5.45%), age groups (-13.94%), and sex (-22.22%). For AD detection, the model trained with augmented MRI images was shown 53.11% and 31.01% reduction of disparities in error rate among age and sex groups, respectively. Image augmentation led to a reduction in the model's ability to identify demographic attributes and resulted in the model trained for clinical purposes incorporating fewer demographic features. INTERPRETATION The model trained using the augmented images was less likely to be influenced by demographic information in detecting image labels. These results demonstrate that the proposed augmentation scheme could enhance the fairness of interpretations by DL models when dealing with data from patients with different demographic backgrounds. FUNDING National Science and Technology Council (Taiwan), National Institutes of Health.
Collapse
Affiliation(s)
- Ryan Wang
- Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan
| | - Po-Chih Kuo
- Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan.
| | - Li-Ching Chen
- Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan
| | - Kenneth Patrick Seastedt
- Department of Surgery, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA; Department of Thoracic Surgery, Roswell Park Comprehensive Cancer Center, Buffalo, NY, USA
| | | | - Leo Anthony Celi
- Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, MA, USA; Division of Pulmonary Critical Care and Sleep Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA; Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, USA
| |
Collapse
|
12
|
Guo LL, Morse KE, Aftandilian C, Steinberg E, Fries J, Posada J, Fleming SL, Lemmon J, Jessa K, Shah N, Sung L. Characterizing the limitations of using diagnosis codes in the context of machine learning for healthcare. BMC Med Inform Decis Mak 2024; 24:51. [PMID: 38355486 PMCID: PMC10868117 DOI: 10.1186/s12911-024-02449-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2023] [Accepted: 01/30/2024] [Indexed: 02/16/2024] Open
Abstract
BACKGROUND Diagnostic codes are commonly used as inputs for clinical prediction models, to create labels for prediction tasks, and to identify cohorts for multicenter network studies. However, the coverage rates of diagnostic codes and their variability across institutions are underexplored. The primary objective was to describe lab- and diagnosis-based labels for 7 selected outcomes at three institutions. Secondary objectives were to describe agreement, sensitivity, and specificity of diagnosis-based labels against lab-based labels. METHODS This study included three cohorts: SickKids from The Hospital for Sick Children, and StanfordPeds and StanfordAdults from Stanford Medicine. We included seven clinical outcomes with lab-based definitions: acute kidney injury, hyperkalemia, hypoglycemia, hyponatremia, anemia, neutropenia and thrombocytopenia. For each outcome, we created four lab-based labels (abnormal, mild, moderate and severe) based on test result and one diagnosis-based label. Proportion of admissions with a positive label were presented for each outcome stratified by cohort. Using lab-based labels as the gold standard, agreement using Cohen's Kappa, sensitivity and specificity were calculated for each lab-based severity level. RESULTS The number of admissions included were: SickKids (n = 59,298), StanfordPeds (n = 24,639) and StanfordAdults (n = 159,985). The proportion of admissions with a positive diagnosis-based label was significantly higher for StanfordPeds compared to SickKids across all outcomes, with odds ratio (99.9% confidence interval) for abnormal diagnosis-based label ranging from 2.2 (1.7-2.7) for neutropenia to 18.4 (10.1-33.4) for hyperkalemia. Lab-based labels were more similar by institution. When using lab-based labels as the gold standard, Cohen's Kappa and sensitivity were lower at SickKids for all severity levels compared to StanfordPeds. CONCLUSIONS Across multiple outcomes, diagnosis codes were consistently different between the two pediatric institutions. This difference was not explained by differences in test results. These results may have implications for machine learning model development and deployment.
Collapse
Affiliation(s)
- Lin Lawrence Guo
- Program in Child Health Evaluative Sciences, The Hospital for Sick Children, Toronto, ON, Canada
| | - Keith E Morse
- Division of Pediatric Hospital Medicine, Department of Pediatrics, Stanford University, Palo Alto, CA, USA
| | - Catherine Aftandilian
- Division of Hematology/Oncology, Department of Pediatrics, Stanford University, Palo Alto, CA, USA
| | - Ethan Steinberg
- Stanford Center for Biomedical Informatics Research, Stanford University, Palo Alto, CA, USA
| | - Jason Fries
- Stanford Center for Biomedical Informatics Research, Stanford University, Palo Alto, CA, USA
| | - Jose Posada
- Universidad del Norte, Barranquilla, Colombia
| | - Scott Lanyon Fleming
- Stanford Center for Biomedical Informatics Research, Stanford University, Palo Alto, CA, USA
| | - Joshua Lemmon
- Program in Child Health Evaluative Sciences, The Hospital for Sick Children, Toronto, ON, Canada
| | - Karim Jessa
- Information Services, The Hospital for Sick Children, Toronto, ON, Canada
| | - Nigam Shah
- Stanford Center for Biomedical Informatics Research, Stanford University, Palo Alto, CA, USA
| | - Lillian Sung
- Program in Child Health Evaluative Sciences, The Hospital for Sick Children, Toronto, ON, Canada.
- Division of Haematology/Oncology, The Hospital for Sick Children, 555 University Avenue, M5G1X8, Toronto, ON, Canada.
| |
Collapse
|
13
|
Weng WH, Sellergen A, Kiraly AP, D'Amour A, Park J, Pilgrim R, Pfohl S, Lau C, Natarajan V, Azizi S, Karthikesalingam A, Cole-Lewis H, Matias Y, Corrado GS, Webster DR, Shetty S, Prabhakara S, Eswaran K, Celi LAG, Liu Y. An intentional approach to managing bias in general purpose embedding models. Lancet Digit Health 2024; 6:e126-e130. [PMID: 38278614 DOI: 10.1016/s2589-7500(23)00227-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/15/2023] [Revised: 10/24/2023] [Accepted: 11/02/2023] [Indexed: 01/28/2024]
Abstract
Advances in machine learning for health care have brought concerns about bias from the research community; specifically, the introduction, perpetuation, or exacerbation of care disparities. Reinforcing these concerns is the finding that medical images often reveal signals about sensitive attributes in ways that are hard to pinpoint by both algorithms and people. This finding raises a question about how to best design general purpose pretrained embeddings (GPPEs, defined as embeddings meant to support a broad array of use cases) for building downstream models that are free from particular types of bias. The downstream model should be carefully evaluated for bias, and audited and improved as appropriate. However, in our view, well intentioned attempts to prevent the upstream components-GPPEs-from learning sensitive attributes can have unintended consequences on the downstream models. Despite producing a veneer of technical neutrality, the resultant end-to-end system might still be biased or poorly performing. We present reasons, by building on previously published data, to support the reasoning that GPPEs should ideally contain as much information as the original data contain, and highlight the perils of trying to remove sensitive attributes from a GPPE. We also emphasise that downstream prediction models trained for specific tasks and settings, whether developed using GPPEs or not, should be carefully designed and evaluated to avoid bias that makes models vulnerable to issues such as distributional shift. These evaluations should be done by a diverse team, including social scientists, on a diverse cohort representing the full breadth of the patient population for which the final model is intended.
Collapse
Affiliation(s)
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Leo A G Celi
- Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA, USA; Department of Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Yun Liu
- Google, Mountain View, CA, USA.
| |
Collapse
|
14
|
Shah NH, Halamka JD, Saria S, Pencina M, Tazbaz T, Tripathi M, Callahan A, Hildahl H, Anderson B. A Nationwide Network of Health AI Assurance Laboratories. JAMA 2024; 331:245-249. [PMID: 38117493 DOI: 10.1001/jama.2023.26930] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/21/2023]
Abstract
Importance Given the importance of rigorous development and evaluation standards needed of artificial intelligence (AI) models used in health care, nationwide accepted procedures to provide assurance that the use of AI is fair, appropriate, valid, effective, and safe are urgently needed. Observations While there are several efforts to develop standards and best practices to evaluate AI, there is a gap between having such guidance and the application of such guidance to both existing and new AI models being developed. As of now, there is no publicly available, nationwide mechanism that enables objective evaluation and ongoing assessment of the consequences of using health AI models in clinical care settings. Conclusion and Relevance The need to create a public-private partnership to support a nationwide health AI assurance labs network is outlined here. In this network, community best practices could be applied for testing health AI models to produce reports on their performance that can be widely shared for managing the lifecycle of AI models over time and across populations and sites where these models are deployed.
Collapse
Affiliation(s)
- Nigam H Shah
- Stanford Medicine, Palo Alto, California
- Coalition for Health AI, Dover, Delaware
| | - John D Halamka
- Coalition for Health AI, Dover, Delaware
- Mayo Clinic Platform, Mayo Clinic, Rochester, Minnesota
| | - Suchi Saria
- Coalition for Health AI, Dover, Delaware
- Bayesian Health, New York, New York
- Johns Hopkins University, Baltimore, Maryland
- Johns Hopkins Medicine, Baltimore, Maryland
| | - Michael Pencina
- Coalition for Health AI, Dover, Delaware
- Duke AI Health, Duke University School of Medicine, Durham, North Carolina
| | - Troy Tazbaz
- US Food and Drug Administration, Silver Spring, Maryland
| | - Micky Tripathi
- US Office of the National Coordinator for Health IT, Washington, DC
| | | | | | - Brian Anderson
- Coalition for Health AI, Dover, Delaware
- MITRE Corporation, Bedford, Massachusetts
| |
Collapse
|
15
|
Pierson E. Accuracy and Equity in Clinical Risk Prediction. N Engl J Med 2024; 390:100-102. [PMID: 38198167 DOI: 10.1056/nejmp2311050] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/11/2024]
Affiliation(s)
- Emma Pierson
- From the Department of Computer Science, Cornell Tech, and the Department of Population Health Sciences, Weill Cornell Medical College - both in New York
| |
Collapse
|
16
|
Yuan C, Linn KA, Hubbard RA. Algorithmic Fairness of Machine Learning Models for Alzheimer Disease Progression. JAMA Netw Open 2023; 6:e2342203. [PMID: 37934495 PMCID: PMC10630899 DOI: 10.1001/jamanetworkopen.2023.42203] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/13/2023] [Accepted: 09/27/2023] [Indexed: 11/08/2023] Open
Abstract
Importance Predictive models using machine learning techniques have potential to improve early detection and management of Alzheimer disease (AD). However, these models potentially have biases and may perpetuate or exacerbate existing disparities. Objective To characterize the algorithmic fairness of longitudinal prediction models for AD progression. Design, Setting, and Participants This prognostic study investigated the algorithmic fairness of logistic regression, support vector machines, and recurrent neural networks for predicting progression to mild cognitive impairment (MCI) and AD using data from participants in the Alzheimer Disease Neuroimaging Initiative evaluated at 57 sites in the US and Canada. Participants aged 54 to 91 years who contributed data on at least 2 visits between September 2005 and May 2017 were included. Data were analyzed in October 2022. Exposures Fairness was quantified across sex, ethnicity, and race groups. Neuropsychological test scores, anatomical features from T1 magnetic resonance imaging, measures extracted from positron emission tomography, and cerebrospinal fluid biomarkers were included as predictors. Main Outcomes and Measures Outcome measures quantified fairness of prediction models (logistic regression [LR], support vector machine [SVM], and recurrent neural network [RNN] models), including equal opportunity, equalized odds, and demographic parity. Specifically, if the model exhibited equal sensitivity for all groups, it aligned with the principle of equal opportunity, indicating fairness in predictive performance. Results A total of 1730 participants in the cohort (mean [SD] age, 73.81 [6.92] years; 776 females [44.9%]; 69 Hispanic [4.0%] and 1661 non-Hispanic [96.0%]; 29 Asian [1.7%], 77 Black [4.5%], 1599 White [92.4%], and 25 other race [1.4%]) were included. Sensitivity for predicting progression to MCI and AD was lower for Hispanic participants compared with non-Hispanic participants; the difference (SD) in true positive rate ranged from 20.9% (5.5%) for the RNN model to 27.8% (9.8%) for the SVM model in MCI and 24.1% (5.4%) for the RNN model to 48.2% (17.3%) for the LR model in AD. Sensitivity was similarly lower for Black and Asian participants compared with non-Hispanic White participants; for example, the difference (SD) in AD true positive rate was 14.5% (51.6%) in the LR model, 12.3% (35.1%) in the SVM model, and 28.4% (16.8%) in the RNN model for Black vs White participants, and the difference (SD) in MCI true positive rate was 25.6% (13.1%) in the LR model, 24.3% (13.1%) in the SVM model, and 6.8% (18.7%) in the RNN model for Asian vs White participants. Models generally satisfied metrics of fairness with respect to sex, with no significant differences by group, except for cognitively normal (CN)-MCI and MCI-AD transitions (eg, an absolute increase [SD] in the true positive rate of CN-MCI transitions of 10.3% [27.8%] for the LR model). Conclusions and Relevance In this study, models were accurate in aggregate but failed to satisfy fairness metrics. These findings suggest that fairness should be considered in the development and use of machine learning models for AD progression.
Collapse
Affiliation(s)
- Chenxi Yuan
- Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia
- Penn Statistics in Imaging and Visualization Endeavor, Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia
| | - Kristin A. Linn
- Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia
- Penn Statistics in Imaging and Visualization Endeavor, Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia
| | - Rebecca A. Hubbard
- Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia
| |
Collapse
|
17
|
Cary MP, Zink A, Wei S, Olson A, Yan M, Senior R, Bessias S, Gadhoumi K, Jean-Pierre G, Wang D, Ledbetter LS, Economou-Zavlanos NJ, Obermeyer Z, Pencina MJ. Mitigating Racial And Ethnic Bias And Advancing Health Equity In Clinical Algorithms: A Scoping Review. Health Aff (Millwood) 2023; 42:1359-1368. [PMID: 37782868 PMCID: PMC10668606 DOI: 10.1377/hlthaff.2023.00553] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/04/2023]
Abstract
In August 2022 the Department of Health and Human Services (HHS) issued a notice of proposed rulemaking prohibiting covered entities, which include health care providers and health plans, from discriminating against individuals when using clinical algorithms in decision making. However, HHS did not provide specific guidelines on how covered entities should prevent discrimination. We conducted a scoping review of literature published during the period 2011-22 to identify health care applications, frameworks, reviews and perspectives, and assessment tools that identify and mitigate bias in clinical algorithms, with a specific focus on racial and ethnic bias. Our scoping review encompassed 109 articles comprising 45 empirical health care applications that included tools tested in health care settings, 16 frameworks, and 48 reviews and perspectives. We identified a wide range of technical, operational, and systemwide bias mitigation strategies for clinical algorithms, but there was no consensus in the literature on a single best practice that covered entities could employ to meet the HHS requirements. Future research should identify optimal bias mitigation methods for various scenarios, depending on factors such as patient population, clinical setting, algorithm design, and types of bias to be addressed.
Collapse
Affiliation(s)
- Michael P Cary
- Michael P. Cary Jr. , Duke University, Durham, North Carolina
| | - Anna Zink
- Anna Zink, University of Chicago, Chicago, Illinois
| | - Sijia Wei
- Sijia Wei, Northwestern University, Chicago, Illinois
| | | | | | | | | | | | | | | | | | | | - Ziad Obermeyer
- Ziad Obermeyer, University of California Berkeley, Berkeley, California
| | | |
Collapse
|
18
|
Wang Y, Song Y, Ma Z, Han X. Multidisciplinary considerations of fairness in medical AI: A scoping review. Int J Med Inform 2023; 178:105175. [PMID: 37595374 DOI: 10.1016/j.ijmedinf.2023.105175] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2023] [Revised: 08/02/2023] [Accepted: 08/04/2023] [Indexed: 08/20/2023]
Abstract
INTRODUCTION Artificial Intelligence (AI) technology has been developed significantly in recent years. The fairness of medical AI is of great concern due to its direct relation to human life and health. This review aims to analyze the existing research literature on fairness in medical AI from the perspectives of computer science, medical science, and social science (including law and ethics). The objective of the review is to examine the similarities and differences in the understanding of fairness, explore influencing factors, and investigate potential measures to implement fairness in medical AI across English and Chinese literature. METHODS This study employed a scoping review methodology and selected the following databases: Web of Science, MEDLINE, Pubmed, OVID, CNKI, WANFANG Data, etc., for the fairness issues in medical AI through February 2023. The search was conducted using various keywords such as "artificial intelligence," "machine learning," "medical," "algorithm," "fairness," "decision-making," and "bias." The collected data were charted, synthesized, and subjected to descriptive and thematic analysis. RESULTS After reviewing 468 English papers and 356 Chinese papers, 53 and 42 were included in the final analysis. Our results show the three different disciplines all show significant differences in the research on the core issues. Data is the foundation that affects medical AI fairness in addition to algorithmic bias and human bias. Legal, ethical, and technological measures all promote the implementation of medical AI fairness. CONCLUSIONS Our review indicates a consensus regarding the importance of data fairness as the foundation for achieving fairness in medical AI across multidisciplinary perspectives. However, there are substantial discrepancies in core aspects such as the concept, influencing factors, and implementation measures of fairness in medical AI. Consequently, future research should facilitate interdisciplinary discussions to bridge the cognitive gaps between different fields and enhance the practical implementation of fairness in medical AI.
Collapse
Affiliation(s)
- Yue Wang
- School of Law, Xi'an Jiaotong University, No.28, Xianning West Road, Xi'an, Shaanxi, 710049, PR China.
| | - Yaxin Song
- School of Law, Xi'an Jiaotong University, No.28, Xianning West Road, Xi'an, Shaanxi, 710049, PR China.
| | - Zhuo Ma
- School of Law, Xi'an Jiaotong University, No.28, Xianning West Road, Xi'an, Shaanxi, 710049, PR China.
| | - Xiaoxue Han
- Xi'an Jiaotong University Library, No.28, Xianning West Road, Xi'an, Shaanxi, 710049, PR China.
| |
Collapse
|
19
|
Timmons AC, Duong JB, Fiallo NS, Lee T, Vo HPQ, Ahle MW, Comer JS, Brewer LC, Frazier SL, Chaspari T. A Call to Action on Assessing and Mitigating Bias in Artificial Intelligence Applications for Mental Health. PERSPECTIVES ON PSYCHOLOGICAL SCIENCE 2023; 18:1062-1096. [PMID: 36490369 PMCID: PMC10250563 DOI: 10.1177/17456916221134490] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
Abstract
Advances in computer science and data-analytic methods are driving a new era in mental health research and application. Artificial intelligence (AI) technologies hold the potential to enhance the assessment, diagnosis, and treatment of people experiencing mental health problems and to increase the reach and impact of mental health care. However, AI applications will not mitigate mental health disparities if they are built from historical data that reflect underlying social biases and inequities. AI models biased against sensitive classes could reinforce and even perpetuate existing inequities if these models create legacies that differentially impact who is diagnosed and treated, and how effectively. The current article reviews the health-equity implications of applying AI to mental health problems, outlines state-of-the-art methods for assessing and mitigating algorithmic bias, and presents a call to action to guide the development of fair-aware AI in psychological science.
Collapse
Affiliation(s)
- Adela C. Timmons
- University of Texas at Austin Institute for Mental Health Research
- Colliga Apps Corporation
| | | | | | | | | | | | | | - LaPrincess C. Brewer
- Department of Cardiovascular Medicine, May Clinic College of Medicine, Rochester, Minnesota, United States
- Center for Health Equity and Community Engagement Research, Mayo Clinic, Rochester, Minnesota, United States
| | | | | |
Collapse
|
20
|
Chohlas-Wood A, Coots M, Goel S, Nyarko J. Designing equitable algorithms. NATURE COMPUTATIONAL SCIENCE 2023; 3:601-610. [PMID: 38177749 DOI: 10.1038/s43588-023-00485-4] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/17/2023] [Accepted: 06/06/2023] [Indexed: 01/06/2024]
Abstract
Predictive algorithms are now commonly used to distribute society's resources and sanctions. But these algorithms can entrench and exacerbate inequities. To guard against this possibility, many have suggested that algorithms be subject to formal fairness constraints. Here we argue, however, that popular constraints-while intuitively appealing-often worsen outcomes for individuals in marginalized groups, and can even leave all groups worse off. We outline a more holistic path forward for improving the equity of algorithmically guided decisions.
Collapse
|
21
|
Banda JM, Shah NH, Periyakoil VS. Characterizing subgroup performance of probabilistic phenotype algorithms within older adults: a case study for dementia, mild cognitive impairment, and Alzheimer's and Parkinson's diseases. JAMIA Open 2023; 6:ooad043. [PMID: 37397506 PMCID: PMC10307941 DOI: 10.1093/jamiaopen/ooad043] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2023] [Revised: 06/06/2023] [Accepted: 06/22/2023] [Indexed: 07/04/2023] Open
Abstract
Objective Biases within probabilistic electronic phenotyping algorithms are largely unexplored. In this work, we characterize differences in subgroup performance of phenotyping algorithms for Alzheimer's disease and related dementias (ADRD) in older adults. Materials and methods We created an experimental framework to characterize the performance of probabilistic phenotyping algorithms under different racial distributions allowing us to identify which algorithms may have differential performance, by how much, and under what conditions. We relied on rule-based phenotype definitions as reference to evaluate probabilistic phenotype algorithms created using the Automated PHenotype Routine for Observational Definition, Identification, Training and Evaluation framework. Results We demonstrate that some algorithms have performance variations anywhere from 3% to 30% for different populations, even when not using race as an input variable. We show that while performance differences in subgroups are not present for all phenotypes, they do affect some phenotypes and groups more disproportionately than others. Discussion Our analysis establishes the need for a robust evaluation framework for subgroup differences. The underlying patient populations for the algorithms showing subgroup performance differences have great variance between model features when compared with the phenotypes with little to no differences. Conclusion We have created a framework to identify systematic differences in the performance of probabilistic phenotyping algorithms specifically in the context of ADRD as a use case. Differences in subgroup performance of probabilistic phenotyping algorithms are not widespread nor do they occur consistently. This highlights the great need for careful ongoing monitoring to evaluate, measure, and try to mitigate such differences.
Collapse
Affiliation(s)
- Juan M Banda
- Corresponding Author: Juan M. Banda, PhD, Department of Computer Science, College of Arts and Sciences, Georgia State University, 25 Park Place, Suite 752, Atlanta, GA 30303, USA;
| | - Nigam H Shah
- Stanford Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California, USA
| | - Vyjeyanthi S Periyakoil
- Stanford Department of Medicine, Palo Alto, California, USA
- VA Palo Alto Health Care System, Palo Alto, California, USA
| |
Collapse
|
22
|
Chen RJ, Wang JJ, Williamson DFK, Chen TY, Lipkova J, Lu MY, Sahai S, Mahmood F. Algorithmic fairness in artificial intelligence for medicine and healthcare. Nat Biomed Eng 2023; 7:719-742. [PMID: 37380750 PMCID: PMC10632090 DOI: 10.1038/s41551-023-01056-8] [Citation(s) in RCA: 30] [Impact Index Per Article: 30.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2021] [Accepted: 04/13/2023] [Indexed: 06/30/2023]
Abstract
In healthcare, the development and deployment of insufficiently fair systems of artificial intelligence (AI) can undermine the delivery of equitable care. Assessments of AI models stratified across subpopulations have revealed inequalities in how patients are diagnosed, treated and billed. In this Perspective, we outline fairness in machine learning through the lens of healthcare, and discuss how algorithmic biases (in data acquisition, genetic variation and intra-observer labelling variability, in particular) arise in clinical workflows and the resulting healthcare disparities. We also review emerging technology for mitigating biases via disentanglement, federated learning and model explainability, and their role in the development of AI-based software as a medical device.
Collapse
Affiliation(s)
- Richard J Chen
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and Massachusetts Institute of Technology, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Judy J Wang
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Boston University School of Medicine, Boston, MA, USA
| | - Drew F K Williamson
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Tiffany Y Chen
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Jana Lipkova
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Ming Y Lu
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and Massachusetts Institute of Technology, Cambridge, MA, USA
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA
- Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Sharifa Sahai
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA
- Cancer Program, Broad Institute of Harvard and Massachusetts Institute of Technology, Cambridge, MA, USA
- Department of Systems Biology, Harvard Medical School, Boston, MA, USA
| | - Faisal Mahmood
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
- Cancer Program, Broad Institute of Harvard and Massachusetts Institute of Technology, Cambridge, MA, USA.
- Cancer Data Science Program, Dana-Farber Cancer Institute, Boston, MA, USA.
- Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA.
- Harvard Data Science Initiative, Harvard University, Cambridge, MA, USA.
| |
Collapse
|
23
|
Pham TH, Yin C, Mehta L, Zhang X, Zhang P. A fair and interpretable network for clinical risk prediction: a regularized multi-view multi-task learning approach. Knowl Inf Syst 2023; 65:1487-1521. [PMID: 36998311 PMCID: PMC10046420 DOI: 10.1007/s10115-022-01813-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/30/2021] [Revised: 12/06/2022] [Accepted: 12/12/2022] [Indexed: 12/24/2022]
Abstract
In healthcare domain, complication risk profiling which can be seen as multiple clinical risk prediction tasks is challenging due to the complex interaction between heterogeneous clinical entities. With the availability of real-world data, many deep learning methods are proposed for complication risk profiling. However, the existing methods face three open challenges. First, they leverage clinical data from a single view and then lead to suboptimal models. Second, most existing methods lack an effective mechanism to interpret predictions. Third, models learned from clinical data may have inherent pre-existing biases and exhibit discrimination against certain social groups. We then propose a multi-view multi-task network (MuViTaNet) to tackle these issues. MuViTaNet complements patient representation by using a multi-view encoder to exploit more information. Moreover, it uses a multi-task learning to generate more generalized representations using both labeled and unlabeled datasets. Last, a fairness variant (F-MuViTaNet) is proposed to mitigate the unfairness issues and promote healthcare equity. The experiments show that MuViTaNet outperforms existing methods for cardiac complication profiling. Its architecture also provides an effective mechanism for interpreting the predictions, which helps clinicians discover the underlying mechanism triggering the complication onsets. F-MuViTaNet can also effectively mitigate the unfairness with only negligible impact on accuracy.
Collapse
Affiliation(s)
- Thai-Hoang Pham
- Department of Computer Science and Engineering, The Ohio State University, Columbus, USA
- Department of Biomedical Informatics, The Ohio State University, Columbus, USA
| | - Changchang Yin
- Department of Computer Science and Engineering, The Ohio State University, Columbus, USA
- Department of Biomedical Informatics, The Ohio State University, Columbus, USA
| | - Laxmi Mehta
- Division of Cardiology, Department of Medicine, The Ohio State University, Columbus, USA
| | - Xueru Zhang
- Department of Computer Science and Engineering, The Ohio State University, Columbus, USA
| | - Ping Zhang
- Department of Computer Science and Engineering, The Ohio State University, Columbus, USA
- Department of Biomedical Informatics, The Ohio State University, Columbus, USA
| |
Collapse
|
24
|
Guo LL, Steinberg E, Fleming SL, Posada J, Lemmon J, Pfohl SR, Shah N, Fries J, Sung L. EHR foundation models improve robustness in the presence of temporal distribution shift. Sci Rep 2023; 13:3767. [PMID: 36882576 PMCID: PMC9992466 DOI: 10.1038/s41598-023-30820-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/13/2022] [Accepted: 03/02/2023] [Indexed: 03/09/2023] Open
Abstract
Temporal distribution shift negatively impacts the performance of clinical prediction models over time. Pretraining foundation models using self-supervised learning on electronic health records (EHR) may be effective in acquiring informative global patterns that can improve the robustness of task-specific models. The objective was to evaluate the utility of EHR foundation models in improving the in-distribution (ID) and out-of-distribution (OOD) performance of clinical prediction models. Transformer- and gated recurrent unit-based foundation models were pretrained on EHR of up to 1.8 M patients (382 M coded events) collected within pre-determined year groups (e.g., 2009-2012) and were subsequently used to construct patient representations for patients admitted to inpatient units. These representations were used to train logistic regression models to predict hospital mortality, long length of stay, 30-day readmission, and ICU admission. We compared our EHR foundation models with baseline logistic regression models learned on count-based representations (count-LR) in ID and OOD year groups. Performance was measured using area-under-the-receiver-operating-characteristic curve (AUROC), area-under-the-precision-recall curve, and absolute calibration error. Both transformer and recurrent-based foundation models generally showed better ID and OOD discrimination relative to count-LR and often exhibited less decay in tasks where there is observable degradation of discrimination performance (average AUROC decay of 3% for transformer-based foundation model vs. 7% for count-LR after 5-9 years). In addition, the performance and robustness of transformer-based foundation models continued to improve as pretraining set size increased. These results suggest that pretraining EHR foundation models at scale is a useful approach for developing clinical prediction models that perform well in the presence of temporal distribution shift.
Collapse
Affiliation(s)
- Lin Lawrence Guo
- Program in Child Health Evaluative Sciences, The Hospital for Sick Children, Toronto, ON, Canada
| | - Ethan Steinberg
- Stanford Center for Biomedical Informatics Research, Stanford University, Palo Alto, CA, USA
| | - Scott Lanyon Fleming
- Stanford Center for Biomedical Informatics Research, Stanford University, Palo Alto, CA, USA
| | - Jose Posada
- Universidad del Norte, Barranquilla, Colombia
| | - Joshua Lemmon
- Program in Child Health Evaluative Sciences, The Hospital for Sick Children, Toronto, ON, Canada
| | - Stephen R Pfohl
- Stanford Center for Biomedical Informatics Research, Stanford University, Palo Alto, CA, USA
| | - Nigam Shah
- Stanford Center for Biomedical Informatics Research, Stanford University, Palo Alto, CA, USA
| | - Jason Fries
- Stanford Center for Biomedical Informatics Research, Stanford University, Palo Alto, CA, USA
| | - Lillian Sung
- Program in Child Health Evaluative Sciences, The Hospital for Sick Children, Toronto, ON, Canada. .,Division of Haematology/Oncology, The Hospital for Sick Children, 555 University Avenue, Toronto, ON, M5G1X8, Canada.
| |
Collapse
|
25
|
de Hond AAH, van Buchem MM, Hernandez-Boussard T. Picture a data scientist: a call to action for increasing diversity, equity, and inclusion in the age of AI. J Am Med Inform Assoc 2022; 29:2178-2181. [PMID: 36048021 PMCID: PMC9667164 DOI: 10.1093/jamia/ocac156] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2022] [Revised: 08/18/2022] [Accepted: 08/26/2022] [Indexed: 11/13/2022] Open
Abstract
The lack of diversity, equity, and inclusion continues to hamper the artificial intelligence (AI) field and is especially problematic for healthcare applications. In this article, we expand on the need for diversity, equity, and inclusion, specifically focusing on the composition of AI teams. We call to action leaders at all levels to make team inclusivity and diversity the centerpieces of AI development, not the afterthought. These recommendations take into consideration mitigation at several levels, including outreach programs at the local level, diversity statements at the academic level, and regulatory steps at the federal level.
Collapse
Affiliation(s)
- Anne A H de Hond
- Clinical AI Implementation and Research Lab, Leiden University Medical Center, Leiden, The Netherlands
- Department of Medicine (Biomedical Informatics), Stanford University, Stanford, California, USA
| | - Marieke M van Buchem
- Clinical AI Implementation and Research Lab, Leiden University Medical Center, Leiden, The Netherlands
- Department of Medicine (Biomedical Informatics), Stanford University, Stanford, California, USA
| | - Tina Hernandez-Boussard
- Department of Medicine (Biomedical Informatics), Stanford University, Stanford, California, USA
- Department of Biomedical Data Science, Stanford University, Stanford, California, USA
- Department of Epidemiology & Population Health (By Courtesy), Stanford University, Stanford, California, USA
| |
Collapse
|
26
|
|
27
|
Leist AK, Klee M, Kim JH, Rehkopf DH, Bordas SPA, Muniz-Terrera G, Wade S. Mapping of machine learning approaches for description, prediction, and causal inference in the social and health sciences. SCIENCE ADVANCES 2022; 8:eabk1942. [PMID: 36260666 PMCID: PMC9581488 DOI: 10.1126/sciadv.abk1942] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/28/2021] [Accepted: 09/01/2022] [Indexed: 05/20/2023]
Abstract
Machine learning (ML) methodology used in the social and health sciences needs to fit the intended research purposes of description, prediction, or causal inference. This paper provides a comprehensive, systematic meta-mapping of research questions in the social and health sciences to appropriate ML approaches by incorporating the necessary requirements to statistical analysis in these disciplines. We map the established classification into description, prediction, counterfactual prediction, and causal structural learning to common research goals, such as estimating prevalence of adverse social or health outcomes, predicting the risk of an event, and identifying risk factors or causes of adverse outcomes, and explain common ML performance metrics. Such mapping may help to fully exploit the benefits of ML while considering domain-specific aspects relevant to the social and health sciences and hopefully contribute to the acceleration of the uptake of ML applications to advance both basic and applied social and health sciences research.
Collapse
Affiliation(s)
- Anja K. Leist
- Department of Social Sciences, Institute for Research on Socio-Economic Inequality (IRSEI), University of Luxembourg, Esch-sur-Alzette, Luxembourg
- Corresponding author.
| | - Matthias Klee
- Department of Social Sciences, Institute for Research on Socio-Economic Inequality (IRSEI), University of Luxembourg, Esch-sur-Alzette, Luxembourg
| | - Jung Hyun Kim
- Department of Social Sciences, Institute for Research on Socio-Economic Inequality (IRSEI), University of Luxembourg, Esch-sur-Alzette, Luxembourg
| | - David H. Rehkopf
- Department of Epidemiology and Population Health, Stanford University, Palo Alto, CA, USA
| | | | - Graciela Muniz-Terrera
- Centre for Dementia Prevention, University of Edinburgh, Edinburgh, UK
- Ohio University, Athens, OH, USA
| | - Sara Wade
- School of Mathematics, University of Edinburgh, Edinburgh, UK
| |
Collapse
|
28
|
Davis SE, Walsh CG, Matheny ME. Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings. Front Digit Health 2022; 4:958284. [PMID: 36120717 PMCID: PMC9478183 DOI: 10.3389/fdgth.2022.958284] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2022] [Accepted: 08/11/2022] [Indexed: 11/15/2022] Open
Abstract
As the implementation of artificial intelligence (AI)-enabled tools is realized across diverse clinical environments, there is a growing understanding of the need for ongoing monitoring and updating of prediction models. Dataset shift-temporal changes in clinical practice, patient populations, and information systems-is now well-documented as a source of deteriorating model accuracy and a challenge to the sustainability of AI-enabled tools in clinical care. While best practices are well-established for training and validating new models, there has been limited work developing best practices for prospective validation and model maintenance. In this paper, we highlight the need for updating clinical prediction models and discuss open questions regarding this critical aspect of the AI modeling lifecycle in three focus areas: model maintenance policies, performance monitoring perspectives, and model updating strategies. With the increasing adoption of AI-enabled tools, the need for such best practices must be addressed and incorporated into new and existing implementations. This commentary aims to encourage conversation and motivate additional research across clinical and data science stakeholders.
Collapse
Affiliation(s)
- Sharon E. Davis
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States,Correspondence: Sharon E. Davis
| | - Colin G. Walsh
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States,Department of Medicine, Vanderbilt University Medical Center, Nashville, TN, United States,Department of Psychiatry, Vanderbilt University Medical Center, Nashville, TN, United States
| | - Michael E. Matheny
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States,Department of Medicine, Vanderbilt University Medical Center, Nashville, TN, United States,Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN, United States,Tennessee Valley Healthcare System VA Medical Center, Veterans Health Administration, Nashville, TN, United States
| |
Collapse
|
29
|
Addressing fairness in artificial intelligence for medical imaging. Nat Commun 2022; 13:4581. [PMID: 35933408 PMCID: PMC9357063 DOI: 10.1038/s41467-022-32186-3] [Citation(s) in RCA: 40] [Impact Index Per Article: 20.0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2022] [Accepted: 07/21/2022] [Indexed: 12/19/2022] Open
|
30
|
Ghanzouri I, Amal S, Ho V, Safarnejad L, Cabot J, Brown-Johnson CG, Leeper N, Asch S, Shah NH, Ross EG. Performance and usability testing of an automated tool for detection of peripheral artery disease using electronic health records. Sci Rep 2022; 12:13364. [PMID: 35922657 PMCID: PMC9349186 DOI: 10.1038/s41598-022-17180-5] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2022] [Accepted: 07/21/2022] [Indexed: 11/18/2022] Open
Abstract
Peripheral artery disease (PAD) is a common cardiovascular disorder that is frequently underdiagnosed, which can lead to poorer outcomes due to lower rates of medical optimization. We aimed to develop an automated tool to identify undiagnosed PAD and evaluate physician acceptance of a dashboard representation of risk assessment. Data were derived from electronic health records (EHR). We developed and compared traditional risk score models to novel machine learning models. For usability testing, primary and specialty care physicians were recruited and interviewed until thematic saturation. Data from 3168 patients with PAD and 16,863 controls were utilized. Results showed a deep learning model that utilized time engineered features outperformed random forest and traditional logistic regression models (average AUCs 0.96, 0.91 and 0.81, respectively), P < 0.0001. Of interviewed physicians, 75% were receptive to an EHR-based automated PAD model. Feedback emphasized workflow optimization, including integrating risk assessments directly into the EHR, using dashboard designs that minimize clicks, and providing risk assessments for clinically complex patients. In conclusion, we demonstrate that EHR-based machine learning models can accurately detect risk of PAD and that physicians are receptive to automated risk detection for PAD. Future research aims to prospectively validate model performance and impact on patient outcomes.
Collapse
Affiliation(s)
- I Ghanzouri
- Division of Vascular Surgery, Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA
| | - S Amal
- Division of Vascular Surgery, Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA
| | - V Ho
- Division of Vascular Surgery, Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA
| | - L Safarnejad
- Division of Vascular Surgery, Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA
| | - J Cabot
- Division of Vascular Surgery, Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA
| | - C G Brown-Johnson
- Department of Medicine, Primary Care and Population Health, Stanford, CA, USA
| | - N Leeper
- Division of Vascular Surgery, Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA
| | - S Asch
- Department of Medicine, Primary Care and Population Health, Stanford, CA, USA
| | - N H Shah
- Department of Medicine, Center for Biomedical Informatics Research, Stanford University School of Medicine, 780 Welch Road, CJ350, Stanford, CA, 94305, USA
| | - E G Ross
- Division of Vascular Surgery, Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA. .,Department of Medicine, Center for Biomedical Informatics Research, Stanford University School of Medicine, 780 Welch Road, CJ350, Stanford, CA, 94305, USA.
| |
Collapse
|
31
|
Lu JH, Callahan A, Patel BS, Morse KE, Dash D, Pfeffer MA, Shah NH. Assessment of Adherence to Reporting Guidelines by Commonly Used Clinical Prediction Models From a Single Vendor: A Systematic Review. JAMA Netw Open 2022; 5:e2227779. [PMID: 35984654 PMCID: PMC9391954 DOI: 10.1001/jamanetworkopen.2022.27779] [Citation(s) in RCA: 15] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/23/2022] Open
Abstract
IMPORTANCE Various model reporting guidelines have been proposed to ensure clinical prediction models are reliable and fair. However, no consensus exists about which model details are essential to report, and commonalities and differences among reporting guidelines have not been characterized. Furthermore, how well documentation of deployed models adheres to these guidelines has not been studied. OBJECTIVES To assess information requested by model reporting guidelines and whether the documentation for commonly used machine learning models developed by a single vendor provides the information requested. EVIDENCE REVIEW MEDLINE was queried using machine learning model card and reporting machine learning from November 4 to December 6, 2020. References were reviewed to find additional publications, and publications without specific reporting recommendations were excluded. Similar elements requested for reporting were merged into representative items. Four independent reviewers and 1 adjudicator assessed how often documentation for the most commonly used models developed by a single vendor reported the items. FINDINGS From 15 model reporting guidelines, 220 unique items were identified that represented the collective reporting requirements. Although 12 items were commonly requested (requested by 10 or more guidelines), 77 items were requested by just 1 guideline. Documentation for 12 commonly used models from a single vendor reported a median of 39% (IQR, 37%-43%; range, 31%-47%) of items from the collective reporting requirements. Many of the commonly requested items had 100% reporting rates, including items concerning outcome definition, area under the receiver operating characteristics curve, internal validation, and intended clinical use. Several items reported half the time or less related to reliability, such as external validation, uncertainty measures, and strategy for handling missing data. Other frequently unreported items related to fairness (summary statistics and subgroup analyses, including for race and ethnicity or sex). CONCLUSIONS AND RELEVANCE These findings suggest that consistent reporting recommendations for clinical predictive models are needed for model developers to share necessary information for model deployment. The many published guidelines would, collectively, require reporting more than 200 items. Model documentation from 1 vendor reported the most commonly requested items from model reporting guidelines. However, areas for improvement were identified in reporting items related to model reliability and fairness. This analysis led to feedback to the vendor, which motivated updates to the documentation for future users.
Collapse
Affiliation(s)
- Jonathan H. Lu
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California
| | - Alison Callahan
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California
| | - Birju S. Patel
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California
| | - Keith E. Morse
- Department of Pediatrics, Stanford University School of Medicine, Stanford, California
- Department of Clinical Informatics, Lucile Packard Children’s Hospital, Palo Alto, California
| | - Dev Dash
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California
| | - Michael A. Pfeffer
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California
- Technology and Digital Solutions, Stanford Medicine, Stanford, California
| | - Nigam H. Shah
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California
- Technology and Digital Solutions, Stanford Medicine, Stanford, California
- Clinical Excellence Research Center, Stanford Medicine, Stanford, California
| |
Collapse
|
32
|
Bias Discovery in Machine Learning Models for Mental Health. INFORMATION 2022. [DOI: 10.3390/info13050237] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
Fairness and bias are crucial concepts in artificial intelligence, yet they are relatively ignored in machine learning applications in clinical psychiatry. We computed fairness metrics and present bias mitigation strategies using a model trained on clinical mental health data. We collected structured data related to the admission, diagnosis, and treatment of patients in the psychiatry department of the University Medical Center Utrecht. We trained a machine learning model to predict future administrations of benzodiazepines on the basis of past data. We found that gender plays an unexpected role in the predictions—this constitutes bias. Using the AI Fairness 360 package, we implemented reweighing and discrimination-aware regularization as bias mitigation strategies, and we explored their implications for model performance. This is the first application of bias exploration and mitigation in a machine learning model trained on real clinical psychiatry data.
Collapse
|
33
|
Bear Don’t Walk OJ, Reyes Nieva H, Lee SSJ, Elhadad N. A scoping review of ethics considerations in clinical natural language processing. JAMIA Open 2022; 5:ooac039. [PMID: 35663112 PMCID: PMC9154253 DOI: 10.1093/jamiaopen/ooac039] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2022] [Revised: 05/05/2022] [Accepted: 05/12/2022] [Indexed: 11/12/2022] Open
Abstract
Abstract
Objectives
To review through an ethics lens the state of research in clinical natural language processing (NLP) for the study of bias and fairness, and to identify gaps in research.
Methods
We queried PubMed and Google Scholar for articles published between 2015 and 2021 concerning clinical NLP, bias, and fairness. We analyzed articles using a framework that combines the machine learning (ML) development process (ie, design, data, algorithm, and critique) and bioethical concepts of beneficence, nonmaleficence, autonomy, justice, as well as explicability. Our approach further differentiated between biases of clinical text (eg, systemic or personal biases in clinical documentation towards patients) and biases in NLP applications.
Results
Out of 1162 articles screened, 22 met criteria for full text review. We categorized articles based on the design (N = 2), data (N = 12), algorithm (N = 14), and critique (N = 17) phases of the ML development process.
Discussion
Clinical NLP can be used to study bias in applications reliant on clinical text data as well as explore biases in the healthcare setting. We identify 3 areas of active research that require unique ethical considerations about the potential for clinical NLP to address and/or perpetuate bias: (1) selecting metrics that interrogate bias in models; (2) opportunities and risks of identifying sensitive patient attributes; and (3) best practices in reconciling individual autonomy, leveraging patient data, and inferring and manipulating sensitive information of subgroups. Finally, we address the limitations of current ethical frameworks to fully address concerns of justice. Clinical NLP is a rapidly advancing field, and assessing current approaches against ethical considerations can help the discipline use clinical NLP to explore both healthcare biases and equitable NLP applications.
Collapse
Affiliation(s)
| | - Harry Reyes Nieva
- Department of Biomedical Informatics, Columbia University , New York, New York, USA
- Department of Medicine, Harvard Medical School , Boston, Massachusetts, USA
| | - Sandra Soo-Jin Lee
- Department of Medical Humanities and Ethics, Columbia University , New York, New York, USA
| | - Noémie Elhadad
- Department of Biomedical Informatics, Columbia University , New York, New York, USA
| |
Collapse
|
34
|
Singh H, Mhasawade V, Chunara R. Generalizability challenges of mortality risk prediction models: A retrospective analysis on a multi-center database. PLOS DIGITAL HEALTH 2022; 1:e0000023. [PMID: 36812510 PMCID: PMC9931319 DOI: 10.1371/journal.pdig.0000023] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/03/2021] [Accepted: 02/17/2022] [Indexed: 12/23/2022]
Abstract
Modern predictive models require large amounts of data for training and evaluation, absence of which may result in models that are specific to certain locations, populations in them and clinical practices. Yet, best practices for clinical risk prediction models have not yet considered such challenges to generalizability. Here we ask whether population- and group-level performance of mortality prediction models vary significantly when applied to hospitals or geographies different from the ones in which they are developed. Further, what characteristics of the datasets explain the performance variation? In this multi-center cross-sectional study, we analyzed electronic health records from 179 hospitals across the US with 70,126 hospitalizations from 2014 to 2015. Generalization gap, defined as difference between model performance metrics across hospitals, is computed for area under the receiver operating characteristic curve (AUC) and calibration slope. To assess model performance by the race variable, we report differences in false negative rates across groups. Data were also analyzed using a causal discovery algorithm "Fast Causal Inference" that infers paths of causal influence while identifying potential influences associated with unmeasured variables. When transferring models across hospitals, AUC at the test hospital ranged from 0.777 to 0.832 (1st-3rd quartile or IQR; median 0.801); calibration slope from 0.725 to 0.983 (IQR; median 0.853); and disparity in false negative rates from 0.046 to 0.168 (IQR; median 0.092). Distribution of all variable types (demography, vitals, and labs) differed significantly across hospitals and regions. The race variable also mediated differences in the relationship between clinical variables and mortality, by hospital/region. In conclusion, group-level performance should be assessed during generalizability checks to identify potential harms to the groups. Moreover, for developing methods to improve model performance in new environments, a better understanding and documentation of provenance of data and health processes are needed to identify and mitigate sources of variation.
Collapse
Affiliation(s)
| | | | - Rumi Chunara
- New York University, Tandon School of Engineering,New York University, School of Global Public Health
| |
Collapse
|
35
|
Foryciarz A, Pfohl SR, Patel B, Shah N. Evaluating algorithmic fairness in the presence of clinical guidelines: the case of atherosclerotic cardiovascular disease risk estimation. BMJ Health Care Inform 2022; 29:bmjhci-2021-100460. [PMID: 35396247 PMCID: PMC8996004 DOI: 10.1136/bmjhci-2021-100460] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2021] [Accepted: 12/17/2021] [Indexed: 12/28/2022] Open
Abstract
ObjectivesThe American College of Cardiology and the American Heart Association guidelines on primary prevention of atherosclerotic cardiovascular disease (ASCVD) recommend using 10-year ASCVD risk estimation models to initiate statin treatment. For guideline-concordant decision-making, risk estimates need to be calibrated. However, existing models are often miscalibrated for race, ethnicity and sex based subgroups. This study evaluates two algorithmic fairness approaches to adjust the risk estimators (group recalibration and equalised odds) for their compatibility with the assumptions underpinning the guidelines’ decision rules.MethodsUsing an updated pooled cohorts data set, we derive unconstrained, group-recalibrated and equalised odds-constrained versions of the 10-year ASCVD risk estimators, and compare their calibration at guideline-concordant decision thresholds.ResultsWe find that, compared with the unconstrained model, group-recalibration improves calibration at one of the relevant thresholds for each group, but exacerbates differences in false positive and false negative rates between groups. An equalised odds constraint, meant to equalise error rates across groups, does so by miscalibrating the model overall and at relevant decision thresholds.DiscussionHence, because of induced miscalibration, decisions guided by risk estimators learned with an equalised odds fairness constraint are not concordant with existing guidelines. Conversely, recalibrating the model separately for each group can increase guideline compatibility, while increasing intergroup differences in error rates. As such, comparisons of error rates across groups can be misleading when guidelines recommend treating at fixed decision thresholds.ConclusionThe illustrated tradeoffs between satisfying a fairness criterion and retaining guideline compatibility underscore the need to evaluate models in the context of downstream interventions.
Collapse
Affiliation(s)
- Agata Foryciarz
- Department of Computer Science, Stanford University School of Engineering, Stanford, California, USA
- Stanford Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California, USA
| | - Stephen R Pfohl
- Stanford Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California, USA
| | - Birju Patel
- Stanford Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California, USA
| | - Nigam Shah
- Stanford Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California, USA
| |
Collapse
|
36
|
Li J, Bzdok D, Chen J, Tam A, Ooi LQR, Holmes AJ, Ge T, Patil KR, Jabbi M, Eickhoff SB, Yeo BTT, Genon S. Cross-ethnicity/race generalization failure of behavioral prediction from resting-state functional connectivity. SCIENCE ADVANCES 2022; 8:eabj1812. [PMID: 35294251 PMCID: PMC8926333 DOI: 10.1126/sciadv.abj1812] [Citation(s) in RCA: 34] [Impact Index Per Article: 17.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/14/2023]
Abstract
Algorithmic biases that favor majority populations pose a key challenge to the application of machine learning for precision medicine. Here, we assessed such bias in prediction models of behavioral phenotypes from brain functional magnetic resonance imaging. We examined the prediction bias using two independent datasets (preadolescent versus adult) of mixed ethnic/racial composition. When predictive models were trained on data dominated by white Americans (WA), out-of-sample prediction errors were generally higher for African Americans (AA) than for WA. This bias toward WA corresponds to more WA-like brain-behavior association patterns learned by the models. When models were trained on AA only, compared to training only on WA or an equal number of AA and WA participants, AA prediction accuracy improved but stayed below that for WA. Overall, the results point to the need for caution and further research regarding the application of current brain-behavior prediction models in minority populations.
Collapse
Affiliation(s)
- Jingwei Li
- Institute of Neuroscience and Medicine, Brain and
Behavior (INM-7), Research Center Jülich, Jülich, Germany
- Institute for Systems Neuroscience, Medical Faculty,
Heinrich-Heine University Düsseldorf, Düsseldorf, Germany
- Department of Electrical and Computer Engineering,
Centre for Sleep and Cognition, Centre for Translational Magnetic Resonance
Research, and N.1 Institute for Health and Institute for Digital Medicine,
National University of Singapore, Singapore, Singapore
- Corresponding author. (J.L.); (S.G.);
(B.T.T.Y.)
| | - Danilo Bzdok
- Department of Biomedical Engineering, Montreal
Neurological Institute (MNI), McConnell Brain Imaging Institute (BIC), McGill
University, Montreal, QC, Canada
- Mila-Quebec Artificial Intelligence Institute,
Montreal, QC, Canada
| | - Jianzhong Chen
- Department of Electrical and Computer Engineering,
Centre for Sleep and Cognition, Centre for Translational Magnetic Resonance
Research, and N.1 Institute for Health and Institute for Digital Medicine,
National University of Singapore, Singapore, Singapore
| | - Angela Tam
- Department of Electrical and Computer Engineering,
Centre for Sleep and Cognition, Centre for Translational Magnetic Resonance
Research, and N.1 Institute for Health and Institute for Digital Medicine,
National University of Singapore, Singapore, Singapore
| | - Leon Qi Rong Ooi
- Department of Electrical and Computer Engineering,
Centre for Sleep and Cognition, Centre for Translational Magnetic Resonance
Research, and N.1 Institute for Health and Institute for Digital Medicine,
National University of Singapore, Singapore, Singapore
| | - Avram J. Holmes
- Departments of Psychology and Psychiatry, Yale
University, New Haven, CT, USA
- Psychiatric and Neurodevelopmental Genetics Unit,
Center for Genomic Medicine, Massachusetts General Hospital, Boston, MA,
USA
| | - Tian Ge
- Psychiatric and Neurodevelopmental Genetics Unit,
Center for Genomic Medicine, Massachusetts General Hospital, Boston, MA,
USA
- Stanley Center for Psychiatric Research, Broad
Institute of MIT and Harvard, Cambridge, MA, USA
- Department of Psychiatry, Massachusetts General
Hospital, Harvard Medical School, Boston, MA, USA
| | - Kaustubh R. Patil
- Institute of Neuroscience and Medicine, Brain and
Behavior (INM-7), Research Center Jülich, Jülich, Germany
- Institute for Systems Neuroscience, Medical Faculty,
Heinrich-Heine University Düsseldorf, Düsseldorf, Germany
| | - Mbemba Jabbi
- Department of Psychiatry, Dell Medical School,
University of Texas at Austin, Austin, TX, USA
- The Mulva Clinic for Neurosciences, Dell Medical
School, University of Texas at Austin, Austin, TX, USA
- Institute of Neuroscience, University of Texas at
Austin, Austin, TX, USA
- Department of Psychology, University of Texas at
Austin, Austin, TX, USA
| | - Simon B. Eickhoff
- Institute of Neuroscience and Medicine, Brain and
Behavior (INM-7), Research Center Jülich, Jülich, Germany
- Institute for Systems Neuroscience, Medical Faculty,
Heinrich-Heine University Düsseldorf, Düsseldorf, Germany
| | - B. T. Thomas Yeo
- Department of Electrical and Computer Engineering,
Centre for Sleep and Cognition, Centre for Translational Magnetic Resonance
Research, and N.1 Institute for Health and Institute for Digital Medicine,
National University of Singapore, Singapore, Singapore
- Integrative Sciences and Engineering Programme
(ISEP), National University of Singapore, Singapore, Singapore
- Corresponding author. (J.L.); (S.G.);
(B.T.T.Y.)
| | - Sarah Genon
- Institute of Neuroscience and Medicine, Brain and
Behavior (INM-7), Research Center Jülich, Jülich, Germany
- Institute for Systems Neuroscience, Medical Faculty,
Heinrich-Heine University Düsseldorf, Düsseldorf, Germany
- Corresponding author. (J.L.); (S.G.);
(B.T.T.Y.)
| |
Collapse
|
37
|
Tucker A, Kannampallil T, Fodeh SJ, Peleg M. New JBI policy emphasizes clinically-meaningful novel machine learning methods. J Biomed Inform 2022; 127:104003. [DOI: 10.1016/j.jbi.2022.104003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2021] [Revised: 01/14/2022] [Accepted: 01/19/2022] [Indexed: 10/19/2022]
|
38
|
Mori M, Jarrin R, Lu Y, Kadakia K, Huang C, Ross JS, Krumholz HM. Sensible regulation and clinical implementation of clinical decision support software as a medical device. BMJ 2022; 376:o525. [PMID: 35228206 DOI: 10.1136/bmj.o525] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Affiliation(s)
- Makoto Mori
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut, USA
- Division of Cardiac Surgery, Yale School of Medicine, New Haven, Connecticut, USA
| | - Robert Jarrin
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut, USA
| | - Yuan Lu
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut, USA
| | | | - Chenxi Huang
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut, USA
| | - Joseph S Ross
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut, USA
- Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut, USA
| | - Harlan M Krumholz
- Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, Connecticut, USA
- Section of Cardiovascular Medicine, Department of Internal Medicine, Yale School of Medicine and the Department of Health Policy and Management, Yale School of Public Health, New Haven, Connecticut, USA
| |
Collapse
|
39
|
Pfohl SR, Zhang H, Xu Y, Foryciarz A, Ghassemi M, Shah NH. A comparison of approaches to improve worst-case predictive model performance over patient subpopulations. Sci Rep 2022; 12:3254. [PMID: 35228563 PMCID: PMC8885701 DOI: 10.1038/s41598-022-07167-7] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2021] [Accepted: 01/31/2022] [Indexed: 12/12/2022] Open
Abstract
Predictive models for clinical outcomes that are accurate on average in a patient population may underperform drastically for some subpopulations, potentially introducing or reinforcing inequities in care access and quality. Model training approaches that aim to maximize worst-case model performance across subpopulations, such as distributionally robust optimization (DRO), attempt to address this problem without introducing additional harms. We conduct a large-scale empirical study of DRO and several variations of standard learning procedures to identify approaches for model development and selection that consistently improve disaggregated and worst-case performance over subpopulations compared to standard approaches for learning predictive models from electronic health records data. In the course of our evaluation, we introduce an extension to DRO approaches that allows for specification of the metric used to assess worst-case performance. We conduct the analysis for models that predict in-hospital mortality, prolonged length of stay, and 30-day readmission for inpatient admissions, and predict in-hospital mortality using intensive care data. We find that, with relatively few exceptions, no approach performs better, for each patient subpopulation examined, than standard learning procedures using the entire training dataset. These results imply that when it is of interest to improve model performance for patient subpopulations beyond what can be achieved with standard practices, it may be necessary to do so via data collection techniques that increase the effective sample size or reduce the level of noise in the prediction problem.
Collapse
Affiliation(s)
- Stephen R Pfohl
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA, 94305, USA.
| | - Haoran Zhang
- Department of Computer Science, University of Toronto, Toronto, ON, Canada
| | - Yizhe Xu
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA, 94305, USA
| | - Agata Foryciarz
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA, 94305, USA
- Department of Computer Science, Stanford University, Stanford, CA, 94305, USA
| | - Marzyeh Ghassemi
- Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
- Institute for Medical and Evaluative Sciences, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | - Nigam H Shah
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA, 94305, USA
| |
Collapse
|
40
|
Farmer N, Osei Baah F, Williams F, Ortiz-Chapparo E, Mitchell VM, Jackson L, Collins B, Graham L, Wallen GR, Powell-Wiley TM, Johnson A. Use of a community advisory board to build equitable algorithms for participation in clinical trials: a protocol paper for HoPeNET. BMJ Health Care Inform 2022; 29:bmjhci-2021-100453. [PMID: 35185011 PMCID: PMC8860013 DOI: 10.1136/bmjhci-2021-100453] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2021] [Accepted: 01/07/2022] [Indexed: 11/25/2022] Open
Abstract
Introduction Participation from racial and ethnic minorities in clinical trials has been burdened by issues surrounding mistrust and access to healthcare. There is emerging use of machine learning (ML) in clinical trial recruitment and evaluation. However, for individuals from groups who are recipients of societal biases, utilisation of ML can lead to the creation and use of biased algorithms. To minimise bias, the design of equitable ML tools that advance health equity could be guided by community engagement processes. The Howard University Partnership with the National Institutes of Health for Equitable Clinical Trial Participation for Racial/Ethnic Communities Underrepresented in Research (HoPeNET) seeks to create an ML-based infrastructure from community advisory board (CAB) experiences to enhance participation of African-Americans/Blacks in clinical trials. Methods and analysis This triphased cross-sectional study (24 months, n=56) will create a CAB of community members and research investigators. The three phases of the study include: (1) identification of perceived barriers/facilitators to clinical trial engagement through qualitative/quantitative methods and systems-based model building participation; (2) operation of CAB meetings and (3) development of a predictive ML tool and outcome evaluation. Identified predictors from the participant-derived systems-based map will be used for the ML tool development. Ethics and dissemination We anticipate minimum risk for participants. Institutional review board approval and informed consent has been obtained and patient confidentiality ensured.
Collapse
Affiliation(s)
- Nicole Farmer
- Translational Biobehavioral and Health Disparities Branch, NIH Clinical Center, Bethesda, Maryland, USA
| | - Foster Osei Baah
- Social Determinants of Obesity and Cardiovascular Risk Laboratory, Cardiovascular Branch, Division of Intramural Research, NHLBI, Bethesda, Maryland, USA
| | | | - Erika Ortiz-Chapparo
- Social Determinants of Obesity and Cardiovascular Risk Laboratory, Cardiovascular Branch, Division of Intramural Research, NHLBI, Bethesda, Maryland, USA
| | - Valerie M Mitchell
- Social Determinants of Obesity and Cardiovascular Risk Laboratory, Cardiovascular Branch, Division of Intramural Research, NHLBI, Bethesda, Maryland, USA
| | - Latifa Jackson
- Department of Pediatrics, Howard University, Washington, DC, USA
| | - Billy Collins
- Social Determinants of Obesity and Cardiovascular Risk Laboratory, Cardiovascular Branch, Division of Intramural Research, NHLBI, Bethesda, Maryland, USA
| | - Lennox Graham
- Department of Health Sciences and Management, College of Nursing and Allied Health Sciences, Howard Unversity, Washington, DC, USA
| | - Gwenyth R Wallen
- Translational Biobehavioral and Health Disparities Branch, NIH Clinical Center, Bethesda, Maryland, USA
| | - Tiffany M Powell-Wiley
- Social Determinants of Obesity and Cardiovascular Risk Laboratory, Cardiovascular Branch, Division of Intramural Research, NHLBI, Bethesda, Maryland, USA .,Intramural Research Program, NIMHD, Bethesda, Maryland, USA
| | - Allan Johnson
- Department of Nurtritional Sciences, College of Nursing and Allied Health Sciences, Howard University, Washington, DC, USA
| |
Collapse
|
41
|
Guo LL, Pfohl SR, Fries J, Johnson AEW, Posada J, Aftandilian C, Shah N, Sung L. Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine. Sci Rep 2022; 12:2726. [PMID: 35177653 PMCID: PMC8854561 DOI: 10.1038/s41598-022-06484-1] [Citation(s) in RCA: 14] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/26/2021] [Accepted: 01/31/2022] [Indexed: 11/24/2022] Open
Abstract
Temporal dataset shift associated with changes in healthcare over time is a barrier to deploying machine learning-based clinical decision support systems. Algorithms that learn robust models by estimating invariant properties across time periods for domain generalization (DG) and unsupervised domain adaptation (UDA) might be suitable to proactively mitigate dataset shift. The objective was to characterize the impact of temporal dataset shift on clinical prediction models and benchmark DG and UDA algorithms on improving model robustness. In this cohort study, intensive care unit patients from the MIMIC-IV database were categorized by year groups (2008-2010, 2011-2013, 2014-2016 and 2017-2019). Tasks were predicting mortality, long length of stay, sepsis and invasive ventilation. Feedforward neural networks were used as prediction models. The baseline experiment trained models using empirical risk minimization (ERM) on 2008-2010 (ERM[08-10]) and evaluated them on subsequent year groups. DG experiment trained models using algorithms that estimated invariant properties using 2008-2016 and evaluated them on 2017-2019. UDA experiment leveraged unlabelled samples from 2017 to 2019 for unsupervised distribution matching. DG and UDA models were compared to ERM[08-16] models trained using 2008-2016. Main performance measures were area-under-the-receiver-operating-characteristic curve (AUROC), area-under-the-precision-recall curve and absolute calibration error. Threshold-based metrics including false-positives and false-negatives were used to assess the clinical impact of temporal dataset shift and its mitigation strategies. In the baseline experiments, dataset shift was most evident for sepsis prediction (maximum AUROC drop, 0.090; 95% confidence interval (CI), 0.080-0.101). Considering a scenario of 100 consecutively admitted patients showed that ERM[08-10] applied to 2017-2019 was associated with one additional false-negative among 11 patients with sepsis, when compared to the model applied to 2008-2010. When compared with ERM[08-16], DG and UDA experiments failed to produce more robust models (range of AUROC difference, - 0.003 to 0.050). In conclusion, DG and UDA failed to produce more robust models compared to ERM in the setting of temporal dataset shift. Alternate approaches are required to preserve model performance over time in clinical medicine.
Collapse
Affiliation(s)
- Lin Lawrence Guo
- Program in Child Health Evaluative Sciences, The Hospital for Sick Children, Toronto, ON, Canada
| | - Stephen R Pfohl
- Biomedical Informatics Research, Stanford University, Palo Alto, USA
| | - Jason Fries
- Biomedical Informatics Research, Stanford University, Palo Alto, USA
| | - Alistair E W Johnson
- Program in Child Health Evaluative Sciences, The Hospital for Sick Children, Toronto, ON, Canada
| | - Jose Posada
- Biomedical Informatics Research, Stanford University, Palo Alto, USA
| | | | - Nigam Shah
- Biomedical Informatics Research, Stanford University, Palo Alto, USA
| | - Lillian Sung
- Program in Child Health Evaluative Sciences, The Hospital for Sick Children, Toronto, ON, Canada.
- Division of Haematology/Oncology, The Hospital for Sick Children, 555 University Avenue, Toronto, ON, M5G1X8, Canada.
| |
Collapse
|
42
|
Crigger E, Reinbold K, Hanson C, Kao A, Blake K, Irons M. Trustworthy Augmented Intelligence in Health Care. J Med Syst 2022; 46:12. [PMID: 35020064 PMCID: PMC8755670 DOI: 10.1007/s10916-021-01790-z] [Citation(s) in RCA: 32] [Impact Index Per Article: 16.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/24/2021] [Accepted: 11/01/2021] [Indexed: 12/26/2022]
Abstract
Augmented Intelligence (AI) systems have the power to transform health care and bring us closer to the quadruple aim: enhancing patient experience, improving population health, reducing costs, and improving the work life of health care providers. Earning physicians' trust is critical for accelerating adoption of AI into patient care. As technology evolves, the medical community will need to develop standards for these innovative technologies and re-visit current regulatory systems that physicians and patients rely on to ensure that health care AI is responsible, evidence-based, free from bias, and designed and deployed to promote equity. To develop actionable guidance for trustworthy AI in health care, the AMA reviewed literature on the challenges health care AI poses and reflected on existing guidance as a starting point for addressing those challenges (including models for regulating the introduction of innovative technologies into clinical care).
Collapse
Affiliation(s)
- Elliott Crigger
- American Medical Association, 330 North Wabash, Chicago, Il, 60611, US.
| | - Karen Reinbold
- American Medical Association, 330 North Wabash, Chicago, Il, 60611, US
| | - Chelsea Hanson
- American Medical Association, 330 North Wabash, Chicago, Il, 60611, US
| | - Audiey Kao
- American Medical Association, 330 North Wabash, Chicago, Il, 60611, US
| | - Kathleen Blake
- American Medical Association, 330 North Wabash, Chicago, Il, 60611, US
| | - Mira Irons
- American Medical Association, 330 North Wabash, Chicago, Il, 60611, US
| |
Collapse
|
43
|
Designing clinically translatable artificial intelligence systems for high-dimensional medical imaging. NAT MACH INTELL 2021. [DOI: 10.1038/s42256-021-00399-8] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/21/2023]
|
44
|
Makridis C, Hurley S, Klote M, Alterovitz G. Ethical Applications of Artificial Intelligence: Evidence From Health Research on Veterans. JMIR Med Inform 2021; 9:e28921. [PMID: 34076584 PMCID: PMC8209529 DOI: 10.2196/28921] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2021] [Revised: 04/23/2021] [Accepted: 04/27/2021] [Indexed: 01/01/2023] Open
Abstract
Background Despite widespread agreement that artificial intelligence (AI) offers significant benefits for individuals and society at large, there are also serious challenges to overcome with respect to its governance. Recent policymaking has focused on establishing principles for the trustworthy use of AI. Adhering to these principles is especially important for ensuring that the development and application of AI raises economic and social welfare, including among vulnerable groups and veterans. Objective We explore the newly developed principles around trustworthy AI and how they can be readily applied at scale to vulnerable groups that are potentially less likely to benefit from technological advances. Methods Using the US Department of Veterans Affairs as a case study, we explore the principles of trustworthy AI that are of particular interest for vulnerable groups and veterans. Results We focus on three principles: (1) designing, developing, acquiring, and using AI so that the benefits of its use significantly outweigh the risks and the risks are assessed and managed; (2) ensuring that the application of AI occurs in well-defined domains and is accurate, effective, and fit for the intended purposes; and (3) ensuring that the operations and outcomes of AI applications are sufficiently interpretable and understandable by all subject matter experts, users, and others. Conclusions These principles and applications apply more generally to vulnerable groups, and adherence to them can allow the VA and other organizations to continue modernizing their technology governance, leveraging the gains of AI while simultaneously managing its risks.
Collapse
Affiliation(s)
- Christos Makridis
- National Artificial Intelligence Institute, Department of Veterans Affairs, Washington, DC, United States.,Stanford Digital Economy Lab, Stanford University, Stanford, CA, United States.,WP Carey School of Business, Arizona State University, Tempe, AZ, United States
| | - Seth Hurley
- National Artificial Intelligence Institute, Department of Veterans Affairs, Washington, DC, United States
| | - Mary Klote
- Office of Research & Development, Department of Veterans Affairs, Washington, DC, United States
| | - Gil Alterovitz
- National Artificial Intelligence Institute, Department of Veterans Affairs, Washington, DC, United States.,Boston Children's Hospital, Harvard Medical School, Boston, MA, United States
| |
Collapse
|