1
|
Ong JCL, Chang SYH, William W, Butte AJ, Shah NH, Chew LST, Liu N, Doshi-Velez F, Lu W, Savulescu J, Ting DSW. Ethical and regulatory challenges of large language models in medicine. Lancet Digit Health 2024:S2589-7500(24)00061-X. [PMID: 38658283 DOI: 10.1016/s2589-7500(24)00061-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2023] [Revised: 03/08/2024] [Accepted: 03/12/2024] [Indexed: 04/26/2024]
Abstract
With the rapid growth of interest in and use of large language models (LLMs) across various industries, we are facing some crucial and profound ethical concerns, especially in the medical field. The unique technical architecture and purported emergent abilities of LLMs differentiate them substantially from other artificial intelligence (AI) models and natural language processing techniques used, necessitating a nuanced understanding of LLM ethics. In this Viewpoint, we highlight ethical concerns stemming from the perspectives of users, developers, and regulators, notably focusing on data privacy and rights of use, data provenance, intellectual property contamination, and broad applications and plasticity of LLMs. A comprehensive framework and mitigating strategies will be imperative for the responsible integration of LLMs into medical practice, ensuring alignment with ethical principles and safeguarding against potential societal risks.
Collapse
Affiliation(s)
- Jasmine Chiat Ling Ong
- Division of Pharmacy, Singapore General Hospital, Singapore; Duke-NUS Medical School, National University of Singapore, Singapore
| | - Shelley Yin-Hsi Chang
- Department of Ophthalmology, Chang Gung Memorial Hospital, Linkou Medical Center, Taoyuan, Taiwan; College of Medicine, Chang Gung University, Taoyuan, Taiwan
| | - Wasswa William
- Department of Biomedical Sciences and Engineering, Mbarara University of Science and Technology, Mbarara, Uganda
| | - Atul J Butte
- Bakar Computational Health Sciences Institute, and Department of Pediatrics, University of California, San Francisco, San Francisco, CA, USA; Center for Data-Driven Insights and Innovation, University of California Health, Oakland, CA, USA
| | - Nigam H Shah
- Stanford Health Care, Palo Alto, CA, USA; Department of Medicine, and Clinical Excellence Research Center, School of Medicine, Stanford University, Stanford, CA, USA
| | - Lita Sui Tjien Chew
- Department of Pharmacy, National University of Singapore, Singapore; Singapore Health Services, Pharmacy and Therapeutics Council Office, Singapore; Department of Pharmacy, National Cancer Centre Singapore, Singapore
| | - Nan Liu
- Duke-NUS Medical School, National University of Singapore, Singapore
| | - Finale Doshi-Velez
- Harvard Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
| | - Wei Lu
- StatNLP Research Group, Singapore University of Technology and Design, Singpore
| | - Julian Savulescu
- Murdoch Children's Research Institute, Melbourne, VIC, Australia; Centre for Biomedical Ethics, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; Oxford Uehiro Centre for Practical Ethics, Faculty of Philosophy, University of Oxford, Oxford, UK
| | - Daniel Shu Wei Ting
- Duke-NUS Medical School, National University of Singapore, Singapore; Artificial Intelligence and Digital Innovation, Singapore Eye Research Institute, Singapore National Eye Center, Singapore Health Service, Singapore; Byers Eye Institute, Stanford University, Palo Alto, CA, USA.
| |
Collapse
|
2
|
Yoo RM, Viggiano BT, Pundi KN, Fries JA, Zahedivash A, Podchiyska T, Din N, Shah NH. Scalable Approach to Consumer Wearable Postmarket Surveillance: Development and Validation Study. JMIR Med Inform 2024; 12:e51171. [PMID: 38596848 PMCID: PMC11024395 DOI: 10.2196/51171] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/29/2023] [Revised: 01/15/2024] [Accepted: 02/04/2024] [Indexed: 04/11/2024] Open
Abstract
Background With the capability to render prediagnoses, consumer wearables have the potential to affect subsequent diagnoses and the level of care in the health care delivery setting. Despite this, postmarket surveillance of consumer wearables has been hindered by the lack of codified terms in electronic health records (EHRs) to capture wearable use. Objective We sought to develop a weak supervision-based approach to demonstrate the feasibility and efficacy of EHR-based postmarket surveillance on consumer wearables that render atrial fibrillation (AF) prediagnoses. Methods We applied data programming, where labeling heuristics are expressed as code-based labeling functions, to detect incidents of AF prediagnoses. A labeler model was then derived from the predictions of the labeling functions using the Snorkel framework. The labeler model was applied to clinical notes to probabilistically label them, and the labeled notes were then used as a training set to fine-tune a classifier called Clinical-Longformer. The resulting classifier identified patients with an AF prediagnosis. A retrospective cohort study was conducted, where the baseline characteristics and subsequent care patterns of patients identified by the classifier were compared against those who did not receive a prediagnosis. Results The labeler model derived from the labeling functions showed high accuracy (0.92; F1-score=0.77) on the training set. The classifier trained on the probabilistically labeled notes accurately identified patients with an AF prediagnosis (0.95; F1-score=0.83). The cohort study conducted using the constructed system carried enough statistical power to verify the key findings of the Apple Heart Study, which enrolled a much larger number of participants, where patients who received a prediagnosis tended to be older, male, and White with higher CHA2DS2-VASc (congestive heart failure, hypertension, age ≥75 years, diabetes, stroke, vascular disease, age 65-74 years, sex category) scores (P<.001). We also made a novel discovery that patients with a prediagnosis were more likely to use anticoagulants (525/1037, 50.63% vs 5936/16,560, 35.85%) and have an eventual AF diagnosis (305/1037, 29.41% vs 262/16,560, 1.58%). At the index diagnosis, the existence of a prediagnosis did not distinguish patients based on clinical characteristics, but did correlate with anticoagulant prescription (P=.004 for apixaban and P=.01 for rivaroxaban). Conclusions Our work establishes the feasibility and efficacy of an EHR-based surveillance system for consumer wearables that render AF prediagnoses. Further work is necessary to generalize these findings for patient populations at other sites.
Collapse
Affiliation(s)
- Richard M Yoo
- Department of Medicine, School of Medicine, Stanford University, Stanford, CA, United States
| | - Ben T Viggiano
- Department of Medicine, School of Medicine, Stanford University, Stanford, CA, United States
| | - Krishna N Pundi
- Department of Cardiovascular Medicine, School of Medicine, Stanford University, Stanford, CA, United States
| | - Jason A Fries
- Department of Medicine, School of Medicine, Stanford University, Stanford, CA, United States
| | - Aydin Zahedivash
- Department of Pediatrics, School of Medicine, Stanford University, Stanford, CA, United States
| | - Tanya Podchiyska
- Veterans Affairs Palo Alto Health Care System, Palo Alto, CA, United States
| | - Natasha Din
- Veterans Affairs Palo Alto Health Care System, Palo Alto, CA, United States
| | - Nigam H Shah
- Department of Medicine, School of Medicine, Stanford University, Stanford, CA, United States
- Clinical Excellence Research Center, School of Medicine, Stanford University, Stanford, CA, United States
- Technology and Digital Services, Stanford Health Care, Stanford, CA, United States
| |
Collapse
|
3
|
Affiliation(s)
- Nigam H Shah
- Technology and Digital Solutions, Stanford Medicine, Palo Alto, California
| | | | | |
Collapse
|
4
|
Jindal JA, Lungren MP, Shah NH. Ensuring useful adoption of generative artificial intelligence in healthcare. J Am Med Inform Assoc 2024:ocae043. [PMID: 38452298 DOI: 10.1093/jamia/ocae043] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2023] [Revised: 02/01/2024] [Accepted: 02/22/2024] [Indexed: 03/09/2024] Open
Abstract
OBJECTIVES This article aims to examine how generative artificial intelligence (AI) can be adopted with the most value in health systems, in response to the Executive Order on AI. MATERIALS AND METHODS We reviewed how technology has historically been deployed in healthcare, and evaluated recent examples of deployments of both traditional AI and generative AI (GenAI) with a lens on value. RESULTS Traditional AI and GenAI are different technologies in terms of their capability and modes of current deployment, which have implications on value in health systems. DISCUSSION Traditional AI when applied with a framework top-down can realize value in healthcare. GenAI in the short term when applied top-down has unclear value, but encouraging more bottom-up adoption has the potential to provide more benefit to health systems and patients. CONCLUSION GenAI in healthcare can provide the most value for patients when health systems adapt culturally to grow with this new technology and its adoption patterns.
Collapse
Affiliation(s)
- Jenelle A Jindal
- Center for Biomedical Informatics Research, Stanford University, Stanford, CA 94305, United States
| | - Matthew P Lungren
- Health and Life Sciences, Microsoft Corporation, Redmond, WA 98052, United States
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA 94305, United States
- Department of Biomedical Imaging, University of California San Francisco, San Francisco, CA 94143, United States
| | - Nigam H Shah
- Department of Medicine, Stanford School of Medicine, Stanford, CA 94304, United States
- Clinical Excellence Research Center, Stanford School of Medicine, Stanford, CA 94304, United States
- Technology and Digital Solutions, Stanford Health Care, Palo Alto, CA 94304, United States
| |
Collapse
|
5
|
Shah NH, Halamka JD, Saria S, Pencina M, Tazbaz T, Tripathi M, Callahan A, Hildahl H, Anderson B. A Nationwide Network of Health AI Assurance Laboratories. JAMA 2024; 331:245-249. [PMID: 38117493 DOI: 10.1001/jama.2023.26930] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/21/2023]
Abstract
Importance Given the importance of rigorous development and evaluation standards needed of artificial intelligence (AI) models used in health care, nationwide accepted procedures to provide assurance that the use of AI is fair, appropriate, valid, effective, and safe are urgently needed. Observations While there are several efforts to develop standards and best practices to evaluate AI, there is a gap between having such guidance and the application of such guidance to both existing and new AI models being developed. As of now, there is no publicly available, nationwide mechanism that enables objective evaluation and ongoing assessment of the consequences of using health AI models in clinical care settings. Conclusion and Relevance The need to create a public-private partnership to support a nationwide health AI assurance labs network is outlined here. In this network, community best practices could be applied for testing health AI models to produce reports on their performance that can be widely shared for managing the lifecycle of AI models over time and across populations and sites where these models are deployed.
Collapse
Affiliation(s)
- Nigam H Shah
- Stanford Medicine, Palo Alto, California
- Coalition for Health AI, Dover, Delaware
| | - John D Halamka
- Coalition for Health AI, Dover, Delaware
- Mayo Clinic Platform, Mayo Clinic, Rochester, Minnesota
| | - Suchi Saria
- Coalition for Health AI, Dover, Delaware
- Bayesian Health, New York, New York
- Johns Hopkins University, Baltimore, Maryland
- Johns Hopkins Medicine, Baltimore, Maryland
| | - Michael Pencina
- Coalition for Health AI, Dover, Delaware
- Duke AI Health, Duke University School of Medicine, Durham, North Carolina
| | - Troy Tazbaz
- US Food and Drug Administration, Silver Spring, Maryland
| | - Micky Tripathi
- US Office of the National Coordinator for Health IT, Washington, DC
| | | | | | - Brian Anderson
- Coalition for Health AI, Dover, Delaware
- MITRE Corporation, Bedford, Massachusetts
| |
Collapse
|
6
|
Mello MM, Shah NH, Char DS. President Biden's Executive Order on Artificial Intelligence-Implications for Health Care Organizations. JAMA 2024; 331:17-18. [PMID: 38032634 DOI: 10.1001/jama.2023.25051] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/01/2023]
Abstract
This Viewpoint discusses a recent executive order by US President Joe Biden about the development and implementation of AI, including the role of government vs the private sector and how the order may affect health care.
Collapse
Affiliation(s)
- Michelle M Mello
- Department of Health Policy, Stanford University School of Medicine, Stanford, California
- Stanford Law School and The Freeman Spogli Institute for International Studies, Stanford, California
| | - Nigam H Shah
- Departments of Medicine and Biomedical Data Science, and the Clinical Excellence Research Center, Stanford University School of Medicine, Stanford, California
| | - Danton S Char
- Department of Anesthesiology, Stanford University School of Medicine, Stanford, California
- Stanford Center for Biomedical Ethics, Stanford, California
| |
Collapse
|
7
|
Seevaratnam B, Wang S, Fong R, Hui F, Callahan A, Chobot S, Gensheimer MF, Li RC, Nguyen D, Ramchandran K, Shah NH, Shieh L, Zeng JGQ, Teuteberg W. Lessons Learned from a Multi-Site, Team-Based Serious Illness Care Program Implementation at an Academic Medical Center. J Palliat Med 2024; 27:83-89. [PMID: 37935036 DOI: 10.1089/jpm.2023.0254] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2023] Open
Abstract
Background: Patients with serious illness benefit from conversations to share prognosis and explore goals and values. To address this, we implemented Ariadne Labs' Serious Illness Care Program (SICP) at Stanford Health Care. Objective: Improve quantity, timing, and quality of serious illness conversations. Methods: Initial implementation followed Ariadne Labs' SICP framework. We later incorporated a team-based approach that included nonphysician care team members. Outcomes included number of patients with documented conversations according to clinician role and practice location. Machine learning algorithms were used in some settings to identify eligible patients. Results: Ambulatory oncology and hospital medicine were our largest implementation sites, engaging 4707 and 642 unique patients in conversations, respectively. Clinicians across eight disciplines engaged in these conversations. Identified barriers that included leadership engagement, complex workflows, and patient identification. Conclusion: Several factors contributed to successful SICP implementation across clinical sites: innovative clinical workflows, machine learning based predictive algorithms, and nonphysician care team member engagement.
Collapse
Affiliation(s)
- Briththa Seevaratnam
- Department of Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Samantha Wang
- Division of Hospital Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Rebecca Fong
- Department of Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Felicia Hui
- Section of Palliative Care, Stanford University School of Medicine, Stanford, California, USA
| | - Alison Callahan
- Biomedical Informatics, Stanford University School of Medicine, Stanford, California, USA
| | | | - Michael F Gensheimer
- Department of Radiation Oncology, Stanford University School of Medicine, Stanford, California, USA
| | - Ron C Li
- Division of Hospital Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Duy Nguyen
- Department of Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Kavitha Ramchandran
- Section of Palliative Care, Stanford University School of Medicine, Stanford, California, USA
- Division of Oncology, Stanford University School of Medicine, Stanford, California, USA
| | - Nigam H Shah
- Biomedical Informatics, Stanford University School of Medicine, Stanford, California, USA
| | - Lisa Shieh
- Division of Hospital Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Jack Guo-Qing Zeng
- Department of Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Winifred Teuteberg
- Section of Palliative Care, Stanford University School of Medicine, Stanford, California, USA
| |
Collapse
|
8
|
Youssef A, Pencina M, Thakur A, Zhu T, Clifton D, Shah NH. External validation of AI models in health should be replaced with recurring local validation. Nat Med 2023; 29:2686-2687. [PMID: 37853136 DOI: 10.1038/s41591-023-02540-z] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2023]
Affiliation(s)
- Alexey Youssef
- Stanford Bioengineering Department, Stanford University, Stanford, CA, USA.
- Department of Engineering Science, University of Oxford, Oxford, UK.
| | | | - Anshul Thakur
- Department of Engineering Science, University of Oxford, Oxford, UK
| | - Tingting Zhu
- Department of Engineering Science, University of Oxford, Oxford, UK
| | - David Clifton
- Department of Engineering Science, University of Oxford, Oxford, UK
- Oxford-Suzhou Centre for Advanced Research, Suzhou, China
| | - Nigam H Shah
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, CA, USA
- Technology and Digital Solutions, Stanford Medicine, Stanford, CA, USA
- Clinical Excellence Research Center, Stanford Medicine, Stanford, CA, USA
| |
Collapse
|
9
|
Callahan A, Ashley E, Datta S, Desai P, Ferris TA, Fries JA, Halaas M, Langlotz CP, Mackey S, Posada JD, Pfeffer MA, Shah NH. The Stanford Medicine data science ecosystem for clinical and translational research. JAMIA Open 2023; 6:ooad054. [PMID: 37545984 PMCID: PMC10397535 DOI: 10.1093/jamiaopen/ooad054] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2022] [Revised: 03/14/2023] [Accepted: 07/19/2023] [Indexed: 08/08/2023] Open
Abstract
Objective To describe the infrastructure, tools, and services developed at Stanford Medicine to maintain its data science ecosystem and research patient data repository for clinical and translational research. Materials and Methods The data science ecosystem, dubbed the Stanford Data Science Resources (SDSR), includes infrastructure and tools to create, search, retrieve, and analyze patient data, as well as services for data deidentification, linkage, and processing to extract high-value information from healthcare IT systems. Data are made available via self-service and concierge access, on HIPAA compliant secure computing infrastructure supported by in-depth user training. Results The Stanford Medicine Research Data Repository (STARR) functions as the SDSR data integration point, and includes electronic medical records, clinical images, text, bedside monitoring data and HL7 messages. SDSR tools include tools for electronic phenotyping, cohort building, and a search engine for patient timelines. The SDSR supports patient data collection, reproducible research, and teaching using healthcare data, and facilitates industry collaborations and large-scale observational studies. Discussion Research patient data repositories and their underlying data science infrastructure are essential to realizing a learning health system and advancing the mission of academic medical centers. Challenges to maintaining the SDSR include ensuring sufficient financial support while providing researchers and clinicians with maximal access to data and digital infrastructure, balancing tool development with user training, and supporting the diverse needs of users. Conclusion Our experience maintaining the SDSR offers a case study for academic medical centers developing data science and research informatics infrastructure.
Collapse
Affiliation(s)
- Alison Callahan
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA
| | - Euan Ashley
- Department of Medicine, School of Medicine, Stanford University, Stanford, California, USA
- Department of Genetics, School of Medicine, Stanford University, Stanford, California, USA
- Department of Biomedical Data Science, School of Medicine, Stanford University, Stanford, California, USA
| | - Somalee Datta
- Technology and Digital Solutions, Stanford Medicine, Stanford University, Stanford, California, USA
| | - Priyamvada Desai
- Technology and Digital Solutions, Stanford Medicine, Stanford University, Stanford, California, USA
| | - Todd A Ferris
- Technology and Digital Solutions, Stanford Medicine, Stanford University, Stanford, California, USA
| | - Jason A Fries
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA
| | - Michael Halaas
- Technology and Digital Solutions, Stanford Medicine, Stanford University, Stanford, California, USA
| | - Curtis P Langlotz
- Department of Radiology, School of Medicine, Stanford University, Stanford, California, USA
| | - Sean Mackey
- Department of Anesthesia, School of Medicine, Stanford University, Stanford, California, USA
| | - José D Posada
- Technology and Digital Solutions, Stanford Medicine, Stanford University, Stanford, California, USA
| | - Michael A Pfeffer
- Technology and Digital Solutions, Stanford Medicine, Stanford University, Stanford, California, USA
| | - Nigam H Shah
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA
- Technology and Digital Solutions, Stanford Medicine, Stanford University, Stanford, California, USA
- Clinical Excellence Research Center, School of Medicine, Stanford University, Stanford, California, USA
| |
Collapse
|
10
|
You SC, Seo SI, Falconer T, Yanover C, Duarte-Salles T, Seager S, Posada JD, Shah NH, Nguyen PA, Kim Y, Hsu JC, Van Zandt M, Hsu MH, Lee HL, Ko H, Shin WG, Pratt N, Park RW, Reich CG, Suchard MA, Hripcsak G, Park CH, Prieto-Alhambra D. Ranitidine Use and Incident Cancer in a Multinational Cohort. JAMA Netw Open 2023; 6:e2333495. [PMID: 37725377 PMCID: PMC10509724 DOI: 10.1001/jamanetworkopen.2023.33495] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/27/2023] [Accepted: 08/02/2023] [Indexed: 09/21/2023] Open
Abstract
Importance Ranitidine, the most widely used histamine-2 receptor antagonist (H2RA), was withdrawn because of N-nitrosodimethylamine impurity in 2020. Given the worldwide exposure to this drug, the potential risk of cancer development associated with the intake of known carcinogens is an important epidemiological concern. Objective To examine the comparative risk of cancer associated with the use of ranitidine vs other H2RAs. Design, Setting, and Participants This new-user active comparator international network cohort study was conducted using 3 health claims and 9 electronic health record databases from the US, the United Kingdom, Germany, Spain, France, South Korea, and Taiwan. Large-scale propensity score (PS) matching was used to minimize confounding of the observed covariates with negative control outcomes. Empirical calibration was performed to account for unobserved confounding. All databases were mapped to a common data model. Database-specific estimates were combined using random-effects meta-analysis. Participants included individuals aged at least 20 years with no history of cancer who used H2RAs for more than 30 days from January 1986 to December 2020, with a 1-year washout period. Data were analyzed from April to September 2021. Exposure The main exposure was use of ranitidine vs other H2RAs (famotidine, lafutidine, nizatidine, and roxatidine). Main Outcomes and Measures The primary outcome was incidence of any cancer, except nonmelanoma skin cancer. Secondary outcomes included all cancer except thyroid cancer, 16 cancer subtypes, and all-cause mortality. Results Among 1 183 999 individuals in 11 databases, 909 168 individuals (mean age, 56.1 years; 507 316 [55.8%] women) were identified as new users of ranitidine, and 274 831 individuals (mean age, 58.0 years; 145 935 [53.1%] women) were identified as new users of other H2RAs. Crude incidence rates of cancer were 14.30 events per 1000 person-years (PYs) in ranitidine users and 15.03 events per 1000 PYs among other H2RA users. After PS matching, cancer risk was similar in ranitidine compared with other H2RA users (incidence, 15.92 events per 1000 PYs vs 15.65 events per 1000 PYs; calibrated meta-analytic hazard ratio, 1.04; 95% CI, 0.97-1.12). No significant associations were found between ranitidine use and any secondary outcomes after calibration. Conclusions and Relevance In this cohort study, ranitidine use was not associated with an increased risk of cancer compared with the use of other H2RAs. Further research is needed on the long-term association of ranitidine with cancer development.
Collapse
Affiliation(s)
- Seng Chan You
- Department of Biomedical Systems Informatics, Yonsei University College of Medicine, Seoul, Korea
- Institute for Innovation in Digital Healthcare, Yonsei University, Seoul, Korea
| | - Seung In Seo
- Department of Internal Medicine, Kangdong Sacred Heart Hospital, Hallym University College of Medicine, Seoul, Korea
- Institute for Liver and Digestive Diseases, Hallym University, Chuncheon, Korea
| | - Thomas Falconer
- Department of Biomedical Informatics, Columbia University, New York, New York
| | | | - Talita Duarte-Salles
- Fundació Institut Universitari per a la recerca a l’Atenció Primària de Salut Jordi Gol i Gurina, Barcelona, Spain
| | | | - Jose D. Posada
- Department of Medicine, Stanford University School of Medicine, Stanford, California
| | - Nigam H. Shah
- Department of Medicine, Stanford University School of Medicine, Stanford, California
| | - Phung-Anh Nguyen
- Graduate Institute of Data Science, College of Management, Taipei Medical University, Taiwan
| | - Yeesuk Kim
- Department of Orthopaedic Surgery, College of Medicine, Hanyang University, Seoul, Korea
| | - Jason C. Hsu
- International PhD Program in Biotech and Healthcare Management, College of Management, Taipei Medical University, Taipei, Taiwan
| | | | - Min-Huei Hsu
- Graduate Institute of Data Science, College of Management, Taipei Medical University, Taiwan
| | - Hang Lak Lee
- Department of Internal Medicine, Hanyang University College of Medicine, Seoul, Korea
| | - Heejoo Ko
- College of Medicine, The Catholic University of Korea, Seoul, Korea
| | - Woon Geon Shin
- Department of Internal Medicine, Kangdong Sacred Heart Hospital, Hallym University College of Medicine, Seoul, Korea
- Institute for Liver and Digestive Diseases, Hallym University, Chuncheon, Korea
| | - Nicole Pratt
- Quality Use of Medicines and Pharmacy Research Centre, Clinical and Health Sciences, University of South Australia, Adelaide, Australia
| | - Rae Woong Park
- Department of Biomedical Informatics, Ajou University School of Medicine, Suwon, Korea
- Department of Biomedical Informatics, Ajou University School of Medicine, Suwon, Gyeonggi-do, Korea
| | | | - Marc A. Suchard
- Department of Biostatistics, Fielding School of Public Health, University of California, Los Angeles
- VA Informatics and Computing Infrastructure, US Department of Veterans Affairs, Salt Lake City, Utah
| | - George Hripcsak
- Department of Biomedical Informatics, Columbia University, New York, New York
- Medical Informatics Services, New York-Presbyterian Hospital, New York, New York
| | - Chan Hyuk Park
- Department of Internal Medicine, Hanyang University Guri Hospital, Hanyang University College of Medicine, Guri, Korea
| | - Daniel Prieto-Alhambra
- Pharmaco- and Device Epidemiology, Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, United Kingdom
- Department of Medical Informatics, Erasmus Medical Center University, Rotterdam, Netherlands
| |
Collapse
|
11
|
Abstract
Importance There is increased interest in and potential benefits from using large language models (LLMs) in medicine. However, by simply wondering how the LLMs and the applications powered by them will reshape medicine instead of getting actively involved, the agency in shaping how these tools can be used in medicine is lost. Observations Applications powered by LLMs are increasingly used to perform medical tasks without the underlying language model being trained on medical records and without verifying their purported benefit in performing those tasks. Conclusions and Relevance The creation and use of LLMs in medicine need to be actively shaped by provisioning relevant training data, specifying the desired benefits, and evaluating the benefits via testing in real-world deployments.
Collapse
Affiliation(s)
- Nigam H Shah
- Stanford Health Care, Palo Alto, California
- Department of Medicine, School of Medicine, Stanford University, Stanford, California
- Clinical Excellence Research Center, School of Medicine, Stanford University, Stanford, California
| | | | - Michael A Pfeffer
- Stanford Health Care, Palo Alto, California
- Department of Medicine, School of Medicine, Stanford University, Stanford, California
| |
Collapse
|
12
|
Corbin CK, Maclay R, Acharya A, Mony S, Punnathanam S, Thapa R, Kotecha N, Shah NH, Chen JH. DEPLOYR: a technical framework for deploying custom real-time machine learning models into the electronic medical record. J Am Med Inform Assoc 2023; 30:1532-1542. [PMID: 37369008 PMCID: PMC10436147 DOI: 10.1093/jamia/ocad114] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2023] [Revised: 05/16/2023] [Accepted: 06/13/2023] [Indexed: 06/29/2023] Open
Abstract
OBJECTIVE Heatlhcare institutions are establishing frameworks to govern and promote the implementation of accurate, actionable, and reliable machine learning models that integrate with clinical workflow. Such governance frameworks require an accompanying technical framework to deploy models in a resource efficient, safe and high-quality manner. Here we present DEPLOYR, a technical framework for enabling real-time deployment and monitoring of researcher-created models into a widely used electronic medical record system. MATERIALS AND METHODS We discuss core functionality and design decisions, including mechanisms to trigger inference based on actions within electronic medical record software, modules that collect real-time data to make inferences, mechanisms that close-the-loop by displaying inferences back to end-users within their workflow, monitoring modules that track performance of deployed models over time, silent deployment capabilities, and mechanisms to prospectively evaluate a deployed model's impact. RESULTS We demonstrate the use of DEPLOYR by silently deploying and prospectively evaluating 12 machine learning models trained using electronic medical record data that predict laboratory diagnostic results, triggered by clinician button-clicks in Stanford Health Care's electronic medical record. DISCUSSION Our study highlights the need and feasibility for such silent deployment, because prospectively measured performance varies from retrospective estimates. When possible, we recommend using prospectively estimated performance measures during silent trials to make final go decisions for model deployment. CONCLUSION Machine learning applications in healthcare are extensively researched, but successful translations to the bedside are rare. By describing DEPLOYR, we aim to inform machine learning deployment best practices and help bridge the model implementation gap.
Collapse
Affiliation(s)
- Conor K Corbin
- Department of Biomedical Data Science, Stanford, California, USA
| | - Rob Maclay
- Stanford Children’s Health, Palo Alto, California, USA
| | | | | | | | - Rahul Thapa
- Stanford Health Care, Palo Alto, California, USA
| | | | - Nigam H Shah
- Center for Biomedical Informatics Research, Division of Hospital Medicine, Department of Medicine, Stanford University, School of Medicine, Stanford, California, USA
| | - Jonathan H Chen
- Center for Biomedical Informatics Research, Division of Hospital Medicine, Department of Medicine, Stanford University, School of Medicine, Stanford, California, USA
| |
Collapse
|
13
|
Davis SE, Zabotka L, Desai RJ, Wang SV, Maro JC, Coughlin K, Hernández-Muñoz JJ, Stojanovic D, Shah NH, Smith JC. Use of Electronic Health Record Data for Drug Safety Signal Identification: A Scoping Review. Drug Saf 2023; 46:725-742. [PMID: 37340238 DOI: 10.1007/s40264-023-01325-0] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 05/31/2023] [Indexed: 06/22/2023]
Abstract
INTRODUCTION Pharmacovigilance programs protect patient health and safety by identifying adverse event signals through postmarketing surveillance of claims data and spontaneous reports. Electronic health records (EHRs) provide new opportunities to address limitations of traditional approaches and promote discovery-oriented pharmacovigilance. METHODS To evaluate the current state of EHR-based medication safety signal identification, we conducted a scoping literature review of studies aimed at identifying safety signals from routinely collected patient-level EHR data. We extracted information on study design, EHR data elements utilized, analytic methods employed, drugs and outcomes evaluated, and key statistical and data analysis choices. RESULTS We identified 81 eligible studies. Disproportionality methods were the predominant analytic approach, followed by data mining and regression. Variability in study design makes direct comparisons difficult. Studies varied widely in terms of data, confounding adjustment, and statistical considerations. CONCLUSION Despite broad interest in utilizing EHRs for safety signal identification, current efforts fail to leverage the full breadth and depth of available data or to rigorously control for confounding. The development of best practices and application of common data models would promote the expansion of EHR-based pharmacovigilance.
Collapse
Affiliation(s)
- Sharon E Davis
- Department of Biomedical Informatics, Vanderbilt University Medical Center, 2525 West End Ave, Suite 1475, Nashville, TN, 37203, USA
- Vanderbilt University School of Medicine, Nashville, TN, USA
| | | | - Rishi J Desai
- Brigham and Women's Hospital, Boston, MA, USA
- Harvard Medical School, Boston, MA, USA
| | - Shirley V Wang
- Brigham and Women's Hospital, Boston, MA, USA
- Harvard Medical School, Boston, MA, USA
| | - Judith C Maro
- Harvard Medical School, Boston, MA, USA
- Harvard Pilgrim Health Care Institute, Boston, MA, USA
| | | | | | | | - Nigam H Shah
- School of Medicine, Stanford University, Stanford, CA, USA
- Stanford Health Care, Palo Alto, CA, USA
| | - Joshua C Smith
- Department of Biomedical Informatics, Vanderbilt University Medical Center, 2525 West End Ave, Suite 1475, Nashville, TN, 37203, USA.
- Vanderbilt University School of Medicine, Nashville, TN, USA.
| |
Collapse
|
14
|
Wornow M, Xu Y, Thapa R, Patel B, Steinberg E, Fleming S, Pfeffer MA, Fries J, Shah NH. The shaky foundations of large language models and foundation models for electronic health records. NPJ Digit Med 2023; 6:135. [PMID: 37516790 PMCID: PMC10387101 DOI: 10.1038/s41746-023-00879-8] [Citation(s) in RCA: 23] [Impact Index Per Article: 23.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2023] [Accepted: 07/13/2023] [Indexed: 07/31/2023] Open
Abstract
The success of foundation models such as ChatGPT and AlphaFold has spurred significant interest in building similar models for electronic medical records (EMRs) to improve patient care and hospital operations. However, recent hype has obscured critical gaps in our understanding of these models' capabilities. In this narrative review, we examine 84 foundation models trained on non-imaging EMR data (i.e., clinical text and/or structured data) and create a taxonomy delineating their architectures, training data, and potential use cases. We find that most models are trained on small, narrowly-scoped clinical datasets (e.g., MIMIC-III) or broad, public biomedical corpora (e.g., PubMed) and are evaluated on tasks that do not provide meaningful insights on their usefulness to health systems. Considering these findings, we propose an improved evaluation framework for measuring the benefits of clinical foundation models that is more closely grounded to metrics that matter in healthcare.
Collapse
Affiliation(s)
- Michael Wornow
- Department of Computer Science, Stanford University, Stanford, CA, USA.
| | - Yizhe Xu
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, CA, USA
| | - Rahul Thapa
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, CA, USA
| | - Birju Patel
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, CA, USA
| | - Ethan Steinberg
- Department of Computer Science, Stanford University, Stanford, CA, USA
| | - Scott Fleming
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, CA, USA
| | - Michael A Pfeffer
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, CA, USA
- Technology and Digital Services, Stanford Health Care, Palo Alto, CA, USA
| | - Jason Fries
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, CA, USA
| | - Nigam H Shah
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, CA, USA
- Technology and Digital Services, Stanford Health Care, Palo Alto, CA, USA
- Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA
- Clinical Excellence Research Center, Stanford University School of Medicine, Stanford, CA, USA
| |
Collapse
|
15
|
Shah NH, Mello MM. Discrepancies Between Clearance Summaries and Marketing Materials of Software-Enabled Medical Devices Cleared by the US Food and Drug Administration. JAMA Netw Open 2023; 6:e2321753. [PMID: 37405777 DOI: 10.1001/jamanetworkopen.2023.21753] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 07/06/2023] Open
Affiliation(s)
- Nigam H Shah
- Clinical Excellence Research Center, Department of Medicine, Stanford University, Stanford, California
- Technology and Digital Solutions, Stanford Healthcare, Palo Alto, California
| | - Michelle M Mello
- Department of Health Research and Policy, Stanford Law School and Stanford School of Medicine, Stanford, California
| |
Collapse
|
16
|
Banda JM, Shah NH, Periyakoil VS. Characterizing subgroup performance of probabilistic phenotype algorithms within older adults: a case study for dementia, mild cognitive impairment, and Alzheimer's and Parkinson's diseases. JAMIA Open 2023; 6:ooad043. [PMID: 37397506 PMCID: PMC10307941 DOI: 10.1093/jamiaopen/ooad043] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2023] [Revised: 06/06/2023] [Accepted: 06/22/2023] [Indexed: 07/04/2023] Open
Abstract
Objective Biases within probabilistic electronic phenotyping algorithms are largely unexplored. In this work, we characterize differences in subgroup performance of phenotyping algorithms for Alzheimer's disease and related dementias (ADRD) in older adults. Materials and methods We created an experimental framework to characterize the performance of probabilistic phenotyping algorithms under different racial distributions allowing us to identify which algorithms may have differential performance, by how much, and under what conditions. We relied on rule-based phenotype definitions as reference to evaluate probabilistic phenotype algorithms created using the Automated PHenotype Routine for Observational Definition, Identification, Training and Evaluation framework. Results We demonstrate that some algorithms have performance variations anywhere from 3% to 30% for different populations, even when not using race as an input variable. We show that while performance differences in subgroups are not present for all phenotypes, they do affect some phenotypes and groups more disproportionately than others. Discussion Our analysis establishes the need for a robust evaluation framework for subgroup differences. The underlying patient populations for the algorithms showing subgroup performance differences have great variance between model features when compared with the phenotypes with little to no differences. Conclusion We have created a framework to identify systematic differences in the performance of probabilistic phenotyping algorithms specifically in the context of ADRD as a use case. Differences in subgroup performance of probabilistic phenotyping algorithms are not widespread nor do they occur consistently. This highlights the great need for careful ongoing monitoring to evaluate, measure, and try to mitigate such differences.
Collapse
Affiliation(s)
- Juan M Banda
- Corresponding Author: Juan M. Banda, PhD, Department of Computer Science, College of Arts and Sciences, Georgia State University, 25 Park Place, Suite 752, Atlanta, GA 30303, USA;
| | - Nigam H Shah
- Stanford Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California, USA
| | - Vyjeyanthi S Periyakoil
- Stanford Department of Medicine, Palo Alto, California, USA
- VA Palo Alto Health Care System, Palo Alto, California, USA
| |
Collapse
|
17
|
Bechler KK, Stolyar L, Steinberg E, Posada J, Minty E, Shah NH. Predicting patients who are likely to develop Lupus Nephritis of those newly diagnosed with Systemic Lupus Erythematosus. AMIA Annu Symp Proc 2023; 2022:221-230. [PMID: 37128416 PMCID: PMC10148321] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 05/03/2023]
Abstract
Patients diagnosed with systemic lupus erythematosus (SLE) suffer from a decreased quality of life, an increased risk of medical complications, and an increased risk of death. In particular, approximately 50% of SLE patients progress to develop lupus nephritis, which oftentimes leads to life-threatening end stage renal disease (ESRD) and requires dialysis or kidney transplant1. The challenge is that lupus nephritis is diagnosed via a kidney biopsy, which is typically performed only after noticeable decreased kidney function, leaving little room for proactive or preventative measures. The ability to predict which patients are most likely to develop lupus nephritis has the potential to shift lupus nephritis disease management from reactive to proactive. We present a clinically useful prediction model to predict which patients with newly diagnosed SLE will go on to develop lupus nephritis in the next five years.
Collapse
Affiliation(s)
- Katelyn K Bechler
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA
| | - Liya Stolyar
- Department of Medicine, Stanford University School of Medicine, Stanford, CA
| | - Ethan Steinberg
- Department of Computer Science, Stanford University, Stanford, CA
| | - Jose Posada
- Stanford Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford CA
- Department of Systems Engineering and Computing, Universidad del Norte, Barranquilla, Colombia
| | - Evan Minty
- O'Brien Institute for Public Health, Faculty of Medicine, University of Calgary, Canada
| | - Nigam H Shah
- Department of Medicine, Stanford University School of Medicine, Stanford, CA
- Stanford Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford CA
| |
Collapse
|
18
|
Xu Y, Foryciarz A, Steinberg E, Shah NH. Clinical utility gains from incorporating comorbidity and geographic location information into risk estimation equations for atherosclerotic cardiovascular disease. J Am Med Inform Assoc 2023; 30:878-887. [PMID: 36795076 PMCID: PMC10114071 DOI: 10.1093/jamia/ocad017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2022] [Revised: 01/17/2023] [Accepted: 02/11/2023] [Indexed: 02/17/2023] Open
Abstract
OBJECTIVE There are over 363 customized risk models of the American College of Cardiology and the American Heart Association (ACC/AHA) pooled cohort equations (PCE) in the literature, but their gains in clinical utility are rarely evaluated. We build new risk models for patients with specific comorbidities and geographic locations and evaluate whether performance improvements translate to gains in clinical utility. MATERIALS AND METHODS We retrain a baseline PCE using the ACC/AHA PCE variables and revise it to incorporate subject-level information of geographic location and 2 comorbidity conditions. We apply fixed effects, random effects, and extreme gradient boosting (XGB) models to handle the correlation and heterogeneity induced by locations. Models are trained using 2 464 522 claims records from Optum©'s Clinformatics® Data Mart and validated in the hold-out set (N = 1 056 224). We evaluate models' performance overall and across subgroups defined by the presence or absence of chronic kidney disease (CKD) or rheumatoid arthritis (RA) and geographic locations. We evaluate models' expected utility using net benefit and models' statistical properties using several discrimination and calibration metrics. RESULTS The revised fixed effects and XGB models yielded improved discrimination, compared to baseline PCE, overall and in all comorbidity subgroups. XGB improved calibration for the subgroups with CKD or RA. However, the gains in net benefit are negligible, especially under low exchange rates. CONCLUSIONS Common approaches to revising risk calculators incorporating extra information or applying flexible models may enhance statistical performance; however, such improvement does not necessarily translate to higher clinical utility. Thus, we recommend future works to quantify the consequences of using risk calculators to guide clinical decisions.
Collapse
Affiliation(s)
- Yizhe Xu
- Stanford Center for Biomedical Informatics Research, Department of Medicine, Stanford University, Stanford, California, USA
| | - Agata Foryciarz
- Stanford Center for Biomedical Informatics Research, Department of Medicine, Stanford University, Stanford, California, USA
| | - Ethan Steinberg
- Stanford Center for Biomedical Informatics Research, Department of Medicine, Stanford University, Stanford, California, USA
| | - Nigam H Shah
- Stanford Center for Biomedical Informatics Research, Department of Medicine, Stanford University, Stanford, California, USA
- Department of Medicine, School of Medicine, Stanford University, Stanford, California, USA
- Clinical Excellence Research Center, Department of Medicine, Stanford University, Stanford, California, USA
- Technology and Digital Solutions, Stanford Healthcare, Stanford, California, USA
| |
Collapse
|
19
|
Singh K, Shah NH, Vickers AJ. Assessing the net benefit of machine learning models in the presence of resource constraints. J Am Med Inform Assoc 2023; 30:668-673. [PMID: 36810659 PMCID: PMC10018264 DOI: 10.1093/jamia/ocad006] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2022] [Revised: 11/01/2022] [Accepted: 01/22/2023] [Indexed: 02/24/2023] Open
Abstract
OBJECTIVE The objective of this study is to provide a method to calculate model performance measures in the presence of resource constraints, with a focus on net benefit (NB). MATERIALS AND METHODS To quantify a model's clinical utility, the Equator Network's TRIPOD guidelines recommend the calculation of the NB, which reflects whether the benefits conferred by intervening on true positives outweigh the harms conferred by intervening on false positives. We refer to the NB achievable in the presence of resource constraints as the realized net benefit (RNB), and provide formulae for calculating the RNB. RESULTS Using 4 case studies, we demonstrate the degree to which an absolute constraint (eg, only 3 available intensive care unit [ICU] beds) diminishes the RNB of a hypothetical ICU admission model. We show how the introduction of a relative constraint (eg, surgical beds that can be converted to ICU beds for very high-risk patients) allows us to recoup some of the RNB but with a higher penalty for false positives. DISCUSSION RNB can be calculated in silico before the model's output is used to guide care. Accounting for the constraint changes the optimal strategy for ICU bed allocation. CONCLUSIONS This study provides a method to account for resource constraints when planning model-based interventions, either to avoid implementations where constraints are expected to play a larger role or to design more creative solutions (eg, converted ICU beds) to overcome absolute constraints when possible.
Collapse
Affiliation(s)
- Karandeep Singh
- Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, Michigan, USA
- Department of Internal Medicine, University of Michigan Medical School, Ann Arbor, Michigan, USA
- Department of Urology, University of Michigan Medical School, Ann Arbor, Michigan, USA
- School of Information, University of Michigan, Ann Arbor, Michigan, USA
| | - Nigam H Shah
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA
| | - Andrew J Vickers
- Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, New York, USA
| |
Collapse
|
20
|
Gensheimer MF, Gupta D, Patel MI, Fardeen T, Hildebrand R, Teuteberg W, Seevaratnam B, Asuncion MK, Alves N, Rogers B, Hansen J, DeNofrio J, Shah NH, Parikh D, Neal J, Fan AC, Moore K, Ruiz S, Li C, Khaki AR, Pagtama J, Chien J, Brown T, Tisch AH, Das M, Srinivas S, Roy M, Wakelee H, Myall NJ, Huang J, Shah S, Lee H, Ramchandran K. Use of Machine Learning and Lay Care Coaches to Increase Advance Care Planning Conversations for Patients With Metastatic Cancer. JCO Oncol Pract 2023; 19:e176-e184. [PMID: 36395436 DOI: 10.1200/op.22.00128] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022] Open
Abstract
PURPOSE Patients with metastatic cancer benefit from advance care planning (ACP) conversations. We aimed to improve ACP using a computer model to select high-risk patients, with shorter predicted survival, for conversations with providers and lay care coaches. Outcomes included ACP documentation frequency and end-of-life quality measures. METHODS In this study of a quality improvement initiative, providers in four medical oncology clinics received Serious Illness Care Program training. Two clinics (thoracic/genitourinary) participated in an intervention, and two (cutaneous/sarcoma) served as controls. ACP conversations were documented in a centralized form in the electronic medical record. In the intervention, providers and care coaches received weekly e-mails highlighting upcoming clinic patients with < 2 year computer-predicted survival and no prior prognosis documentation. Care coaches contacted these patients for an ACP conversation (excluding prognosis). Providers were asked to discuss and document prognosis. RESULTS In the four clinics, 4,968 clinic visits by 1,251 patients met inclusion criteria (metastatic cancer with no prognosis previously documented). In their first visit, 28% of patients were high-risk (< 2 year predicted survival). Preintervention, 3% of both intervention and control clinic patients had ACP documentation during a visit. By intervention end (February 2021), 35% of intervention clinic patients had ACP documentation compared with 3% of control clinic patients. Providers' prognosis documentation rate also increased in intervention clinics after the intervention (2%-27% in intervention clinics, P < .0001; 0%-1% in control clinics). End-of-life care intensity was similar in intervention versus control clinics, but patients with ≥ 1 provider ACP edit met fewer high-intensity care measures (P = .04). CONCLUSION Combining a computer prognosis model with care coaches increased ACP documentation.
Collapse
Affiliation(s)
| | - Divya Gupta
- Stanford University School of Medicine, Stanford CA
| | - Manali I Patel
- Stanford University School of Medicine, Stanford CA.,VA Palo Alto Health Care System, Palo Alto, CA
| | | | | | | | | | | | - Nina Alves
- Stanford University School of Medicine, Stanford CA
| | - Brian Rogers
- Stanford University School of Medicine, Stanford CA
| | | | - Jan DeNofrio
- Stanford University School of Medicine, Stanford CA
| | - Nigam H Shah
- Stanford University School of Medicine, Stanford CA
| | - Divya Parikh
- Stanford University School of Medicine, Stanford CA
| | - Joel Neal
- Stanford University School of Medicine, Stanford CA
| | - Alice C Fan
- Stanford University School of Medicine, Stanford CA
| | - Kaidi Moore
- Stanford University School of Medicine, Stanford CA
| | - Shann Ruiz
- Stanford University School of Medicine, Stanford CA
| | - Connie Li
- Stanford University School of Medicine, Stanford CA
| | | | - Judy Pagtama
- Stanford University School of Medicine, Stanford CA
| | - Joanne Chien
- Stanford University School of Medicine, Stanford CA
| | | | | | - Millie Das
- Stanford University School of Medicine, Stanford CA
| | | | - Mohana Roy
- Stanford University School of Medicine, Stanford CA
| | | | | | - Jane Huang
- Stanford University School of Medicine, Stanford CA
| | - Sumit Shah
- Stanford University School of Medicine, Stanford CA
| | - Howard Lee
- Stanford University School of Medicine, Stanford CA
| | | |
Collapse
|
21
|
Yoo RM, Dash D, Lu JH, Genkins JZ, Rabbani N, Fries JA, Shah NH. Investigating real-world consequences of biases in commonly used clinical calculators. Am J Manag Care 2023; 29:e1-e7. [PMID: 36716157 DOI: 10.37765/ajmc.2023.89306] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/28/2023]
Abstract
OBJECTIVES To evaluate whether one summary metric of calculator performance sufficiently conveys equity across different demographic subgroups, as well as to evaluate how calculator predictive performance affects downstream health outcomes. STUDY DESIGN We evaluate 3 commonly used clinical calculators-Model for End-Stage Liver Disease (MELD), CHA2DS2-VASc, and simplified Pulmonary Embolism Severity Index (sPESI)-on the cohort extracted from the Stanford Medicine Research Data Repository, following the cohort selection process as described in respective calculator derivation papers. METHODS We quantified the predictive performance of the 3 clinical calculators across sex and race. Then, using the clinical guidelines that guide care based on these calculators' output, we quantified potential disparities in subsequent health outcomes. RESULTS Across the examined subgroups, the MELD calculator exhibited worse performance for female and White populations, CHA2DS2-VASc calculator for the male population, and sPESI for the Black population. The extent to which such performance differences translated into differential health outcomes depended on the distribution of the calculators' scores around the thresholds used to trigger a care action via the corresponding guidelines. In particular, under the old guideline for CHA2DS2-VASc, among those who would not have been offered anticoagulant therapy, the Hispanic subgroup exhibited the highest rate of stroke. CONCLUSIONS Clinical calculators, even when they do not include variables such as sex and race as inputs, can have very different care consequences across those subgroups. These differences in health care outcomes across subgroups can be explained by examining the distribution of scores and their calibration around the thresholds encoded in the accompanying care guidelines.
Collapse
Affiliation(s)
- Richard M Yoo
- Stanford University, 1265 Welch Rd, Stanford, CA 94305.
| | | | | | | | | | | | | |
Collapse
|
22
|
Miner AS, Fleming SL, Haque A, Fries JA, Althoff T, Wilfley DE, Agras WS, Milstein A, Hancock J, Asch SM, Stirman SW, Arnow BA, Shah NH. A computational approach to measure the linguistic characteristics of psychotherapy timing, responsiveness, and consistency. Npj Ment Health Res 2022; 1:19. [PMID: 38609510 PMCID: PMC10956022 DOI: 10.1038/s44184-022-00020-9] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/12/2022] [Accepted: 10/18/2022] [Indexed: 04/14/2024]
Abstract
Although individual psychotherapy is generally effective for a range of mental health conditions, little is known about the moment-to-moment language use of effective therapists. Increased access to computational power, coupled with a rise in computer-mediated communication (telehealth), makes feasible the large-scale analyses of language use during psychotherapy. Transparent methodological approaches are lacking, however. Here we present novel methods to increase the efficiency of efforts to examine language use in psychotherapy. We evaluate three important aspects of therapist language use - timing, responsiveness, and consistency - across five clinically relevant language domains: pronouns, time orientation, emotional polarity, therapist tactics, and paralinguistic style. We find therapist language is dynamic within sessions, responds to patient language, and relates to patient symptom diagnosis but not symptom severity. Our results demonstrate that analyzing therapist language at scale is feasible and may help answer longstanding questions about specific behaviors of effective therapists.
Collapse
Affiliation(s)
- Adam S Miner
- Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA.
- Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA.
| | - Scott L Fleming
- Department of Biomedical Data Science, Stanford University, Stanford, CA, USA
- Department of Computer Science, Stanford University, Stanford, CA, USA
| | - Albert Haque
- Department of Computer Science, Stanford University, Stanford, CA, USA
| | - Jason A Fries
- Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA
| | - Tim Althoff
- Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA
| | - Denise E Wilfley
- Departments of Psychiatry, Medicine, Pediatrics, and Psychological & Brain Sciences, Washington University in St. Louis, St. Louis, MO, USA
| | - W Stewart Agras
- Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA
| | - Arnold Milstein
- Clinical Excellence Research Center, Stanford University, Stanford, CA, USA
| | - Jeff Hancock
- Department of Communication, Stanford University, Stanford, CA, USA
| | - Steven M Asch
- VA Palo Alto Health Care System, Palo Alto, CA, USA
- Division of Primary Care and Population Health, Stanford University School of Medicine, Stanford, CA, USA
| | - Shannon Wiltsey Stirman
- Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA
- VA Palo Alto Health Care System, Palo Alto, CA, USA
- National Center for Posttraumatic Stress Disorders, Dissemination and Training Division, VA Palo Alto Healthcare System, Menlo Park, CA, USA
| | - Bruce A Arnow
- Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA
| | - Nigam H Shah
- Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA
- Department of Biomedical Data Science, Stanford University, Stanford, CA, USA
- Clinical Excellence Research Center, Stanford University, Stanford, CA, USA
- Technology and Digital Solutions, Stanford Healthcare, Stanford, CA, USA
| |
Collapse
|
23
|
Lu J, Sattler A, Wang S, Khaki AR, Callahan A, Fleming S, Fong R, Ehlert B, Li RC, Shieh L, Ramchandran K, Gensheimer MF, Chobot S, Pfohl S, Li S, Shum K, Parikh N, Desai P, Seevaratnam B, Hanson M, Smith M, Xu Y, Gokhale A, Lin S, Pfeffer MA, Teuteberg W, Shah NH. Considerations in the reliability and fairness audits of predictive models for advance care planning. Front Digit Health 2022; 4:943768. [PMID: 36339512 PMCID: PMC9634737 DOI: 10.3389/fdgth.2022.943768] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/14/2022] [Accepted: 08/17/2022] [Indexed: 11/30/2022] Open
Abstract
Multiple reporting guidelines for artificial intelligence (AI) models in healthcare recommend that models be audited for reliability and fairness. However, there is a gap of operational guidance for performing reliability and fairness audits in practice. Following guideline recommendations, we conducted a reliability audit of two models based on model performance and calibration as well as a fairness audit based on summary statistics, subgroup performance and subgroup calibration. We assessed the Epic End-of-Life (EOL) Index model and an internally developed Stanford Hospital Medicine (HM) Advance Care Planning (ACP) model in 3 practice settings: Primary Care, Inpatient Oncology and Hospital Medicine, using clinicians' answers to the surprise question (“Would you be surprised if [patient X] passed away in [Y years]?”) as a surrogate outcome. For performance, the models had positive predictive value (PPV) at or above 0.76 in all settings. In Hospital Medicine and Inpatient Oncology, the Stanford HM ACP model had higher sensitivity (0.69, 0.89 respectively) than the EOL model (0.20, 0.27), and better calibration (O/E 1.5, 1.7) than the EOL model (O/E 2.5, 3.0). The Epic EOL model flagged fewer patients (11%, 21% respectively) than the Stanford HM ACP model (38%, 75%). There were no differences in performance and calibration by sex. Both models had lower sensitivity in Hispanic/Latino male patients with Race listed as “Other.” 10 clinicians were surveyed after a presentation summarizing the audit. 10/10 reported that summary statistics, overall performance, and subgroup performance would affect their decision to use the model to guide care; 9/10 said the same for overall and subgroup calibration. The most commonly identified barriers for routinely conducting such reliability and fairness audits were poor demographic data quality and lack of data access. This audit required 115 person-hours across 8–10 months. Our recommendations for performing reliability and fairness audits include verifying data validity, analyzing model performance on intersectional subgroups, and collecting clinician-patient linkages as necessary for label generation by clinicians. Those responsible for AI models should require such audits before model deployment and mediate between model auditors and impacted stakeholders.
Collapse
Affiliation(s)
- Jonathan Lu
- Center for Biomedical Informatics Research, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
- Correspondence: Jonathan Hsijing Lu
| | - Amelia Sattler
- Stanford Healthcare AI Applied Research Team, Division of Primary Care and Population Health, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Samantha Wang
- Division of Hospital Medicine, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Ali Raza Khaki
- Division of Oncology, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Alison Callahan
- Center for Biomedical Informatics Research, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Scott Fleming
- Center for Biomedical Informatics Research, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Rebecca Fong
- Serious Illness Care Program, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Benjamin Ehlert
- Center for Biomedical Informatics Research, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Ron C. Li
- Division of Hospital Medicine, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Lisa Shieh
- Division of Hospital Medicine, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Kavitha Ramchandran
- Division of Oncology, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Michael F. Gensheimer
- Department of Radiation Oncology, Stanford University School of Medicine, Palo Alto, United States
| | - Sarah Chobot
- Inpatient Palliative Care, Stanford Health Care, Palo Alto, United States
| | - Stephen Pfohl
- Center for Biomedical Informatics Research, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Siyun Li
- Center for Biomedical Informatics Research, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Kenny Shum
- Technology / Digital Solutions, Stanford Health Care and Stanford University School of Medicine, Palo Alto, United States
| | - Nitin Parikh
- Technology / Digital Solutions, Stanford Health Care and Stanford University School of Medicine, Palo Alto, United States
| | - Priya Desai
- Technology / Digital Solutions, Stanford Health Care and Stanford University School of Medicine, Palo Alto, United States
| | - Briththa Seevaratnam
- Serious Illness Care Program, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Melanie Hanson
- Serious Illness Care Program, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Margaret Smith
- Stanford Healthcare AI Applied Research Team, Division of Primary Care and Population Health, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Yizhe Xu
- Center for Biomedical Informatics Research, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Arjun Gokhale
- Center for Biomedical Informatics Research, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Steven Lin
- Stanford Healthcare AI Applied Research Team, Division of Primary Care and Population Health, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Michael A. Pfeffer
- Division of Hospital Medicine, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
- Technology / Digital Solutions, Stanford Health Care and Stanford University School of Medicine, Palo Alto, United States
| | - Winifred Teuteberg
- Serious Illness Care Program, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
| | - Nigam H. Shah
- Center for Biomedical Informatics Research, Department of Medicine, Stanford University School of Medicine, Palo Alto, United States
- Technology / Digital Solutions, Stanford Health Care and Stanford University School of Medicine, Palo Alto, United States
- Clinical Excellence Research Center, Stanford University School of Medicine, Palo Alto, United States
| |
Collapse
|
24
|
Ghanzouri I, Amal S, Ho V, Safarnejad L, Cabot J, Brown-Johnson CG, Leeper N, Asch S, Shah NH, Ross EG. Performance and usability testing of an automated tool for detection of peripheral artery disease using electronic health records. Sci Rep 2022; 12:13364. [PMID: 35922657 PMCID: PMC9349186 DOI: 10.1038/s41598-022-17180-5] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2022] [Accepted: 07/21/2022] [Indexed: 11/18/2022] Open
Abstract
Peripheral artery disease (PAD) is a common cardiovascular disorder that is frequently underdiagnosed, which can lead to poorer outcomes due to lower rates of medical optimization. We aimed to develop an automated tool to identify undiagnosed PAD and evaluate physician acceptance of a dashboard representation of risk assessment. Data were derived from electronic health records (EHR). We developed and compared traditional risk score models to novel machine learning models. For usability testing, primary and specialty care physicians were recruited and interviewed until thematic saturation. Data from 3168 patients with PAD and 16,863 controls were utilized. Results showed a deep learning model that utilized time engineered features outperformed random forest and traditional logistic regression models (average AUCs 0.96, 0.91 and 0.81, respectively), P < 0.0001. Of interviewed physicians, 75% were receptive to an EHR-based automated PAD model. Feedback emphasized workflow optimization, including integrating risk assessments directly into the EHR, using dashboard designs that minimize clicks, and providing risk assessments for clinically complex patients. In conclusion, we demonstrate that EHR-based machine learning models can accurately detect risk of PAD and that physicians are receptive to automated risk detection for PAD. Future research aims to prospectively validate model performance and impact on patient outcomes.
Collapse
Affiliation(s)
- I Ghanzouri
- Division of Vascular Surgery, Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA
| | - S Amal
- Division of Vascular Surgery, Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA
| | - V Ho
- Division of Vascular Surgery, Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA
| | - L Safarnejad
- Division of Vascular Surgery, Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA
| | - J Cabot
- Division of Vascular Surgery, Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA
| | - C G Brown-Johnson
- Department of Medicine, Primary Care and Population Health, Stanford, CA, USA
| | - N Leeper
- Division of Vascular Surgery, Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA
| | - S Asch
- Department of Medicine, Primary Care and Population Health, Stanford, CA, USA
| | - N H Shah
- Department of Medicine, Center for Biomedical Informatics Research, Stanford University School of Medicine, 780 Welch Road, CJ350, Stanford, CA, 94305, USA
| | - E G Ross
- Division of Vascular Surgery, Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA. .,Department of Medicine, Center for Biomedical Informatics Research, Stanford University School of Medicine, 780 Welch Road, CJ350, Stanford, CA, 94305, USA.
| |
Collapse
|
25
|
Lu JH, Callahan A, Patel BS, Morse KE, Dash D, Pfeffer MA, Shah NH. Assessment of Adherence to Reporting Guidelines by Commonly Used Clinical Prediction Models From a Single Vendor: A Systematic Review. JAMA Netw Open 2022; 5:e2227779. [PMID: 35984654 PMCID: PMC9391954 DOI: 10.1001/jamanetworkopen.2022.27779] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/23/2022] Open
Abstract
IMPORTANCE Various model reporting guidelines have been proposed to ensure clinical prediction models are reliable and fair. However, no consensus exists about which model details are essential to report, and commonalities and differences among reporting guidelines have not been characterized. Furthermore, how well documentation of deployed models adheres to these guidelines has not been studied. OBJECTIVES To assess information requested by model reporting guidelines and whether the documentation for commonly used machine learning models developed by a single vendor provides the information requested. EVIDENCE REVIEW MEDLINE was queried using machine learning model card and reporting machine learning from November 4 to December 6, 2020. References were reviewed to find additional publications, and publications without specific reporting recommendations were excluded. Similar elements requested for reporting were merged into representative items. Four independent reviewers and 1 adjudicator assessed how often documentation for the most commonly used models developed by a single vendor reported the items. FINDINGS From 15 model reporting guidelines, 220 unique items were identified that represented the collective reporting requirements. Although 12 items were commonly requested (requested by 10 or more guidelines), 77 items were requested by just 1 guideline. Documentation for 12 commonly used models from a single vendor reported a median of 39% (IQR, 37%-43%; range, 31%-47%) of items from the collective reporting requirements. Many of the commonly requested items had 100% reporting rates, including items concerning outcome definition, area under the receiver operating characteristics curve, internal validation, and intended clinical use. Several items reported half the time or less related to reliability, such as external validation, uncertainty measures, and strategy for handling missing data. Other frequently unreported items related to fairness (summary statistics and subgroup analyses, including for race and ethnicity or sex). CONCLUSIONS AND RELEVANCE These findings suggest that consistent reporting recommendations for clinical predictive models are needed for model developers to share necessary information for model deployment. The many published guidelines would, collectively, require reporting more than 200 items. Model documentation from 1 vendor reported the most commonly requested items from model reporting guidelines. However, areas for improvement were identified in reporting items related to model reliability and fairness. This analysis led to feedback to the vendor, which motivated updates to the documentation for future users.
Collapse
Affiliation(s)
- Jonathan H. Lu
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California
| | - Alison Callahan
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California
| | - Birju S. Patel
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California
| | - Keith E. Morse
- Department of Pediatrics, Stanford University School of Medicine, Stanford, California
- Department of Clinical Informatics, Lucile Packard Children’s Hospital, Palo Alto, California
| | - Dev Dash
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California
| | - Michael A. Pfeffer
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California
- Technology and Digital Solutions, Stanford Medicine, Stanford, California
| | - Nigam H. Shah
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California
- Technology and Digital Solutions, Stanford Medicine, Stanford, California
- Clinical Excellence Research Center, Stanford Medicine, Stanford, California
| |
Collapse
|
26
|
Vilendrer S, Lough ME, Garvert DW, Lambert MH, Lu JH, Patel B, Shah NH, Williams MY, Kling SMR. Nursing Workflow Change in a COVID-19 Inpatient Unit Following the Deployment of Inpatient Telehealth: Observational Study Using a Real-Time Locating System. J Med Internet Res 2022; 24:e36882. [PMID: 35635840 PMCID: PMC9208574 DOI: 10.2196/36882] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2022] [Revised: 04/13/2022] [Accepted: 05/11/2022] [Indexed: 01/29/2023] Open
Abstract
BACKGROUND The COVID-19 pandemic prompted widespread implementation of telehealth, including in the inpatient setting, with the goals to reduce potential pathogen exposure events and personal protective equipment (PPE) utilization. Nursing workflow adaptations in these novel environments are of particular interest given the association between nursing time at the bedside and patient safety. Understanding the frequency and duration of nurse-patient encounters following the introduction of a novel telehealth platform in the context of COVID-19 may therefore provide insight into downstream impacts on patient safety, pathogen exposure, and PPE utilization. OBJECTIVE The aim of this study was to evaluate changes in nursing workflow relative to prepandemic levels using a real-time locating system (RTLS) following the deployment of inpatient telehealth on a COVID-19 unit. METHODS In March 2020, telehealth was installed in patient rooms in a COVID-19 unit and on movable carts in 3 comparison units. The existing RTLS captured nurse movement during 1 pre- and 5 postpandemic stages (January-December 2020). Change in direct nurse-patient encounters, time spent in patient rooms per encounter, and total time spent with patients per shift relative to baseline were calculated. Generalized linear models assessed difference-in-differences in outcomes between COVID-19 and comparison units. Telehealth adoption was captured and reported at the unit level. RESULTS Change in frequency of encounters and time spent per encounter from baseline differed between the COVID-19 and comparison units at all stages of the pandemic (all P<.001). Frequency of encounters decreased (difference-in-differences range -6.6 to -14.1 encounters) and duration of encounters increased (difference-in-differences range 1.8 to 6.2 minutes) from baseline to a greater extent in the COVID-19 units relative to the comparison units. At most stages of the pandemic, the change in total time nurses spent in patient rooms per patient per shift from baseline did not differ between the COVID-19 and comparison units (all P>.17). The primary COVID-19 unit quickly adopted telehealth technology during the observation period, initiating 15,088 encounters that averaged 6.6 minutes (SD 13.6) each. CONCLUSIONS RTLS movement data suggest that total nursing time at the bedside remained unchanged following the deployment of inpatient telehealth in a COVID-19 unit. Compared to other units with shared mobile telehealth units, the frequency of nurse-patient in-person encounters decreased and the duration lengthened on a COVID-19 unit with in-room telehealth availability, indicating "batched" redistribution of work to maintain total time at bedside relative to prepandemic periods. The simultaneous adoption of telehealth suggests that virtual care was a complement to, rather than a replacement for, in-person care. However, study limitations preclude our ability to draw a causal link between nursing workflow change and telehealth adoption. Thus, further evaluation is needed to determine potential downstream implications on disease transmission, PPE utilization, and patient safety.
Collapse
Affiliation(s)
- Stacie Vilendrer
- Evaluation Sciences Unit, Division of Primary Care and Population Health, Department of Medicine, Stanford University School of Medicine, Stanford, CA, United States
| | - Mary E Lough
- Evaluation Sciences Unit, Division of Primary Care and Population Health, Department of Medicine, Stanford University School of Medicine, Stanford, CA, United States.,Office of Research Patient Care Services, Stanford Health Care, Palo Alto, CA, United States
| | - Donn W Garvert
- Evaluation Sciences Unit, Division of Primary Care and Population Health, Department of Medicine, Stanford University School of Medicine, Stanford, CA, United States
| | - Monique H Lambert
- Office of Research Patient Care Services, Stanford Health Care, Palo Alto, CA, United States
| | - Jonathan Hsijing Lu
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, CA, United States
| | - Birju Patel
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, CA, United States
| | - Nigam H Shah
- Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, CA, United States
| | - Michelle Y Williams
- Evaluation Sciences Unit, Division of Primary Care and Population Health, Department of Medicine, Stanford University School of Medicine, Stanford, CA, United States.,Office of Research Patient Care Services, Stanford Health Care, Palo Alto, CA, United States
| | - Samantha M R Kling
- Evaluation Sciences Unit, Division of Primary Care and Population Health, Department of Medicine, Stanford University School of Medicine, Stanford, CA, United States
| |
Collapse
|
27
|
Morse KE, Brown C, Fleming S, Todd I, Powell A, Russell A, Scheinker D, Sutherland SM, Lu J, Watkins B, Shah NH, Pageler NM, Palma JP. Monitoring Approaches for a Pediatric Chronic Kidney Disease Machine Learning Model. Appl Clin Inform 2022; 13:431-438. [PMID: 35508197 PMCID: PMC9068274 DOI: 10.1055/s-0042-1746168] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/29/2022] Open
Abstract
OBJECTIVE The purpose of this study is to evaluate the ability of three metrics to monitor for a reduction in performance of a chronic kidney disease (CKD) model deployed at a pediatric hospital. METHODS The CKD risk model estimates a patient's risk of developing CKD 3 to 12 months following an inpatient admission. The model was developed on a retrospective dataset of 4,879 admissions from 2014 to 2018, then run silently on 1,270 admissions from April to October, 2019. Three metrics were used to monitor its performance during the silent phase: (1) standardized mean differences (SMDs); (2) performance of a "membership model"; and (3) response distribution analysis. Observed patient outcomes for the 1,270 admissions were used to calculate prospective model performance and the ability of the three metrics to detect performance changes. RESULTS The deployed model had an area under the receiver-operator curve (AUROC) of 0.63 in the prospective evaluation, which was a significant decrease from an AUROC of 0.76 on retrospective data (p = 0.033). Among the three metrics, SMDs were significantly different for 66/75 (88%) of the model's input variables (p <0.05) between retrospective and deployment data. The membership model was able to discriminate between the two settings (AUROC = 0.71, p <0.0001) and the response distributions were significantly different (p <0.0001) for the two settings. CONCLUSION This study suggests that the three metrics examined could provide early indication of performance deterioration in deployed models' performance.
Collapse
Affiliation(s)
- Keith E Morse
- Division of Pediatric Hospital Medicine, Department of Pediatrics, Stanford University School of Medicine, Stanford, California, United States
| | - Conner Brown
- Information Services Department, Lucile Packard Children's Hospital, Stanford, Palo Alto, California, United States
| | - Scott Fleming
- Department of Biomedical Data Science, Stanford University, Palo Alto, California, United States
| | - Irene Todd
- Information Services Department, Lucile Packard Children's Hospital, Stanford, Palo Alto, California, United States
| | - Austin Powell
- Information Services Department, Lucile Packard Children's Hospital, Stanford, Palo Alto, California, United States
| | - Alton Russell
- Harvard Medical School, Boston, Massachusetts, United States
| | - David Scheinker
- Information Services Department, Lucile Packard Children's Hospital, Stanford, Palo Alto, California, United States
| | - Scott M Sutherland
- Division of Nephrology, Department of Pediatrics, Stanford University, Stanford, California, United States
| | - Jonathan Lu
- Department of Biomedical Data Science, Stanford University, Palo Alto, California, United States
| | - Brendan Watkins
- Information Services Department, Lucile Packard Children's Hospital, Stanford, Palo Alto, California, United States
| | - Nigam H Shah
- Department of Biomedical Data Science, Stanford University, Palo Alto, California, United States
| | - Natalie M Pageler
- Division of Pediatric Critical Care Medicine, Department of Pediatrics, Stanford University School of Medicine, Stanford, California, United States.,Division of Systems Medicine, Department of Pediatrics, Stanford University School of Medicine, Stanford, California, United States
| | - Jonathan P Palma
- Division of Neonatology, Department of Pediatrics, Orlando Health, Orlando, Florida, United States
| |
Collapse
|
28
|
Luo C, Islam MN, Sheils NE, Buresh J, Reps J, Schuemie MJ, Ryan PB, Edmondson M, Duan R, Tong J, Marks-Anglin A, Bian J, Chen Z, Duarte-Salles T, Fernández-Bertolín S, Falconer T, Kim C, Park RW, Pfohl SR, Shah NH, Williams AE, Xu H, Zhou Y, Lautenbach E, Doshi JA, Werner RM, Asch DA, Chen Y. DLMM as a lossless one-shot algorithm for collaborative multi-site distributed linear mixed models. Nat Commun 2022; 13:1678. [PMID: 35354802 PMCID: PMC8967932 DOI: 10.1038/s41467-022-29160-4] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2021] [Accepted: 03/03/2022] [Indexed: 12/21/2022] Open
Abstract
Linear mixed models are commonly used in healthcare-based association analyses for analyzing multi-site data with heterogeneous site-specific random effects. Due to regulations for protecting patients’ privacy, sensitive individual patient data (IPD) typically cannot be shared across sites. We propose an algorithm for fitting distributed linear mixed models (DLMMs) without sharing IPD across sites. This algorithm achieves results identical to those achieved using pooled IPD from multiple sites (i.e., the same effect size and standard error estimates), hence demonstrating the lossless property. The algorithm requires each site to contribute minimal aggregated data in only one round of communication. We demonstrate the lossless property of the proposed DLMM algorithm by investigating the associations between demographic and clinical characteristics and length of hospital stay in COVID-19 patients using administrative claims from the UnitedHealth Group Clinical Discovery Database. We extend this association study by incorporating 120,609 COVID-19 patients from 11 collaborative data sources worldwide. A lossless, one-shot and privacy-preserving distributed algorithm was revealed for fitting linear mixed models on multi-site data. The algorithm was applied to a study of 120,609 COVID-19 patients using only minimal aggregated data from each of 14 sites.
Collapse
Affiliation(s)
- Chongliang Luo
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA, USA.,Division of Public Health Sciences, Washington University School of Medicine in St. Louis, St. Louis, MO, USA
| | | | | | | | - Jenna Reps
- Janssen Research and Development LLC, Titusville, NJ, USA
| | | | - Patrick B Ryan
- Janssen Research and Development LLC, Titusville, NJ, USA
| | - Mackenzie Edmondson
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA, USA
| | - Rui Duan
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA, USA.,Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, USA
| | - Jiayi Tong
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA, USA
| | - Arielle Marks-Anglin
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA, USA
| | - Jiang Bian
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA
| | - Zhaoyi Chen
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA
| | - Talita Duarte-Salles
- Fundacio Institut Universitari per a la recerca a l'Atencio Primaria de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
| | - Sergio Fernández-Bertolín
- Fundacio Institut Universitari per a la recerca a l'Atencio Primaria de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
| | - Thomas Falconer
- Department of Biomedical Informatics, Columbia University, New York, NY, USA
| | - Chungsoo Kim
- Department of Biomedical Sciences, Ajou University Graduate School of Medicine, Suwon, Republic of Korea
| | - Rae Woong Park
- Department of Biomedical Sciences, Ajou University Graduate School of Medicine, Suwon, Republic of Korea.,Department of Biomedical Informatics, Ajou University School of Medicine, Suwon, Republic of Korea
| | - Stephen R Pfohl
- Stanford Center for Biomedical Informatics Research, Stanford, CA, USA
| | - Nigam H Shah
- Stanford Center for Biomedical Informatics Research, Stanford, CA, USA
| | - Andrew E Williams
- Institute for Clinical Research and Health Policy Studies, Tufts University School of Medicine, Boston, MA, USA
| | - Hua Xu
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Yujia Zhou
- School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, USA
| | - Ebbing Lautenbach
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA, USA.,Division of Infectious Diseases, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA.,Center for Clinical Epidemiology and Biostatistics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Jalpa A Doshi
- Division of General Internal Medicine, University of Pennsylvania, Philadelphia, PA, USA.,Leonard Davis Institute of Health Economics, Philadelphia, PA, USA
| | - Rachel M Werner
- Division of General Internal Medicine, University of Pennsylvania, Philadelphia, PA, USA.,Leonard Davis Institute of Health Economics, Philadelphia, PA, USA.,Cpl Michael J Crescenz VA Medical Center, Philadelphia, PA, USA
| | - David A Asch
- Division of General Internal Medicine, University of Pennsylvania, Philadelphia, PA, USA.,Leonard Davis Institute of Health Economics, Philadelphia, PA, USA
| | - Yong Chen
- Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA, USA.
| |
Collapse
|
29
|
Dash D, Gokhale A, Patel BS, Callahan A, Posada J, Krishnan G, Collins W, Li R, Schulman K, Ren L, Shah NH. Building a Learning Health System: Creating an Analytical Workflow for Evidence Generation to Inform Institutional Clinical Care Guidelines. Appl Clin Inform 2022; 13:315-321. [PMID: 35235994 PMCID: PMC8890914 DOI: 10.1055/s-0042-1743241] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
Background
One key aspect of a learning health system (LHS) is utilizing data generated during care delivery to inform clinical care. However, institutional guidelines that utilize observational data are rare and require months to create, making current processes impractical for more urgent scenarios such as those posed by the COVID-19 pandemic. There exists a need to rapidly analyze institutional data to drive guideline creation where evidence from randomized control trials are unavailable.
Objectives
This article provides a background on the current state of observational data generation in institutional guideline creation and details our institution's experience in creating a novel workflow to (1) demonstrate the value of such a workflow, (2) demonstrate a real-world example, and (3) discuss difficulties encountered and future directions.
Methods
Utilizing a multidisciplinary team of database specialists, clinicians, and informaticists, we created a workflow for identifying and translating a clinical need into a queryable format in our clinical data warehouse, creating data summaries and feeding this information back into clinical guideline creation.
Results
Clinical questions posed by the hospital medicine division were answered in a rapid time frame and informed creation of institutional guidelines for the care of patients with COVID-19. The cost of setting up a workflow, answering the questions, and producing data summaries required around 300 hours of effort and $300,000 USD.
Conclusion
A key component of an LHS is the ability to learn from data generated during care delivery. There are rare examples in the literature and we demonstrate one such example along with proposed thoughts of ideal multidisciplinary team formation and deployment.
Collapse
Affiliation(s)
- Dev Dash
- Department of Medicine, Stanford University School of Medicine Stanford, California, United States
| | - Arjun Gokhale
- Department of Medicine, Stanford University School of Medicine Stanford, California, United States
| | - Birju S Patel
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, United States
| | - Alison Callahan
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, United States
| | - Jose Posada
- Department of Medicine, Stanford University School of Medicine Stanford, California, United States
| | - Gomathi Krishnan
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, United States
| | - William Collins
- Department of Medicine, Stanford University School of Medicine Stanford, California, United States
| | - Ron Li
- Department of Medicine, Stanford University School of Medicine Stanford, California, United States
| | - Kevin Schulman
- Department of Medicine, Stanford University School of Medicine Stanford, California, United States
| | - Lily Ren
- Department of Medicine, Stanford University School of Medicine Stanford, California, United States
| | - Nigam H Shah
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, United States
| |
Collapse
|
30
|
Nestsiarovich A, Reps JM, Matheny ME, DuVall SL, Lynch KE, Beaton M, Jiang X, Spotnitz M, Pfohl SR, Shah NH, Torre CO, Reich CG, Lee DY, Son SJ, You SC, Park RW, Ryan PB, Lambert CG. Predictors of diagnostic transition from major depressive disorder to bipolar disorder: a retrospective observational network study. Transl Psychiatry 2021; 11:642. [PMID: 34930903 PMCID: PMC8688463 DOI: 10.1038/s41398-021-01760-6] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/12/2021] [Revised: 11/25/2021] [Accepted: 12/01/2021] [Indexed: 12/02/2022] Open
Abstract
Many patients with bipolar disorder (BD) are initially misdiagnosed with major depressive disorder (MDD) and are treated with antidepressants, whose potential iatrogenic effects are widely discussed. It is unknown whether MDD is a comorbidity of BD or its earlier stage, and no consensus exists on individual conversion predictors, delaying BD's timely recognition and treatment. We aimed to build a predictive model of MDD to BD conversion and to validate it across a multi-national network of patient databases using the standardization afforded by the Observational Medical Outcomes Partnership (OMOP) common data model. Five "training" US databases were retrospectively analyzed: IBM MarketScan CCAE, MDCR, MDCD, Optum EHR, and Optum Claims. Cyclops regularized logistic regression models were developed on one-year MDD-BD conversion with all standard covariates from the HADES PatientLevelPrediction package. Time-to-conversion Kaplan-Meier analysis was performed up to a decade after MDD, stratified by model-estimated risk. External validation of the final prediction model was performed across 9 patient record databases within the Observational Health Data Sciences and Informatics (OHDSI) network internationally. The model's area under the curve (AUC) varied 0.633-0.745 (µ = 0.689) across the five US training databases. Nine variables predicted one-year MDD-BD transition. Factors that increased risk were: younger age, severe depression, psychosis, anxiety, substance misuse, self-harm thoughts/actions, and prior mental disorder. AUCs of the validation datasets ranged 0.570-0.785 (µ = 0.664). An assessment algorithm was built for MDD to BD conversion that allows distinguishing as much as 100-fold risk differences among patients and validates well across multiple international data sources.
Collapse
Affiliation(s)
- Anastasiya Nestsiarovich
- University of New Mexico Health Sciences Center, Department of Internal Medicine, Center for Global Health, Albuquerque, NM, USA
| | - Jenna M Reps
- Janssen Research and Development, Raritan, NJ, USA
| | - Michael E Matheny
- Vanderbilt University, Department of Biomedical Informatics, Department of Medicine, Department of Biostatistics, Nashville, TN, USA
- Tennessee Valley Healthcare System VA, Nashville, TN, USA
| | - Scott L DuVall
- Veterans Affairs Informatics and Computing Infrastructure, VA Salt Lake City Health Care System, Salt Lake City, UT, USA
- University of Utah, Department of Internal Medicine, Salt Lake City, UT, USA
| | - Kristine E Lynch
- Veterans Affairs Informatics and Computing Infrastructure, VA Salt Lake City Health Care System, Salt Lake City, UT, USA
- University of Utah, Department of Internal Medicine, Salt Lake City, UT, USA
| | - Maura Beaton
- Columbia University Irving Medical Center, Department of Biomedical Informatics, New York, NY, USA
| | - Xinzhuo Jiang
- Columbia University Irving Medical Center, Department of Biomedical Informatics, New York, NY, USA
| | - Matthew Spotnitz
- Columbia University Irving Medical Center, Department of Biomedical Informatics, New York, NY, USA
| | - Stephen R Pfohl
- Stanford University, Stanford Center for Biomedical Informatics Research, Stanford, CA, USA
| | - Nigam H Shah
- Stanford University, Stanford Center for Biomedical Informatics Research, Stanford, CA, USA
| | | | | | - Dong Yun Lee
- Ajou University School of Medicine, Department of Psychiatry, Suwon, Republic of Korea
| | - Sang Joon Son
- Ajou University School of Medicine, Department of Psychiatry, Suwon, Republic of Korea
| | - Seng Chan You
- Ajou University School of Medicine, Department of Biomedical Informatics, Suwon, Republic of Korea
| | - Rae Woong Park
- Ajou University School of Medicine, Department of Biomedical Informatics, Suwon, Republic of Korea
| | - Patrick B Ryan
- Janssen Research and Development, Raritan, NJ, USA
- Columbia University Irving Medical Center, Department of Biomedical Informatics, New York, NY, USA
| | - Christophe G Lambert
- University of New Mexico Health Sciences Center, Department of Internal Medicine, Center for Global Health, Albuquerque, NM, USA.
- University of New Mexico Health Sciences Center, Department of Internal Medicine, Center for Global Health, Division of Translational Informatics, Albuquerque, NM, USA.
| |
Collapse
|
31
|
Abstract
Background The promise of precision population health includes the ability to use robust patient data to tailor prevention and care to specific groups. Advanced analytics may allow for automated detection of clinically informative subgroups that account for clinical, genetic, and environmental variability. This study sought to evaluate whether unsupervised machine learning approaches could interpret heterogeneous and missing clinical data to discover clinically important coronary artery disease subgroups. Methods and Results The Genetic Determinants of Peripheral Arterial Disease study is a prospective cohort that includes individuals with newly diagnosed and/or symptomatic coronary artery disease. We applied generalized low rank modeling and K-means cluster analysis using 155 phenotypic and genetic variables from 1329 participants. Cox proportional hazard models were used to examine associations between clusters and major adverse cardiovascular and cerebrovascular events and all-cause mortality. We then compared performance of risk stratification based on clusters and the American College of Cardiology/American Heart Association pooled cohort equations. Unsupervised analysis identified 4 phenotypically and prognostically distinct clusters. All-cause mortality was highest in cluster 1 (oldest/most comorbid; 26%), whereas major adverse cardiovascular and cerebrovascular event rates were highest in cluster 2 (youngest/multiethnic; 41%). Cluster 4 (middle-aged/healthiest behaviors) experienced more incident major adverse cardiovascular and cerebrovascular events (30%) than cluster 3 (middle-aged/lowest medication adherence; 23%), despite apparently similar risk factor and lifestyle profiles. In comparison with the pooled cohort equations, cluster membership was more informative for risk assessment of myocardial infarction, stroke, and mortality. Conclusions Unsupervised clustering identified 4 unique coronary artery disease subgroups with distinct clinical trajectories. Flexible unsupervised machine learning algorithms offer the ability to meaningfully process heterogeneous patient data and provide sharper insights into disease characterization and risk assessment. Registration URL: https://www.clinicaltrials.gov; Unique identifier: NCT00380185.
Collapse
Affiliation(s)
- Alyssa M Flores
- Division of Vascular Surgery Department of Surgery Stanford University School of Medicine Stanford CA
| | - Alejandro Schuler
- Center for Biomedical Informatics Research Stanford University Stanford CA
| | - Anne Verena Eberhard
- Division of Vascular Surgery Department of Surgery Stanford University School of Medicine Stanford CA
| | - Jeffrey W Olin
- Zena and Michael A. Wiener Cardiovascular InstituteMarie-Josée and Henry R. Kravis Center for Cardiovascular HealthIcahn School of Medicine at Mount Sinai New York NY
| | - John P Cooke
- Department of Cardiovascular Sciences Houston Methodist Research Institute Houston TX
| | - Nicholas J Leeper
- Division of Vascular Surgery Department of Surgery Stanford University School of Medicine Stanford CA.,Division of Cardiovascular Medicine Department of Medicine Stanford University School of Medicine Stanford CA.,Stanford Cardiovascular Institute Stanford CA
| | - Nigam H Shah
- Center for Biomedical Informatics Research Stanford University Stanford CA
| | - Elsie G Ross
- Division of Vascular Surgery Department of Surgery Stanford University School of Medicine Stanford CA.,Center for Biomedical Informatics Research Stanford University Stanford CA.,Stanford Cardiovascular Institute Stanford CA
| |
Collapse
|
32
|
Bai L, Scott MKD, Steinberg E, Kalesinskas L, Habtezion A, Shah NH, Khatri P. Computational drug repositioning of atorvastatin for ulcerative colitis. J Am Med Inform Assoc 2021; 28:2325-2335. [PMID: 34529084 PMCID: PMC8510297 DOI: 10.1093/jamia/ocab165] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2021] [Revised: 06/22/2021] [Accepted: 07/20/2021] [Indexed: 02/07/2023] Open
Abstract
OBJECTIVE Ulcerative colitis (UC) is a chronic inflammatory disorder with limited effective therapeutic options for long-term treatment and disease maintenance. We hypothesized that a multi-cohort analysis of independent cohorts representing real-world heterogeneity of UC would identify a robust transcriptomic signature to improve identification of FDA-approved drugs that can be repurposed to treat patients with UC. MATERIALS AND METHODS We performed a multi-cohort analysis of 272 colon biopsy transcriptome samples across 11 publicly available datasets to identify a robust UC disease gene signature. We compared the gene signature to in vitro transcriptomic profiles induced by 781 FDA-approved drugs to identify potential drug targets. We used a retrospective cohort study design modeled after a target trial to evaluate the protective effect of predicted drugs on colectomy risk in patients with UC from the Stanford Research Repository (STARR) database and Optum Clinformatics DataMart. RESULTS Atorvastatin treatment had the highest inverse-correlation with the UC gene signature among non-oncolytic FDA-approved therapies. In both STARR (n = 827) and Optum (n = 7821), atorvastatin intake was significantly associated with a decreased risk of colectomy, a marker of treatment-refractory disease, compared to patients prescribed a comparator drug (STARR: HR = 0.47, P = .03; Optum: HR = 0.66, P = .03), irrespective of age and length of atorvastatin treatment. DISCUSSION & CONCLUSION These findings suggest that atorvastatin may serve as a novel therapeutic option for ameliorating disease in patients with UC. Importantly, we provide a systematic framework for integrating publicly available heterogeneous molecular data with clinical data at a large scale to repurpose existing FDA-approved drugs for a wide range of human diseases.
Collapse
Affiliation(s)
- Lawrence Bai
- Immunology Program, Stanford University School of Medicine, Stanford, California, USA.,Institute for Immunity, Transplantation and Infection, Stanford University School of Medicine, Stanford, California, USA.,Center for Biomedical Informatics Research, Department of Medicine, Stanford University, Stanford, California, USA
| | - Madeleine K D Scott
- Institute for Immunity, Transplantation and Infection, Stanford University School of Medicine, Stanford, California, USA.,Center for Biomedical Informatics Research, Department of Medicine, Stanford University, Stanford, California, USA.,Biophysics Program, Stanford University School of Medicine, Stanford, California, USA
| | - Ethan Steinberg
- Computer Science Program, Department of Computer Science, Stanford University, Stanford, California, USA
| | - Laurynas Kalesinskas
- Biomedical Informatics Training Program, Stanford University School of Medicine, Stanford, California, USA
| | - Aida Habtezion
- Immunology Program, Stanford University School of Medicine, Stanford, California, USA.,Institute for Immunity, Transplantation and Infection, Stanford University School of Medicine, Stanford, California, USA.,Division of Gastroenterology and Hepatology, Department of Medicine, Stanford University School of Medicine, Stanford, California, USA
| | - Nigam H Shah
- Center for Biomedical Informatics Research, Department of Medicine, Stanford University, Stanford, California, USA
| | - Purvesh Khatri
- Institute for Immunity, Transplantation and Infection, Stanford University School of Medicine, Stanford, California, USA.,Center for Biomedical Informatics Research, Department of Medicine, Stanford University, Stanford, California, USA
| |
Collapse
|
33
|
Lai AG, Chang WH, Parisinos CA, Katsoulis M, Blackburn RM, Shah AD, Nguyen V, Denaxas S, Davey Smith G, Gaunt TR, Nirantharakumar K, Cox MP, Forde D, Asselbergs FW, Harris S, Richardson S, Sofat R, Dobson RJB, Hingorani A, Patel R, Sterne J, Banerjee A, Denniston AK, Ball S, Sebire NJ, Shah NH, Foster GR, Williams B, Hemingway H. An informatics consult approach for generating clinical evidence for treatment decisions. BMC Med Inform Decis Mak 2021; 21:281. [PMID: 34641870 PMCID: PMC8506488 DOI: 10.1186/s12911-021-01638-z] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2021] [Accepted: 09/27/2021] [Indexed: 11/20/2022] Open
Abstract
BACKGROUND An Informatics Consult has been proposed in which clinicians request novel evidence from large scale health data resources, tailored to the treatment of a specific patient. However, the availability of such consultations is lacking. We seek to provide an Informatics Consult for a situation where a treatment indication and contraindication coexist in the same patient, i.e., anti-coagulation use for stroke prevention in a patient with both atrial fibrillation (AF) and liver cirrhosis. METHODS We examined four sources of evidence for the effect of warfarin on stroke risk or all-cause mortality from: (1) randomised controlled trials (RCTs), (2) meta-analysis of prior observational studies, (3) trial emulation (using population electronic health records (N = 3,854,710) and (4) genetic evidence (Mendelian randomisation). We developed prototype forms to request an Informatics Consult and return of results in electronic health record systems. RESULTS We found 0 RCT reports and 0 trials recruiting for patients with AF and cirrhosis. We found broad concordance across the three new sources of evidence we generated. Meta-analysis of prior observational studies showed that warfarin use was associated with lower stroke risk (hazard ratio [HR] = 0.71, CI 0.39-1.29). In a target trial emulation, warfarin was associated with lower all-cause mortality (HR = 0.61, CI 0.49-0.76) and ischaemic stroke (HR = 0.27, CI 0.08-0.91). Mendelian randomisation served as a drug target validation where we found that lower levels of vitamin K1 (warfarin is a vitamin K1 antagonist) are associated with lower stroke risk. A pilot survey with an independent sample of 34 clinicians revealed that 85% of clinicians found information on prognosis useful and that 79% thought that they should have access to the Informatics Consult as a service within their healthcare systems. We identified candidate steps for automation to scale evidence generation and to accelerate the return of results. CONCLUSION We performed a proof-of-concept Informatics Consult for evidence generation, which may inform treatment decisions in situations where there is dearth of randomised trials. Patients are surprised to know that their clinicians are currently not able to learn in clinic from data on 'patients like me'. We identify the key challenges in offering such an Informatics Consult as a service.
Collapse
Affiliation(s)
- Alvina G Lai
- Institute of Health Informatics, University College London, London, UK.
- Health Data Research UK, London, UK.
| | - Wai Hoong Chang
- Institute of Health Informatics, University College London, London, UK
- Health Data Research UK, London, UK
| | | | - Michail Katsoulis
- Institute of Health Informatics, University College London, London, UK
| | - Ruth M Blackburn
- Institute of Health Informatics, University College London, London, UK
- Health Data Research UK, London, UK
| | - Anoop D Shah
- Institute of Health Informatics, University College London, London, UK
- University College London Hospitals NIHR Biomedical Research Centre, London, UK
- University College London Hospitals NHS Trust, London, UK
| | - Vincent Nguyen
- Institute of Health Informatics, University College London, London, UK
| | - Spiros Denaxas
- Institute of Health Informatics, University College London, London, UK
- Health Data Research UK, London, UK
- University College London Hospitals NIHR Biomedical Research Centre, London, UK
- The Alan Turing Institute, London, UK
| | - George Davey Smith
- Medical Research Council Integrative Epidemiology Unit, University of Bristol, Bristol, UK
- Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
| | - Tom R Gaunt
- Medical Research Council Integrative Epidemiology Unit, University of Bristol, Bristol, UK
- Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
| | - Krishnarajah Nirantharakumar
- Health Data Research UK, London, UK
- Institute of Applies Health Research, University of Birmingham, Birmingham, UK
| | - Murray P Cox
- Statistics and Bioinformatics Group, School of Fundamental Sciences, Massey University, Palmerston North, New Zealand
| | - Donall Forde
- Public Health Wales, University Hospital of Wales, Cardiff, UK
| | - Folkert W Asselbergs
- Institute of Health Informatics, University College London, London, UK
- Health Data Research UK, London, UK
- University College London Hospitals NIHR Biomedical Research Centre, London, UK
- Department of Cardiology, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
- Institute of Cardiovascular Science, University College London, London, UK
| | - Steve Harris
- University College London Hospitals NHS Trust, London, UK
| | - Sylvia Richardson
- Medical Research Council Biostatistics Unit, University of Cambridge, Cambridge, UK
| | - Reecha Sofat
- Institute of Health Informatics, University College London, London, UK
- University College London Hospitals NIHR Biomedical Research Centre, London, UK
| | - Richard J B Dobson
- Institute of Health Informatics, University College London, London, UK
- Health Data Research UK, London, UK
- Department of Biostatistics and Health Informatics, King's College London, London, UK
| | - Aroon Hingorani
- Health Data Research UK, London, UK
- Institute of Cardiovascular Science, University College London, London, UK
| | - Riyaz Patel
- Institute of Cardiovascular Science, University College London, London, UK
| | - Jonathan Sterne
- Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
| | - Amitava Banerjee
- Institute of Health Informatics, University College London, London, UK
- Barts Health NHS Trust, The Royal London Hospital, Whitechapel Rd, London, UK
| | - Alastair K Denniston
- Health Data Research UK, London, UK
- University Hospitals Birmingham NHSFT, Birmingham, UK
| | - Simon Ball
- Health Data Research UK, London, UK
- University Hospitals Birmingham NHSFT, Birmingham, UK
| | - Neil J Sebire
- UCL Great Ormond Street Institute of Child Health, London, UK
- NIHR Great Ormond Street Hospital Biomedical Research Centre, London, UK
| | - Nigam H Shah
- Department of Medicine, School of Medicine, Stanford University, Stanford, CA, USA
| | - Graham R Foster
- Barts Liver Centre, Blizard Institute, Queen Mary University of London, London, UK
| | - Bryan Williams
- University College London Hospitals NIHR Biomedical Research Centre, London, UK
- Institute of Cardiovascular Science, University College London, London, UK
- University College London Hospitals NHS Trust, London, UK
| | - Harry Hemingway
- Institute of Health Informatics, University College London, London, UK
- Health Data Research UK, London, UK
| |
Collapse
|
34
|
Kashyap S, Morse KE, Patel B, Shah NH. A survey of extant organizational and computational setups for deploying predictive models in health systems. J Am Med Inform Assoc 2021; 28:2445-2450. [PMID: 34423364 PMCID: PMC8510384 DOI: 10.1093/jamia/ocab154] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2021] [Revised: 06/07/2021] [Accepted: 07/11/2021] [Indexed: 11/13/2022] Open
Abstract
OBJECTIVE Artificial intelligence (AI) and machine learning (ML) enabled healthcare is now feasible for many health systems, yet little is known about effective strategies of system architecture and governance mechanisms for implementation. Our objective was to identify the different computational and organizational setups that early-adopter health systems have utilized to integrate AI/ML clinical decision support (AI-CDS) and scrutinize their trade-offs. MATERIALS AND METHODS We conducted structured interviews with health systems with AI deployment experience about their organizational and computational setups for deploying AI-CDS at point of care. RESULTS We contacted 34 health systems and interviewed 20 healthcare sites (58% response rate). Twelve (60%) sites used the native electronic health record vendor configuration for model development and deployment, making it the most common shared infrastructure. Nine (45%) sites used alternative computational configurations which varied significantly. Organizational configurations for managing AI-CDS were distinguished by how they identified model needs, built and implemented models, and were separable into 3 major types: Decentralized translation (n = 10, 50%), IT Department led (n = 2, 10%), and AI in Healthcare (AIHC) Team (n = 8, 40%). DISCUSSION No singular computational configuration enables all current use cases for AI-CDS. Health systems need to consider their desired applications for AI-CDS and whether investment in extending the off-the-shelf infrastructure is needed. Each organizational setup confers trade-offs for health systems planning strategies to implement AI-CDS. CONCLUSION Health systems will be able to use this framework to understand strengths and weaknesses of alternative organizational and computational setups when designing their strategy for artificial intelligence.
Collapse
Affiliation(s)
- Sehj Kashyap
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA
| | - Keith E Morse
- Division of Pediatric Hospital Medicine, Department of Pediatrics, Stanford University School of Medicine, Stanford, California, USA
| | - Birju Patel
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA
| | - Nigam H Shah
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA
| |
Collapse
|
35
|
Tan EH, Sena AG, Prats-Uribe A, You SC, Ahmed WUR, Kostka K, Reich C, Duvall SL, Lynch KE, Matheny ME, Duarte-Salles T, Bertolin SF, Hripcsak G, Natarajan K, Falconer T, Spotnitz M, Ostropolets A, Blacketer C, Alshammari TM, Alghoul H, Alser O, Lane JCE, Dawoud DM, Shah K, Yang Y, Zhang L, Areia C, Golozar A, Recalde M, Casajust P, Jonnagaddala J, Subbian V, Vizcaya D, Lai LYH, Nyberg F, Morales DR, Posada JD, Shah NH, Gong M, Vivekanantham A, Abend A, Minty EP, Suchard M, Rijnbeek P, Ryan PB, Prieto-Alhambra D. COVID-19 in patients with autoimmune diseases: characteristics and outcomes in a multinational network of cohorts across three countries. Rheumatology (Oxford) 2021; 60:SI37-SI50. [PMID: 33725121 PMCID: PMC7989171 DOI: 10.1093/rheumatology/keab250] [Citation(s) in RCA: 21] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/10/2020] [Accepted: 03/07/2021] [Indexed: 12/28/2022] Open
Abstract
OBJECTIVE Patients with autoimmune diseases were advised to shield to avoid coronavirus disease 2019 (COVID-19), but information on their prognosis is lacking. We characterized 30-day outcomes and mortality after hospitalization with COVID-19 among patients with prevalent autoimmune diseases, and compared outcomes after hospital admissions among similar patients with seasonal influenza. METHODS A multinational network cohort study was conducted using electronic health records data from Columbia University Irving Medical Center [USA, Optum (USA), Department of Veterans Affairs (USA), Information System for Research in Primary Care-Hospitalization Linked Data (Spain) and claims data from IQVIA Open Claims (USA) and Health Insurance and Review Assessment (South Korea). All patients with prevalent autoimmune diseases, diagnosed and/or hospitalized between January and June 2020 with COVID-19, and similar patients hospitalized with influenza in 2017-18 were included. Outcomes were death and complications within 30 days of hospitalization. RESULTS We studied 133 589 patients diagnosed and 48 418 hospitalized with COVID-19 with prevalent autoimmune diseases. Most patients were female, aged ≥50 years with previous comorbidities. The prevalence of hypertension (45.5-93.2%), chronic kidney disease (14.0-52.7%) and heart disease (29.0-83.8%) was higher in hospitalized vs diagnosed patients with COVID-19. Compared with 70 660 hospitalized with influenza, those admitted with COVID-19 had more respiratory complications including pneumonia and acute respiratory distress syndrome, and higher 30-day mortality (2.2-4.3% vs 6.32-24.6%). CONCLUSION Compared with influenza, COVID-19 is a more severe disease, leading to more complications and higher mortality.
Collapse
Affiliation(s)
- Eng Hooi Tan
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, OX3 7LD, UK
| | - Anthony G Sena
- Janssen Research and Development, Titusville, NJ USA
- Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, The Netherlands
| | - Albert Prats-Uribe
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, OX3 7LD, UK
| | - Seng Chan You
- Department of Biomedical Informatics, Ajou University School of Medicine, Suwon, Korea
| | - Waheed-Ul-Rahman Ahmed
- Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Botnar Research Centre, Oxford, OX3, 7LD, UK
- College of Medicine and Health, University of Exeter, St Luke’s, 2LU, USA
| | | | | | - Scott L Duvall
- VA Informatics and Computing Infrastructure, VA Salt Lake City Health Care System, UT, USA
- Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, USA
| | - Kristine E Lynch
- VA Informatics and Computing Infrastructure, VA Salt Lake City Health Care System, UT, USA
- Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, USA
| | - Michael E Matheny
- Tennessee Valley Healthcare System, Veterans Affairs Medical Center, Nashville, TN, USA
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Talita Duarte-Salles
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
| | - Sergio Fernandez Bertolin
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
| | - George Hripcsak
- Department of Biomedical Informatics, Columbia University, New York, NY, USA
- New York-Presbyterian Hospital, New York, NY, USA
| | - Karthik Natarajan
- Department of Biomedical Informatics, Columbia University, New York, NY, USA
- New York-Presbyterian Hospital, New York, NY, USA
| | - Thomas Falconer
- Department of Biomedical Informatics, Columbia University, New York, NY, USA
| | - Matthew Spotnitz
- Department of Biomedical Informatics, Columbia University, New York, NY, USA
| | - Anna Ostropolets
- Department of Biomedical Informatics, Columbia University, New York, NY, USA
| | - Clair Blacketer
- Janssen Research and Development, Titusville, NJ USA
- Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, The Netherlands
| | - Thamir M Alshammari
- Medication Safety Research Chair, King Saud University, Riyadh, Saudi Arabia
| | - Heba Alghoul
- Faculty of Medicine, Islamic University of Gaza, Palestine
| | - Osaid Alser
- Massachusetts General Hospital, Harvard Medical School, Boston, 02114, MA, USA
| | - Jennifer C E Lane
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, OX3 7LD, UK
| | | | - Karishma Shah
- Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Botnar Research Centre, Oxford, OX3, 7LD, UK
| | - Yue Yang
- Digital China Health Technologies Co., LTD, Beijing 100085, China
| | - Lin Zhang
- School of Population Medicine and Public Health, Chinese Academy of Medical Science & Peking Union Medical College, Beijing 100730, China
- Melbourne School of Population and Global Health, The University of Melbourne, Victoria 3015, Australia
| | - Carlos Areia
- Nuffield Department of Clinical Neurosciences, University of Oxford, OX3 9DU, UK
| | - Asieh Golozar
- Regeneron Pharmaceuticals, NY, USA
- Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health,, Baltimore, MD, USA
| | - Martina Recalde
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
- Universitat Autonoma de Barcelona, Bellaterra, Spain
| | - Paula Casajust
- Real-World Evidence, Trial Form Support, Barcelona, Spain
| | | | - Vignesh Subbian
- College of Engineering, The University of Arizona Tucson, Arizona, USA
| | - David Vizcaya
- Bayer Pharmaceuticals, Sant Joan Despi, Barcelona, Spain
| | - Lana Y H Lai
- School of Medical Sciences, University of Manchester, UK
| | - Fredrik Nyberg
- School of Public Health and Community Medicine, Institute of Medicine, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
| | - Daniel R Morales
- Division of Population Health Sciences, University of Dundee, Dundee, Scotland, UK
| | - Jose D Posada
- Stanford Center for Biomedical Informatics Research, Department of Medicine, School of Medicine, Stanford University, Stanford, CA, USA
| | - Nigam H Shah
- Stanford Center for Biomedical Informatics Research, Department of Medicine, School of Medicine, Stanford University, Stanford, CA, USA
| | - Mengchun Gong
- Health Management Institute, Southern Medical University, Guangzhou, China
| | - Arani Vivekanantham
- Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Botnar Research Centre, Oxford, OX3, 7LD, UK
| | - Aaron Abend
- Autoimmune Registry Inc., Guilford, CT 06437, USA
| | - Evan P Minty
- O’Brien School for Public Health, Faculty of Medicine, University of Calgary, Calgary, Alberta, T2N, 1N4, Canada
| | - Marc Suchard
- Department of Biostatistics, UCLA Fielding School of Public Health, University of California, Los Angeles, CA, USA
| | - Peter Rijnbeek
- Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, The Netherlands
| | - Patrick B Ryan
- Janssen Research and Development, Titusville, NJ USA
- Department of Biomedical Informatics, Columbia University, New York, NY, USA
| | - Daniel Prieto-Alhambra
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, OX3 7LD, UK
| |
Collapse
|
36
|
Sounderajah V, Ashrafian H, Rose S, Shah NH, Ghassemi M, Golub R, Kahn CE, Esteva A, Karthikesalingam A, Mateen B, Webster D, Milea D, Ting D, Treanor D, Cushnan D, King D, McPherson D, Glocker B, Greaves F, Harling L, Ordish J, Cohen JF, Deeks J, Leeflang M, Diamond M, McInnes MDF, McCradden M, Abràmoff MD, Normahani P, Markar SR, Chang S, Liu X, Mallett S, Shetty S, Denniston A, Collins GS, Moher D, Whiting P, Bossuyt PM, Darzi A. A quality assessment tool for artificial intelligence-centered diagnostic test accuracy studies: QUADAS-AI. Nat Med 2021; 27:1663-1665. [PMID: 34635854 DOI: 10.1038/s41591-021-01517-0] [Citation(s) in RCA: 58] [Impact Index Per Article: 19.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Affiliation(s)
- Viknesh Sounderajah
- Institute of Global Health Innovation, Imperial College London, London, UK.
- Department of Surgery and Cancer, Imperial College London, London, UK.
| | - Hutan Ashrafian
- Institute of Global Health Innovation, Imperial College London, London, UK
- Department of Surgery and Cancer, Imperial College London, London, UK
| | - Sherri Rose
- Center for Health Policy and Center for Primary Care and Outcomes Research, Stanford University, Stanford, CA, USA
| | - Nigam H Shah
- Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA
| | - Marzyeh Ghassemi
- Institute for Medical Engineering & Science, Massachusetts Institute of Technology, Cambridge, MA, USA
| | - Robert Golub
- Journal of the American Medical Association (JAMA), Chicago, IL, USA
| | - Charles E Kahn
- University of Pennsylvania, Philadelphia, Pennsylvania, PA, USA
| | | | | | | | | | - Dan Milea
- Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, Singapore
| | - Daniel Ting
- Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, Singapore
| | - Darren Treanor
- Leeds Teaching Hospitals NHS Trust, Leeds, UK
- University of Leeds, Leeds, UK
- Department of Clinical Pathology, and Department of Clinical and Experimental Medicine, Linköping University, Linköping, Sweden
- Center for Medical Image Science and Visualization (CMIV), Linköping University, Linköping, Sweden
| | | | - Dominic King
- Institute of Global Health Innovation, Imperial College London, London, UK
- Optum, London, UK
| | | | - Ben Glocker
- Faculty of Engineering, Department of Computing, Imperial College London, London, UK
| | - Felix Greaves
- National Institute for Health and Care Excellence, London, UK
| | - Leanne Harling
- Institute of Global Health Innovation, Imperial College London, London, UK
- Department of Surgery and Cancer, Imperial College London, London, UK
| | - Johan Ordish
- Medicines and Healthcare Products Regulatory Agency, London, UK
| | - Jérémie F Cohen
- Department of Pediatrics, Centre of Research in Epidemiology and Statistics, Inserm UMR 1153, Necker- Enfants Malades Hospital, Assistance Publique-Hôpitaux de Paris, Université de Paris, Paris, France
| | - Jon Deeks
- Institute of Applied Health Research, University of Birmingham, Birmingham, UK
| | - Mariska Leeflang
- Department of Epidemiology and Data Science, Amsterdam University Medical Centres, University of Amsterdam, Amsterdam, The Netherlands
| | | | - Matthew D F McInnes
- Departments of Radiology and Epidemiology, University of Ottawa, The Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
| | - Melissa McCradden
- Department of Bioethics, The Hospital for Sick Kids, Toronto, Ontario, Canada
| | - Michael D Abràmoff
- Department of Ophthalmology and Visual Sciences, University of Iowa, Iowa City, IA, USA
| | - Pasha Normahani
- Department of Surgery and Cancer, Imperial College London, London, UK
| | - Sheraz R Markar
- Department of Surgery and Cancer, Imperial College London, London, UK
| | - Stephanie Chang
- Annals of Internal Medicine, American College of Physicians, Philadelphia, PA, USA
| | - Xiaoxuan Liu
- Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
- University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Health Data Research UK, London, UK
| | - Susan Mallett
- Centre for Medical Imaging, University College London, London, UK
| | | | - Alastair Denniston
- Institute of Inflammation and Ageing, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK
- University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- Health Data Research UK, London, UK
| | - Gary S Collins
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
- NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
| | - David Moher
- Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
| | - Penny Whiting
- Bristol Medical School, University of Bristol, Bristol, UK
| | - Patrick M Bossuyt
- Department of Epidemiology and Data Science, Amsterdam University Medical Centres, University of Amsterdam, Amsterdam, The Netherlands.
| | - Ara Darzi
- Institute of Global Health Innovation, Imperial College London, London, UK.
- Department of Surgery and Cancer, Imperial College London, London, UK.
| |
Collapse
|
37
|
Roel E, Pistillo A, Recalde M, Sena AG, Fernández-Bertolín S, Aragón M, Puente D, Ahmed WUR, Alghoul H, Alser O, Alshammari TM, Areia C, Blacketer C, Carter W, Casajust P, Culhane AC, Dawoud D, DeFalco F, DuVall SL, Falconer T, Golozar A, Gong M, Hester L, Hripcsak G, Tan EH, Jeon H, Jonnagaddala J, Lai LYH, Lynch KE, Matheny ME, Morales DR, Natarajan K, Nyberg F, Ostropolets A, Posada JD, Prats-Uribe A, Reich CG, Rivera DR, Schilling LM, Soerjomataram I, Shah K, Shah NH, Shen Y, Spotniz M, Subbian V, Suchard MA, Trama A, Zhang L, Zhang Y, Ryan PB, Prieto-Alhambra D, Kostka K, Duarte-Salles T. Characteristics and Outcomes of Over 300,000 Patients with COVID-19 and History of Cancer in the United States and Spain. Cancer Epidemiol Biomarkers Prev 2021; 30:1884-1894. [PMID: 34272262 PMCID: PMC8974356 DOI: 10.1158/1055-9965.epi-21-0266] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2021] [Revised: 04/26/2021] [Accepted: 07/07/2021] [Indexed: 01/07/2023] Open
Abstract
BACKGROUND We described the demographics, cancer subtypes, comorbidities, and outcomes of patients with a history of cancer and coronavirus disease 2019 (COVID-19). Second, we compared patients hospitalized with COVID-19 to patients diagnosed with COVID-19 and patients hospitalized with influenza. METHODS We conducted a cohort study using eight routinely collected health care databases from Spain and the United States, standardized to the Observational Medical Outcome Partnership common data model. Three cohorts of patients with a history of cancer were included: (i) diagnosed with COVID-19, (ii) hospitalized with COVID-19, and (iii) hospitalized with influenza in 2017 to 2018. Patients were followed from index date to 30 days or death. We reported demographics, cancer subtypes, comorbidities, and 30-day outcomes. RESULTS We included 366,050 and 119,597 patients diagnosed and hospitalized with COVID-19, respectively. Prostate and breast cancers were the most frequent cancers (range: 5%-18% and 1%-14% in the diagnosed cohort, respectively). Hematologic malignancies were also frequent, with non-Hodgkin's lymphoma being among the five most common cancer subtypes in the diagnosed cohort. Overall, patients were aged above 65 years and had multiple comorbidities. Occurrence of death ranged from 2% to 14% and from 6% to 26% in the diagnosed and hospitalized COVID-19 cohorts, respectively. Patients hospitalized with influenza (n = 67,743) had a similar distribution of cancer subtypes, sex, age, and comorbidities but lower occurrence of adverse events. CONCLUSIONS Patients with a history of cancer and COVID-19 had multiple comorbidities and a high occurrence of COVID-19-related events. Hematologic malignancies were frequent. IMPACT This study provides epidemiologic characteristics that can inform clinical care and etiologic studies.
Collapse
Affiliation(s)
- Elena Roel
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
- Universitat Autònoma de Barcelona, Spain
| | - Andrea Pistillo
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
| | - Martina Recalde
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
- Universitat Autònoma de Barcelona, Spain
| | - Anthony G Sena
- Janssen Research and Development, Titusville, New Jersey
- Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, the Netherlands
| | - Sergio Fernández-Bertolín
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
| | - Maria Aragón
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
| | - Diana Puente
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
- Universitat Autònoma de Barcelona, Spain
| | - Waheed-Ul-Rahman Ahmed
- NDORMS, University of Oxford, Botnar Research Centre, Windmill Road, Oxford, United Kingdom
- College of Medicine and Health, University of Exeter, St Luke's Campus, Heavitree Road, Exeter, United Kingdom
| | - Heba Alghoul
- Faculty of Medicine, Islamic University of Gaza, Gaza, Palestine
| | - Osaid Alser
- Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
| | - Thamir M Alshammari
- Medication Safety Research Chair, King Saud University, Riyadh, Saudi Arabia
| | - Carlos Areia
- Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, United Kingdom
| | | | - William Carter
- Data Science to Patient Value Program, Department of Medicine, University of Colorado Anschutz Medical Campus, Aurora, Colorado
| | - Paula Casajust
- Real-World Evidence, Trial Form Support, Barcelona, Spain
| | - Aedin C Culhane
- Department of Data Science, Dana-Farber Cancer Institute, Boston, Massachusetts
- Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, Massachusetts
| | - Dalia Dawoud
- Faculty of Pharmacy, Cairo University, Cairo, Egypt
| | - Frank DeFalco
- Janssen Research and Development, Titusville, New Jersey
| | - Scott L DuVall
- VA Informatics and Computing Infrastructure, VA Salt Lake City Health Care System, Salt Lake City, Utah
- Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, Utah
| | - Thomas Falconer
- Department of Biomedical Informatics, Columbia University, New York, New York
- New York-Presbyterian Hospital, New York, New York
| | - Asieh Golozar
- Department of Epidemiology, Johns Hopkins School of Public Health, Baltimore, Maryland
- Pharmacoepidemiology, Regeneron Pharmaceuticals, Westchester County, New York
| | - Mengchun Gong
- Digital Health China Technologies Co., Ltd., Beijing, China
| | - Laura Hester
- Janssen Research and Development, LLC, Raritan, New Jersey
| | - George Hripcsak
- Department of Biomedical Informatics, Columbia University, New York, New York
- New York-Presbyterian Hospital, New York, New York
| | - Eng Hooi Tan
- Centre for Statistics in Medicine, NDORMS, University of Oxford, Oxford, United Kingdom
| | - Hokyun Jeon
- Department of Biomedical Sciences, Ajou University Graduate School of Medicine, Suwon, Gyeonggi-do, Republic of Korea
| | | | - Lana Y H Lai
- School of Medical Sciences, University of Manchester, Manchester, United Kingdom
| | - Kristine E Lynch
- VA Informatics and Computing Infrastructure, VA Salt Lake City Health Care System, Salt Lake City, Utah
- Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, Utah
| | - Michael E Matheny
- Tennessee Valley Healthcare System, Veterans Affairs Medical Center, Nashville, Tennessee
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee
| | - Daniel R Morales
- Division of Population Health and Genomics, University of Dundee, Dundee, United Kingdom
- University of Southern Denmark, Odense, Denmark
| | - Karthik Natarajan
- Department of Biomedical Informatics, Columbia University, New York, New York
- New York-Presbyterian Hospital, New York, New York
| | - Fredrik Nyberg
- School of Public Health and Community Medicine, Institute of Medicine, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
| | - Anna Ostropolets
- Department of Biomedical Informatics, Columbia University, New York, New York
| | - José D Posada
- Department of Medicine, School of Medicine, Stanford University, Redwood City, California
| | - Albert Prats-Uribe
- Centre for Statistics in Medicine, NDORMS, University of Oxford, Oxford, United Kingdom
| | | | - Donna R Rivera
- Division of Cancer Control and Population Sciences, National Cancer Institute, Rockville, Maryland
| | - Lisa M Schilling
- Data Science to Patient Value Program, Department of Medicine, University of Colorado Anschutz Medical Campus, Aurora, Colorado
| | - Isabelle Soerjomataram
- Section of Cancer Surveillance, International Agency for Research on Cancer, Lyon, France
| | - Karishma Shah
- NDORMS, University of Oxford, Botnar Research Centre, Windmill Road, Oxford, United Kingdom
| | - Nigam H Shah
- Department of Medicine, School of Medicine, Stanford University, Redwood City, California
| | - Yang Shen
- Digital Health China Technologies Co., Ltd., Beijing, China
| | - Matthew Spotniz
- Department of Biomedical Informatics, Columbia University, New York, New York
| | - Vignesh Subbian
- College of Engineering, University of Arizona, Tucson, Arizona
| | - Marc A Suchard
- Fielding School of Public Health, University of California, Los Angeles, California
| | - Annalisa Trama
- Fondazione IRCSS Istituto Nazionale dei Tumori, Milan, Italy
| | - Lin Zhang
- School of Population Medicine and Public Health, Peking Union Medical College, Chinese Academy of Medical Sciences, Beijing, China
- School of Population Health and Global Health, University of Melbourne, Melbourne, Victoria, Australia
| | - Ying Zhang
- Digital Health China Technologies Co., Ltd., Beijing, China
| | - Patrick B Ryan
- Janssen Research and Development, Titusville, New Jersey
- Department of Biomedical Informatics, Columbia University, New York, New York
| | | | | | - Talita Duarte-Salles
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain.
| |
Collapse
|
38
|
Duarte-Salles T, Vizcaya D, Pistillo A, Casajust P, Sena AG, Lai LYH, Prats-Uribe A, Ahmed WUR, Alshammari TM, Alghoul H, Alser O, Burn E, You SC, Areia C, Blacketer C, DuVall S, Falconer T, Fernandez-Bertolin S, Fortin S, Golozar A, Gong M, Tan EH, Huser V, Iveli P, Morales DR, Nyberg F, Posada JD, Recalde M, Roel E, Schilling LM, Shah NH, Shah K, Suchard MA, Zhang L, Zhang Y, Williams AE, Reich CG, Hripcsak G, Rijnbeek P, Ryan P, Kostka K, Prieto-Alhambra D. Thirty-Day Outcomes of Children and Adolescents With COVID-19: An International Experience. Pediatrics 2021; 148:peds.2020-042929. [PMID: 34049958 DOI: 10.1542/peds.2020-042929] [Citation(s) in RCA: 29] [Impact Index Per Article: 9.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 04/13/2021] [Indexed: 11/24/2022] Open
Abstract
OBJECTIVES To characterize the demographics, comorbidities, symptoms, in-hospital treatments, and health outcomes among children and adolescents diagnosed or hospitalized with coronavirus disease 2019 (COVID-19) and to compare them in secondary analyses with patients diagnosed with previous seasonal influenza in 2017-2018. METHODS International network cohort using real-world data from European primary care records (France, Germany, and Spain), South Korean claims and US claims, and hospital databases. We included children and adolescents diagnosed and/or hospitalized with COVID-19 at age <18 between January and June 2020. We described baseline demographics, comorbidities, symptoms, 30-day in-hospital treatments, and outcomes including hospitalization, pneumonia, acute respiratory distress syndrome, multisystem inflammatory syndrome in children, and death. RESULTS A total of 242 158 children and adolescents diagnosed and 9769 hospitalized with COVID-19 and 2 084 180 diagnosed with influenza were studied. Comorbidities including neurodevelopmental disorders, heart disease, and cancer were more common among those hospitalized with versus diagnosed with COVID-19. Dyspnea, bronchiolitis, anosmia, and gastrointestinal symptoms were more common in COVID-19 than influenza. In-hospital prevalent treatments for COVID-19 included repurposed medications (<10%) and adjunctive therapies: systemic corticosteroids (6.8%-7.6%), famotidine (9.0%-28.1%), and antithrombotics such as aspirin (2.0%-21.4%), heparin (2.2%-18.1%), and enoxaparin (2.8%-14.8%). Hospitalization was observed in 0.3% to 1.3% of the cohort diagnosed with COVID-19, with undetectable (n < 5 per database) 30-day fatality. Thirty-day outcomes including pneumonia and hypoxemia were more frequent in COVID-19 than influenza. CONCLUSIONS Despite negligible fatality, complications including hospitalization, hypoxemia, and pneumonia were more frequent in children and adolescents with COVID-19 than with influenza. Dyspnea, anosmia, and gastrointestinal symptoms could help differentiate diagnoses. A wide range of medications was used for the inpatient management of pediatric COVID-19.
Collapse
Affiliation(s)
- Talita Duarte-Salles
- Fundació Institut Universitari per a la Recerca a l'Atenció Primària de Salut Jordi Gol i Gurina, Barcelona, Spain
| | | | - Andrea Pistillo
- Fundació Institut Universitari per a la Recerca a l'Atenció Primària de Salut Jordi Gol i Gurina, Barcelona, Spain
| | - Paula Casajust
- Real-World Evidence, Trial Form Support, Barcelona, Spain
| | - Anthony G Sena
- Janssen Research & Development, Titusville, New Jersey.,Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, Netherlands
| | - Lana Yin Hui Lai
- School of Medical Sciences, University of Manchester, Manchester, United Kingdom
| | - Albert Prats-Uribe
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences
| | - Waheed-Ul-Rahman Ahmed
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences.,College of Medicine and Health, St Luke's Campus, University of Exeter, Exeter, United Kingdom
| | | | - Heba Alghoul
- Faculty of Medicine, Islamic University of Gaza, Gaza, Palestine
| | - Osaid Alser
- Massachusetts General Hospital and Harvard Medical School, Harvard University, Boston, Massachusetts
| | - Edward Burn
- Fundació Institut Universitari per a la Recerca a l'Atenció Primària de Salut Jordi Gol i Gurina, Barcelona, Spain.,Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences
| | - Seng Chan You
- Department of Biomedical Informatics, School of Medicine, Ajou University, Suwon, South Korea
| | - Carlos Areia
- Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, United Kingdom
| | - Clair Blacketer
- Janssen Research & Development, Titusville, New Jersey.,Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, Netherlands
| | - Scott DuVall
- Department of Veterans Affairs, Salt Lake City, Utah.,School of Medicine, University of Utah, Salt Lake City, Utah
| | - Thomas Falconer
- Department of Biomedical Informatics, Columbia University, New York, New York
| | - Sergio Fernandez-Bertolin
- Fundació Institut Universitari per a la Recerca a l'Atenció Primària de Salut Jordi Gol i Gurina, Barcelona, Spain
| | | | - Asieh Golozar
- Regeneron Pharmaceuticals, Tarrytown, New York.,Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Johns Hopkins University, Baltimore, Maryland
| | | | - Eng Hooi Tan
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences
| | - Vojtech Huser
- Lister Hill National Center for Biomedical Communications, National Library of Medicine, National Institutes of Health, Bethesda, Maryland
| | | | - Daniel R Morales
- Division of Population Health and Genomics, School of Medicine, University of Dundee, Dundee, United Kingdom
| | - Fredrik Nyberg
- School of Public Health and Community Medicine, Institute of Medicine, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
| | - Jose D Posada
- Department of Medicine, School of Medicine, Stanford University, Stanford, California
| | - Martina Recalde
- Fundació Institut Universitari per a la Recerca a l'Atenció Primària de Salut Jordi Gol i Gurina, Barcelona, Spain.,Universitat Autònoma de Barcelona, Barcelona, Spain
| | - Elena Roel
- Fundació Institut Universitari per a la Recerca a l'Atenció Primària de Salut Jordi Gol i Gurina, Barcelona, Spain
| | - Lisa M Schilling
- Data Science to Patient Value Program, Department of Medicine, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, Colorado
| | - Nigam H Shah
- Department of Medicine, School of Medicine, Stanford University, Stanford, California
| | - Karishma Shah
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences
| | - Marc A Suchard
- Fundació Institut Universitari per a la Recerca a l'Atenció Primària de Salut Jordi Gol i Gurina, Barcelona, Spain.,Fundació Institut Universitari per a la Recerca a l'Atenció Primària de Salut Jordi Gol i Gurina, Barcelona, Spain
| | - Lin Zhang
- Bayer Pharmaceuticals, Sant Joan Despi, Spain.,Bayer Pharmaceuticals, Sant Joan Despi, Spain.,Real-World Evidence, Trial Form Support, Barcelona, Spain.,Real-World Evidence, Trial Form Support, Barcelona, Spain
| | - Ying Zhang
- DHC Technologies, Co, Ltd, Beijing, China
| | - Andrew E Williams
- Janssen Research & Development, Titusville, New Jersey.,Janssen Research & Development, Titusville, New Jersey
| | - Christian G Reich
- Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, Netherlands.,Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, Netherlands
| | - George Hripcsak
- Department of Biomedical Informatics, Columbia University, New York, New York
| | - Peter Rijnbeek
- Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, Netherlands
| | - Patrick Ryan
- Janssen Research & Development, Titusville, New Jersey.,Department of Biomedical Informatics, Columbia University, New York, New York
| | - Kristin Kostka
- Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, Netherlands.,Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, Netherlands
| | - Daniel Prieto-Alhambra
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences
| |
Collapse
|
39
|
Caswell-Jin JL, Callahan A, Purington N, Han SS, Itakura H, John EM, Blayney DW, Sledge GW, Shah NH, Kurian AW. Treatment and Monitoring Variability in US Metastatic Breast Cancer Care. JCO Clin Cancer Inform 2021; 5:600-614. [PMID: 34043432 DOI: 10.1200/cci.21.00031] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022] Open
Abstract
PURPOSE Treatment and monitoring options for patients with metastatic breast cancer (MBC) are increasing, but little is known about variability in care. We sought to improve understanding of MBC care and its correlates by analyzing real-world claims data using a search engine with a novel query language to enable temporal electronic phenotyping. METHODS Using the Advanced Cohort Engine, we identified 6,180 women who met criteria for having estrogen receptor-positive, human epidermal growth factor receptor 2-negative MBC from IBM MarketScan US insurance claims (2007-2014). We characterized treatment, monitoring, and hospice usage, along with clinical and nonclinical factors affecting care. RESULTS We observed wide variability in treatment modality and monitoring across patients and geography. Most women received first-recorded therapy with endocrine (67%) versus chemotherapy, underwent more computed tomography (CT) (76%) than positron emission tomography-CT, and were monitored using tumor markers (58%). Nearly half (46%) met criteria for aggressive disease, which were associated with receiving chemotherapy first, monitoring primarily with CT, and more frequent imaging. Older age was associated with endocrine therapy first, less frequent imaging, and less use of tumor markers. After controlling for clinical factors, care strategies varied significantly by nonclinical factors (median regional income with first-recorded therapy and imaging type, geographic region with these and with imaging frequency and use of tumor markers; P < .0001). CONCLUSION Variability in US MBC care is explained by patient and disease factors and by nonclinical factors such as geographic region, suggesting that treatment decisions are influenced by local practice patterns and/or resources. A search engine designed to express complex electronic phenotypes from longitudinal patient records enables the identification of variability in patient care, helping to define disparities and areas for improvement.
Collapse
Affiliation(s)
| | - Alison Callahan
- Department of Medicine, Stanford University School of Medicine, Stanford, CA
| | - Natasha Purington
- Department of Medicine, Stanford University School of Medicine, Stanford, CA.,Department of Neurosurgery, Stanford University School of Medicine, Stanford, CA
| | - Summer S Han
- Department of Medicine, Stanford University School of Medicine, Stanford, CA.,Department of Neurosurgery, Stanford University School of Medicine, Stanford, CA
| | - Haruka Itakura
- Department of Medicine, Stanford University School of Medicine, Stanford, CA
| | - Esther M John
- Department of Medicine, Stanford University School of Medicine, Stanford, CA.,Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA
| | - Douglas W Blayney
- Department of Medicine, Stanford University School of Medicine, Stanford, CA
| | - George W Sledge
- Department of Medicine, Stanford University School of Medicine, Stanford, CA
| | - Nigam H Shah
- Department of Medicine, Stanford University School of Medicine, Stanford, CA
| | - Allison W Kurian
- Department of Medicine, Stanford University School of Medicine, Stanford, CA.,Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA
| |
Collapse
|
40
|
Gensheimer MF, Aggarwal S, Benson KRK, Carter JN, Henry AS, Wood DJ, Soltys SG, Hancock S, Pollom E, Shah NH, Chang DT. Automated model versus treating physician for predicting survival time of patients with metastatic cancer. J Am Med Inform Assoc 2021; 28:1108-1116. [PMID: 33313792 DOI: 10.1093/jamia/ocaa290] [Citation(s) in RCA: 22] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/07/2020] [Accepted: 11/09/2020] [Indexed: 02/06/2023] Open
Abstract
OBJECTIVE Being able to predict a patient's life expectancy can help doctors and patients prioritize treatments and supportive care. For predicting life expectancy, physicians have been shown to outperform traditional models that use only a few predictor variables. It is possible that a machine learning model that uses many predictor variables and diverse data sources from the electronic medical record can improve on physicians' performance. For patients with metastatic cancer, we compared accuracy of life expectancy predictions by the treating physician, a machine learning model, and a traditional model. MATERIALS AND METHODS A machine learning model was trained using 14 600 metastatic cancer patients' data to predict each patient's distribution of survival time. Data sources included note text, laboratory values, and vital signs. From 2015-2016, 899 patients receiving radiotherapy for metastatic cancer were enrolled in a study in which their radiation oncologist estimated life expectancy. Survival predictions were also made by the machine learning model and a traditional model using only performance status. Performance was assessed with area under the curve for 1-year survival and calibration plots. RESULTS The radiotherapy study included 1190 treatment courses in 899 patients. A total of 879 treatment courses in 685 patients were included in this analysis. Median overall survival was 11.7 months. Physicians, machine learning model, and traditional model had area under the curve for 1-year survival of 0.72 (95% CI 0.63-0.81), 0.77 (0.73-0.81), and 0.68 (0.65-0.71), respectively. CONCLUSIONS The machine learning model's predictions were more accurate than those of the treating physician or a traditional model.
Collapse
Affiliation(s)
| | - Sonya Aggarwal
- Department of Radiation Oncology, Stanford University, Stanford, CA, USA
| | - Kathryn R K Benson
- Department of Radiation Oncology, Stanford University, Stanford, CA, USA
| | - Justin N Carter
- Department of Radiation Oncology, Stanford University, Stanford, CA, USA
| | - A Solomon Henry
- Department of Biomedical Data Science, Stanford University, Stanford, CA, USA
| | - Douglas J Wood
- Department of Biomedical Data Science, Stanford University, Stanford, CA, USA
| | - Scott G Soltys
- Department of Radiation Oncology, Stanford University, Stanford, CA, USA
| | - Steven Hancock
- Department of Radiation Oncology, Stanford University, Stanford, CA, USA
| | - Erqi Pollom
- Department of Radiation Oncology, Stanford University, Stanford, CA, USA
| | - Nigam H Shah
- Department of Biomedical Data Science, Stanford University, Stanford, CA, USA
| | - Daniel T Chang
- Department of Radiation Oncology, Stanford University, Stanford, CA, USA
| |
Collapse
|
41
|
Patel BS, Steinberg E, Pfohl SR, Shah NH. Learning decision thresholds for risk stratification models from aggregate clinician behavior. J Am Med Inform Assoc 2021; 28:2258-2264. [PMID: 34350942 PMCID: PMC8449610 DOI: 10.1093/jamia/ocab159] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/03/2021] [Revised: 06/26/2021] [Accepted: 07/13/2021] [Indexed: 11/22/2022] Open
Abstract
Using a risk stratification model to guide clinical practice often requires the choice of a cutoff—called the decision threshold—on the model’s output to trigger a subsequent action such as an electronic alert. Choosing this cutoff is not always straightforward. We propose a flexible approach that leverages the collective information in treatment decisions made in real life to learn reference decision thresholds from physician practice. Using the example of prescribing a statin for primary prevention of cardiovascular disease based on 10-year risk calculated by the 2013 pooled cohort equations, we demonstrate the feasibility of using real-world data to learn the implicit decision threshold that reflects existing physician behavior. Learning a decision threshold in this manner allows for evaluation of a proposed operating point against the threshold reflective of the community standard of care. Furthermore, this approach can be used to monitor and audit model-guided clinical decision making following model deployment.
Collapse
Affiliation(s)
- Birju S Patel
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA
| | - Ethan Steinberg
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA
| | - Stephen R Pfohl
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA
| | - Nigam H Shah
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA
- Corresponding Author: Nigam H. Shah, MBBS, PhD, Stanford Center for Biomedical Informatics Research, Stanford University, 1265 Welch Road, Stanford, CA 94305, USA;
| |
Collapse
|
42
|
Recalde M, Roel E, Pistillo A, Sena AG, Prats-Uribe A, Ahmed WUR, Alghoul H, Alshammari TM, Alser O, Areia C, Burn E, Casajust P, Dawoud D, DuVall SL, Falconer T, Fernández-Bertolín S, Golozar A, Gong M, Lai LYH, Lane JCE, Lynch KE, Matheny ME, Mehta PP, Morales DR, Natarjan K, Nyberg F, Posada JD, Reich CG, Rijnbeek PR, Schilling LM, Shah K, Shah NH, Subbian V, Zhang L, Zhu H, Ryan P, Prieto-Alhambra D, Kostka K, Duarte-Salles T. Characteristics and outcomes of 627 044 COVID-19 patients living with and without obesity in the United States, Spain, and the United Kingdom. Int J Obes (Lond) 2021; 45:2347-2357. [PMID: 34267326 PMCID: PMC8281807 DOI: 10.1038/s41366-021-00893-4] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/23/2020] [Revised: 06/07/2021] [Accepted: 06/24/2021] [Indexed: 12/18/2022]
Abstract
BACKGROUND A detailed characterization of patients with COVID-19 living with obesity has not yet been undertaken. We aimed to describe and compare the demographics, medical conditions, and outcomes of COVID-19 patients living with obesity (PLWO) to those of patients living without obesity. METHODS We conducted a cohort study based on outpatient/inpatient care and claims data from January to June 2020 from Spain, the UK, and the US. We used six databases standardized to the OMOP common data model. We defined two non-mutually exclusive cohorts of patients diagnosed and/or hospitalized with COVID-19; patients were followed from index date to 30 days or death. We report the frequency of demographics, prior medical conditions, and 30-days outcomes (hospitalization, events, and death) by obesity status. RESULTS We included 627 044 (Spain: 122 058, UK: 2336, and US: 502 650) diagnosed and 160 013 (Spain: 18 197, US: 141 816) hospitalized patients with COVID-19. The prevalence of obesity was higher among patients hospitalized (39.9%, 95%CI: 39.8-40.0) than among those diagnosed with COVID-19 (33.1%; 95%CI: 33.0-33.2). In both cohorts, PLWO were more often female. Hospitalized PLWO were younger than patients without obesity. Overall, COVID-19 PLWO were more likely to have prior medical conditions, present with cardiovascular and respiratory events during hospitalization, or require intensive services compared to COVID-19 patients without obesity. CONCLUSION We show that PLWO differ from patients without obesity in a wide range of medical conditions and present with more severe forms of COVID-19, with higher hospitalization rates and intensive services requirements. These findings can help guiding preventive strategies of COVID-19 infection and complications and generating hypotheses for causal inference studies.
Collapse
Affiliation(s)
- Martina Recalde
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain.,Universitat Autònoma de Barcelona, Bellaterra, Spain
| | - Elena Roel
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain.,Universitat Autònoma de Barcelona, Bellaterra, Spain
| | - Andrea Pistillo
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
| | - Anthony G Sena
- Janssen Research & Development, Titusville, NJ, USA.,Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, The Netherlands
| | - Albert Prats-Uribe
- Centre for Statistics in Medicine, NDORMS, University of Oxford, Oxford, UK
| | - Waheed-Ul-Rahman Ahmed
- Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Botnar Research Centre, Oxford, UK.,College of Medicine and Health, University of Exeter, St Luke's Campus, Exeter, UK
| | - Heba Alghoul
- Faculty of Medicine, Islamic University of Gaza, Gaza, Palestine
| | | | - Osaid Alser
- Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Carlos Areia
- Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
| | - Edward Burn
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain.,Centre for Statistics in Medicine, NDORMS, University of Oxford, Oxford, UK
| | - Paula Casajust
- Real-World Evidence, Trial Form Support, Barcelona, Spain
| | - Dalia Dawoud
- Cairo University, Faculty of Pharmacy, Cairo, Egypt
| | - Scott L DuVall
- VA Informatics and Computing Infrastructure, VA Salt Lake City Health Care System, Salt Lake City, UT, USA.,Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, USA
| | - Thomas Falconer
- Department of Biomedical Informatics, Columbia University, New York, NY, USA
| | - Sergio Fernández-Bertolín
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
| | - Asieh Golozar
- Department of Epidemiology, Johns Hopkins School of Public, Baltimore, MD, USA.,Pharmacoepidemiology, Regeneron Pharmaceuticals, Tarrytown, NY, USA
| | | | - Lana Yin Hui Lai
- Division of Cancer Sciences, School of Medical Sciences, University of Manchester, Manchester, UK
| | - Jennifer C E Lane
- Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Botnar Research Centre, Oxford, UK
| | - Kristine E Lynch
- VA Informatics and Computing Infrastructure, VA Salt Lake City Health Care System, Salt Lake City, UT, USA.,Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, USA
| | - Michael E Matheny
- Tennessee Valley Healthcare System, Veterans Affairs Medical Center, Nashville, TN, USA.,Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Paras P Mehta
- College of Medicine, The University of Arizona, Tucson, AZ, USA
| | - Daniel R Morales
- Division of Population Health and Genomics, University of Dundee, Dundee, UK
| | - Karthik Natarjan
- Department of Biomedical Informatics, Columbia University, New York, NY, USA.,New York-Presbyterian Hospital, New York, NY, USA
| | - Fredrik Nyberg
- School of Public Health and Community Medicine, Institute of Medicine, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
| | - Jose D Posada
- Department of Medicine, Stanford University, Palo Alto, CA, USA
| | | | - Peter R Rijnbeek
- Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, The Netherlands
| | - Lisa M Schilling
- Data Science to Patient Value Program, Department of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO, USA
| | - Karishma Shah
- Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Botnar Research Centre, Oxford, UK
| | - Nigam H Shah
- Department of Medicine, Stanford University, Palo Alto, CA, USA
| | - Vignesh Subbian
- College of Engineering, The University of Arizona, Tucson, AZ, USA
| | - Lin Zhang
- School of Population Medicine and Public Health, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing, China.,Melbourne School of Population and Global Health, The University of Melbourne, Melbourne, VIC, Australia
| | - Hong Zhu
- Institute of Health Management, Southern Medical University, Guangzhou, China.,Nanfang Hospital, Southern Medical University, Guangzhou, China
| | - Patrick Ryan
- Janssen Research & Development, Titusville, NJ, USA.,Department of Biomedical Informatics, Columbia University, New York, NY, USA
| | - Daniel Prieto-Alhambra
- Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, The Netherlands.,Centre for Statistics in Medicine, NDORMS, University of Oxford, Oxford, UK
| | - Kristin Kostka
- Real World Solutions, IQVIA, Cambridge, MA, USA.,The OHDSI Center at the Roux Institute, Northeastern University, Portland, ME, USA
| | - Talita Duarte-Salles
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain.
| |
Collapse
|
43
|
Callahan A, Polony V, Posada JD, Banda JM, Gombar S, Shah NH. ACE: the Advanced Cohort Engine for searching longitudinal patient records. J Am Med Inform Assoc 2021; 28:1468-1479. [PMID: 33712854 PMCID: PMC8279796 DOI: 10.1093/jamia/ocab027] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/08/2020] [Accepted: 02/23/2021] [Indexed: 01/02/2023] Open
Abstract
OBJECTIVE To propose a paradigm for a scalable time-aware clinical data search, and to describe the design, implementation and use of a search engine realizing this paradigm. MATERIALS AND METHODS The Advanced Cohort Engine (ACE) uses a temporal query language and in-memory datastore of patient objects to provide a fast, scalable, and expressive time-aware search. ACE accepts data in the Observational Medicine Outcomes Partnership Common Data Model, and is configurable to balance performance with compute cost. ACE's temporal query language supports automatic query expansion using clinical knowledge graphs. The ACE API can be used with R, Python, Java, HTTP, and a Web UI. RESULTS ACE offers an expressive query language for complex temporal search across many clinical data types with multiple output options. ACE enables electronic phenotyping and cohort-building with subsecond response times in searching the data of millions of patients for a variety of use cases. DISCUSSION ACE enables fast, time-aware search using a patient object-centric datastore, thereby overcoming many technical and design shortcomings of relational algebra-based querying. Integrating electronic phenotype development with cohort-building enables a variety of high-value uses for a learning health system. Tradeoffs include the need to learn a new query language and the technical setup burden. CONCLUSION ACE is a tool that combines a unique query language for time-aware search of longitudinal patient records with a patient object datastore for rapid electronic phenotyping, cohort extraction, and exploratory data analyses.
Collapse
Affiliation(s)
- Alison Callahan
- Center for Biomedical Informatics Research, School of Medicine, School of Medicine, Stanford University, Stanford, California, USA
| | - Vladimir Polony
- Center for Biomedical Informatics Research, School of Medicine, School of Medicine, Stanford University, Stanford, California, USA
| | - José D Posada
- Center for Biomedical Informatics Research, School of Medicine, School of Medicine, Stanford University, Stanford, California, USA
| | - Juan M Banda
- Department of Computer Science, Georgia State University, Atlanta, Georgia, USA
| | - Saurabh Gombar
- Department of Pathology, School of Medicine, Stanford University, Stanford, California, USA
| | - Nigam H Shah
- Center for Biomedical Informatics Research, School of Medicine, School of Medicine, Stanford University, Stanford, California, USA
| |
Collapse
|
44
|
Ko M, Chen E, Agrawal A, Rajpurkar P, Avati A, Ng A, Basu S, Shah NH. Improving hospital readmission prediction using individualized utility analysis. J Biomed Inform 2021; 119:103826. [PMID: 34087428 DOI: 10.1016/j.jbi.2021.103826] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2021] [Revised: 05/23/2021] [Accepted: 05/28/2021] [Indexed: 11/24/2022]
Abstract
OBJECTIVE Machine learning (ML) models for allocating readmission-mitigating interventions are typically selected according to their discriminative ability, which may not necessarily translate into utility in allocation of resources. Our objective was to determine whether ML models for allocating readmission-mitigating interventions have different usefulness based on their overall utility and discriminative ability. MATERIALS AND METHODS We conducted a retrospective utility analysis of ML models using claims data acquired from the Optum Clinformatics Data Mart, including 513,495 commercially-insured inpatients (mean [SD] age 69 [19] years; 294,895 [57%] Female) over the period January 2016 through January 2017 from all 50 states with mean 90 day cost of $11,552. Utility analysis estimates the cost, in dollars, of allocating interventions for lowering readmission risk based on the reduction in the 90-day cost. RESULTS Allocating readmission-mitigating interventions based on a GBDT model trained to predict readmissions achieved an estimated utility gain of $104 per patient, and an AUC of 0.76 (95% CI 0.76, 0.77); allocating interventions based on a model trained to predict cost as a proxy achieved a higher utility of $175.94 per patient, and an AUC of 0.62 (95% CI 0.61, 0.62). A hybrid model combining both intervention strategies is comparable with the best models on either metric. Estimated utility varies by intervention cost and efficacy, with each model performing the best under different intervention settings. CONCLUSION We demonstrate that machine learning models may be ranked differently based on overall utility and discriminative ability. Machine learning models for allocation of limited health resources should consider directly optimizing for utility.
Collapse
Affiliation(s)
- Michael Ko
- Department of Computer Science, Stanford University, CA, USA
| | - Emma Chen
- Department of Computer Science, Stanford University, CA, USA
| | - Ashwin Agrawal
- Department of Computer Science, Stanford University, CA, USA
| | | | - Anand Avati
- Department of Computer Science, Stanford University, CA, USA
| | - Andrew Ng
- Department of Computer Science, Stanford University, CA, USA
| | - Sanjay Basu
- Center for Primary Care, Harvard Medical School, MA, USA
| | - Nigam H Shah
- Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA
| |
Collapse
|
45
|
Prats-Uribe A, Sena AG, Lai LYH, Ahmed WUR, Alghoul H, Alser O, Alshammari TM, Areia C, Carter W, Casajust P, Dawoud D, Golozar A, Jonnagaddala J, Mehta PP, Gong M, Morales DR, Nyberg F, Posada JD, Recalde M, Roel E, Shah K, Shah NH, Schilling LM, Subbian V, Vizcaya D, Zhang L, Zhang Y, Zhu H, Liu L, Cho J, Lynch KE, Matheny ME, You SC, Rijnbeek PR, Hripcsak G, Lane JC, Burn E, Reich C, Suchard MA, Duarte-Salles T, Kostka K, Ryan PB, Prieto-Alhambra D. Use of repurposed and adjuvant drugs in hospital patients with covid-19: multinational network cohort study. BMJ 2021; 373:n1038. [PMID: 33975825 PMCID: PMC8111167 DOI: 10.1136/bmj.n1038] [Citation(s) in RCA: 36] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 04/16/2021] [Indexed: 12/12/2022]
Abstract
OBJECTIVE To investigate the use of repurposed and adjuvant drugs in patients admitted to hospital with covid-19 across three continents. DESIGN Multinational network cohort study. SETTING Hospital electronic health records from the United States, Spain, and China, and nationwide claims data from South Korea. PARTICIPANTS 303 264 patients admitted to hospital with covid-19 from January 2020 to December 2020. MAIN OUTCOME MEASURES Prescriptions or dispensations of any drug on or 30 days after the date of hospital admission for covid-19. RESULTS Of the 303 264 patients included, 290 131 were from the US, 7599 from South Korea, 5230 from Spain, and 304 from China. 3455 drugs were identified. Common repurposed drugs were hydroxychloroquine (used in from <5 (<2%) patients in China to 2165 (85.1%) in Spain), azithromycin (from 15 (4.9%) in China to 1473 (57.9%) in Spain), combined lopinavir and ritonavir (from 156 (<2%) in the VA-OMOP US to 2,652 (34.9%) in South Korea and 1285 (50.5%) in Spain), and umifenovir (0% in the US, South Korea, and Spain and 238 (78.3%) in China). Use of adjunctive drugs varied greatly, with the five most used treatments being enoxaparin, fluoroquinolones, ceftriaxone, vitamin D, and corticosteroids. Hydroxychloroquine use increased rapidly from March to April 2020 but declined steeply in May to June and remained low for the rest of the year. The use of dexamethasone and corticosteroids increased steadily during 2020. CONCLUSIONS Multiple drugs were used in the first few months of the covid-19 pandemic, with substantial geographical and temporal variation. Hydroxychloroquine, azithromycin, lopinavir-ritonavir, and umifenovir (in China only) were the most prescribed repurposed drugs. Antithrombotics, antibiotics, H2 receptor antagonists, and corticosteroids were often used as adjunctive treatments. Research is needed on the comparative risk and benefit of these treatments in the management of covid-19.
Collapse
Affiliation(s)
- Albert Prats-Uribe
- Pharmaco- and Device Epidemiology, Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - Anthony G Sena
- Janssen Research and Development, Titusville, NJ, USA
- Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, Netherlands
| | - Lana Yin Hui Lai
- Division of Cancer Sciences, School of Medical Sciences, University of Manchester, Manchester, UK
| | - Waheed-Ul-Rahman Ahmed
- Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK
- College of Medicine and Health, University of Exeter, Exeter, UK
| | - Heba Alghoul
- Faculty of Medicine, Islamic University of Gaza, Gaza City, Palestine
| | - Osaid Alser
- Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | | | - Carlos Areia
- Critical Care Research Group, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
| | - William Carter
- University of Colorado Anschutz Medical Campus, Aurora, CO, USA
| | - Paula Casajust
- Real-World Evidence, Trial Form Support, Barcelona, Spain
| | - Dalia Dawoud
- Faculty of Pharmacy, Cairo University, Cairo, Egypt
- National Institute for Health and Care Excellence, London, UK
| | - Asieh Golozar
- Regeneron Pharmaceuticals, Tarrytown, NY, US
- Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA
| | | | - Paras P Mehta
- College of Medicine, University of Arizona, Tucson, AZ, USA
| | | | - Daniel R Morales
- Division of Population Health and Genomics, University of Dundee, Dundee, UK
- Department of Public Health, University of Southern Denmark, Odense, Denmark
| | - Fredrik Nyberg
- School of Public Health and Community Medicine, Institute of Medicine, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden
| | - Jose D Posada
- Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA
| | - Martina Recalde
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
- Universitat Autònoma de Barcelona, Barcelona, Spain
| | - Elena Roel
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
- Universitat Autònoma de Barcelona, Barcelona, Spain
| | - Karishma Shah
- Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - Nigam H Shah
- Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA
| | | | - Vignesh Subbian
- College of Engineering, University of Arizona Tucson, AZ, USA
| | | | - Lin Zhang
- School of Population Medicine and Public Health, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
- School of Population and Global Health, University of Melbourne, Melbourne, VIC, Australia
| | | | - Hong Zhu
- Nanfang University, Southern Medical University, Guangzhou, China
| | - Li Liu
- Nanfang University, Southern Medical University, Guangzhou, China
| | - Jaehyeong Cho
- Department of Biomedical Informatics, Ajou University School of Medicine, Suwon, South Korea
| | - Kristine E Lynch
- VA Informatics and Computing Infrastructure, VA Salt Lake City Healthcare System, Salt Lake City, Utah, USA; Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, UT, USA
| | - Michael E Matheny
- VA Informatics and Computing Infrastructure, Tennessee Valley Healthcare System, VA Medical Center, Nashville, TN, USA
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Seng Chan You
- Department of Preventive Medicine and Public Health, Yonsei University College of Medicine, Seoul, South Korea
| | - Peter R Rijnbeek
- Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, Netherlands
| | - George Hripcsak
- Department of Biomedical Informatics, Columbia University, New York, NY, USA
| | - Jennifer Ce Lane
- Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| | - Edward Burn
- Pharmaco- and Device Epidemiology, Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
| | | | - Marc A Suchard
- Department of Biostatistics, UCLA Fielding School of Public Health, University of California, Los Angeles, Los Angeles, CA, USA
| | - Talita Duarte-Salles
- Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), Barcelona, Spain
| | - Kristin Kostka
- IQVIA, Cambridge, MA, USA
- OHDSI Center at The Roux Institute, Northeastern University, Portland, ME, USA
| | - Patrick B Ryan
- Janssen Research and Development, Titusville, NJ, USA
- Columbia University Irving Medical Center, New York, NY, USA
| | - Daniel Prieto-Alhambra
- Pharmaco- and Device Epidemiology, Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology, and Musculoskeletal Sciences, University of Oxford, Oxford, UK
| |
Collapse
|
46
|
Hernandez-Boussard T, Bozkurt S, Ioannidis JPA, Shah NH. MINIMAR (MINimum Information for Medical AI Reporting): Developing reporting standards for artificial intelligence in health care. J Am Med Inform Assoc 2021; 27:2011-2015. [PMID: 32594179 PMCID: PMC7727333 DOI: 10.1093/jamia/ocaa088] [Citation(s) in RCA: 123] [Impact Index Per Article: 41.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/16/2020] [Revised: 04/24/2020] [Accepted: 04/29/2020] [Indexed: 12/23/2022] Open
Abstract
The rise of digital data and computing power have contributed to significant advancements in artificial intelligence (AI), leading to the use of classification and prediction models in health care to enhance clinical decision-making for diagnosis, treatment and prognosis. However, such advances are limited by the lack of reporting standards for the data used to develop those models, the model architecture, and the model evaluation and validation processes. Here, we present MINIMAR (MINimum Information for Medical AI Reporting), a proposal describing the minimum information necessary to understand intended predictions, target populations, and hidden biases, and the ability to generalize these emerging technologies. We call for a standard to accurately and responsibly report on AI in health care. This will facilitate the design and implementation of these models and promote the development and use of associated clinical decision support tools, as well as manage concerns regarding accuracy and bias.
Collapse
Affiliation(s)
- Tina Hernandez-Boussard
- Department of Medicine, Stanford University, Stanford, California, USA.,Department of Biomedical Data Science, Stanford University, Stanford, California, USA.,Department of Surgery, Stanford University, Stanford, California, USA
| | - Selen Bozkurt
- Department of Medicine, Stanford University, Stanford, California, USA
| | - John P A Ioannidis
- Department of Medicine, Stanford University, Stanford, California, USA.,Department of Statistics, Stanford University, Stanford, California, USA.,Meta-Research Innovation Center at Stanford, Stanford University, Stanford, California, USA
| | - Nigam H Shah
- Department of Medicine, Stanford University, Stanford, California, USA.,Department of Biomedical Data Science, Stanford University, Stanford, California, USA
| |
Collapse
|
47
|
Fries JA, Steinberg E, Khattar S, Fleming SL, Posada J, Callahan A, Shah NH. Ontology-driven weak supervision for clinical entity classification in electronic health records. Nat Commun 2021; 12:2017. [PMID: 33795682 PMCID: PMC8016863 DOI: 10.1038/s41467-021-22328-4] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/07/2020] [Accepted: 02/26/2021] [Indexed: 02/07/2023] Open
Abstract
In the electronic health record, using clinical notes to identify entities such as disorders and their temporality (e.g. the order of an event relative to a time index) can inform many important analyses. However, creating training data for clinical entity tasks is time consuming and sharing labeled data is challenging due to privacy concerns. The information needs of the COVID-19 pandemic highlight the need for agile methods of training machine learning models for clinical notes. We present Trove, a framework for weakly supervised entity classification using medical ontologies and expert-generated rules. Our approach, unlike hand-labeled notes, is easy to share and modify, while offering performance comparable to learning from manually labeled training data. In this work, we validate our framework on six benchmark tasks and demonstrate Trove's ability to analyze the records of patients visiting the emergency department at Stanford Health Care for COVID-19 presenting symptoms and risk factors.
Collapse
Affiliation(s)
- Jason A Fries
- Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA.
| | - Ethan Steinberg
- Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA
- Department of Computer Science, Stanford University, Stanford, CA, USA
| | - Saelig Khattar
- Department of Computer Science, Stanford University, Stanford, CA, USA
| | - Scott L Fleming
- Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA
| | - Jose Posada
- Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA
| | - Alison Callahan
- Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA
| | - Nigam H Shah
- Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA
| |
Collapse
|
48
|
Kashyap M, Seneviratne M, Banda JM, Falconer T, Ryu B, Yoo S, Hripcsak G, Shah NH. Development and validation of phenotype classifiers across multiple sites in the observational health data sciences and informatics network. J Am Med Inform Assoc 2021; 27:877-883. [PMID: 32374408 PMCID: PMC7309227 DOI: 10.1093/jamia/ocaa032] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2019] [Revised: 12/17/2019] [Accepted: 03/12/2020] [Indexed: 11/16/2022] Open
Abstract
Objective Accurate electronic phenotyping is essential to support collaborative observational research. Supervised machine learning methods can be used to train phenotype classifiers in a high-throughput manner using imperfectly labeled data. We developed 10 phenotype classifiers using this approach and evaluated performance across multiple sites within the Observational Health Data Sciences and Informatics (OHDSI) network. Materials and Methods We constructed classifiers using the Automated PHenotype Routine for Observational Definition, Identification, Training and Evaluation (APHRODITE) R-package, an open-source framework for learning phenotype classifiers using datasets in the Observational Medical Outcomes Partnership Common Data Model. We labeled training data based on the presence of multiple mentions of disease-specific codes. Performance was evaluated on cohorts derived using rule-based definitions and real-world disease prevalence. Classifiers were developed and evaluated across 3 medical centers, including 1 international site. Results Compared to the multiple mentions labeling heuristic, classifiers showed a mean recall boost of 0.43 with a mean precision loss of 0.17. Performance decreased slightly when classifiers were shared across medical centers, with mean recall and precision decreasing by 0.08 and 0.01, respectively, at a site within the USA, and by 0.18 and 0.10, respectively, at an international site. Discussion and Conclusion We demonstrate a high-throughput pipeline for constructing and sharing phenotype classifiers across sites within the OHDSI network using APHRODITE. Classifiers exhibit good portability between sites within the USA, however limited portability internationally, indicating that classifier generalizability may have geographic limitations, and, consequently, sharing the classifier-building recipe, rather than the pretrained classifiers, may be more useful for facilitating collaborative observational research.
Collapse
Affiliation(s)
- Mehr Kashyap
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA
| | - Martin Seneviratne
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA
| | - Juan M Banda
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA.,Department of Computer Science, Georgia State University, Atlanta, Georgia, USA
| | - Thomas Falconer
- Department of Biomedical Informatics, Columbia University, New York, New York, USA
| | - Borim Ryu
- Office of eHealth and Business, Seoul National University Bundang Hospital, Gyeonggi-do, South Korea
| | - Sooyoung Yoo
- Office of eHealth and Business, Seoul National University Bundang Hospital, Gyeonggi-do, South Korea
| | - George Hripcsak
- Department of Biomedical Informatics, Columbia University, New York, New York, USA
| | - Nigam H Shah
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA
| |
Collapse
|
49
|
Giori NJ, Radin J, Callahan A, Fries JA, Halilaj E, Ré C, Delp SL, Shah NH, Harris AHS. Assessment of Extractability and Accuracy of Electronic Health Record Data for Joint Implant Registries. JAMA Netw Open 2021; 4:e211728. [PMID: 33720372 PMCID: PMC7961313 DOI: 10.1001/jamanetworkopen.2021.1728] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/15/2022] Open
Abstract
IMPORTANCE Implant registries provide valuable information on the performance of implants in a real-world setting, yet they have traditionally been expensive to establish and maintain. Electronic health records (EHRs) are widely used and may include the information needed to generate clinically meaningful reports similar to a formal implant registry. OBJECTIVES To quantify the extractability and accuracy of registry-relevant data from the EHR and to assess the ability of these data to track trends in implant use and the durability of implants (hereafter referred to as implant survivorship), using data stored since 2000 in the EHR of the largest integrated health care system in the United States. DESIGN, SETTING, AND PARTICIPANTS Retrospective cohort study of a large EHR of veterans who had 45 351 total hip arthroplasty procedures in Veterans Health Administration hospitals from 2000 to 2017. Data analysis was performed from January 1, 2000, to December 31, 2017. EXPOSURES Total hip arthroplasty. MAIN OUTCOMES AND MEASURES Number of total hip arthroplasty procedures extracted from the EHR, trends in implant use, and relative survivorship of implants. RESULTS A total of 45 351 total hip arthroplasty procedures were identified from 2000 to 2017 with 192 805 implant parts. Data completeness improved over the time. After 2014, 85% of prosthetic heads, 91% of shells, 81% of stems, and 85% of liners used in the Veterans Health Administration health care system were identified by part number. Revision burden and trends in metal vs ceramic prosthetic femoral head use were found to reflect data from the American Joint Replacement Registry. Recalled implants were obvious negative outliers in implant survivorship using Kaplan-Meier curves. CONCLUSIONS AND RELEVANCE Although loss to follow-up remains a challenge that requires additional attention to improve the quantitative nature of calculated implant survivorship, we conclude that data collected during routine clinical care and stored in the EHR of a large health system over 18 years were sufficient to provide clinically meaningful data on trends in implant use and to identify poor implants that were subsequently recalled. This automated approach was low cost and had no reporting burden. This low-cost, low-overhead method to assess implant use and performance within a large health care setting may be useful to internal quality assurance programs and, on a larger scale, to postmarket surveillance of implant performance.
Collapse
Affiliation(s)
- Nicholas J. Giori
- Center for Innovation to Implementation, VA Palo Alto Health Care System, Palo Alto, California
- Department of Orthopedic Surgery, Stanford University, Stanford, California
| | - John Radin
- Center for Innovation to Implementation, VA Palo Alto Health Care System, Palo Alto, California
| | - Alison Callahan
- Center for Biomedical Informatics Research, Stanford University, Stanford, California
| | - Jason A. Fries
- Center for Biomedical Informatics Research, Stanford University, Stanford, California
- Department of Computer Science, Stanford University, Stanford, California
| | - Eni Halilaj
- Department of Bioengineering, Stanford University, Stanford, California
| | - Christopher Ré
- Department of Computer Science, Stanford University, Stanford, California
| | - Scott L. Delp
- Department of Bioengineering, Stanford University, Stanford, California
| | - Nigam H. Shah
- Center for Biomedical Informatics Research, Stanford University, Stanford, California
| | - Alex H. S. Harris
- Center for Innovation to Implementation, VA Palo Alto Health Care System, Palo Alto, California
- Department of Surgery, Stanford University, Stanford, California
| |
Collapse
|
50
|
Liu VX, Bates DW, Wiens J, Shah NH. The number needed to benefit: estimating the value of predictive analytics in healthcare. J Am Med Inform Assoc 2021; 26:1655-1659. [PMID: 31192367 DOI: 10.1093/jamia/ocz088] [Citation(s) in RCA: 27] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2019] [Revised: 05/10/2019] [Accepted: 05/17/2019] [Indexed: 12/21/2022] Open
Abstract
Predictive analytics in health care has generated increasing enthusiasm recently, as reflected in a rapidly growing body of predictive models reported in literature and in real-time embedded models using electronic health record data. However, estimating the benefit of applying any single model to a specific clinical problem remains challenging today. Developing a shared framework for estimating model value is therefore critical to facilitate the effective, safe, and sustainable use of predictive tools into the future. We highlight key concepts within the prediction-action dyad that together are expected to impact model benefit. These include factors relevant to model prediction (including the number needed to screen) as well as those relevant to the subsequent action (number needed to treat). In the simplest terms, a number needed to benefit contextualizes the numbers needed to screen and treat, offering an opportunity to estimate the value of a clinical predictive model in action.
Collapse
Affiliation(s)
- Vincent X Liu
- Division of Research, Kaiser Permanente, Oakland, California, USA
| | - David W Bates
- Division of General Internal Medicine, Brigham and Women's Hospital, and Harvard Medical School, Boston, Massachusetts, USA
| | - Jenna Wiens
- Division of Computer Science and Engineering, College of Engineering, University of Michigan, Ann Arbor, Michigan, USA
| | - Nigam H Shah
- Division of Biomedical Informatics Research, Stanford University, Stanford, California, USA
| |
Collapse
|