1
|
Kiraly AP, Cunningham CA, Najafi R, Nabulsi Z, Yang J, Lau C, Ledsam JR, Ye W, Ardila D, McKinney SM, Pilgrim R, Liu Y, Saito H, Shimamura Y, Etemadi M, Melnick D, Jansen S, Corrado GS, Peng L, Tse D, Shetty S, Prabhakara S, Nadich DP, Beladia N, Eswaran K. Assistive AI in Lung Cancer Screening: A Retrospective Multinational Study in the United States and Japan. Radiol Artif Intell 2024; 6:e230079. [PMID: 38477661 DOI: 10.1148/ryai.230079] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/14/2024]
Abstract
Purpose To evaluate the impact of an artificial intelligence (AI) assistant for lung cancer screening on multinational clinical workflows. Materials and Methods An AI assistant for lung cancer screening was evaluated on two retrospective randomized multireader multicase studies where 627 (141 cancer-positive cases) low-dose chest CT cases were each read twice (with and without AI assistance) by experienced thoracic radiologists (six U.S.-based or six Japan-based radiologists), resulting in a total of 7524 interpretations. Positive cases were defined as those within 2 years before a pathology-confirmed lung cancer diagnosis. Negative cases were defined as those without any subsequent cancer diagnosis for at least 2 years and were enriched for a spectrum of diverse nodules. The studies measured the readers' level of suspicion (on a 0-100 scale), country-specific screening system scoring categories, and management recommendations. Evaluation metrics included the area under the receiver operating characteristic curve (AUC) for level of suspicion and sensitivity and specificity of recall recommendations. Results With AI assistance, the radiologists' AUC increased by 0.023 (0.70 to 0.72; P = .02) for the U.S. study and by 0.023 (0.93 to 0.96; P = .18) for the Japan study. Scoring system specificity for actionable findings increased 5.5% (57% to 63%; P < .001) for the U.S. study and 6.7% (23% to 30%; P < .001) for the Japan study. There was no evidence of a difference in corresponding sensitivity between unassisted and AI-assisted reads for the U.S. (67.3% to 67.5%; P = .88) and Japan (98% to 100%; P > .99) studies. Corresponding stand-alone AI AUC system performance was 0.75 (95% CI: 0.70, 0.81) and 0.88 (95% CI: 0.78, 0.97) for the U.S.- and Japan-based datasets, respectively. Conclusion The concurrent AI interface improved lung cancer screening specificity in both U.S.- and Japan-based reader studies, meriting further study in additional international screening environments. Keywords: Assistive Artificial Intelligence, Lung Cancer Screening, CT Supplemental material is available for this article. Published under a CC BY 4.0 license.
Collapse
Affiliation(s)
- Atilla P Kiraly
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Corbin A Cunningham
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Ryan Najafi
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Zaid Nabulsi
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Jie Yang
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Charles Lau
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Joseph R Ledsam
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Wenxing Ye
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Diego Ardila
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Scott M McKinney
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Rory Pilgrim
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Yun Liu
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Hiroaki Saito
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Yasuteru Shimamura
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Mozziyar Etemadi
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - David Melnick
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Sunny Jansen
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Greg S Corrado
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Lily Peng
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Daniel Tse
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Shravya Shetty
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Shruthi Prabhakara
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - David P Nadich
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Neeral Beladia
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| | - Krish Eswaran
- From Google Health Research, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (A.P.K., C.A.C., R.N., Z.N., C.L., J.R.L., D.A., S.M.M., R.P., Y.L., S.J., G.S.C., L.P., D.T., S.S., S.P., K.E.); Waymo, Mountain View, Calif (J.Y., N.B.), David Geffen School of Medicine at UCLA, Los Angeles, Calif (C.L.); Google, Mountain View, Calif (W.Y.); Department of Gastroenterology, Sendai Kousei Hospital, Sendai, Japan (H.S.); MNES Inc, Hiroshima, Japan (Y.S.); Department of Telemedicine, Northwestern University Feinberg School of Medicine, Chicago, Ill (M.E., D.M.); and Center for Biological Imaging, New York University-Langone Medical Center, New York, NY (D.P.N.)
| |
Collapse
|
2
|
Weng WH, Sellergen A, Kiraly AP, D'Amour A, Park J, Pilgrim R, Pfohl S, Lau C, Natarajan V, Azizi S, Karthikesalingam A, Cole-Lewis H, Matias Y, Corrado GS, Webster DR, Shetty S, Prabhakara S, Eswaran K, Celi LAG, Liu Y. An intentional approach to managing bias in general purpose embedding models. Lancet Digit Health 2024; 6:e126-e130. [PMID: 38278614 DOI: 10.1016/s2589-7500(23)00227-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/15/2023] [Revised: 10/24/2023] [Accepted: 11/02/2023] [Indexed: 01/28/2024]
Abstract
Advances in machine learning for health care have brought concerns about bias from the research community; specifically, the introduction, perpetuation, or exacerbation of care disparities. Reinforcing these concerns is the finding that medical images often reveal signals about sensitive attributes in ways that are hard to pinpoint by both algorithms and people. This finding raises a question about how to best design general purpose pretrained embeddings (GPPEs, defined as embeddings meant to support a broad array of use cases) for building downstream models that are free from particular types of bias. The downstream model should be carefully evaluated for bias, and audited and improved as appropriate. However, in our view, well intentioned attempts to prevent the upstream components-GPPEs-from learning sensitive attributes can have unintended consequences on the downstream models. Despite producing a veneer of technical neutrality, the resultant end-to-end system might still be biased or poorly performing. We present reasons, by building on previously published data, to support the reasoning that GPPEs should ideally contain as much information as the original data contain, and highlight the perils of trying to remove sensitive attributes from a GPPE. We also emphasise that downstream prediction models trained for specific tasks and settings, whether developed using GPPEs or not, should be carefully designed and evaluated to avoid bias that makes models vulnerable to issues such as distributional shift. These evaluations should be done by a diverse team, including social scientists, on a diverse cohort representing the full breadth of the patient population for which the final model is intended.
Collapse
Affiliation(s)
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Leo A G Celi
- Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA, USA; Department of Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Yun Liu
- Google, Mountain View, CA, USA.
| |
Collapse
|
3
|
Lee C, Willis A, Chen C, Sieniek M, Watters A, Stetson B, Uddin A, Wong J, Pilgrim R, Chou K, Tse D, Shetty S, Gomes RG. Development of a Machine Learning Model for Sonographic Assessment of Gestational Age. JAMA Netw Open 2023; 6:e2248685. [PMID: 36598790 PMCID: PMC9857195 DOI: 10.1001/jamanetworkopen.2022.48685] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/05/2023] Open
Abstract
IMPORTANCE Fetal ultrasonography is essential for confirmation of gestational age (GA), and accurate GA assessment is important for providing appropriate care throughout pregnancy and for identifying complications, including fetal growth disorders. Derivation of GA from manual fetal biometry measurements (ie, head, abdomen, and femur) is operator dependent and time-consuming. OBJECTIVE To develop artificial intelligence (AI) models to estimate GA with higher accuracy and reliability, leveraging standard biometry images and fly-to ultrasonography videos. DESIGN, SETTING, AND PARTICIPANTS To improve GA estimates, this diagnostic study used AI to interpret standard plane ultrasonography images and fly-to ultrasonography videos, which are 5- to 10-second videos that can be automatically recorded as part of the standard of care before the still image is captured. Three AI models were developed and validated: (1) an image model using standard plane images, (2) a video model using fly-to videos, and (3) an ensemble model (combining both image and video models). The models were trained and evaluated on data from the Fetal Age Machine Learning Initiative (FAMLI) cohort, which included participants from 2 study sites at Chapel Hill, North Carolina (US), and Lusaka, Zambia. Participants were eligible to be part of this study if they received routine antenatal care at 1 of these sites, were aged 18 years or older, had a viable intrauterine singleton pregnancy, and could provide written consent. They were not eligible if they had known uterine or fetal abnormality, or had any other conditions that would make participation unsafe or complicate interpretation. Data analysis was performed from January to July 2022. MAIN OUTCOMES AND MEASURES The primary analysis outcome for GA was the mean difference in absolute error between the GA model estimate and the clinical standard estimate, with the ground truth GA extrapolated from the initial GA estimated at an initial examination. RESULTS Of the total cohort of 3842 participants, data were calculated for a test set of 404 participants with a mean (SD) age of 28.8 (5.6) years at enrollment. All models were statistically superior to standard fetal biometry-based GA estimates derived from images captured by expert sonographers. The ensemble model had the lowest mean absolute error compared with the clinical standard fetal biometry (mean [SD] difference, -1.51 [3.96] days; 95% CI, -1.90 to -1.10 days). All 3 models outperformed standard biometry by a more substantial margin on fetuses that were predicted to be small for their GA. CONCLUSIONS AND RELEVANCE These findings suggest that AI models have the potential to empower trained operators to estimate GA with higher accuracy.
Collapse
Affiliation(s)
- Chace Lee
- Google Health, Palo Alto, California
| | | | | | | | - Amber Watters
- Department of Obstetrics and Gynecology, Northwestern University Feinberg School of Medicine, Chicago, Illinois
| | - Bethany Stetson
- Department of Obstetrics and Gynecology, Northwestern University Feinberg School of Medicine, Chicago, Illinois
| | | | | | | | | | | | | | | |
Collapse
|
4
|
Kazemzadeh S, Yu J, Jamshy S, Pilgrim R, Nabulsi Z, Chen C, Beladia N, Lau C, McKinney SM, Hughes T, Kiraly AP, Kalidindi SR, Muyoyeta M, Malemela J, Shih T, Corrado GS, Peng L, Chou K, Chen PHC, Liu Y, Eswaran K, Tse D, Shetty S, Prabhakara S. Deep Learning Detection of Active Pulmonary Tuberculosis at Chest Radiography Matched the Clinical Performance of Radiologists. Radiology 2023; 306:124-137. [PMID: 36066366 DOI: 10.1148/radiol.212213] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
Background The World Health Organization (WHO) recommends chest radiography to facilitate tuberculosis (TB) screening. However, chest radiograph interpretation expertise remains limited in many regions. Purpose To develop a deep learning system (DLS) to detect active pulmonary TB on chest radiographs and compare its performance to that of radiologists. Materials and Methods A DLS was trained and tested using retrospective chest radiographs (acquired between 1996 and 2020) from 10 countries. To improve generalization, large-scale chest radiograph pretraining, attention pooling, and semisupervised learning ("noisy-student") were incorporated. The DLS was evaluated in a four-country test set (China, India, the United States, and Zambia) and in a mining population in South Africa, with positive TB confirmed with microbiological tests or nucleic acid amplification testing (NAAT). The performance of the DLS was compared with that of 14 radiologists. The authors studied the efficacy of the DLS compared with that of nine radiologists using the Obuchowski-Rockette-Hillis procedure. Given WHO targets of 90% sensitivity and 70% specificity, the operating point of the DLS (0.45) was prespecified to favor sensitivity. Results A total of 165 754 images in 22 284 subjects (mean age, 45 years; 21% female) were used for model development and testing. In the four-country test set (1236 subjects, 17% with active TB), the receiver operating characteristic (ROC) curve of the DLS was higher than those for all nine India-based radiologists, with an area under the ROC curve of 0.89 (95% CI: 0.87, 0.91). Compared with these radiologists, at the prespecified operating point, the DLS sensitivity was higher (88% vs 75%, P < .001) and specificity was noninferior (79% vs 84%, P = .004). Trends were similar within other patient subgroups, in the South Africa data set, and across various TB-specific chest radiograph findings. In simulations, the use of the DLS to identify likely TB-positive chest radiographs for NAAT confirmation reduced the cost by 40%-80% per TB-positive patient detected. Conclusion A deep learning method was found to be noninferior to radiologists for the determination of active tuberculosis on digital chest radiographs. © RSNA, 2022 Online supplemental material is available for this article. See also the editorial by van Ginneken in this issue.
Collapse
Affiliation(s)
- Sahar Kazemzadeh
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Jin Yu
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Shahar Jamshy
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Rory Pilgrim
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Zaid Nabulsi
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Christina Chen
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Neeral Beladia
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Charles Lau
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Scott Mayer McKinney
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Thad Hughes
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Atilla P Kiraly
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Sreenivasa Raju Kalidindi
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Monde Muyoyeta
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Jameson Malemela
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Ting Shih
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Greg S Corrado
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Lily Peng
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Katherine Chou
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Po-Hsuan Cameron Chen
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Yun Liu
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Krish Eswaran
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Daniel Tse
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Shravya Shetty
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| | - Shruthi Prabhakara
- From Google Health, 1600 Amphitheatre Pkwy, Mountain View, CA 94043 (S.K., J.Y., S.J., R.P., Z.N., C.C., N.B., S.M.M., T.H., A.P.K., G.S.C., L.P., K.C., P.H.C.C., Y.L., K.E., D.T., S.S., S.P.); Advanced Clinical, Deerfield, Ill (C.L.); Apollo Radiology International, Hyderabad, India (S.R.K.); TB Department, Center of Infectious Disease Research in Zambia, Lusaka, Zambia (M.M.); Sibanye Stillwater, Weltevreden Park, Roodepoort, South Africa (J.M.); and Clickmedix, Gaithersburg, Md (T.S.)
| |
Collapse
|
5
|
Gomes RG, Vwalika B, Lee C, Willis A, Sieniek M, Price JT, Chen C, Kasaro MP, Taylor JA, Stringer EM, McKinney SM, Sindano N, Dahl GE, Goodnight W, Gilmer J, Chi BH, Lau C, Spitz T, Saensuksopa T, Liu K, Tiyasirichokchai T, Wong J, Pilgrim R, Uddin A, Corrado G, Peng L, Chou K, Tse D, Stringer JSA, Shetty S. A mobile-optimized artificial intelligence system for gestational age and fetal malpresentation assessment. Commun Med (Lond) 2022; 2:128. [PMID: 36249461 PMCID: PMC9553916 DOI: 10.1038/s43856-022-00194-5] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/10/2022] [Accepted: 09/28/2022] [Indexed: 11/05/2022] Open
Abstract
Background Fetal ultrasound is an important component of antenatal care, but shortage of adequately trained healthcare workers has limited its adoption in low-to-middle-income countries. This study investigated the use of artificial intelligence for fetal ultrasound in under-resourced settings. Methods Blind sweep ultrasounds, consisting of six freehand ultrasound sweeps, were collected by sonographers in the USA and Zambia, and novice operators in Zambia. We developed artificial intelligence (AI) models that used blind sweeps to predict gestational age (GA) and fetal malpresentation. AI GA estimates and standard fetal biometry estimates were compared to a previously established ground truth, and evaluated for difference in absolute error. Fetal malpresentation (non-cephalic vs cephalic) was compared to sonographer assessment. On-device AI model run-times were benchmarked on Android mobile phones. Results Here we show that GA estimation accuracy of the AI model is non-inferior to standard fetal biometry estimates (error difference -1.4 ± 4.5 days, 95% CI -1.8, -0.9, n = 406). Non-inferiority is maintained when blind sweeps are acquired by novice operators performing only two of six sweep motion types. Fetal malpresentation AUC-ROC is 0.977 (95% CI, 0.949, 1.00, n = 613), sonographers and novices have similar AUC-ROC. Software run-times on mobile phones for both diagnostic models are less than 3 s after completion of a sweep. Conclusions The gestational age model is non-inferior to the clinical standard and the fetal malpresentation model has high AUC-ROCs across operators and devices. Our AI models are able to run on-device, without internet connectivity, and provide feedback scores to assist in upleveling the capabilities of lightly trained ultrasound operators in low resource settings.
Collapse
Affiliation(s)
| | - Bellington Vwalika
- Department of Obstetrics and Gynaecology, University of Zambia School of Medicine, Lusaka, Zambia
- Department of Obstetrics and Gynecology, University of North Carolina School of Medicine, Chapel Hill, NC USA
| | | | | | | | - Joan T. Price
- Department of Obstetrics and Gynecology, University of North Carolina School of Medicine, Chapel Hill, NC USA
- UNC Global Projects—Zambia, LLC, Lusaka, Zambia
| | | | - Margaret P. Kasaro
- Department of Obstetrics and Gynaecology, University of Zambia School of Medicine, Lusaka, Zambia
- UNC Global Projects—Zambia, LLC, Lusaka, Zambia
| | | | - Elizabeth M. Stringer
- Department of Obstetrics and Gynecology, University of North Carolina School of Medicine, Chapel Hill, NC USA
| | | | | | | | - William Goodnight
- Department of Obstetrics and Gynaecology, University of Zambia School of Medicine, Lusaka, Zambia
| | | | - Benjamin H. Chi
- Department of Obstetrics and Gynecology, University of North Carolina School of Medicine, Chapel Hill, NC USA
- UNC Global Projects—Zambia, LLC, Lusaka, Zambia
| | | | | | | | - Kris Liu
- Google Health, Palo Alto, CA USA
| | | | | | | | | | | | | | | | | | - Jeffrey S. A. Stringer
- Department of Obstetrics and Gynecology, University of North Carolina School of Medicine, Chapel Hill, NC USA
- UNC Global Projects—Zambia, LLC, Lusaka, Zambia
| | | |
Collapse
|
6
|
Nabulsi Z, Sellergren A, Jamshy S, Lau C, Santos E, Kiraly AP, Ye W, Yang J, Pilgrim R, Kazemzadeh S, Yu J, Kalidindi SR, Etemadi M, Garcia-Vicente F, Melnick D, Corrado GS, Peng L, Eswaran K, Tse D, Beladia N, Liu Y, Chen PHC, Shetty S. Deep learning for distinguishing normal versus abnormal chest radiographs and generalization to two unseen diseases tuberculosis and COVID-19. Sci Rep 2021; 11:15523. [PMID: 34471144 PMCID: PMC8410908 DOI: 10.1038/s41598-021-93967-2] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2021] [Accepted: 07/01/2021] [Indexed: 01/20/2023] Open
Abstract
Chest radiography (CXR) is the most widely-used thoracic clinical imaging modality and is crucial for guiding the management of cardiothoracic conditions. The detection of specific CXR findings has been the main focus of several artificial intelligence (AI) systems. However, the wide range of possible CXR abnormalities makes it impractical to detect every possible condition by building multiple separate systems, each of which detects one or more pre-specified conditions. In this work, we developed and evaluated an AI system to classify CXRs as normal or abnormal. For training and tuning the system, we used a de-identified dataset of 248,445 patients from a multi-city hospital network in India. To assess generalizability, we evaluated our system using 6 international datasets from India, China, and the United States. Of these datasets, 4 focused on diseases that the AI was not trained to detect: 2 datasets with tuberculosis and 2 datasets with coronavirus disease 2019. Our results suggest that the AI system trained using a large dataset containing a diverse array of CXR abnormalities generalizes to new patient populations and unseen diseases. In a simulated workflow where the AI system prioritized abnormal cases, the turnaround time for abnormal cases reduced by 7-28%. These results represent an important step towards evaluating whether AI can be safely used to flag cases in a general setting where previously unseen abnormalities exist. Lastly, to facilitate the continued development of AI models for CXR, we release our collected labels for the publicly available dataset.
Collapse
Affiliation(s)
| | | | | | - Charles Lau
- Google Health Via Advanced Clinical, Deerfield, USA
| | | | | | | | - Jie Yang
- Google Health, Google, Palo Alto, USA
| | | | | | - Jin Yu
- Google Health, Google, Palo Alto, USA
| | | | | | | | | | | | - Lily Peng
- Google Health, Google, Palo Alto, USA
| | | | | | | | - Yun Liu
- Google Health, Google, Palo Alto, USA
| | | | | |
Collapse
|
7
|
Abstract
Hurricanes cause serious and long-term damage to the Agricultural sectors of Caribbean countries. Bananas and tree crops are defoliated, snapped or uprooted and food crops may be flooded or washed away. Recovery takes time and money as both the production bases and the infrastructure are damaged or destroyed. National economies do not have the resources to expedite recovery without aid. An account is given of the actions taken to estimate losses and prepare rehabilitation plans after Hurricanes David (1979) in Dominica and Allen (1980) in St. Lucia and St. Vincent. The implementation of funding agency-assisted agricultural rehabilitation programmes is also described. Some steps that farmers can take to reduce loss of food are suggested.
Collapse
Affiliation(s)
- J L Hammerton
- Caribbean Agricultural Research and Development Institute (CARDI) P.O. Box 971 Castries St. Lucia
| | | | | |
Collapse
|
8
|
Sigel A, Schrott KM, Leibold W, Baenkler HW, Heidler R, Pilgrim R, Seybold D. [Kidney transplantation. Indications, conditions and results]. Fortschr Med 1982; 100:203-221. [PMID: 7044930] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 05/21/2023]
|
9
|
Pilgrim R. [Antibacterial medications, antibiotics and antitubercular agents in kidney insufficiency]. Fortschr Med 1982; 100:218-20. [PMID: 7084870] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 01/23/2023]
|
10
|
Pilgrim R. [Pyelonephritis]. Fortschr Med 1982; 100:212-6. [PMID: 7084869] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 01/23/2023]
|
11
|
Pilgrim R, Lux E, Gessler U. [Therapy of arterial hypertension]. Fortschr Med 1979; 97:1357-62. [PMID: 39883] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
|
12
|
Schulz W, Chlepas S, Heidler R, Pilgrim R, Seybold D, Sigel A, Gessler U. [Kidney transplantation from a nephrological-urological viewpoint--results and problems. 2. Diagnosis and therapy after transplantation, complications, long-term results]. Fortschr Med 1977; 95:2813-8. [PMID: 338452] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
Abstract
Indications, selection of donor and recipient, medical and surgical management and complications, problems of organ procurement. Renal transplantation has become routine therapy. Organs are predominantly obtained from cadavers, transplantations from living donors are rarely indicated. Advances in preservation methods have improved organ quality and prolonged storage time. Selection of the most suitable recipient is based on histocompatibility matching. Blood transfusions before transplantation seem to improve the results. Recognition of a rejection crisis is primarily based on clinical symptoms. Persistent rejection calls for prompt explantation and the patient has to return to dialysis. Infections, serum-hepatitis and gastro-intestinal bleeding are the most common complications. Late complicatons are diabetes mellitus, cirrhosis of the liver, osteopathy, recurring glomerulonephritis, and, rarely, malignomas. Transplantation frequency in the Federal Republic of Germany could be increased by more awareness of physicians and a better knowledge of the general public about the need for cadaver donors.
Collapse
|
13
|
Schulz W, Chlepas S, Heidler R, Pilgrim R, Seybold D, Sigel A, Gessler U. [Kidney transplantation from a nephrological-urological viewpoint; results and problems. 1. Indications, selection of donors and patients, measures before transplantation]. Fortschr Med 1977; 95:2766-70. [PMID: 145400] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
|
14
|
Gessler U, Pilgrim R. [Pharmacotherapy of arterial hypertension]. Fortschr Med 1977; 95:1912-5. [PMID: 892686] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 12/24/2022]
|