3
|
Schaffter T, Buist DSM, Lee CI, Nikulin Y, Ribli D, Guan Y, Lotter W, Jie Z, Du H, Wang S, Feng J, Feng M, Kim HE, Albiol F, Albiol A, Morrell S, Wojna Z, Ahsen ME, Asif U, Jimeno Yepes A, Yohanandan S, Rabinovici-Cohen S, Yi D, Hoff B, Yu T, Chaibub Neto E, Rubin DL, Lindholm P, Margolies LR, McBride RB, Rothstein JH, Sieh W, Ben-Ari R, Harrer S, Trister A, Friend S, Norman T, Sahiner B, Strand F, Guinney J, Stolovitzky G. Evaluation of Combined Artificial Intelligence and Radiologist Assessment to Interpret Screening Mammograms. JAMA Netw Open 2020; 3:e200265. [PMID: 32119094 PMCID: PMC7052735 DOI: 10.1001/jamanetworkopen.2020.0265] [Citation(s) in RCA: 157] [Impact Index Per Article: 39.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/12/2019] [Accepted: 12/26/2019] [Indexed: 12/18/2022] Open
Abstract
Importance Mammography screening currently relies on subjective human interpretation. Artificial intelligence (AI) advances could be used to increase mammography screening accuracy by reducing missed cancers and false positives. Objective To evaluate whether AI can overcome human mammography interpretation limitations with a rigorous, unbiased evaluation of machine learning algorithms. Design, Setting, and Participants In this diagnostic accuracy study conducted between September 2016 and November 2017, an international, crowdsourced challenge was hosted to foster AI algorithm development focused on interpreting screening mammography. More than 1100 participants comprising 126 teams from 44 countries participated. Analysis began November 18, 2016. Main Outcomes and Measurements Algorithms used images alone (challenge 1) or combined images, previous examinations (if available), and clinical and demographic risk factor data (challenge 2) and output a score that translated to cancer yes/no within 12 months. Algorithm accuracy for breast cancer detection was evaluated using area under the curve and algorithm specificity compared with radiologists' specificity with radiologists' sensitivity set at 85.9% (United States) and 83.9% (Sweden). An ensemble method aggregating top-performing AI algorithms and radiologists' recall assessment was developed and evaluated. Results Overall, 144 231 screening mammograms from 85 580 US women (952 cancer positive ≤12 months from screening) were used for algorithm training and validation. A second independent validation cohort included 166 578 examinations from 68 008 Swedish women (780 cancer positive). The top-performing algorithm achieved an area under the curve of 0.858 (United States) and 0.903 (Sweden) and 66.2% (United States) and 81.2% (Sweden) specificity at the radiologists' sensitivity, lower than community-practice radiologists' specificity of 90.5% (United States) and 98.5% (Sweden). Combining top-performing algorithms and US radiologist assessments resulted in a higher area under the curve of 0.942 and achieved a significantly improved specificity (92.0%) at the same sensitivity. Conclusions and Relevance While no single AI algorithm outperformed radiologists, an ensemble of AI algorithms combined with radiologist assessment in a single-reader screening environment improved overall accuracy. This study underscores the potential of using machine learning methods for enhancing mammography screening interpretation.
Collapse
Affiliation(s)
| | - Diana S. M. Buist
- Kaiser Permanente Washington Health Research Institute, Seattle, Washington
| | | | | | - Dezső Ribli
- Department of Physics of Complex Systems, ELTE Eötvös Loránd University, Budapest, Hungary
| | - Yuanfang Guan
- Department of Computational Medicine and Bioinformatics, Michigan Medicine, University of Michigan, Ann Arbor
| | | | | | - Hao Du
- National University of Singapore, Singapore
| | - Sijia Wang
- Integrated Health Information Systems Pte Ltd, Singapore
| | - Jiashi Feng
- Department of Electrical and Computer Engineering, National University of Singapore, Singapore
| | | | | | - Francisco Albiol
- Instituto de Física Corpuscular (IFIC), CSIC–Universitat de València, Valencia, Spain
| | - Alberto Albiol
- Universitat Politecnica de Valencia, Valencia, Valenciana, Spain
| | - Stephen Morrell
- Centre for Medical Image Computing, University College London, Bloomsbury, London, United Kingdom
| | | | | | - Umar Asif
- IBM Research Australia, Melbourne, Australia
| | | | | | | | - Darvin Yi
- Stanford University, Stanford, California
| | - Bruce Hoff
- Computational Oncology, Sage Bionetworks, Seattle, Washington
| | - Thomas Yu
- Computational Oncology, Sage Bionetworks, Seattle, Washington
| | | | - Daniel L. Rubin
- Department of Biomedical Data Science, Radiology, and Medicine (Biomedical Informatics), Stanford University, Stanford, California
| | - Peter Lindholm
- Department of Physiology and Pharmacology, Karolinska Institutet, Stockholm, Sweden
| | - Laurie R. Margolies
- Department of Diagnostic, Molecular and Interventional Radiology, Icahn School of Medicine at Mount Sinai, New York, New York
| | - Russell Bailey McBride
- Department of Pathology, Molecular and Cell-Based Medicine, Icahn School of Medicine at Mount Sinai, New York, New York
| | - Joseph H. Rothstein
- Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, New York
| | - Weiva Sieh
- Department of Population Health Science and Policy, Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, New York
| | - Rami Ben-Ari
- IBM Research Haifa, Haifa University Campus, Mount Carmel, Haifa, Israel
| | | | - Andrew Trister
- Fred Hutchinson Cancer Research Center, Seattle, Washington
| | - Stephen Friend
- Computational Oncology, Sage Bionetworks, Seattle, Washington
| | - Thea Norman
- Bill and Melinda Gates Foundation, Seattle, Washington
| | - Berkman Sahiner
- Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, Maryland
| | - Fredrik Strand
- Department of Oncology-Pathology, Karolinska Institutet, Stockholm, Sweden
- Breast Radiology, Karolinska University Hospital, Stockholm, Sweden
| | - Justin Guinney
- Computational Oncology, Sage Bionetworks, Seattle, Washington
| | - Gustavo Stolovitzky
- IBM Research, Translational Systems Biology and Nanobiotechnology, Thomas J. Watson Research Center, Yorktown Heights, New York
| | | |
Collapse
|
8
|
Adachi K, Sasaki H, Nagahisa S, Yoshida K, Hattori N, Nishiyama Y, Kawase T, Hasegawa M, Abe M, Hirose Y, Alentorn A, Marie Y, Poggioli S, Alshehhi H, Boisselier B, Carpentier C, Mokhtari K, Capelle L, Figarella-Branger D, Hoang-Xuan K, Sanson M, Delattre JY, Idbaih A, Yust-Katz S, Anderson M, Olar A, Eterovic A, Ezzeddine N, Chen K, Zhao H, Fuller G, Aldape K, de Groot J, Andor N, Harness J, Lopez SG, Fung TL, Mewes HW, Petritsch C, Arivazhagan A, Somasundaram K, Thennarasu K, Pandey P, Anandh B, Santosh V, Chandramouli B, Hegde A, Kondaiah P, Rao M, Bell R, Kang R, Hong C, Song J, Costello J, Bell R, Nagarajan R, Zhang B, Diaz A, Wang T, Song J, Costello J, Bie L, Li Y, Li Y, Liu H, Luyo WFC, Carnero MH, Iruegas MEP, Morell AR, Figueiras MC, Lopez RL, Valverde CF, Chan AKY, Pang JCS, Chung NYF, Li KKW, Poon WS, Chan DTM, Wang Y, Ng HAK, Chaumeil M, Larson P, Yoshihara H, Vigneron D, Nelson S, Pieper R, Phillips J, Ronen S, Clark V, Omay ZE, Serin A, Gunel J, Omay B, Grady C, Youngblood M, Bilguvar K, Baehring J, Piepmeier J, Gutin P, Vortmeyer A, Brennan C, Pamir MN, Kilic T, Krischek B, Simon M, Yasuno K, Gunel M, Cohen AL, Sato M, Aldape KD, Mason C, Diefes K, Heathcock L, Abegglen L, Shrieve D, Couldwell W, Schiffman JD, Colman H, D'Alessandris QG, Cenci T, Martini M, Ricci-Vitiani L, De Maria R, Larocca LM, Pallini R, de Groot J, Theeler B, Aldape K, Lang F, Rao G, Gilbert M, Sulman E, Luthra R, Eterovic K, Chen K, Routbort M, Verhaak R, Mills G, Mendelsohn J, Meric-Bernstam F, Yung A, MacArthur K, Hahn S, Kao G, Lustig R, Alonso-Basanta M, Chandrasekaran S, Wileyto EP, Reyes E, Dorsey J, Fujii K, Kurozumi K, Ichikawa T, Onishi M, Ishida J, Shimazu Y, Kaur B, Chiocca EA, Date I, Geisenberger C, Mock A, Warta R, Schwager C, Hartmann C, von Deimling A, Abdollahi A, Herold-Mende C, Gevaert O, Achrol A, Gholamin S, Mitra S, Westbroek E, Loya J, Mitchell L, Chang S, Steinberg G, Plevritis S, Cheshier S, Gevaert O, Mitchell L, Achrol A, Xu J, Steinberg G, Cheshier S, Napel S, Zaharchuk G, Plevritis S, Gevaert O, Achrol A, Chang S, Harsh G, Steinberg G, Cheshier S, Plevritis S, Gutman D, Holder C, Colen R, Dunn W, Jain R, Cooper L, Hwang S, Flanders A, Brat D, Hayes J, Droop A, Thygesen H, Boissinot M, Westhead D, Short S, Lawler S, Bady P, Kurscheid S, Delorenzi M, Hegi ME, Crosby C, Faulkner C, Smye-Rumsby T, Kurian K, Williams M, Hopkins K, Faulkner C, Palmer A, Williams H, Wragg C, Haynes HR, Williams M, Hopkins K, Kurian KM, Haynes HR, Crosby C, Williams H, White P, Hopkins K, Williams M, Kurian KM, Ishida J, Kurozumi K, Ichikawa T, Onishi M, Fujii K, Shimazu Y, Oka T, Date I, Jalbert L, Elkhaled A, Phillips J, Chang S, Nelson S, Jensen R, Salzman K, Schabel M, Gillespie D, Mumert M, Johnson B, Mazor T, Hong C, Barnes M, Yamamoto S, Ueda H, Tatsuno K, Aihara K, Jalbert L, Nelson S, Bollen A, Hirst M, Marra M, Mukasa A, Saito N, Aburatani H, Berger M, Chang S, Taylor B, Costello J, Popov S, Mackay A, Ingram W, Burford A, Jury A, Vinci M, Jones C, Jones DTW, Hovestadt V, Picelli S, Wang W, Northcott PA, Kool M, Reifenberger G, Pietsch T, Sultan M, Lehrach H, Yaspo ML, Borkhardt A, Landgraf P, Eils R, Korshunov A, Zapatka M, Radlwimmer B, Pfister SM, Lichter P, Joy A, Smirnov I, Reiser M, Shapiro W, Mills G, Kim S, Feuerstein B, Jungk C, Mock A, Geisenberger C, Warta R, Friauf S, Unterberg A, Herold-Mende C, Juratli TA, McElroy J, Meng W, Huebner A, Geiger KD, Krex D, Schackert G, Chakravarti A, Lautenschlaeger T, Kim BY, Jiang W, Beiko J, Prabhu S, DeMonte F, Lang F, Gilbert M, Aldape K, Sawaya R, Cahill D, McCutcheon I, Lau C, Wang L, Terashima K, Yamaguchi S, Burstein M, Sun J, Suzuki T, Nishikawa R, Nakamura H, Natsume A, Terasaka S, Ng HK, Muzny D, Gibbs R, Wheeler D, Lautenschlaeger T, Juratli TA, McElroy J, Meng W, Huebner A, Geiger KD, Krex D, Schackert G, Chakravarti A, Zhang XQ, Sun S, Lam KF, Kiang KMY, Pu JKS, Ho ASW, Leung GKK, Loebel F, Curry WT, Barker FG, Lelic N, Chi AS, Cahill DP, Lu D, Yin J, Teo C, McDonald K, Madhankumar A, Weston C, Slagle-Webb B, Sheehan J, Patel A, Glantz M, Connor J, Maire C, Francis J, Zhang CZ, Jung J, Manzo V, Adalsteinsson V, Homer H, Blumenstiel B, Pedamallu CS, Nickerson E, Ligon A, Love C, Meyerson M, Ligon K, Mazor T, Johnson B, Hong C, Barnes M, Jalbert LE, Nelson SJ, Bollen AW, Smirnov IV, Song JS, Olshen AB, Berger MS, Chang SM, Taylor BS, Costello JF, Mehta S, Armstrong B, Peng S, Bapat A, Berens M, Melendez B, Mollejo M, Mur P, Hernandez-Iglesias T, Fiano C, Ruiz J, Rey JA, Mock A, Stadler V, Schulte A, Lamszus K, Schichor C, Westphal M, Tonn JC, Unterberg A, Herold-Mende C, Morozova O, Katzman S, Grifford M, Salama S, Haussler D, Nagarajan R, Zhang B, Johnson B, Bell R, Olshen A, Fouse S, Diaz A, Smirnov I, Kang R, Wang T, Costello J, Nakamizo S, Sasayama T, Tanaka H, Tanaka K, Mizukawa K, Yoshida M, Kohmura E, Northcott P, Hovestadt V, Jones D, Kool M, Korshunov A, Lichter P, Pfister S, Otani R, Mukasa A, Takayanagi S, Saito K, Tanaka S, Shin M, Saito N, Ozawa T, Riester M, Cheng YK, Huse J, Helmy K, Charles N, Squatrito M, Michor F, Holland E, Perrech M, Dreher L, Rohn G, Goldbrunner R, Timmer M, Pollo B, Palumbo V, Calatozzolo C, Patane M, Nunziata R, Farinotti M, Silvani A, Lodrini S, Finocchiaro G, Lopez E, Rioscovian A, Ruiz R, Siordia G, de Leon AP, Rostomily C, Rostomily R, Silbergeld D, Kolstoe D, Chamberlain M, Silber J, Roth P, Keller A, Hoheisel J, Codo P, Bauer A, Backes C, Leidinger P, Meese E, Thiel E, Korfel A, Weller M, Saito K, Mukasa A, Nagae G, Nagane M, Aihara K, Takayanagi S, Tanaka S, Aburatani H, Saito N, Salama S, Sanborn JZ, Grifford M, Brennan C, Mikkelsen T, Jhanwar S, Chin L, Haussler D, Sasayama T, Tanaka K, Nakamizo S, Nishihara M, Tanaka H, Mizukawa K, Kohmura E, Schliesser M, Grimm C, Weiss E, Claus R, Weichenhan D, Weiler M, Hielscher T, Sahm F, Wiestler B, Klein AC, Blaes J, Weller M, Plass C, Wick W, Stragliotto G, Rahbar A, Soderberg-Naucler C, Sulman E, Won M, Ezhilarasan R, Sun P, Blumenthal D, Vogelbaum M, Colman H, Jenkins R, Chakravarti A, Jeraj R, Brown P, Jaeckle K, Schiff D, Dignam J, Atkins J, Brachman D, Werner-Wasik M, Gilbert M, Mehta M, Aldape K, Terashima K, Shen J, Luan J, Yu A, Suzuki T, Nishikawa R, Matsutani M, Liang Y, Man TK, Lau C, Trister A, Tokita M, Mikheeva S, Mikheev A, Friend S, Rostomily R, van den Bent M, Erdem L, Gorlia T, Taphoorn M, Kros J, Wesseling P, Dubbink H, Ibdaih A, Sanson M, French P, van Thuijl H, Mazor T, Johnson B, Fouse S, Heimans J, Wesseling P, Ylstra B, Reijneveld J, Taylor B, Berger M, Chang S, Costello J, Prabowo A, van Thuijl H, Scheinin I, van Essen H, Spliet W, Ferrier C, van Rijen P, Veersema T, Thom M, Meeteren ASV, Reijneveld J, Ylstra B, Wesseling P, Aronica E, Kim H, Zheng S, Mikkelsen T, Brat DJ, Virk S, Amini S, Sougnez C, Chin L, Barnholtz-Sloan J, Verhaak RGW, Watts C, Sottoriva A, Spiteri I, Piccirillo S, Touloumis A, Collins P, Marioni J, Curtis C, Tavare S, Weiss E, Grimm C, Schliesser M, Hielscher T, Claus R, Sahm F, Wiestler B, Klein AC, Blaes J, Tews B, Weiler M, Weichenhan D, Hartmann C, Weller M, Plass C, Wick W, Yeung TPC, Al-Khazraji B, Morrison L, Hoffman L, Jackson D, Lee TY, Yartsev S, Bauman G, Zheng S, Fu J, Vegesna R, Mao Y, Heathcock LE, Torres-Garcia W, Ezhilarasan R, Wang S, McKenna A, Chin L, Brennan CW, Yung WKA, Weinstein JN, Aldape KD, Sulman EP, Chen K, Koul D, Verhaak RGW. OMICS AND PROGNSTIC MARKERS. Neuro Oncol 2013; 15:iii136-iii155. [PMCID: PMC3823898 DOI: 10.1093/neuonc/not183] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/21/2023] Open
|