1
|
Kataoka Y, Taito S, Yamamoto N, So R, Tsutsumi Y, Anan K, Banno M, Tsujimoto Y, Wada Y, Sagami S, Tsujimoto H, Nihashi T, Takeuchi M, Terasawa T, Iguchi M, Kumasawa J, Ichikawa T, Furukawa R, Yamabe J, Furukawa TA. An open competition involving thousands of competitors failed to construct useful abstract classifiers for new diagnostic test accuracy systematic reviews. Res Synth Methods 2023; 14:707-717. [PMID: 37337729 DOI: 10.1002/jrsm.1649] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2022] [Revised: 06/05/2023] [Accepted: 06/07/2023] [Indexed: 06/21/2023]
Abstract
There are currently no abstract classifiers, which can be used for new diagnostic test accuracy (DTA) systematic reviews to select primary DTA study abstracts from database searches. Our goal was to develop machine-learning-based abstract classifiers for new DTA systematic reviews through an open competition. We prepared a dataset of abstracts obtained through database searches from 11 reviews in different clinical areas. As the reference standard, we used the abstract lists that required manual full-text review. We randomly splitted the datasets into a train set, a public test set, and a private test set. Competition participants used the training set to develop classifiers and validated their classifiers using the public test set. The classifiers were refined based on the performance of the public test set. They could submit as many times as they wanted during the competition. Finally, we used the private test set to rank the submitted classifiers. To reduce false exclusions, we used the Fbeta measure with a beta set to seven for evaluating classifiers. After the competition, we conducted the external validation using a dataset from a cardiology DTA review. We received 13,774 submissions from 1429 teams or persons over 4 months. The top-honored classifier achieved a Fbeta score of 0.4036 and a recall of 0.2352 in the external validation. In conclusion, we were unable to develop an abstract classifier with sufficient recall for immediate application to new DTA systematic reviews. Further studies are needed to update and validate classifiers with datasets from other clinical areas.
Collapse
Affiliation(s)
- Yuki Kataoka
- Department of Internal Medicine, Kyoto Min-iren Asukai Hospital, Kyoto, Japan
- Scientific Research Works Peer Support Group (SRWS-PSG), Osaka, Japan
- Section of Clinical Epidemiology, Department of Community Medicine, Kyoto University Graduate School of Medicine, Kyoto, Japan
- Department of Healthcare Epidemiology, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan
| | - Shunsuke Taito
- Scientific Research Works Peer Support Group (SRWS-PSG), Osaka, Japan
- Division of Rehabilitation, Department of Clinical Practice and Support, Hiroshima University Hospital, Hiroshima, Japan
| | - Norio Yamamoto
- Scientific Research Works Peer Support Group (SRWS-PSG), Osaka, Japan
- Department of Orthopedic Surgery, Miyamoto Orthopedic Hospital, Okayama, Japan
- Department of Epidemiology, Graduate School of Medicine, Dentistry and Pharmaceutical Sciences, Okayama University, Okayama, Japan
| | - Ryuhei So
- Scientific Research Works Peer Support Group (SRWS-PSG), Osaka, Japan
- Department of Psychiatry, Okayama Psychiatric Medical Center, Okayama, Japan
- CureApp, Inc., Tokyo, Japan
| | - Yusuke Tsutsumi
- Scientific Research Works Peer Support Group (SRWS-PSG), Osaka, Japan
- Department of Healthcare Epidemiology, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan
- Department of Emergency Medicine, National Hospital Organization Mito Medical Center, Ibaraki, Japan
| | - Keisuke Anan
- Scientific Research Works Peer Support Group (SRWS-PSG), Osaka, Japan
- Division of Respiratory Medicine, Saiseikai Kumamoto Hospital, Kumamoto, Japan
- Department of Healthcare Epidemiology, Graduate School of Medicine and Public Health, Kyoto University, Kyoto, Japan
| | - Masahiro Banno
- Scientific Research Works Peer Support Group (SRWS-PSG), Osaka, Japan
- Department of Psychiatry, Seichiryo Hospital, Nagoya, Japan
- Department of Psychiatry, Nagoya University Graduate School of Medicine, Nagoya, Japan
| | - Yasushi Tsujimoto
- Scientific Research Works Peer Support Group (SRWS-PSG), Osaka, Japan
- Oku Medical Clinic, Osaka, Japan
- Department of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto University, Kyoto, Japan
| | - Yoshitaka Wada
- Scientific Research Works Peer Support Group (SRWS-PSG), Osaka, Japan
- Department of Rehabilitation Medicine, School of Medicine, Fujita Health University, Toyoake, Japan
| | - Shintaro Sagami
- Center for Advanced IBD Research and Treatment, Kitasato University Kitasato Institute Hospital, Tokyo, Japan
- Department of Gastroenterology and Hepatology, Kitasato University Kitasato Institute Hospital, Tokyo, Japan
| | - Hiraku Tsujimoto
- Hospital Care Research Unit, Hyogo Prefectural Amagasaki General Medical Center, Amagasaki, Japan
| | - Takashi Nihashi
- Department of Radiology, Komaki City Hospital, Komaki, Japan
| | - Motoki Takeuchi
- Department of Emergency and General Internal Medicine, Fujita Health University School of Medicine, Toyoake, Japan
| | - Teruhiko Terasawa
- Section of General Internal Medicine, Department of Emergency and General Internal Medicine, Fujita Health University School of Medicine, Toyoake, Japan
| | - Masahiro Iguchi
- Department of Neurology, Fukushima Medical University, Fukushima, Japan
| | - Junji Kumasawa
- Human Health Sciences, Kyoto University Graduate School of Medicine, Kyoto, Japan
- Department of Critical Care Medicine, Sakai City Medical Center, Sakai, Japan
| | | | | | | | - Toshi A Furukawa
- Department of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto University, Kyoto, Japan
| |
Collapse
|
2
|
Boyle RJ, Shamji MH. Developments in the field of allergy in 2020 through the eyes of Clinical and Experimental Allergy. Clin Exp Allergy 2021; 51:1531-1537. [PMID: 34750898 DOI: 10.1111/cea.14046] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2021] [Accepted: 11/08/2021] [Indexed: 11/28/2022]
Abstract
While 2020 will be remembered for the global coronavirus pandemic, there were also important advances in the field of allergy. In this review article, we summarize key findings reported in Clinical and Experimental Allergy during 2020. We hope this provides readers with an accessible snapshot of the work published in our journal during this time.
Collapse
Affiliation(s)
- Robert J Boyle
- National Heart and Lung Institute, Imperial College London, London, UK
| | - Mohamed H Shamji
- National Heart and Lung Institute, Imperial College London, London, UK.,NIHR Imperial Biomedical Research Centre, London, UK
| |
Collapse
|