1
|
Wang J, Wang K, Yu Y, Lu Y, Xiao W, Sun Z, Liu F, Zou Z, Gao Y, Yang L, Zhou HY, Miao H, Zhao W, Huang L, Zeng L, Guo R, Chong I, Deng B, Cheng L, Chen X, Luo J, Zhu MH, Baptista-Hon D, Monteiro O, Li M, Ke Y, Li J, Zeng S, Guan T, Zeng J, Xue K, Oermann E, Luo H, Yin Y, Zhang K, Qu J. Self-improving generative foundation model for synthetic medical image generation and clinical applications. Nat Med 2024:10.1038/s41591-024-03359-y. [PMID: 39663467 DOI: 10.1038/s41591-024-03359-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2024] [Accepted: 10/15/2024] [Indexed: 12/13/2024]
Abstract
In many clinical and research settings, the scarcity of high-quality medical imaging datasets has hampered the potential of artificial intelligence (AI) clinical applications. This issue is particularly pronounced in less common conditions, underrepresented populations and emerging imaging modalities, where the availability of diverse and comprehensive datasets is often inadequate. To address this challenge, we introduce a unified medical image-text generative model called MINIM that is capable of synthesizing medical images of various organs across various imaging modalities based on textual instructions. Clinician evaluations and rigorous objective measurements validate the high quality of MINIM's synthetic images. MINIM exhibits an enhanced generative capability when presented with previously unseen data domains, demonstrating its potential as a generalist medical AI (GMAI). Our findings show that MINIM's synthetic images effectively augment existing datasets, boosting performance across multiple medical applications such as diagnostics, report generation and self-supervised learning. On average, MINIM enhances performance by 12% for ophthalmic, 15% for chest, 13% for brain and 17% for breast-related tasks. Furthermore, we demonstrate MINIM's potential clinical utility in the accurate prediction of HER2-positive breast cancer from MRI images. Using a large retrospective simulation analysis, we demonstrate MINIM's clinical potential by accurately identifying targeted therapy-sensitive EGFR mutations using lung cancer computed tomography images, which could potentially lead to improved 5-year survival rates. Although these results are promising, further validation and refinement in more diverse and prospective settings would greatly enhance the model's generalizability and robustness.
Collapse
Affiliation(s)
- Jinzhuo Wang
- Department of Big Data and Biomedical AI, National Biomedical Imaging Center, College of Future Technology, Peking University and Peking-Tsinghua Center for Life Sciences, Beijing, China.
| | - Kai Wang
- Department of Big Data and Biomedical AI, National Biomedical Imaging Center, College of Future Technology, Peking University and Peking-Tsinghua Center for Life Sciences, Beijing, China
- National Clinical Research Center for Ocular Diseases, Eye Hospital, Wenzhou Medical University, Wenzhou, China
| | - Yunfang Yu
- Institute for AI in Medicine, Faculty of Medicine, Macau University of Science and Technology, Taipa, Macau, China
- Sun Yat-sen Memorial Hospital, Sun Yat-sen University, Guangzhou, China
| | - Yuxing Lu
- Department of Big Data and Biomedical AI, National Biomedical Imaging Center, College of Future Technology, Peking University and Peking-Tsinghua Center for Life Sciences, Beijing, China
| | - Wenchao Xiao
- Department of Ophthalmology, Zhuhai People's Hospital and the First Affiliated Hospital of Faculty of Medicine, Macau University of Science and Technology, Zhuhai, China
| | - Zhuo Sun
- Department of Ophthalmology, The Third People's Hospital of Changzhou, Changzhou, China
- Advanced Institute for Eye Health and Diseases, Wenzhou Medical University, Wenzhou, China
| | - Fei Liu
- Institute for AI in Medicine, Faculty of Medicine, Macau University of Science and Technology, Taipa, Macau, China
- National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
| | - Zixing Zou
- Guangzhou National Laboratory, Guangzhou, China
| | - Yuanxu Gao
- National Clinical Research Center for Ocular Diseases, Eye Hospital, Wenzhou Medical University, Wenzhou, China
- Institute for AI in Medicine, Faculty of Medicine, Macau University of Science and Technology, Taipa, Macau, China
| | - Lei Yang
- Department of Thoracic Surgery, The First Affiliated Hospital of Sun Yat-sen University, Guangzhou, China
| | - Hong-Yu Zhou
- National Clinical Research Center for Ocular Diseases, Eye Hospital, Wenzhou Medical University, Wenzhou, China
| | - Hanpei Miao
- Dongguan People's Hospital, Southern Medical University, Dongguan, China
| | - Wenting Zhao
- Department of Ophthalmology, Zhuhai People's Hospital and the First Affiliated Hospital of Faculty of Medicine, Macau University of Science and Technology, Zhuhai, China
| | - Lisha Huang
- Department of Ophthalmology, Zhuhai People's Hospital and the First Affiliated Hospital of Faculty of Medicine, Macau University of Science and Technology, Zhuhai, China
| | - Lingchao Zeng
- Department of Ophthalmology, Zhuhai People's Hospital and the First Affiliated Hospital of Faculty of Medicine, Macau University of Science and Technology, Zhuhai, China
| | - Rui Guo
- Department of Ophthalmology, Zhuhai People's Hospital and the First Affiliated Hospital of Faculty of Medicine, Macau University of Science and Technology, Zhuhai, China
| | - Ieng Chong
- Faculty of Innovation Engineering, Macau University of Science and Technology, Macau, China
| | - Boyu Deng
- Guangzhou National Laboratory, Guangzhou, China
| | - Linling Cheng
- Institute for AI in Medicine, Faculty of Medicine, Macau University of Science and Technology, Taipa, Macau, China
| | - Xiaoniao Chen
- National Clinical Research Center for Ocular Diseases, Eye Hospital, Wenzhou Medical University, Wenzhou, China
| | - Jing Luo
- Department of Ophthalmology, The Second Xiangya Hospital affiliated to Central South University, Changsha, China
| | - Meng-Hua Zhu
- Institute for AI in Medicine, Faculty of Medicine, Macau University of Science and Technology, Taipa, Macau, China
- Space Science Institute, Macau University of Science and Technology, Taipa, Macau, China
| | - Daniel Baptista-Hon
- Institute for AI in Medicine, Faculty of Medicine, Macau University of Science and Technology, Taipa, Macau, China
| | - Olivia Monteiro
- Institute for AI in Medicine, Faculty of Medicine, Macau University of Science and Technology, Taipa, Macau, China
| | - Ming Li
- National Clinical Research Center for Ocular Diseases, Eye Hospital, Wenzhou Medical University, Wenzhou, China
| | - Yu Ke
- Dongguan People's Hospital, Southern Medical University, Dongguan, China
| | - Jiahui Li
- Guangzhou Women and Children's Medical Center, Guangzhou Medical University, Guangzhou, China
| | - Simiao Zeng
- Guangzhou Women and Children's Medical Center, Guangzhou Medical University, Guangzhou, China
| | - Taihua Guan
- Guangzhou National Laboratory, Guangzhou, China
| | - Jin Zeng
- Guangzhou National Laboratory, Guangzhou, China
| | - Kanmin Xue
- Nuffield Department of Clinical Neurosciences, University of Oxford & University of Oxford Hospitals NHS Foundation Trust, Oxford, UK
| | - Eric Oermann
- NYU Langone Medical Center, New York University, New York, NY, USA
| | - Huiyan Luo
- State Key Laboratory of Oncology in South China, Collaborative Innovation Center of Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China
| | - Yun Yin
- Advanced Institute for Eye Health and Diseases, Wenzhou Medical University, Wenzhou, China
| | - Kang Zhang
- National Clinical Research Center for Ocular Diseases, Eye Hospital, Wenzhou Medical University, Wenzhou, China.
- Institute for AI in Medicine, Faculty of Medicine, Macau University of Science and Technology, Taipa, Macau, China.
- Department of Ophthalmology, Zhuhai People's Hospital and the First Affiliated Hospital of Faculty of Medicine, Macau University of Science and Technology, Zhuhai, China.
- Advanced Institute for Eye Health and Diseases, Wenzhou Medical University, Wenzhou, China.
- Guangzhou National Laboratory, Guangzhou, China.
- Eye and Vision Innovation Center, Eye Valley, Wenzhou, China.
| | - Jia Qu
- National Clinical Research Center for Ocular Diseases, Eye Hospital, Wenzhou Medical University, Wenzhou, China.
- Oujiang Laboratory, Zhejiang Lab for Regenerative Medicine, Vision and Brain Health, Wenzhou, China.
| |
Collapse
|
2
|
Fok WYR, Fieselmann A, Huemmer C, Biniazan R, Beister M, Geiger B, Kappler S, Saalfeld S. Adversarial robustness improvement for X-ray bone segmentation using synthetic data created from computed tomography scans. Sci Rep 2024; 14:25813. [PMID: 39468116 PMCID: PMC11519576 DOI: 10.1038/s41598-024-73363-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2024] [Accepted: 09/17/2024] [Indexed: 10/30/2024] Open
Abstract
Deep learning-based image analysis offers great potential in clinical practice. However, it faces mainly two challenges: scarcity of large-scale annotated clinical data for training and susceptibility to adversarial data in inference. As an example, an artificial intelligence (AI) system could check patient positioning, by segmenting and evaluating relative positions of anatomical structures in medical images. Nevertheless, data to train such AI system might be highly imbalanced with mostly well-positioned images being available. Thus, we propose the use of synthetic X-ray images and annotation masks forward projected from 3D photon-counting CT volumes to create realistic non-optimally positioned X-ray images for training. An open-source model (TotalSegmentator) was used to annotate the clavicles in 3D CT volumes. We evaluated model robustness with respect to the internal (simulated) patient rotation α on real-data-trained models and real&synthetic-data-trained models. Our results showed that real&synthetic- data-trained models have Dice score percentage improvements of 3% to 15% across different α groups compared to the real-data-trained model. Therefore, we demonstrated that synthetic data could be supplementary used to train and enrich heavily underrepresented conditions to increase model robustness.
Collapse
Affiliation(s)
- Wai Yan Ryana Fok
- Faculty of Computer Science, Otto-von-Guericke-University Magdeburg, 39106, Magdeburg, Germany.
- X-ray Products, Siemens Healthineers AG, 91301, Forchheim, Germany.
| | | | | | - Ramyar Biniazan
- X-ray Products, Siemens Healthineers AG, 91301, Forchheim, Germany
| | - Marcel Beister
- X-ray Products, Siemens Healthineers AG, 91301, Forchheim, Germany
| | - Bernhard Geiger
- X-ray Products, Siemens Healthineers AG, 91301, Forchheim, Germany
| | - Steffen Kappler
- X-ray Products, Siemens Healthineers AG, 91301, Forchheim, Germany
| | - Sylvia Saalfeld
- Faculty of Computer Science, Otto-von-Guericke-University Magdeburg, 39106, Magdeburg, Germany
- Institute for Medical Informatics and Statistics, University Hospital Schleswig-Holstein Campus Kiel, 24105, Kiel, Germany
| |
Collapse
|
3
|
Wang X, Zhang Z, Xu S, Luo X, Zhang B, Wu XJ. Contrastive learning based method for X-ray and CT registration under surgical equipment occlusion. Comput Biol Med 2024; 180:108946. [PMID: 39106676 DOI: 10.1016/j.compbiomed.2024.108946] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/23/2024] [Revised: 07/25/2024] [Accepted: 07/25/2024] [Indexed: 08/09/2024]
Abstract
Deep learning-based 3D/2D surgical navigation registration techniques achieved excellent results. However, these methods are limited by the occlusion of surgical equipment resulting in poor accuracy. We designed a contrastive learning method that treats occluded and unoccluded X-rays as positive samples, maximizing the similarity between the positive samples and reducing interference from occlusion. The designed registration model has Transformer's residual connection (ResTrans), which enhances the long-sequence mapping capability, combined with the contrast learning strategy, ResTrans can adaptively retrieve the valid features in the global range to ensure the performance in the case of occlusion. Further, a learning-based region of interest (RoI) fine-tuning method is designed to refine the misalignment. We conducted experiments on occluded X-rays that contained different surgical devices. The experiment results show that the mean target registration error (mTRE) of ResTrans is 3.25 mm and the running time is 1.59 s. Compared with the state-of-the-art (SOTA) 3D/2D registration methods, our method offers better performance on occluded 3D/2D registration tasks.
Collapse
Affiliation(s)
- Xiyuan Wang
- School of Electronics and Information Engineering at University of Science and Technology Suzhou, SuZhou, 215009, China
| | - Zhancheng Zhang
- School of Electronics and Information Engineering at University of Science and Technology Suzhou, SuZhou, 215009, China.
| | - Shaokang Xu
- School of Electronics and Information Engineering at University of Science and Technology Suzhou, SuZhou, 215009, China; Shanghai Jirui Maestro Surgical Technology Co, ShangHai, 200000, China
| | - Xiaoqing Luo
- Jiangsu Provincial Engineering Laboratory of Pattern Recognition and Computational Intelligence, School of Artificial Intelligence and Computer Science at Jiangnan University, WuXi, 214122, China
| | - Baocheng Zhang
- Department of Orthopaedics, General Hospital of Central Theater Command of PLA, WuHan, 430012, China
| | - Xiao-Jun Wu
- Jiangsu Provincial Engineering Laboratory of Pattern Recognition and Computational Intelligence, School of Artificial Intelligence and Computer Science at Jiangnan University, WuXi, 214122, China
| |
Collapse
|
4
|
Que W, Bian Y, Chen S, Zhao X, Ji Z, Hu P, Han C, Shi L. Efficient electrocardiogram generation based on cardiac electric vector simulation model. Comput Biol Med 2024; 177:108629. [PMID: 38820778 DOI: 10.1016/j.compbiomed.2024.108629] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/22/2024] [Revised: 04/27/2024] [Accepted: 05/18/2024] [Indexed: 06/02/2024]
Abstract
This study introduces a novel Cardiac Electric Vector Simulation Model (CEVSM) to address the computational inefficiencies and low fidelity of traditional electrophysiological models in generating electrocardiograms (ECGs). Our approach leverages CEVSM to efficiently produce reliable ECG samples, facilitating data augmentation essential for the computer-aided diagnosis of myocardial infarction (MI). Significantly, experimental results show that our model dramatically reduces computation time compared to conventional models, with the self-adapting regression transformation matrix method (SRTM) providing clear advantages. SRTM not only achieves high fidelity in ECG simulations but also ensures exceptional consistency with the gold standard method, greatly enhancing MI localization accuracy by data augmentation. These advancements highlight the potential of our model to generate dependable ECG training samples, making it highly suitable for data augmentation and significantly advancing the development and validation of intelligent MI diagnostic systems. Furthermore, this study demonstrates the feasibility of applying life system simulations in the training of medical big models.
Collapse
Affiliation(s)
- Wenge Que
- Department of Automation, Tsinghua University, Beijing, 100084, China.
| | - Yingnan Bian
- School of Logistics, Henan College of Transportation, Zhengzhou, 450000, China.
| | - Shengjie Chen
- Department of Automation, Tsinghua University, Beijing, 100084, China.
| | - Xiliang Zhao
- Center for Coronary Artery Disease, Division of Cardiology, Beijing Anzhen Hospital, Capital Medical University, Beijing, 100029, China.
| | - Zehua Ji
- Department of Automation, Tsinghua University, Beijing, 100084, China.
| | - Pingge Hu
- Department of Automation, Tsinghua University, Beijing, 100084, China.
| | - Chuang Han
- School of Computer Science and Technology, Zhengzhou University of Light Industry, Zhengzhou, 450000, China.
| | - Li Shi
- Department of Automation, Tsinghua University, Beijing, 100084, China; Beijing National Research Center for Information Science and Technology, Beijing, 100084, China.
| |
Collapse
|
5
|
Wu S, Kurugol S, Kleinman PK, Ecklund K, Walters M, Connolly SA, Johnston P, Tsai A. Deep generative model of the distal tibial classic metaphyseal lesion in infants: assessment of synthetic images. RADIOLOGY ADVANCES 2024; 1:umae018. [PMID: 39171131 PMCID: PMC11335364 DOI: 10.1093/radadv/umae018] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/14/2024] [Revised: 06/04/2024] [Accepted: 06/25/2024] [Indexed: 08/23/2024]
Abstract
Background The classic metaphyseal lesion (CML) is a distinctive fracture highly specific to infant abuse. To increase the size and diversity of the training CML database for automated deep-learning detection of this fracture, we developed a mask conditional diffusion model (MaC-DM) to generate synthetic images with and without CMLs. Purpose To objectively and subjectively assess the synthetic radiographic images with and without CMLs generated by MaC-DM. Materials and Methods For retrospective testing, we randomly chose 100 real images (50 normals and 50 with CMLs; 39 infants, male = 22, female = 17; mean age = 4.1 months; SD = 3.1 months) from an existing distal tibia dataset (177 normal, 73 with CMLs), and generated 100 synthetic distal tibia images via MaC-DM (50 normals and 50 with CMLs). These test images were shown to 3 blinded radiologists. In the first session, radiologists determined if the images were normal or had CMLs. In the second session, they determined if the images were real or synthetic. We analyzed the radiologists' interpretations and employed t-distributed stochastic neighbor embedding technique to analyze the data distribution of the test images. Results When presented with the 200 images (100 synthetic, 100 with CMLs), radiologists reliably and accurately diagnosed CMLs (kappa = 0.90, 95% CI = [0.88-0.92]; accuracy = 92%, 95% CI = [89-97]). However, they were inaccurate in differentiating between real and synthetic images (kappa = 0.05, 95% CI = [0.03-0.07]; accuracy = 53%, 95% CI = [49-59]). The t-distributed stochastic neighbor embedding analysis showed substantial differences in the data distribution between normal images and those with CMLs (area under the curve = 0.996, 95% CI = [0.992-1.000], P < .01), but minor differences between real and synthetic images (area under the curve = 0.566, 95% CI = [0.486-0.647], P = .11). Conclusion Radiologists accurately diagnosed images with distal tibial CMLs but were unable to distinguish real from synthetically generated ones, indicating that our generative model could synthesize realistic images. Thus, MaC-DM holds promise as an effective strategy for data augmentation in training machine-learning models for diagnosis of distal tibial CMLs.
Collapse
Affiliation(s)
- Shaoju Wu
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, United States
| | - Sila Kurugol
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, United States
| | - Paul K Kleinman
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, United States
| | - Kirsten Ecklund
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, United States
| | - Michele Walters
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, United States
| | - Susan A Connolly
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, United States
| | - Patrick Johnston
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, United States
| | - Andy Tsai
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, United States
| |
Collapse
|
6
|
Killeen BD, Chaudhary S, Osgood G, Unberath M. Take a shot! Natural language control of intelligent robotic X-ray systems in surgery. Int J Comput Assist Radiol Surg 2024; 19:1165-1173. [PMID: 38619790 PMCID: PMC11178437 DOI: 10.1007/s11548-024-03120-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2024] [Accepted: 03/22/2024] [Indexed: 04/16/2024]
Abstract
PURPOSE The expanding capabilities of surgical systems bring with them increasing complexity in the interfaces that humans use to control them. Robotic C-arm X-ray imaging systems, for instance, often require manipulation of independent axes via joysticks, while higher-level control options hide inside device-specific menus. The complexity of these interfaces hinder "ready-to-hand" use of high-level functions. Natural language offers a flexible, familiar interface for surgeons to express their desired outcome rather than remembering the steps necessary to achieve it, enabling direct access to task-aware, patient-specific C-arm functionality. METHODS We present an English language voice interface for controlling a robotic X-ray imaging system with task-aware functions for pelvic trauma surgery. Our fully integrated system uses a large language model (LLM) to convert natural spoken commands into machine-readable instructions, enabling low-level commands like "Tilt back a bit," to increase the angular tilt or patient-specific directions like, "Go to the obturator oblique view of the right ramus," based on automated image analysis. RESULTS We evaluate our system with 212 prompts provided by an attending physician, in which the system performed satisfactory actions 97% of the time. To test the fully integrated system, we conduct a real-time study in which an attending physician placed orthopedic hardware along desired trajectories through an anthropomorphic phantom, interacting solely with an X-ray system via voice. CONCLUSION Voice interfaces offer a convenient, flexible way for surgeons to manipulate C-arms based on desired outcomes rather than device-specific processes. As LLMs grow increasingly capable, so too will their applications in supporting higher-level interactions with surgical assistance systems.
Collapse
Affiliation(s)
- Benjamin D Killeen
- Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD, 21218, USA.
| | - Shreayan Chaudhary
- Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD, 21218, USA
| | - Greg Osgood
- Department of Orthopaedic Surgery, Johns Hopkins University, Baltimore, MD, 212187, USA
| | - Mathias Unberath
- Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD, 21218, USA
| |
Collapse
|
7
|
Killeen BD, Zhang H, Wang LJ, Liu Z, Kleinbeck C, Rosen M, Taylor RH, Osgood G, Unberath M. Stand in surgeon's shoes: virtual reality cross-training to enhance teamwork in surgery. Int J Comput Assist Radiol Surg 2024; 19:1213-1222. [PMID: 38642297 PMCID: PMC11178441 DOI: 10.1007/s11548-024-03138-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2024] [Accepted: 03/28/2024] [Indexed: 04/22/2024]
Abstract
PURPOSE Teamwork in surgery depends on a shared mental model of success, i.e., a common understanding of objectives in the operating room. A shared model leads to increased engagement among team members and is associated with fewer complications and overall better outcomes for patients. However, clinical training typically focuses on role-specific skills, leaving individuals to acquire a shared model indirectly through on-the-job experience. METHODS We investigate whether virtual reality (VR) cross-training, i.elet@tokeneonedotexposure to other roles, can enhance a shared mental model for non-surgeons more directly. Our study focuses on X-ray guided pelvic trauma surgery, a procedure where successful communication depends on the shared model between the surgeon and a C-arm technologist. We present a VR environment supporting both roles and evaluate a cross-training curriculum in which non-surgeons swap roles with the surgeon. RESULTS Exposure to the surgical task resulted in higher engagement with the C-arm technologist role in VR, as measured by the mental demand and effort expended by participants ( p < 0.001 ). It also has a significant effect on non-surgeon's mental model of the overall task; novice participants' estimation of the mental demand and effort required for the surgeon's task increases after training, while their perception of overall performance decreases ( p < 0.05 ), indicating a gap in understanding based solely on observation. This phenomenon was also present for a professional C-arm technologist. CONCLUSION Until now, VR applications for clinical training have focused on virtualizing existing curricula. We demonstrate how novel approaches which are not possible outside of a virtual environment, such as role swapping, may enhance the shared mental model of surgical teams by contextualizing each individual's role within the overall task in a time- and cost-efficient manner. As workflows grow increasingly sophisticated, we see VR curricula as being able to directly foster a shared model for success, ultimately benefiting patient outcomes through more effective teamwork in surgery.
Collapse
Affiliation(s)
| | - Han Zhang
- Johns Hopkins University, Baltimore, MD, 21218, USA
| | - Liam J Wang
- Johns Hopkins University, Baltimore, MD, 21218, USA
| | - Zixuan Liu
- Johns Hopkins University, Baltimore, MD, 21218, USA
| | - Constantin Kleinbeck
- Johns Hopkins University, Baltimore, MD, 21218, USA
- Friedrich-Alexander-Universität, Erlangen, Germany
| | | | | | - Greg Osgood
- Department of Orthopaedic Surgery, Johns Hopkins Medicine, Baltimore, MD, 21218, USA
| | | |
Collapse
|
8
|
Fan F, Ritschl L, Beister M, Biniazan R, Wagner F, Kreher B, Gottschalk TM, Kappler S, Maier A. Simulation-driven training of vision transformers enables metal artifact reduction of highly truncated CBCT scans. Med Phys 2024; 51:3360-3375. [PMID: 38150576 DOI: 10.1002/mp.16919] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2023] [Revised: 11/17/2023] [Accepted: 12/13/2023] [Indexed: 12/29/2023] Open
Abstract
BACKGROUND Due to the high attenuation of metals, severe artifacts occur in cone beam computed tomography (CBCT). The metal segmentation in CBCT projections usually serves as a prerequisite for metal artifact reduction (MAR) algorithms. PURPOSE The occurrence of truncation caused by the limited detector size leads to the incomplete acquisition of metal masks from the threshold-based method in CBCT volume. Therefore, segmenting metal directly in CBCT projections is pursued in this work. METHODS Since the generation of high quality clinical training data is a constant challenge, this study proposes to generate simulated digital radiographs (data I) based on real CT data combined with self-designed computer aided design (CAD) implants. In addition to the simulated projections generated from 3D volumes, 2D x-ray images combined with projections of implants serve as the complementary data set (data II) to improve the network performance. In this work, SwinConvUNet consisting of shift window (Swin) vision transformers (ViTs) with patch merging as encoder is proposed for metal segmentation. RESULTS The model's performance is evaluated on accurately labeled test datasets obtained from cadaver scans as well as the unlabeled clinical projections. When trained on the data I only, the convolutional neural network (CNN) encoder-based networks UNet and TransUNet achieve only limited performance on the cadaver test data, with an average dice score of 0.821 and 0.850. After using both data II and data I during training, the average dice scores for the two models increase to 0.906 and 0.919, respectively. By replacing the CNN encoder with Swin transformer, the proposed SwinConvUNet reaches an average dice score of 0.933 for cadaver projections when only trained on the data I. Furthermore, SwinConvUNet has the largest average dice score of 0.953 for cadaver projections when trained on the combined data set. CONCLUSIONS Our experiments quantitatively demonstrate the effectiveness of the combination of the projections simulated under two pathways for network training. Besides, the proposed SwinConvUNet trained on the simulated projections performs state-of-the-art, robust metal segmentation as demonstrated on experiments on cadaver and clinical data sets. With the accurate segmentations from the proposed model, MAR can be conducted even for highly truncated CBCT scans.
Collapse
Affiliation(s)
- Fuxin Fan
- Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | | | | | | | - Fabian Wagner
- Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | | | | | | | - Andreas Maier
- Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| |
Collapse
|
9
|
Huang Z, Li H, Shao S, Zhu H, Hu H, Cheng Z, Wang J, Kevin Zhou S. PELE scores: pelvic X-ray landmark detection with pelvis extraction and enhancement. Int J Comput Assist Radiol Surg 2024; 19:939-950. [PMID: 38491244 DOI: 10.1007/s11548-024-03089-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2023] [Accepted: 02/27/2024] [Indexed: 03/18/2024]
Abstract
PURPOSE Pelvic X-ray (PXR) is widely utilized in clinical decision-making associated with the pelvis, the lower part of the trunk that supports and balances the trunk. In particular, PXR-based landmark detection facilitates downstream analysis and computer-assisted diagnosis and treatment of pelvic diseases. Although PXR has the advantages of low radiation and reduced cost compared to computed tomography (CT), it characterizes the 2D pelvis-tissue superposition of 3D structures, which may affect the accuracy of landmark detection in some cases. However, the superposition nature of PXR is implicitly handled by existing deep learning-based landmark detection methods, which mainly design the deep network structures for better detection performances. Explicit handling of the superposition nature of PXR is rarely done. METHODS In this paper, we explicitly focus on the superposition of X-ray images. Specifically, we propose a pelvis extraction (PELE) module that consists of a decomposition network, a domain adaptation network, and an enhancement module, which utilizes 3D prior anatomical knowledge in CT to guide and well isolate the pelvis from PXR, thereby eliminating the influence of soft tissue for landmark detection. The extracted pelvis image, after enhancement, is then used for landmark detection. RESULTS We conduct an extensive evaluation based on two public and one private dataset, totaling 850 PXRs. The experimental results show that the proposed PELE module significantly improves the accuracy of PXRs landmark detection and achieves state-of-the-art performances in several benchmark metrics. CONCLUSION The design of PELE module can improve the accuracy of different pelvic landmark detection baselines, which we believe is obviously conducive to the positioning and inspection of clinical landmarks and critical structures, thus better serving downstream tasks. Our project has been open-sourced at https://github.com/ECNUACRush/PELEscores .
Collapse
Affiliation(s)
- Zhen Huang
- Computer Science Department, University of Science and Technology of China (USTC), Hefei, 230026, China
- Center for Medical Imaging, Robotics, Analytic Computing and Learning (MIRACLE), Suzhou Institute for Advanced Research, USTC, Suzhou, 215123, China
| | - Han Li
- School of Biomedical Engineering, Division of Life Sciences and Medicine, USTC, Hefei, 230026, China
- Center for Medical Imaging, Robotics, Analytic Computing and Learning (MIRACLE), Suzhou Institute for Advanced Research, USTC, Suzhou, 215123, China
| | | | - Heqin Zhu
- School of Biomedical Engineering, Division of Life Sciences and Medicine, USTC, Hefei, 230026, China
- Center for Medical Imaging, Robotics, Analytic Computing and Learning (MIRACLE), Suzhou Institute for Advanced Research, USTC, Suzhou, 215123, China
| | - Huijie Hu
- Computer Science Department, University of Science and Technology of China (USTC), Hefei, 230026, China
- Center for Medical Imaging, Robotics, Analytic Computing and Learning (MIRACLE), Suzhou Institute for Advanced Research, USTC, Suzhou, 215123, China
| | | | - Jianji Wang
- Affiliated Hospital of Guizhou Medical University, Guiyang, 550000, China
| | - S Kevin Zhou
- School of Biomedical Engineering, Division of Life Sciences and Medicine, USTC, Hefei, 230026, China.
- Center for Medical Imaging, Robotics, Analytic Computing and Learning (MIRACLE), Suzhou Institute for Advanced Research, USTC, Suzhou, 215123, China.
- Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing, 100190, China.
| |
Collapse
|
10
|
Wang G, Zhou M, Ning X, Tiwari P, Zhu H, Yang G, Yap CH. US2Mask: Image-to-mask generation learning via a conditional GAN for cardiac ultrasound image segmentation. Comput Biol Med 2024; 172:108282. [PMID: 38503085 DOI: 10.1016/j.compbiomed.2024.108282] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/05/2024] [Revised: 02/29/2024] [Accepted: 03/12/2024] [Indexed: 03/21/2024]
Abstract
Cardiac ultrasound (US) image segmentation is vital for evaluating clinical indices, but it often demands a large dataset and expert annotations, resulting in high costs for deep learning algorithms. To address this, our study presents a framework utilizing artificial intelligence generation technology to produce multi-class RGB masks for cardiac US image segmentation. The proposed approach directly performs semantic segmentation of the heart's main structures in US images from various scanning modes. Additionally, we introduce a novel learning approach based on conditional generative adversarial networks (CGAN) for cardiac US image segmentation, incorporating a conditional input and paired RGB masks. Experimental results from three cardiac US image datasets with diverse scan modes demonstrate that our approach outperforms several state-of-the-art models, showcasing improvements in five commonly used segmentation metrics, with lower noise sensitivity. Source code is available at https://github.com/energy588/US2mask.
Collapse
Affiliation(s)
- Gang Wang
- School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, Chongqing; Department of Bioengineering, Imperial College London, London, UK
| | - Mingliang Zhou
- School of Computer Science, Chongqing University, Chongqing, Chongqing.
| | - Xin Ning
- Institute of Semiconductors, Chinese Academy of Sciences, Beijing, China
| | - Prayag Tiwari
- School of Information Technology, Halmstad University, Halmstad, Sweden
| | | | - Guang Yang
- Department of Bioengineering, Imperial College London, London, UK; Cardiovascular Research Centre, Royal Brompton Hospital, London, UK; National Heart and Lung Institute, Imperial College London, London, UK
| | - Choon Hwai Yap
- Department of Bioengineering, Imperial College London, London, UK
| |
Collapse
|
11
|
Walton NA, Nagarajan R, Wang C, Sincan M, Freimuth RR, Everman DB, Walton DC, McGrath SP, Lemas DJ, Benos PV, Alekseyenko AV, Song Q, Gamsiz Uzun E, Taylor CO, Uzun A, Person TN, Rappoport N, Zhao Z, Williams MS. Enabling the clinical application of artificial intelligence in genomics: a perspective of the AMIA Genomics and Translational Bioinformatics Workgroup. J Am Med Inform Assoc 2024; 31:536-541. [PMID: 38037121 PMCID: PMC10797281 DOI: 10.1093/jamia/ocad211] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2023] [Revised: 10/09/2023] [Accepted: 10/26/2023] [Indexed: 12/02/2023] Open
Abstract
OBJECTIVE Given the importance AI in genomics and its potential impact on human health, the American Medical Informatics Association-Genomics and Translational Biomedical Informatics (GenTBI) Workgroup developed this assessment of factors that can further enable the clinical application of AI in this space. PROCESS A list of relevant factors was developed through GenTBI workgroup discussions in multiple in-person and online meetings, along with review of pertinent publications. This list was then summarized and reviewed to achieve consensus among the group members. CONCLUSIONS Substantial informatics research and development are needed to fully realize the clinical potential of such technologies. The development of larger datasets is crucial to emulating the success AI is achieving in other domains. It is important that AI methods do not exacerbate existing socio-economic, racial, and ethnic disparities. Genomic data standards are critical to effectively scale such technologies across institutions. With so much uncertainty, complexity and novelty in genomics and medicine, and with an evolving regulatory environment, the current focus should be on using these technologies in an interface with clinicians that emphasizes the value each brings to clinical decision-making.
Collapse
Affiliation(s)
- Nephi A Walton
- Division of Medical Genetics, University of Utah School of Medicine, Salt Lake City, UT 84112 ,United States
| | - Radha Nagarajan
- Enterprise Information Services, Cedars-Sinai Medical Center, Los Angeles, CA 90025, United States
- Information Services Department, Children’s Hospital of Orange County, Orange, CA 92868, United States
| | - Chen Wang
- Division of Computational Biology, Department of Quantitative Health Sciences, Center for Individualized Medicine, Mayo Clinic, Rochester, MN 55905, United States
| | - Murat Sincan
- Flatiron Health, New York, NY 10013, United States
- Department of Internal Medicine, Sanford School of Medicine, University of South Dakota, Sioux Falls, SD 57107, United States
| | - Robert R Freimuth
- Department of Artificial Intelligence and Informatics, Center for Individualized Medicine, Mayo Clinic, Rochester, MN 55905, United States
| | - David B Everman
- EverMed Genetics and Genomics Consulting LLC, Greenville, SC 29607, United States
| | | | - Scott P McGrath
- CITRIS Health, CITRIS and Banatao Institute, University of California Berkeley, Berkeley, CA 94720, United States
| | - Dominick J Lemas
- Department of Health Outcomes and Biomedical Informatics, University of Florida, Gainesville, FL 32610, United States
| | - Panayiotis V Benos
- Department of Epidemiology, University of Florida, Gainesville, FL 32610, United States
| | - Alexander V Alekseyenko
- Department of Public Health Sciences, Biomedical Informatics Center, Medical University of South Carolina, Charleston, SC 29403, United States
| | - Qianqian Song
- Department of Health Outcomes and Biomedical Informatics, University of Florida, Gainesville, FL 32610, United States
| | - Ece Gamsiz Uzun
- Department of Pathology and Laboratory Medicine, Rhode Island Hospital and Lifespan Medical Center, Providence, RI 02915, United States
- Department of Pathology and Laboratory Medicine, Warren Alpert Medical School of Brown University, Providence, RI 02915, United States
| | - Casey Overby Taylor
- Departments of Medicine and Biomedical Engineering, Johns Hopkins School of Medicine, Baltimore, MD 21205, United States
| | - Alper Uzun
- Department of Pathology and Laboratory Medicine, Warren Alpert Medical School of Brown University, Providence, RI 02915, United States
- Legorreta Cancer Center, Brown University, Providence, RI 02915, United States
| | - Thomas Nate Person
- Department of Bioinformatics and Genomics, Huck Institutes of the Life Sciences, Penn State University, Bloomsburg, PA 16802, United States
| | - Nadav Rappoport
- Department of Software and Information Systems Engineering, Ben-Gurion University of the Negev, Beer Sheva 8410501, Israel
| | - Zhongming Zhao
- Center for Precision Health, McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX 77030, United States
| | - Marc S Williams
- Department of Genomic Health, Geisinger, Danville, PA 17822, United States
| |
Collapse
|
12
|
Koohi-Moghadam M, Bae KT. Generative AI in Medical Imaging: Applications, Challenges, and Ethics. J Med Syst 2023; 47:94. [PMID: 37651022 DOI: 10.1007/s10916-023-01987-4] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/24/2023] [Accepted: 08/21/2023] [Indexed: 09/01/2023]
Abstract
Medical imaging is playing an important role in diagnosis and treatment of diseases. Generative artificial intelligence (AI) have shown great potential in enhancing medical imaging tasks such as data augmentation, image synthesis, image-to-image translation, and radiology report generation. This commentary aims to provide an overview of generative AI in medical imaging, discussing applications, challenges, and ethical considerations, while highlighting future research directions in this rapidly evolving field.
Collapse
Affiliation(s)
- Mohamad Koohi-Moghadam
- Department of Diagnostic Radiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Pok Fu Lam, Hong Kong.
| | - Kyongtae Ty Bae
- Department of Diagnostic Radiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Pok Fu Lam, Hong Kong.
| |
Collapse
|