1
|
Wu R, He C, Liang P, Liu Y, Huang Y, Liu W, Shu B, Xu P, Chang Q. MCF-SMSIS: Multi-tasking with complementary functions for stereo matching and surgical instrument segmentation. Comput Biol Med 2024; 179:108923. [PMID: 39053335 DOI: 10.1016/j.compbiomed.2024.108923] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2024] [Revised: 06/03/2024] [Accepted: 07/17/2024] [Indexed: 07/27/2024]
Abstract
Stereo matching and instrument segmentation of laparoscopic surgical scenarios are key tasks in robotic surgical automation. Many researchers have been studying the two tasks separately for stereo matching and instrument segmentation. However, the relationship between these two tasks is often neglected. In this paper, we propose a model framework for multi-tasking with complementary functions for stereo matching and surgical instrument segmentation (MCF-SMSIS). We aim to complement the features of instrument prediction segmentation to the parallax matching block of stereo matching. We also propose two new evaluation metrics (MINPD and MAXPD) for assessing how well the parallax range matches the migrated domain when the model used for the stereo matching task undergoes domain migration. We performed stereo matching experiments on the SCARED , SERV-CT dataset as well as instrumentation segmentation experiments on the AutoLaparo dataset. The results demonstrate the effectiveness of the proposed method. In particular, stereo matching supplemented with instrument features reduced EPE, >3px and RMSE Depth in the surgical instrument section by 9.5%, 12.7% and 6.51%, respectively. The instrumentation segmentation performance also achieves a DSC value of 0.9233. Moreover, MCF-SMSIS takes only 0.14 s to infer a set of images. The model code and model weights for each stage are available from https://github.com/wurenkai/MCF-SMSIS.
Collapse
Affiliation(s)
- Renkai Wu
- School of Microelectronics, Shanghai University, Shanghai, China; Department of Surgery, Shanghai Key Laboratory of Gastric Neoplasms, Shanghai Institute of Digestive Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Changyu He
- Department of Surgery, Shanghai Key Laboratory of Gastric Neoplasms, Shanghai Institute of Digestive Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Pengchen Liang
- School of Microelectronics, Shanghai University, Shanghai, China; Department of Surgery, Shanghai Key Laboratory of Gastric Neoplasms, Shanghai Institute of Digestive Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Yinghao Liu
- Department of Surgery, Shanghai Key Laboratory of Gastric Neoplasms, Shanghai Institute of Digestive Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Yiqi Huang
- Department of Mathematics, Imperial College London, London, United Kingdom
| | - Weiping Liu
- Shanghai Microport Medbot (Group) Co., Ltd., Shanghai, China
| | - Biao Shu
- Shanghai Microport Medbot (Group) Co., Ltd., Shanghai, China
| | - Panlong Xu
- Shanghai Microport Medbot (Group) Co., Ltd., Shanghai, China
| | - Qing Chang
- School of Microelectronics, Shanghai University, Shanghai, China; Department of Surgery, Shanghai Key Laboratory of Gastric Neoplasms, Shanghai Institute of Digestive Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China.
| |
Collapse
|
2
|
Rueckert T, Rueckert D, Palm C. Methods and datasets for segmentation of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art. Comput Biol Med 2024; 169:107929. [PMID: 38184862 DOI: 10.1016/j.compbiomed.2024.107929] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2023] [Revised: 12/02/2023] [Accepted: 01/01/2024] [Indexed: 01/09/2024]
Abstract
In the field of computer- and robot-assisted minimally invasive surgery, enormous progress has been made in recent years based on the recognition of surgical instruments in endoscopic images and videos. In particular, the determination of the position and type of instruments is of great interest. Current work involves both spatial and temporal information, with the idea that predicting the movement of surgical tools over time may improve the quality of the final segmentations. The provision of publicly available datasets has recently encouraged the development of new methods, mainly based on deep learning. In this review, we identify and characterize datasets used for method development and evaluation and quantify their frequency of use in the literature. We further present an overview of the current state of research regarding the segmentation and tracking of minimally invasive surgical instruments in endoscopic images and videos. The paper focuses on methods that work purely visually, without markers of any kind attached to the instruments, considering both single-frame semantic and instance segmentation approaches, as well as those that incorporate temporal information. The publications analyzed were identified through the platforms Google Scholar, Web of Science, and PubMed. The search terms used were "instrument segmentation", "instrument tracking", "surgical tool segmentation", and "surgical tool tracking", resulting in a total of 741 articles published between 01/2015 and 07/2023, of which 123 were included using systematic selection criteria. A discussion of the reviewed literature is provided, highlighting existing shortcomings and emphasizing the available potential for future developments.
Collapse
Affiliation(s)
- Tobias Rueckert
- Regensburg Medical Image Computing (ReMIC), Ostbayerische Technische Hochschule Regensburg (OTH Regensburg), Germany.
| | - Daniel Rueckert
- Artificial Intelligence in Healthcare and Medicine, Klinikum rechts der Isar, Technical University of Munich, Germany; Department of Computing, Imperial College London, UK
| | - Christoph Palm
- Regensburg Medical Image Computing (ReMIC), Ostbayerische Technische Hochschule Regensburg (OTH Regensburg), Germany; Regensburg Center of Health Sciences and Technology (RCHST), OTH Regensburg, Germany
| |
Collapse
|