1
|
Wang T, Dremel J, Richter S, Polanski W, Uckermann O, Eyüpoglu I, Czarske JW, Kuschmierz R. Resolution-enhanced multi-core fiber imaging learned on a digital twin for cancer diagnosis. NEUROPHOTONICS 2024; 11:S11505. [PMID: 38298866 PMCID: PMC10828892 DOI: 10.1117/1.nph.11.s1.s11505] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/20/2023] [Revised: 01/04/2024] [Accepted: 01/08/2024] [Indexed: 02/02/2024]
Abstract
Significance Deep learning enables label-free all-optical biopsies and automated tissue classification. Endoscopic systems provide intraoperative diagnostics to deep tissue and speed up treatment without harmful tissue removal. However, conventional multi-core fiber (MCF) endoscopes suffer from low resolution and artifacts, which hinder tumor diagnostics. Aim We introduce a method to enable unpixelated, high-resolution tumor imaging through a given MCF with a diameter of around 0.65 mm and arbitrary core arrangement and inhomogeneous transmissivity. Approach Image reconstruction is based on deep learning and the digital twin concept of the single-reference-based simulation with inhomogeneous optical properties of MCF and transfer learning on a small experimental dataset of biological tissue. The reference provided physical information about the MCF during the training processes. Results For the simulated data, hallucination caused by the MCF inhomogeneity was eliminated, and the averaged peak signal-to-noise ratio and structural similarity were increased from 11.2 dB and 0.20 to 23.4 dB and 0.74, respectively. By transfer learning, the metrics of independent test images experimentally acquired on glioblastoma tissue ex vivo can reach up to 31.6 dB and 0.97 with 14 fps computing speed. Conclusions With the proposed approach, a single reference image was required in the pre-training stage and laborious acquisition of training data was bypassed. Validation on glioblastoma cryosections with transfer learning on only 50 image pairs showed the capability for high-resolution deep tissue retrieval and high clinical feasibility.
Collapse
Affiliation(s)
- Tijue Wang
- TU Dresden, Laboratory of Measurement and Sensor System Technique, Dresden, Germany
- TU Dresden, Competence Center BIOLAS, Dresden, Germany
- TU Dresden, Else Kröner Fresenius Center for Digital Health, Germany
| | - Jakob Dremel
- TU Dresden, Laboratory of Measurement and Sensor System Technique, Dresden, Germany
- TU Dresden, Competence Center BIOLAS, Dresden, Germany
- TU Dresden, Else Kröner Fresenius Center for Digital Health, Germany
| | - Sven Richter
- TU Dresden, Else Kröner Fresenius Center for Digital Health, Germany
- University Hospital Carl Gustav Carus, TU Dresden, Department of Neurosurgery, Dresden, Germany
| | - Witold Polanski
- TU Dresden, Else Kröner Fresenius Center for Digital Health, Germany
- University Hospital Carl Gustav Carus, TU Dresden, Department of Neurosurgery, Dresden, Germany
| | - Ortrud Uckermann
- TU Dresden, Else Kröner Fresenius Center for Digital Health, Germany
- University Hospital Carl Gustav Carus, TU Dresden, Department of Neurosurgery, Dresden, Germany
- University Hospital Carl Gustav Carus, TU Dresden, Division of Medical Biology, Department of Psychiatry, Faculty of Medicine, Dresden, Germany
| | - Ilker Eyüpoglu
- TU Dresden, Else Kröner Fresenius Center for Digital Health, Germany
- University Hospital Carl Gustav Carus, TU Dresden, Department of Neurosurgery, Dresden, Germany
| | - Jürgen W. Czarske
- TU Dresden, Laboratory of Measurement and Sensor System Technique, Dresden, Germany
- TU Dresden, Competence Center BIOLAS, Dresden, Germany
- TU Dresden, Else Kröner Fresenius Center for Digital Health, Germany
- TU Dresden, Excellence Cluster Physics of Life, Dresden, Germany
- TU Dresden, School of Science, Faculty of Physics, Dresden, Germany
| | - Robert Kuschmierz
- TU Dresden, Laboratory of Measurement and Sensor System Technique, Dresden, Germany
- TU Dresden, Competence Center BIOLAS, Dresden, Germany
- TU Dresden, Else Kröner Fresenius Center for Digital Health, Germany
| |
Collapse
|
2
|
Wang J, Chen C, You W, Jiao Y, Liu X, Jiang X, Lu W. Honeycomb effect elimination in differential phase fiber-bundle-based endoscopy. OPTICS EXPRESS 2024; 32:20682-20694. [PMID: 38859444 DOI: 10.1364/oe.526033] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/10/2024] [Accepted: 05/10/2024] [Indexed: 06/12/2024]
Abstract
Fiber-bundle-based endoscopy, with its ultrathin probe and micrometer-level resolution, has become a widely adopted imaging modality for in vivo imaging. However, the fiber bundles introduce a significant honeycomb effect, primarily due to the multi-core structure and crosstalk of adjacent fiber cores, which superposes the honeycomb pattern image on the original image. To tackle this issue, we propose an iterative-free spatial pixel shifting (SPS) algorithm, designed to suppress the honeycomb effect and enhance real-time imaging performance. The process involves the creation of three additional sub-images by shifting the original image by one pixel at 0, 45, and 90 degree angles. These four sub-images are then used to compute differential maps in the x and y directions. By performing spiral integration on these differential maps, we reconstruct a honeycomb-free image with improved details. Our simulations and experimental results, conducted on a self-built fiber bundle-based endoscopy system, demonstrate the effectiveness of the SPS algorithm. SPS significantly improves the image quality of reflective objects and unlabeled transparent scattered objects, laying a solid foundation for biomedical endoscopic applications.
Collapse
|
3
|
Kim J, Lee H, Oh SR, Yang S. Real-Time Endomicroscopic Image Mosaicking with an EKF-based Sensor Fusion Approach. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023; 2023:1-5. [PMID: 38083648 DOI: 10.1109/embc40787.2023.10340903] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/18/2023]
Abstract
This study presents a real-time sensor fusion framework based on the extended Kalman filter (EKF) for accurate and robust endomicroscopic image mosaicking. The sensor fusion framework incorporates an optical tracking system that can track 6-DOF pose of the imaging probe with high accuracy in real time in conjunction with 2D local image registration from image feature matching between two consecutive frames. We evaluated the performance of the real-time image mosaicking based on the sensor fusion compared with the image or tracker only approach. As a result, it could retain the microscopic level of image detail from the image-based approach and also achieve a robust image mosaic without any drift by using the accurate optical tracking system.
Collapse
|