1
|
Wang T, Dremel J, Richter S, Polanski W, Uckermann O, Eyüpoglu I, Czarske JW, Kuschmierz R. Resolution-enhanced multi-core fiber imaging learned on a digital twin for cancer diagnosis. NEUROPHOTONICS 2024; 11:S11505. [PMID: 38298866 PMCID: PMC10828892 DOI: 10.1117/1.nph.11.s1.s11505] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/20/2023] [Revised: 01/04/2024] [Accepted: 01/08/2024] [Indexed: 02/02/2024]
Abstract
Significance Deep learning enables label-free all-optical biopsies and automated tissue classification. Endoscopic systems provide intraoperative diagnostics to deep tissue and speed up treatment without harmful tissue removal. However, conventional multi-core fiber (MCF) endoscopes suffer from low resolution and artifacts, which hinder tumor diagnostics. Aim We introduce a method to enable unpixelated, high-resolution tumor imaging through a given MCF with a diameter of around 0.65 mm and arbitrary core arrangement and inhomogeneous transmissivity. Approach Image reconstruction is based on deep learning and the digital twin concept of the single-reference-based simulation with inhomogeneous optical properties of MCF and transfer learning on a small experimental dataset of biological tissue. The reference provided physical information about the MCF during the training processes. Results For the simulated data, hallucination caused by the MCF inhomogeneity was eliminated, and the averaged peak signal-to-noise ratio and structural similarity were increased from 11.2 dB and 0.20 to 23.4 dB and 0.74, respectively. By transfer learning, the metrics of independent test images experimentally acquired on glioblastoma tissue ex vivo can reach up to 31.6 dB and 0.97 with 14 fps computing speed. Conclusions With the proposed approach, a single reference image was required in the pre-training stage and laborious acquisition of training data was bypassed. Validation on glioblastoma cryosections with transfer learning on only 50 image pairs showed the capability for high-resolution deep tissue retrieval and high clinical feasibility.
Collapse
Affiliation(s)
- Tijue Wang
- TU Dresden, Laboratory of Measurement and Sensor System Technique, Dresden, Germany
- TU Dresden, Competence Center BIOLAS, Dresden, Germany
- TU Dresden, Else Kröner Fresenius Center for Digital Health, Germany
| | - Jakob Dremel
- TU Dresden, Laboratory of Measurement and Sensor System Technique, Dresden, Germany
- TU Dresden, Competence Center BIOLAS, Dresden, Germany
- TU Dresden, Else Kröner Fresenius Center for Digital Health, Germany
| | - Sven Richter
- TU Dresden, Else Kröner Fresenius Center for Digital Health, Germany
- University Hospital Carl Gustav Carus, TU Dresden, Department of Neurosurgery, Dresden, Germany
| | - Witold Polanski
- TU Dresden, Else Kröner Fresenius Center for Digital Health, Germany
- University Hospital Carl Gustav Carus, TU Dresden, Department of Neurosurgery, Dresden, Germany
| | - Ortrud Uckermann
- TU Dresden, Else Kröner Fresenius Center for Digital Health, Germany
- University Hospital Carl Gustav Carus, TU Dresden, Department of Neurosurgery, Dresden, Germany
- University Hospital Carl Gustav Carus, TU Dresden, Division of Medical Biology, Department of Psychiatry, Faculty of Medicine, Dresden, Germany
| | - Ilker Eyüpoglu
- TU Dresden, Else Kröner Fresenius Center for Digital Health, Germany
- University Hospital Carl Gustav Carus, TU Dresden, Department of Neurosurgery, Dresden, Germany
| | - Jürgen W. Czarske
- TU Dresden, Laboratory of Measurement and Sensor System Technique, Dresden, Germany
- TU Dresden, Competence Center BIOLAS, Dresden, Germany
- TU Dresden, Else Kröner Fresenius Center for Digital Health, Germany
- TU Dresden, Excellence Cluster Physics of Life, Dresden, Germany
- TU Dresden, School of Science, Faculty of Physics, Dresden, Germany
| | - Robert Kuschmierz
- TU Dresden, Laboratory of Measurement and Sensor System Technique, Dresden, Germany
- TU Dresden, Competence Center BIOLAS, Dresden, Germany
- TU Dresden, Else Kröner Fresenius Center for Digital Health, Germany
| |
Collapse
|
2
|
Du Y, Dylda E, Stibůrek M, Gomes AD, Turtaev S, Pakan JMP, Čižmár T. Advancing the path to in-vivo imaging in freely moving mice via multimode-multicore fiber based holographic endoscopy. NEUROPHOTONICS 2024; 11:S11506. [PMID: 38352728 PMCID: PMC10863504 DOI: 10.1117/1.nph.11.s1.s11506] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/30/2023] [Revised: 01/17/2024] [Accepted: 01/22/2024] [Indexed: 02/16/2024]
Abstract
Significance Hair-thin multimode optical fiber-based holographic endoscopes have gained considerable interest in modern neuroscience for their ability to achieve cellular and even subcellular resolution during in-vivo deep brain imaging. However, the application of multimode fibers in freely moving animals presents a persistent challenge as it is difficult to maintain optimal imaging performance while the fiber undergoes deformations. Aim We propose a fiber solution for challenging in-vivo applications with the capability of deep brain high spatial resolution imaging and neuronal activity monitoring in anesthetized as well as awake behaving mice. Approach We used our previously developed M 3 CF multimode-multicore fiber to record fluorescently labeled neurons in anesthetized mice. Our M 3 CF exhibits a cascaded refractive index structure, enabling two distinct regimes of light transport that imitate either a multimode or a multicore fiber. The M 3 CF has been specifically designed for use in the initial phase of an in-vivo experiment, allowing for the navigation of the endoscope's distal end toward the targeted brain structure. The multicore regime enables the transfer of light to and from each individual neuron within the field of view. For chronic experiments in awake behaving mice, it is crucial to allow for disconnecting the fiber and the animal between experiments. Therefore, we provide here an effective solution and establish a protocol for reconnection of two segments of M 3 CF with hexagonally arranged corelets. Results We successfully utilized the M 3 CF to image neurons in anaesthetized transgenic mice expressing enhanced green fluorescent protein. Additionally, we compared imaging results obtained with the M 3 CF with larger numerical aperture (NA) fibers in fixed whole-brain tissue. Conclusions This study focuses on addressing challenges and providing insights into the use of multimode-multicore fibers as imaging solutions for in-vivo applications. We suggest that the upcoming version of the M 3 CF increases the overall NA between the two cladding layers to allow for access to high resolution spatial imaging. As the NA increases in the multimode regime, the fiber diameter and ring structure must be reduced to minimize the computational burden and invasiveness.
Collapse
Affiliation(s)
- Yang Du
- University of Chinese Academy of Sciences, Hangzhou Institute for Advanced Study, Hangzhou, China
- Leibniz Institute of Photonic Technology, Jena, Germany
| | - Evelyn Dylda
- Otto-von-Guericke-University Magdeburg, Institute of Cognitive Neurology and Dementia Research, Magdeburg, Germany
- Center for Behavioral Brain Sciences, Magdeburg, Germany
| | | | - André D Gomes
- Leibniz Institute of Photonic Technology, Jena, Germany
| | | | - Janelle M. P. Pakan
- Otto-von-Guericke-University Magdeburg, Institute of Cognitive Neurology and Dementia Research, Magdeburg, Germany
- Center for Behavioral Brain Sciences, Magdeburg, Germany
- German Centre for Neurodegenerative Diseases, Magdeburg, Germany
- Leibniz Institute for Neurobiology, Magdeburg, Germany
| | - Tomáš Čižmár
- Leibniz Institute of Photonic Technology, Jena, Germany
- Institute of Scientific Instruments of CAS, Brno, Czechia
- Friedrich Schiller University Jena, Institute of Applied Optics, Jena, Germany
| |
Collapse
|
3
|
Song P, Wang R, Loetgering L, Liu J, Vouras P, Lee Y, Jiang S, Feng B, Maiden A, Yang C, Zheng G. Ptycho-endoscopy on a lensless ultrathin fiber bundle tip. LIGHT, SCIENCE & APPLICATIONS 2024; 13:168. [PMID: 39019852 PMCID: PMC11255264 DOI: 10.1038/s41377-024-01510-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/30/2024] [Revised: 06/10/2024] [Accepted: 06/25/2024] [Indexed: 07/19/2024]
Abstract
Synthetic aperture radar (SAR) utilizes an aircraft-carried antenna to emit electromagnetic pulses and detect the returning echoes. As the aircraft travels across a designated area, it synthesizes a large virtual aperture to improve image resolution. Inspired by SAR, we introduce synthetic aperture ptycho-endoscopy (SAPE) for micro-endoscopic imaging beyond the diffraction limit. SAPE operates by hand-holding a lensless fiber bundle tip to record coherent diffraction patterns from specimens. The fiber cores at the distal tip modulate the diffracted wavefield within a confined area, emulating the role of the 'airborne antenna' in SAR. The handheld operation introduces positional shifts to the tip, analogous to the aircraft's movement. These shifts facilitate the acquisition of a ptychogram and synthesize a large virtual aperture extending beyond the bundle's physical limit. We mitigate the influences of hand motion and fiber bending through a low-rank spatiotemporal decomposition of the bundle's modulation profile. Our tests demonstrate the ability to resolve a 548-nm linewidth on a resolution target. The achieved space-bandwidth product is ~1.1 million effective pixels, representing a 36-fold increase compared to that of the original fiber bundle. Furthermore, SAPE's refocusing capability enables imaging over an extended depth of field exceeding 2 cm. The aperture synthesizing process in SAPE surpasses the diffraction limit set by the probe's maximum collection angle, opening new opportunities for both fiber-based and distal-chip endoscopy in applications such as medical diagnostics and industrial inspection.
Collapse
Affiliation(s)
- Pengming Song
- Department of Biomedical Engineering, University of Connecticut, Storrs, CT, 06269, USA.
| | - Ruihai Wang
- Department of Biomedical Engineering, University of Connecticut, Storrs, CT, 06269, USA
| | - Lars Loetgering
- CarlZeiss AG, Carl Zeiss Promenade, Jena, Thuringia, 07745, Germany
| | - Jia Liu
- Department of Biomedical Engineering, University of Connecticut, Storrs, CT, 06269, USA
| | - Peter Vouras
- United States Department of Defense, Washington, DC, 20301, USA
| | - Yujin Lee
- School of Electrical and Electronic Engineering, Yonsei University, Seoul, 03722, Republic of Korea
| | - Shaowei Jiang
- Department of Biomedical Engineering, University of Connecticut, Storrs, CT, 06269, USA
| | - Bin Feng
- Department of Biomedical Engineering, University of Connecticut, Storrs, CT, 06269, USA
| | - Andrew Maiden
- Department of Electronic and Electrical Engineering, University of Sheffield, Sheffield, South Yorkshire S1 3JD, UK
- Diamond Light Source, Harwell, Oxfordshire, OX11 0DE, UK
| | - Changhuei Yang
- Department of Electrical Engineering, California Institute of Technology, Pasadena, CA, 91125, USA
| | - Guoan Zheng
- Department of Biomedical Engineering, University of Connecticut, Storrs, CT, 06269, USA.
- Center for Biomedical and Bioengineering Innovation, University of Connecticut, Storrs, CT, 06269, USA.
| |
Collapse
|
4
|
Wang J, Chen C, You W, Jiao Y, Liu X, Jiang X, Lu W. Honeycomb effect elimination in differential phase fiber-bundle-based endoscopy. OPTICS EXPRESS 2024; 32:20682-20694. [PMID: 38859444 DOI: 10.1364/oe.526033] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/10/2024] [Accepted: 05/10/2024] [Indexed: 06/12/2024]
Abstract
Fiber-bundle-based endoscopy, with its ultrathin probe and micrometer-level resolution, has become a widely adopted imaging modality for in vivo imaging. However, the fiber bundles introduce a significant honeycomb effect, primarily due to the multi-core structure and crosstalk of adjacent fiber cores, which superposes the honeycomb pattern image on the original image. To tackle this issue, we propose an iterative-free spatial pixel shifting (SPS) algorithm, designed to suppress the honeycomb effect and enhance real-time imaging performance. The process involves the creation of three additional sub-images by shifting the original image by one pixel at 0, 45, and 90 degree angles. These four sub-images are then used to compute differential maps in the x and y directions. By performing spiral integration on these differential maps, we reconstruct a honeycomb-free image with improved details. Our simulations and experimental results, conducted on a self-built fiber bundle-based endoscopy system, demonstrate the effectiveness of the SPS algorithm. SPS significantly improves the image quality of reflective objects and unlabeled transparent scattered objects, laying a solid foundation for biomedical endoscopic applications.
Collapse
|
5
|
Weinberg G, Kang M, Choi W, Choi W, Katz O. Ptychographic lensless coherent endomicroscopy through a flexible fiber bundle. OPTICS EXPRESS 2024; 32:20421-20431. [PMID: 38859424 DOI: 10.1364/oe.503963] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/22/2023] [Accepted: 04/18/2024] [Indexed: 06/12/2024]
Abstract
Conventional fiber-bundle-based endoscopes allow minimally invasive imaging through flexible multi-core fiber (MCF) bundles by placing a miniature lens at the distal tip and using each core as an imaging pixel. In recent years, lensless imaging through MCFs was made possible by correcting the core-to-core phase distortions pre-measured in a calibration procedure. However, temporally varying wavefront distortions, for instance, due to dynamic fiber bending, pose a challenge for such approaches. Here, we demonstrate a coherent lensless imaging technique based on intensity-only measurements insensitive to core-to-core phase distortions. We leverage a ptychographic reconstruction algorithm to retrieve the phase and amplitude profiles of reflective objects placed at a distance from the fiber tip, using as input a set of diffracted intensity patterns reflected from the object when the illumination is scanned over the MCF cores. Our approach thus utilizes an acquisition process equivalent to confocal microendoscopy, only replacing the single detector with a camera.
Collapse
|
6
|
Hughes MR, McCall C. Improved resolution in fiber bundle inline holographic microscopy using multiple illumination sources. BIOMEDICAL OPTICS EXPRESS 2024; 15:1500-1514. [PMID: 38495718 PMCID: PMC10942680 DOI: 10.1364/boe.516030] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/14/2023] [Revised: 01/17/2024] [Accepted: 01/23/2024] [Indexed: 03/19/2024]
Abstract
Recent work has shown that high-quality inline holographic microscopy images can be captured through fiber imaging bundles. Speckle patterns arising from modal interference within the bundle cores can be minimized by use of a partially-coherent optical source such as an LED delivered via a multimode fiber. This allows numerical refocusing of holograms from samples at working distances of up to approximately 1 mm from the fiber bundle before the finite coherence begins to degrade the lateral resolution. However, at short working distances the lateral resolution is limited not by coherence, but by sampling effects due to core-to-core spacing in the bundle. In this article we demonstrate that multiple shifted holograms can be combined to improve the resolution by a factor of two. The shifted holograms can be rapidly acquired by sequentially firing LEDs, which are each coupled to their own, mutually offset, illumination fiber. Following a one-time calibration, resolution-enhanced images are created in real-time at an equivalent net frame rate of up to 7.5 Hz. The resolution improvement is demonstrated quantitatively using a resolution target and qualitatively using mounted biological slides. At longer working distances, beyond 0.6 mm, the improvement is reduced as resolution becomes limited by the source spatial and temporal coherence.
Collapse
Affiliation(s)
- Michael R. Hughes
- Applied Optics Group, School of Physics and Astronomy, University of Kent, Canterbury, Kent, CT2 7NH, United Kingdom
| | - Callum McCall
- Applied Optics Group, School of Physics and Astronomy, University of Kent, Canterbury, Kent, CT2 7NH, United Kingdom
| |
Collapse
|
7
|
Sun J, Zhao B, Wang D, Wang Z, Zhang J, Koukourakis N, Czarske JW, Li X. Calibration-free quantitative phase imaging in multi-core fiber endoscopes using end-to-end deep learning. OPTICS LETTERS 2024; 49:342-345. [PMID: 38194563 DOI: 10.1364/ol.509772] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/23/2023] [Accepted: 12/08/2023] [Indexed: 01/11/2024]
Abstract
Quantitative phase imaging (QPI) through multi-core fibers (MCFs) has been an emerging in vivo label-free endoscopic imaging modality with minimal invasiveness. However, the computational demands of conventional iterative phase retrieval algorithms have limited their real-time imaging potential. We demonstrate a learning-based MCF phase imaging method that significantly reduced the phase reconstruction time to 5.5 ms, enabling video-rate imaging at 181 fps. Moreover, we introduce an innovative optical system that automatically generated the first, to the best of our knowledge, open-source dataset tailored for MCF phase imaging, comprising 50,176 paired speckles and phase images. Our trained deep neural network (DNN) demonstrates a robust phase reconstruction performance in experiments with a mean fidelity of up to 99.8%. Such an efficient fiber phase imaging approach can broaden the applications of QPI in hard-to-reach areas.
Collapse
|
8
|
Wakayama T, Higuchi Y, Kondo R, Mizutani Y, Higashiguchi T. Lensless single-fiber ghost imaging. APPLIED OPTICS 2023; 62:9559-9567. [PMID: 38108781 DOI: 10.1364/ao.507550] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/06/2023] [Accepted: 11/20/2023] [Indexed: 12/19/2023]
Abstract
We demonstrate lensless single-fiber ghost imaging, which allows illumination and collection using a single optical fiber without a transmission-type system. Speckle patterns with relative coincidence degrees of 0.14 were formed by image reconstruction using improved differential ghost imaging. Employing fiber with a diameter of 105 µm, we achieved a spatial resolution of 0.05 mm in an observing area of 9m m 2, at a working distance of 10 mm. Compared to a conventional neuroendoscope at a power density of 94m W/c m 2, our imaging could be realized by extremely weak illumination at a laser power density of 0.10m W/c m 2. Using our lensless single-fiber ghost imaging, with 30,000 speckle patterns and implementing a diffuser, we attained an average coincidence degree of 0.45.
Collapse
|
9
|
Collard L, Piscopo L, Pisano F, Zheng D, De Vittorio M, Pisanello F. Optimizing the internal phase reference to shape the output of a multimode optical fiber. PLoS One 2023; 18:e0290300. [PMID: 37682976 PMCID: PMC10490902 DOI: 10.1371/journal.pone.0290300] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2023] [Accepted: 08/04/2023] [Indexed: 09/10/2023] Open
Abstract
Pre-shaping light to achieve desired amplitude distributions at the tip of a multimode fiber (MMF) has emerged as a powerful method allowing a wide range of imaging techniques to be implemented at the distal facet. Such techniques rely on measuring the transmission matrix of the optically turbid waveguide which scrambles the coherent input light into an effectively random speckle pattern. Typically, this is done by measuring the interferogram between the output speckle and a reference beam. In recent years, an optical setup where the reference beam passes through the MMF has become an attractive configuration because of the high interferometric stability of the common optical path. However, the merits and drawbacks of an internal reference beam remain controversial. The measurement of the transmission matrix is known to depend on the choice of internal reference and has been reported to result in "blind spots" due to phase singularities of the reference beam. Here, we describe how the focussing efficiency of the calibration can be increased by several percent by optimising the choice of internal reference beam.
Collapse
Affiliation(s)
- Liam Collard
- Istituto Italiano di Tecnologia, Center for Biomolecular Nanotechnologies, Arnesano, Italy
- RAISE Ecosystem, Genova, Italy
| | - Linda Piscopo
- Istituto Italiano di Tecnologia, Center for Biomolecular Nanotechnologies, Arnesano, Italy
- Dipartimento di Ingegneria Dell’Innovazione, Università del Salento, Lecce, Italy
| | - Filippo Pisano
- Istituto Italiano di Tecnologia, Center for Biomolecular Nanotechnologies, Arnesano, Italy
- Department of Physics and Astronomy “G. Galilei”, University of Padova, Padova, Italy
| | - Di Zheng
- Istituto Italiano di Tecnologia, Center for Biomolecular Nanotechnologies, Arnesano, Italy
| | - Massimo De Vittorio
- Istituto Italiano di Tecnologia, Center for Biomolecular Nanotechnologies, Arnesano, Italy
- RAISE Ecosystem, Genova, Italy
- Dipartimento di Ingegneria Dell’Innovazione, Università del Salento, Lecce, Italy
| | - Ferruccio Pisanello
- Istituto Italiano di Tecnologia, Center for Biomolecular Nanotechnologies, Arnesano, Italy
- RAISE Ecosystem, Genova, Italy
| |
Collapse
|
10
|
Kang M, Choi W, Choi W, Choi Y. Fourier holographic endoscopy for imaging continuously moving objects. OPTICS EXPRESS 2023; 31:11705-11716. [PMID: 37155799 DOI: 10.1364/oe.482923] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/10/2023]
Abstract
Coherent fiber bundles are widely used for endoscopy, but conventional approaches require distal optics to form an object image and acquire pixelated information owing to the geometry of the fiber cores. Recently, holographic recording of a reflection matrix enables a bare fiber bundle to perform pixelation-free microscopic imaging as well as allows a flexible mode operation, because the random core-to-core phase retardations due to any fiber bending and twisting could be removed in situ from the recorded matrix. Despite its flexibility, the method is not suitable for a moving object because the fiber probe should remain stationary during the matrix recording to avoid the alteration of the phase retardations. Here, we acquire a reflection matrix of a Fourier holographic endoscope equipped with a fiber bundle and explore the effect of fiber bending on the recorded matrix. By removing the motion effect, we develop a method that can resolve the perturbation of the reflection matrix caused by a continuously moving fiber bundle. Thus, we demonstrate high-resolution endoscopic imaging through a fiber bundle, even when the fiber probe changes its shape along with the moving objects. The proposed method can be used for minimally invasive monitoring of behaving animals.
Collapse
|