Full-color structured illumination optical sectioning microscopy

Deep learning-assisted low-cost autofluorescence microscopy for rapid slide-free imaging with virtual histological staining

Slide-free imaging techniques have shown great promise in improving the histological workflow. For example, computational high-throughput autofluorescence microscopy by pattern illumination (CHAMP) has achieved high resolution with a long depth of field, which, however, requires a costly ultraviolet laser. Here, simply using a low-cost light-emitting diode (LED), we propose a deep learning-assisted framework of enhanced widefield microscopy, termed EW-LED, to generate results similar to CHAMP (the learning target). Comparing EW-LED and CHAMP, EW-LED reduces the cost by 85×, shortening the image acquisition time and computation time by 36× and 17×, respectively. This framework can be applied to other imaging modalities, enhancing widefield images for better virtual histology.

Easily scalable multi-color DMD-based structured illumination microscopy

Structured illumination microscopy (SIM) achieves super-resolution imaging using a series of phase-shifted sinusoidal illumination patterns to down-modulate high spatial-frequency information of samples. Digital micromirror devices (DMDs) have been increasingly used to generate SIM illumination patterns due to their high speed and moderate cost. However, a DMD micromirror array's blazed grating structure causes strong angular dispersion for different wavelengths of light, thus severely hampering its application in multicolor imaging. We developed a multi-color DMD-SIM setup that employs a diffraction grating to compensate the DMD's dispersion and demonstrate super-resolution SIM imaging of both fluorescent beads and live cells samples with four color channels. This simple but effective approach can be readily scaled to more color channels, thereby greatly expanding the application of SIM in the study of complex multi-component structures and dynamics in soft matter systems.

ML-SIM: universal reconstruction of structured illumination microscopy images using transfer learning

Structured illumination microscopy (SIM) has become an important technique for optical super-resolution imaging because it allows a doubling of image resolution at speeds compatible with live-cell imaging. However, the reconstruction of SIM images is often slow, prone to artefacts, and requires multiple parameter adjustments to reflect different hardware or experimental conditions. Here, we introduce a versatile reconstruction method, ML-SIM, which makes use of transfer learning to obtain a parameter-free model that generalises beyond the task of reconstructing data recorded by a specific imaging system for a specific sample type. We demonstrate the generality of the model and the high quality of the obtained reconstructions by application of ML-SIM on raw data obtained for multiple sample types acquired on distinct SIM microscopes. ML-SIM is an end-to-end deep residual neural network that is trained on an auxiliary domain consisting of simulated images, but is transferable to the target task of reconstructing experimental SIM images. By generating the training data to reflect challenging imaging conditions encountered in real systems, ML-SIM becomes robust to noise and irregularities in the illumination patterns of the raw SIM input frames. Since ML-SIM does not require the acquisition of experimental training data, the method can be efficiently adapted to any specific experimental SIM implementation. We compare the reconstruction quality enabled by ML-SIM with current state-of-the-art SIM reconstruction methods and demonstrate advantages in terms of generality and robustness to noise for both simulated and experimental inputs, thus making ML-SIM a useful alternative to traditional methods for challenging imaging conditions. Additionally, reconstruction of a SIM stack is accomplished in less than 200 ms on a modern graphics processing unit, enabling future applications for real-time imaging. Source code and ready-to-use software for the method are available at http://ML-SIM.github.io.

High-resolution and large field-of-view Fourier ptychographic microscopy and its applications in biomedicine

Fourier ptychographic microscopy (FPM) is a promising and fast-growing computational imaging technique with high resolution, wide field-of-view (FOV) and quantitative phase recovery, which effectively tackles the problems of phase loss, aberration-introduced artifacts, narrow depth-of-field and the trade-off between resolution and FOV in conventional microscopy simultaneously. In this review, we provide a comprehensive roadmap of microscopy, the fundamental principles, advantages, and drawbacks of existing imaging techniques, and the significant roles that FPM plays in the development of science. Since FPM is an optimization problem in nature, we discuss the framework and related work. We also reveal the connection of Euler's formula between FPM and structured illumination microscopy. We review recent advances in FPM, including the implementation of high-precision quantitative phase imaging, high-throughput imaging, high-speed imaging, three-dimensional imaging, mixed-state decoupling, and introduce the prosperous biomedical applications. We conclude by discussing the challenging problems and future applications. FPM can be extended to a kind of framework to tackle the phase loss and system limits in the imaging system. This insight can be used easily in speckle imaging, incoherent imaging for retina imaging, large-FOV fluorescence imaging, etc.

Full-color optically-sectioned imaging by wide-field microscopy via deep-learning

Wide-field microscopy (WFM) is broadly used in experimental studies of biological specimens. However, combining the out-of-focus signals with the in-focus plane reduces the signal-to-noise ratio (SNR) and axial resolution of the image. Therefore, structured illumination microscopy (SIM) with white light illumination has been used to obtain full-color 3D images, which can capture high SNR optically-sectioned images with improved axial resolution and natural specimen colors. Nevertheless, this full-color SIM (FC-SIM) has a data acquisition burden for 3D-image reconstruction with a shortened depth-of-field, especially for thick samples such as insects and large-scale 3D imaging using stitching techniques. In this paper, we propose a deep-learning-based method for full-color WFM, i.e., FC-WFM-Deep, which can reconstruct high-quality full-color 3D images with an extended optical sectioning capability directly from the FC-WFM z-stack data. Case studies of different specimens with a specific imaging system are used to illustrate this method. Consequently, the image quality achievable with this FC-WFM-Deep method is comparable to the FC-SIM method in terms of 3D information and spatial resolution, while the reconstruction data size is 21-fold smaller and the in-focus depth is doubled. This technique significantly reduces the 3D data acquisition requirements without losing detail and improves the 3D imaging speed by extracting the optical sectioning in the depth-of-field. This cost-effective and convenient method offers a promising tool to observe high-precision color 3D spatial distributions of biological samples.

Four-dimensional visualization of zebrafish cardiovascular and vessel dynamics by a structured illumination microscope with electrically tunable lens

We established a four-dimensional (4D) microscopy method using structured illumination for optical axial imaging with an electrically tunable lens. With its fast imaging capability, the dynamics of the cardiovascular system of the zebrafish and cerebral vessels were imaged based on the coverage of two stacks (25 layers) per second with lateral /axial resolutions of 0.6 µm and 1.8 µm, respectively. Time lapse imaging clearly shows the contractile-relaxation response of the beating heart at different cardiac phases and with different mobilities of blood vessels in different regions. This new 4D technique will facilitate in vivo imaging of organ function, generation, as well as drug responses in small-sized animals.

Quadrature demodulation in line-scan focal modulation microscopy for imaging three-dimensional zebrafish neural structure

Visualizing biological processes in neuroscience requires in vivo functional imaging at single-neuron resolution, high image acquisition speed and strong optical sectioning ability. However, due to light scattering of in tissue, very often conventional wide-field fluorescence microscopes are unable to resolve cells in the presence of a strong out-of-focus background. Line-scan focal modulation microscopy enables high temporal resolution and good optical sectioning ability at the same time. Here we demonstrate a quadrature demodulation method to extract the focal information with an extended frequency bandwidth and therefore higher spatial resolution. The performance of the demodulation scheme in line-scan focal modulation microscope has been evaluated by performing imaging experiments with fluorescence beads and zebrafish neural structure. Reduced background, reduced artifacts and more detailed morphological information are evident in the obtained images.

Artifact-free objective-type multicolor total internal reflection fluorescence microscopy with light-emitting diode light sources-Part I

Total internal reflection fluorescence excitation (TIRF) microscopy allows the selective observation of fluorescent molecules in immediate proximity to an interface between different refractive indices. Objective-type or prism-less TIRF excitation is typically achieved with laser light sources. We here propose a simple, yet optically advantageous light-emitting diode (LED)-based implementation of objective-type TIRF (LED-TIRF). The proposed LED-TIRF condenser is affordable and easy to set up at any epifluorescence microscope to perform multicolor TIRF and/or combined TIRF-epifluorescence imaging with even illumination of the entire field of view. Electrical control of LED light sources replaces mechanical shutters or optical modulators. LED-TIRF microscopy eliminates safety burdens that are associated with laser sources, offers favorable instrument lifetime and stability without active cooling. The non-coherent light source and the type of projection eliminate interference fringing and local scattering artifacts that are associated with conventional laser-TIRF. Unlike azimuthal spinning laser-TIRF, LED-TIRF does not require synchronization between beam rotation and the camera and can be monitored with either global or rolling shutter cameras. Typical implementations, such as live cell multicolor imaging in TIRF and epifluorescence of imaging of short-lived, localized translocation events of a Ca²⁺ -sensitive protein kinase C α fusion protein are demonstrated.

Multi-view object topography measurement with optical sectioning structured illumination microscopy

Various optical instruments have been developed for three-dimensional (3D) surface topography, including the white light interference, reflectance confocal microscopes, and digital holographic microscopes, etc. However, the steep local slope of objects may cause the light to be reflected in a way that it will not be captured by the objective lens because of the finite collection angle of the objective. To solve this "shadow problem," we report a method to enlarge the collection angle range of optical sectioning structured illumination microscopy by capturing sectioned images of the objects from multiple angle of views. We develop a multi-view image fusion algorithm to reconstruct a single 3D image. Using this method, we detect previously invisible details whose slopes are beyond the collection angle of the objective. The proposed approach is useful for height map measurement and quantitative analyses in a variety of fields, such as biology, materials science, microelectronics, etc.

Low-loss metal-dielectric waveguide mode enabled structured illumination microscopy with 0.18λ0 resolution

Structured illumination microscopy (SIM) is a powerful super-resolved imaging technique which enables to perform fast and in vivo imaging of bio-samples. In order to achieve a better resolution of a SIM system, evanescent waves with larger in-plane wave-vector are preferred for SIM, among which the total internal reflection (TIRF-SIM) and the plasmonic SIM (pSIM) configurations are widely studied. Here, we demonstrated a metal-dielectric waveguide (MDW) based SIM system - termed as MDW-SIM, which can achieve a good compromise between TIRF-SIM and pSIM. The MDW can support a low-loss waveguide mode at an aqueous environment, with an evanescent tail existing above the water/dielectric interface for SIM. A proof-of-concept imaging experiment was performed on fluorescent beads, where a spatial resolution of 86nm was achieved at a 473nm illumination wavelength and a 1.45 numerical aperture objective lens. The proposed MDW-SIM has a great potential for the bio-imaging applications.

Large-scale 3D imaging of insects with natural color

High-resolution 3D imaging technology has found a number of applications in many biological fields. However, the existing 3D imaging tools are often too time-consuming to use on large-scale specimens, such as centimeter-sized insects. In addition, most 3D imaging systems discard the natural color information of the specimens. To surmount these limitations, we present a structured illumination-based approach capable of delivering large field-of-view three-dimensional images. With this approach, 580nm lateral resolution full-color 3D images and 3D morphological data in the size range of typical insect samples can be obtained. This method provides a promising approach that can be used to support many different types of entomological investigations, including taxonomy, evolution, bionics, developmental biology, functional morphology, paleontology, forestry, etc.

Structured illumination imaging without grating rotation based on mirror operation on 1D Fourier spectrum

Structured illumination microscopy (SIM) is a rapidly developing a super-resolution optical microscopy technique. With SIM, the grating is needed in order to rotate several angles for illuminating the sample in different directions. Multiple rotations reduce the imaging speed and grating rotation angle errors damage the image recovery quality. We introduce mirror transformation on one-dimension (1D) Fourier spectrum to SIM for resolving the problems of low imaging speed and severe impact on image reconstruction quality by grating rotation angle errors. When mirror operation and SIM are combined, the grating is placed at an orientation for obtaining three shadow images. The three shadow images are acquired by CCD at three different phase shift for a direction of grating. Thus, the SIM imaging speed is faster and the effect on image reconstruction quality by grating rotation angle errors is greatly reduced.

Computational coherent imaging by rotating a cylindrical lens

To overcome longitudinal sampling rate fluctuation in axial multi-image computational imaging, an effortless and high-efficient optical scanning imaging system via the rotation of single cylindrical lens (RSCL) is proposed for reconstructing the amplitude and phase information of sample. Here the cylinder is a non-axial-symmetry phase modulator for generating diffracted intensity patterns. To determine the explicit rotation angle as a core parameter in the perfect reconstruction of the light field, two-step Radon transform (TsRT) is designed to obtain accurate rotation status of the cylindrical lens with diffraction patterns at the focal plane. All rotation operations are at a lateral plane for keeping the same scale in this multiple parameter computational imaging system. As a kind of scanning imaging approach, the experiment of RSCL is greatly simplified.

Airborne Optical Sectioning

Drones are becoming increasingly popular for remote sensing of landscapes in archeology, cultural heritage, forestry, and other disciplines. They are more efficient than airplanes for capturing small areas, of up to several hundred square meters. LiDAR (light detection and ranging) and photogrammetry have been applied together with drones to achieve 3D reconstruction. With airborne optical sectioning (AOS), we present a radically different approach that is based on an old idea: synthetic aperture imaging. Rather than measuring, computing, and rendering 3D point clouds or triangulated 3D meshes, we apply image-based rendering for 3D visualization. In contrast to photogrammetry, AOS does not suffer from inaccurate correspondence matches and long processing times. It is cheaper than LiDAR, delivers surface color information, and has the potential to achieve high sampling resolutions. AOS samples the optical signal of wide synthetic apertures (30–100 m diameter) with unstructured video images recorded from a low-cost camera drone to support optical sectioning by image integration. The wide aperture signal results in a shallow depth of field and consequently in a strong blur of out-of-focus occluders, while images of points in focus remain clearly visible. Shifting focus computationally towards the ground allows optical slicing through dense occluder structures (such as leaves, tree branches, and coniferous trees), and discovery and inspection of concealed artifacts on the surface.

Interactive floating full-parallax digital three-dimensional light-field display based on wavefront recomposing

Advanced three-dimensional (3D) imaging techniques can acquire high-resolution 3D biomedical and biological data, but available digital display methods show this data in restricted two dimensions. 3D light-field displays optically reconstruct realistic 3D image by carefully tailoring light fields, and a natural and comfortable 3D sense of real objects or scenes is expected. An interactive floating full-parallax 3D light-field display with all depth cues is demonstrated with 3D biomedical and biological data, which are capable of achieving high efficiency and high image quality. A compound lens-array with two pieces of lens in each lens unit is designed and fabricated to suppress the aberrations and increase the viewing angle. The optimally designed holographic functional screen is used to recompose the light distribution from the lens-array. The imaging distortion can be decreased to less than 1.9% from more than 20%. The real time interactive floating full-parallax 3D light-field image with the clear displayed depth of 30 cm can be perceived with the right geometric occlusion and smooth parallax in the viewing angle of 45°, where 9216 viewpoints are used.

Development of Three-Dimensional Dental Scanning Apparatus Using Structured Illumination

We demonstrated a three-dimensional (3D) dental scanning apparatus based on structured illumination. A liquid lens was used for tuning focus and a piezomotor stage was used for the shift of structured light. A simple algorithm, which detects intensity modulation, was used to perform optical sectioning with structured illumination. We reconstructed a 3D point cloud, which represents the 3D coordinates of the digitized surface of a dental gypsum cast by piling up sectioned images. We performed 3D registration of an individual 3D point cloud, which includes alignment and merging the 3D point clouds to exhibit a 3D model of the dental cast.

Image recombination transform algorithm for superresolution structured illumination microscopy

Structured illumination microscopy (SIM) is an attractive choice for fast superresolution imaging. The generation of structured illumination patterns made by interference of laser beams is broadly employed to obtain high modulation depth of patterns, while the polarizations of the laser beams must be elaborately controlled to guarantee the high contrast of interference intensity, which brings a more complex configuration for the polarization control. The emerging pattern projection strategy is much more compact, but the modulation depth of patterns is deteriorated by the optical transfer function of the optical system, especially in high spatial frequency near the diffraction limit. Therefore, the traditional superresolution reconstruction algorithm for interference-based SIM will suffer from many artifacts in the case of projection-based SIM that possesses a low modulation depth. Here, we propose an alternative reconstruction algorithm based on image recombination transform, which provides an alternative solution to address this problem even in a weak modulation depth. We demonstrated the effectiveness of this algorithm in the multicolor superresolution imaging of bovine pulmonary arterial endothelial cells in our developed projection-based SIM system, which applies a computer controlled digital micromirror device for fast fringe generation and multicolor light-emitting diodes for illumination. The merit of the system incorporated with the proposed algorithm allows for a low excitation intensity fluorescence imaging even less than 1??W/cm2, which is beneficial for the long-term, in vivo superresolved imaging of live cells and tissues.