Video-rate multi-color structured illumination microscopy with simultaneous real-time reconstruction

Fast volumetric multifocus structured illumination microscopy of subcellular dynamics in living cells

Studying the nanoscale dynamics of subcellular structures is possible with 2D structured illumination microscopy (SIM). The method allows for acquisition with improved resolution over typical widefield. For 3D samples, the acquisition speed is inherently limited by the need to acquire sequential two-dimensional planes to create a volume. Here, we present a development of multifocus SIM designed to provide high volumetric frame rate by using fast synchronized electro-optical components. We demonstrate the high volumetric imaging capacity of the microscope by recording the dynamics of microtubule and endoplasmatic reticulum in living cells at up to 2.3 super resolution volumes per second for a total volume of 30 × 30 × 1.8 µm3.

Facile Conversion and Optimization of Structured Illumination Image Reconstruction Code into the GPU Environment

Superresolution, structured illumination microscopy (SIM) is an ideal modality for imaging live cells due to its relatively high speed and low photon-induced damage to the cells. The rate-limiting step in observing a superresolution image in SIM is often the reconstruction speed of the algorithm used to form a single image from as many as nine raw images. Reconstruction algorithms impose a significant computing burden due to an intricate workflow and a large number of often complex calculations to produce the final image. Further adding to the computing burden is that the code, even within the MATLAB environment, can be inefficiently written by microscopists who are noncomputer science researchers. In addition, they do not take into consideration the processing power of the graphics processing unit (GPU) of the computer. To address these issues, we present simple but efficient approaches to first revise MATLAB code, followed by conversion to GPU-optimized code. When combined with cost-effective, high-performance GPU-enabled computers, a 4- to 500-fold improvement in algorithm execution speed is observed as shown for the image denoising Hessian-SIM algorithm. Importantly, the improved algorithm produces images identical in quality to the original.

Optical tomography in a single camera frame using fringe-encoded deep-learning full-field OCT

Optical coherence tomography is a valuable tool for in vivo examination thanks to its superior combination of axial resolution, field-of-view and working distance. OCT images are reconstructed from several phases that are obtained by modulation/multiplexing of light wavelength or optical path. This paper shows that only one phase (and one camera frame) is sufficient for en face tomography. The idea is to encode a high-frequency fringe patterns into the selected layer of the sample using low-coherence interferometry. These patterns can then be efficiently extracted with a high-pass filter enhanced via deep learning networks to create the tomographic full-field OCT view. This brings 10-fold improvement in imaging speed, considerably reducing the phase errors and incoherent light artifacts related to in vivo movements. Moreover, this work opens a path for low-cost tomography with slow consumer cameras. Optically, the device resembles the conventional time-domain full-field OCT without incurring additional costs or a field-of-view/resolution reduction. The approach is validated by imaging in vivo cornea in human subjects. Open-source and easy-to-follow codes for data generation/training/inference with U-Net/Pix2Pix networks are provided to be used in a variety of image-to-image translation tasks.

Easily scalable multi-color DMD-based structured illumination microscopy

Structured illumination microscopy (SIM) achieves super-resolution imaging using a series of phase-shifted sinusoidal illumination patterns to down-modulate high spatial-frequency information of samples. Digital micromirror devices (DMDs) have been increasingly used to generate SIM illumination patterns due to their high speed and moderate cost. However, a DMD micromirror array's blazed grating structure causes strong angular dispersion for different wavelengths of light, thus severely hampering its application in multicolor imaging. We developed a multi-color DMD-SIM setup that employs a diffraction grating to compensate the DMD's dispersion and demonstrate super-resolution SIM imaging of both fluorescent beads and live cells samples with four color channels. This simple but effective approach can be readily scaled to more color channels, thereby greatly expanding the application of SIM in the study of complex multi-component structures and dynamics in soft matter systems.

Probing the dynamic crosstalk of lysosomes and mitochondria with structured illumination microscopy

Structured illumination microscopy (SIM) is a super-resolution technology for imaging living cells and has been used for studying the dynamics of lysosomes and mitochondria. Recently, new probes and analyzing methods have been developed for SIM imaging, enabling the quantitative analysis of these subcellular structures and their interactions. This review provides an overview of the working principle and advances of SIM, as well as the organelle-targeting principles and types of fluorescence probes, including small molecules, metal complexes, nanoparticles, and fluorescent proteins. Additionally, quantitative methods based on organelle morphology and distribution are outlined. Finally, the review provides an outlook on the current challenges and future directions for improving the combination of SIM imaging and image analysis to further advance the study of organelles. We hope that this review will be useful for researchers working in the field of organelle research and help to facilitate the development of SIM imaging and analysis techniques.

Grazing incidence to total internal reflection fluorescence structured illumination microscopy enabled by a prism telescope

In super-resolution structured illumination microscopy (SR-SIM) the separation between opposing laser spots in the back focal plane of the objective lens affects the pattern periodicity, and, thus, the resulting spatial resolution. Here, we introduce a novel hexagonal prism telescope which allows us to seamlessly change the separation between parallel laser beams for 3 pairs of beams, simultaneously. Each end of the prism telescope is composed of 6 Littrow prisms, which are custom-ground so they can be grouped together in the form of a tight hexagon. By changing the distance between the hexagons, the beam separation can be adjusted. This allows us to easily control the position of opposing laser spots in the back focal plane and seamlessly adjust the spatial frequency of the resulting interference pattern. This also enables the seamless transition from 2D-SIM to total internal reflection fluorescence (TIRF) excitation using objective lenses with a high numerical aperture. In linear SR-SIM the highest spatial resolution can be achieved for extreme TIRF angles. The prism telescope allows us to investigate how the spatial resolution and contrast depend on the angle of incidence near, at, and beyond the critical angle. We demonstrate this by imaging the cytoskeleton and plasma membrane of liver sinusoidal endothelial cells, which have a characteristic morphology consisting of thousands of small, transcellular pores that can only be observed by super-resolution microscopy.

Dependence of ghost on the incident light angle into dichroic mirror

The dichroic mirror (DM) is a key component in microscope. We found a ghost in the reflection channel of a dual-channel fluorescence microscope and studied the relationship between the ghost and the incidence angle θ into the DM. The DM emission surface reflection generated ghost if the θ is not45 ° . We analyzed the distance and intensity relationship between the ghost and the primary image, which is θ -dependent and was demonstrated by imaging live cells and a stage micrometer. The ghost can be eliminated by placing the DM between objective and tube lens, but not between tube lens and detector, ensuring that the incident light into the DM is approximately parallel. Furthermore, the transmitted light of the DM is shifted towards a longer wavelength with increasing θ . Collectively, microscopists must carefully optimize the θ when designing a microscope to avoid the ghost.

Xanthene, cyanine, oxazine and BODIPY: the four pillars of the fluorophore empire for super-resolution bioimaging

In the realm of biological research, the invention of super-resolution microscopy (SRM) has enabled the visualization of ultrafine sub-cellular structures and their functions in live cells at the nano-scale level, beyond the diffraction limit, which has opened up a new window for advanced biomedical studies to unravel the complex unknown details of physiological disorders at the sub-cellular level with unprecedented resolution and clarity. However, most of the SRM techniques are highly reliant on the personalized special photophysical features of the fluorophores. In recent times, there has been an unprecedented surge in the development of robust new fluorophore systems with personalized features for various super-resolution imaging techniques. To date, xanthene, cyanine, oxazine and BODIPY cores have been authoritatively utilized as the basic fluorophore units in most of the small-molecule-based organic fluorescent probe designing strategies for SRM owing to their excellent photophysical characteristics and easy synthetic acquiescence. Since the future of next-generation SRM studies will be decided by the availability of advanced fluorescent probes and these four fluorescent building blocks will play an important role in progressive new fluorophore design, there is an urgent need to review the recent advancements in designing fluorophores for different SRM methods based on these fluorescent dye cores. This review article not only includes a comprehensive discussion about the recent developments in designing fluorescent probes for various SRM techniques based on these four important fluorophore building blocks with special emphasis on their effective integration into live cell super-resolution bio-imaging applications but also critically evaluates the background of each of the fluorescent dye cores to highlight their merits and demerits towards developing newer fluorescent probes for SRM.

Simultaneous Multicolor Multifocal Scanning Microscopy

Super-resolution fluorescence microscopy has revolutionized cell biology over the past decade, enabling the visualization of subcellular complexity with unparalleled clarity and detail. However, the rapid development of image-scanning-based super-resolution systems still restrains convenient access to commonly used instruments such as epi-fluorescence microscopes. Here, we present multifocal scanning microscopy (MSM) for super-resolution imaging with simultaneous multicolor acquisition and minimal instrumental complexity. MSM implements a stationary, interposed multifocal multicolor excitation by exploiting the motion of the specimens, realizing super-resolution microscopy through a general epi-fluorescence platform without compromising the image-scanning mechanism or inducing complex instrument alignment. The system is demonstrated with various phantom and biological specimens, and the results present effective resolution doubling, optical sectioning, and contrast enhancement. We anticipate MSM, as a highly accessible and compatible super-resolution technique, to offer a promising methodological pathway for broad cell biological discoveries.

High-speed TIRF and 2D super-resolution structured illumination microscopy with a large field of view based on fiber optic components

Super-resolved structured illumination microscopy (SR-SIM) is among the most flexible, fast, and least perturbing fluorescence microscopy techniques capable of surpassing the optical diffraction limit. Current custom-built instruments are easily able to deliver two-fold resolution enhancement at video-rate frame rates, but the cost of the instruments is still relatively high, and the physical size of the instruments based on the implementation of their optics is still rather large. Here, we present our latest results towards realizing a new generation of compact, cost-efficient, and high-speed SR-SIM instruments. Tight integration of the fiber-based structured illumination microscope capable of multi-color 2D- and TIRF-SIM imaging, allows us to demonstrate SR-SIM with a field of view of up to 150 × 150 µm2 and imaging rates of up to 44 Hz while maintaining highest spatiotemporal resolution of less than 100 nm. We discuss the overall integration of optics, electronics, and software that allowed us to achieve this, and then present the fiberSIM imaging capabilities by visualizing the intracellular structure of rat liver sinusoidal endothelial cells, in particular by resolving the structure of their trans-cellular nanopores called fenestrations.

Projective oblique plane structured illumination microscopy

Structured illumination microscopy (SIM) can double the spatial resolution of a fluorescence microscope and video rate live cell imaging in a two-dimensional format has been demonstrated. However, rapid implementations of 2D SIM typically only cover a narrow slice of the sample immediately at the coverslip, with most of the cellular volume out of reach. Here we implement oblique plane structured illumination microscopy (OPSIM) in a projection format to rapidly image an entire cell in a 2D SIM framework. As no mechanical scanning of the sample or objective is involved, this technique has the potential for rapid projection imaging with doubled resolution. We characterize the spatial resolution with fluorescent nanospheres, compare projection and 3D imaging using OPSIM and image mitochondria and ER dynamics across an entire cell at up to 2.7 Hz. To our knowledge, this represents the fastest whole cell SIM imaging to date.

Superresolution structured illumination microscopy reconstruction algorithms: a review

Structured illumination microscopy (SIM) has become the standard for next-generation wide-field microscopy, offering ultrahigh imaging speed, superresolution, a large field-of-view, and long-term imaging. Over the past decade, SIM hardware and software have flourished, leading to successful applications in various biological questions. However, unlocking the full potential of SIM system hardware requires the development of advanced reconstruction algorithms. Here, we introduce the basic theory of two SIM algorithms, namely, optical sectioning SIM (OS-SIM) and superresolution SIM (SR-SIM), and summarize their implementation modalities. We then provide a brief overview of existing OS-SIM processing algorithms and review the development of SR-SIM reconstruction algorithms, focusing primarily on 2D-SIM, 3D-SIM, and blind-SIM. To showcase the state-of-the-art development of SIM systems and assist users in selecting a commercial SIM system for a specific application, we compare the features of representative off-the-shelf SIM systems. Finally, we provide perspectives on the potential future developments of SIM.

Confocal Laser Scanning Microscope Imaging of Custom-Made Multi-Cylinder Phantoms: Theory and Experiment

In this work, the image formation in a confocal laser scanning microscope (CLSM) is investigated for custom-made multi-cylinder phantoms. The cylinder structures were fabricated using 3D direct laser writing and consist of parallel cylinders with radii of 5 and 10 μm for the respective multi-cylinder phantom, with overall dimensions of about 200×200×200 μm3. Measurements were performed for different refractive index differences and by varying other parameters of the measurement system, such as pinhole size or numerical aperture (NA). For theoretical comparison, the confocal setup was implemented in an in-house developed tetrahedron-based and GPU-accelerated Monte Carlo (MC) software. The simulation results for a cylindrical single scatterer were first compared with the analytical solution of Maxwell's equations in two dimensions for prior validation. Subsequently, the more complex multi-cylinder structures were simulated using the MC software and compared with the experimental results. For the largest refractive index difference, i.e., air as the surrounding medium, the simulated and measured data show a high degree of agreement, with all the key features of the CLSM image being reproduced by the simulation. Even with a significant reduction in the refractive index difference by the use of immersion oil to values as low as 0.005, a good agreement between simulation and measurement was observed, particularly with respect to the increase in penetration depth.

High-speed spatially re-modulated structured illumination microscopy

Structured illumination microscopy (SIM) allows non-invasive visualization of nanoscale subcellular structures. However, image acquisition and reconstruction become the bottleneck to further improve the imaging speed. Here, we propose a method to accelerate SIM imaging by combining the spatial re-modulation principle with Fourier domain filtering and using measured illumination patterns. This approach enables high-speed, high-quality imaging of dense subcellular structures using a conventional nine-frame SIM modality without phase estimation of the patterns. In addition, seven-frame SIM reconstruction and additional hardware acceleration further improve the imaging speed using our method. Furthermore, our method is also applicable to other spatially uncorrelated illumination patterns, such as distorted sinusoidal, multifocal, and speckle patterns.

Rapid, artifact-reduced, image reconstruction for super-resolution structured illumination microscopy

Super-resolution structured illumination microscopy (SR-SIM) is finding increasing application in biomedical research due to its superior ability to visualize subcellular dynamics in living cells. However, during image reconstruction artifacts can be introduced and when coupled with time-consuming postprocessing procedures, limits this technique from becoming a routine imaging tool for biologists. To address these issues, an accelerated, artifact-reduced reconstruction algorithm termed joint space frequency reconstruction-based artifact reduction algorithm (JSFR-AR-SIM) was developed by integrating a high-speed reconstruction framework with a high-fidelity optimization approach designed to suppress the sidelobe artifact. Consequently, JSFR-AR-SIM produces high-quality, super-resolution images with minimal artifacts, and the reconstruction speed is increased. We anticipate this algorithm to facilitate SR-SIM becoming a routine tool in biomedical laboratories.

Super-resolution fluorescence microscopic imaging in pathogenesis and drug treatment of neurological disease

Since super-resolution fluorescence microscopic technology breaks the diffraction limit that has existed for a long time in optical imaging, it can observe the process of synapses formed between nerve cells and the protein aggregation related to neurological disease. Thus, super-resolution fluorescence microscopic imaging has significantly impacted several industries, including drug development and pathogenesis research, and it is anticipated that it will significantly alter the future of life science research. Here, we focus on several typical super-resolution fluorescence microscopic technologies, introducing their benefits and drawbacks, as well as applications in several common neurological diseases, in the hope that their services will be expanded and improved in the pathogenesis and drug treatment of neurological diseases.

Single Molecule Localization Microscopy for Studying Small Extracellular Vesicles

Small extracellular vesicles (sEVs) are 30-200 nm nanovesicles enriched with unique cargoes of nucleic acids, lipids, and proteins. sEVs are released by all cell types and have emerged as a critical mediator of cell-to-cell communication. Although many studies have dealt with the role of sEVs in health and disease, the exact mechanism of sEVs biogenesis and uptake remain unexplored due to the lack of suitable imaging technologies. For sEVs functional studies, imaging has long relied on conventional fluorescence microscopy that has only 200-300 nm resolution, thereby generating blurred images. To break this resolution limit, recent developments in super-resolution microscopy techniques, specifically single-molecule localization microscopy (SMLM), expanded the understanding of subcellular details at the few nanometer level. SMLM success relies on the use of appropriate fluorophores with excellent blinking properties. In this review, the basic principle of SMLM is highlighted and the state of the art of SMLM use in sEV biology is summarized. Next, how SMLM techniques implemented for cell imaging can be translated to sEV imaging is discussed by applying different labeling strategies to study sEV biogenesis and their biomolecular interaction with the distant recipient cells.

Parameter estimation of the structured illumination pattern based on principal component analysis (PCA): PCA-SIM

Principal component analysis (PCA), a common dimensionality reduction method, is introduced into SIM to identify the frequency vectors and pattern phases of the illumination pattern with precise subpixel accuracy, fast speed, and noise-robustness, which is promising for real-time and long-term live-cell imaging.

Transmission structured illumination microscopy with tunable frequency illumination using tilt mirror assembly

We present experimental demonstration of tilt-mirror assisted transmission structured illumination microscopy (tSIM) that offers a large field of view super resolution imaging. An assembly of custom-designed tilt-mirrors are employed as the illumination module where the sample is excited with the interference of two beams reflected from the opposite pair of mirror facets. Tunable frequency structured patterns are generated by changing the mirror-tilt angle and the hexagonal-symmetric arrangement is considered for the isotropic resolution in three orientations. Utilizing high numerical aperture (NA) objective in standard SIM provides super-resolution compromising with the field-of-view (FOV). Employing low NA (20X/0.4) objective lens detection, we experimentally demonstrate [Formula: see text] (0.56 mm[Formula: see text]0.35 mm) size single FOV image with [Formula: see text]1.7- and [Formula: see text]2.4-fold resolution improvement (exploiting various illumination by tuning tilt-mirrors) over the diffraction limit. The results are verified both for the fluorescent beads as well as biological samples. The tSIM geometry decouples the illumination and the collection light paths consequently enabling free change of the imaging objective lens without influencing the spatial frequency of the illumination pattern that are defined by the tilt-mirrors. The large and scalable FOV supported by tSIM will find usage for applications where scanning large areas are necessary as in pathology and applications where images must be correlated both in space and time.

Surpassing the resolution limitation of structured illumination microscopy by an untrained neural network

Structured illumination microscopy (SIM), as a flexible tool, has been widely applied to observing subcellular dynamics in live cells. It is noted, however, that SIM still encounters a problem with theoretical resolution limitation being only twice over wide-field microscopy, where imaging of finer biological structures and dynamics are significantly constrained. To surpass the resolution limitation of SIM, we developed an image postprocessing method to further improve the lateral resolution of SIM by an untrained neural network, i.e., deep resolution-enhanced SIM (DRE-SIM). DRE-SIM can further extend the spatial frequency components of SIM by employing the implicit priors based on the neural network without training datasets. The further super-resolution capability of DRE-SIM is verified by theoretical simulations as well as experimental measurements. Our experimental results show that DRE-SIM can achieve the resolution enhancement by a factor of about 1.4 compared with conventional SIM. Given the advantages of improving the lateral resolution while keeping the imaging speed, DRE-SIM will have a wide range of applications in biomedical imaging, especially when high-speed imaging mechanisms are integrated into the conventional SIM system.

Miniaturized structured illumination microscopy using two 3-axis MEMS micromirrors

We present the development and performance characterisation of a novel structured illumination microscope (SIM) in which the grating pattern is generated using two optical beams controlled via 2 micro-electro-mechanical system (MEMS) three-axis scanning micromirrors. The implementation of MEMS micromirrors to accurately and repeatably control angular, radial and phase positioning delivers flexible control of the fluorescence excitation illumination, with achromatic beam delivery through the same optical path, reduced spatial footprint and cost-efficient integration being further benefits. Our SIM architecture enables the direct implementation of multi-color imaging in a compact and adaptable package. The two-dimensional SIM system approach is enabled by a pair of 2 mm aperture electrostatically actuated three-axis micromirrors having static angular tilt motion along the x- and y-axes and static piston motion along the z-axis. This allows precise angular, radial and phase positioning of two optical beams, generating a fully controllable spatial interference pattern at the focal plane by adjusting the positions of the beam in the back-aperture of a microscope objective. This MEMS-SIM system was applied to fluorescent bead samples and cell specimens, and was able to obtain a variable lateral resolution improvement between 1.3 and 1.8 times the diffraction limited resolution.

Fluorescence microscopy imaging of a neurotransmitter receptor and its cell membrane lipid milieu

Hampered by the diffraction phenomenon, as expressed in 1873 by Abbe, applications of optical microscopy to image biological structures were for a long time limited to resolutions above the ∼200 nm barrier and restricted to the observation of stained specimens. The introduction of fluorescence was a game changer, and since its inception it became the gold standard technique in biological microscopy. The plasma membrane is a tenuous envelope of 4 nm-10 nm in thickness surrounding the cell. Because of its highly versatile spectroscopic properties and availability of suitable instrumentation, fluorescence techniques epitomize the current approach to study this delicate structure and its molecular constituents. The wide spectral range covered by fluorescence, intimately linked to the availability of appropriate intrinsic and extrinsic probes, provides the ability to dissect membrane constituents at the molecular scale in the spatial domain. In addition, the time resolution capabilities of fluorescence methods provide complementary high precision for studying the behavior of membrane molecules in the time domain. This review illustrates the value of various fluorescence techniques to extract information on the topography and motion of plasma membrane receptors. To this end I resort to a paradigmatic membrane-bound neurotransmitter receptor, the nicotinic acetylcholine receptor (nAChR). The structural and dynamic picture emerging from studies of this prototypic pentameric ligand-gated ion channel can be extrapolated not only to other members of this superfamily of ion channels but to other membrane-bound proteins. I also briefly discuss the various emerging techniques in the field of biomembrane labeling with new organic chemistry strategies oriented to applications in fluorescence nanoscopy, the form of fluorescence microscopy that is expanding the depth and scope of interrogation of membrane-associated phenomena.

Cost-effective high-speed, three-dimensional live-cell imaging of HIV-1 transfer at the T cell virological synapse

The availability of cost-effective, highly portable, and easy to use high-resolution live-cell imaging systems could present a significant technological break-through in challenging environments, such as high-level biosafety laboratories or sites where new viral outbreaks are suspected. We describe and demonstrate a cost-effective high-speed fluorescence microscope enabling the live tracking of virus particles across virological synapses that form between infected and uninfected T cells. The dynamics of HIV-1 proteins studied at the cellular level and the formation of virological synapses in living T cells reveals mechanisms by which cell-cell interactions facilitate infection between immune cells. Dual-color 3D fluorescence deconvolution microscopy of HIV-1 particles at frames rates of 100 frames per second allows us to follow the transfer of HIV-1 particles across the T cell virological synapse between living T cells. We also confirm the successful transfer of virus by imaging T cell samples fixed at specific time points during cell-cell virus transfer by super-resolution structured illumination microscopy.

Compact structured illumination microscopy with high spatial frequency diffractive lattice patterns

Structured illumination microscopy (SIM) enables live-cell super-resolution imaging with wide field of view (FOV) and high imaging speed, but the illumination system is usually bulky. With the advantages of small structure and high efficiency, lattice patterns assisted by diffractive optical elements (DOEs) have been used for structured illumination in SIM. But it is still challenging to raise the spatial frequency of diffractive lattice patterns when using traditional DOE design method, and thus the super-resolution imaging performance is restricted. In this paper, we propose a novel design method for DOE to generate lattice patterns with spatial frequency close to the cut-off frequency. It is the first time to obtain a lattice pattern with such high spatial frequency by diffractive optics. Finally, the proposed SIM achieves a lateral resolution of 131 nm at 519 nm fluorescent light while maintaining an original size as a standard inverted fluorescence microscope by only inserting a single well-designed DOE in the illumination optical path, which may promote this compact SIM applied in super-resolution imaging field.

An ultra-small nine-color spectrometer with a two-layer biparted ten-dichroic-mirror array and an image sensor

An ultra-small (54 × 58 × 8.5 mm) and large aperture (1 × 7 mm) nine-color spectrometer-using an array of ten dichroic mirrors "biparted" as two layers-was developed and used for snapshot spectral imaging. Incident-light flux with a cross section smaller than the aperture size is split into nine color fluxes with 20-nm-width contiguous wavelength bands and central wavelengths of 530, 550, 570, 590, 610, 630, 650, 670, and 690 nm. Images of the nine color fluxes are simultaneously and efficiently measured by an image sensor. Unlike a conventional dichroic-mirror array, the developed dichroic-mirror array has a unique biparted configuration that not only increases the number of colors that can be measured simultaneously but also improves the image resolution of each color flux. The developed nine-color spectrometer was used for four-capillary-array electrophoresis. Eight dyes concurrently migrating in each capillary were simultaneously quantified by nine-color laser-induced fluorescence detection. Since the nine-color spectrometer is not only ultra-small and inexpensive but also has high light throughput and sufficient spectral resolution for most spectral-imaging applications, it has the potential to be widely used in various fields.

An LED-Based structured illumination microscope using a digital micromirror device and GPU accelerated image reconstruction

When combined with computational approaches, fluorescence imaging becomes one of the most powerful tools in biomedical research. It is possible to achieve resolution figures beyond the diffraction limit, and improve the performance and flexibility of high-resolution imaging systems with techniques such as structured illumination microscopy (SIM) reconstruction. In this study, the hardware and software implementation of an LED-based super-resolution imaging system using SIM employing GPU accelerated parallel image reconstruction is presented. The sample is illuminated with two-dimensional sinusoidal patterns with various orientations and lateral phase shifts generated using a digital micromirror device (DMD). SIM reconstruction is carried out in frequency space using parallel CUDA kernel functions. Furthermore, a general purpose toolbox for the parallel image reconstruction algorithm and an infrastructure that allows all users to perform parallel operations on images without developing any CUDA kernel code is presented. The developed image reconstruction algorithm was run separately on a CPU and a GPU. Two different SIM reconstruction algorithms have been developed for the CPU as mono-thread CPU algorithm and multi-thread OpenMP CPU algorithm. SIM reconstruction of 1024 × 1024 px images was achieved in 1.49 s using GPU computation, indicating an enhancement by ∼28 and ∼20 in computation time when compared with mono-thread CPU computation and multi-thread OpenMP CPU computation, respectively.

In vivo super-resolution of the brain – How to visualize the hidden nanoplasticity?

Super-resolution fluorescence microscopy has entered most biological laboratories worldwide and its benefit is undisputable. Its application to brain imaging, for example in living mice, enables the study of sub-cellular structural plasticity and brain function directly in a living mammal. The demands of brain imaging on the different super-resolution microscopy techniques (STED, RESOLFT, SIM, ISM) and labeling strategies are discussed here as well as the challenges of the required cranial window preparation. Applications of super-resolution in the anesthetized mouse brain enlighten the stability and plasticity of synaptic nanostructures. These studies show the potential of in vivo super-resolution imaging and justify its application more widely in vivo to investigate the role of nanostructures in memory and learning.

Resolution-enhanced optical inspection system to examine metallic nanostructures using structured illumination

We developed a structured illumination-based optical inspection system to inspect metallic nanostructures in real time. To address this, we used post-image-processing techniques to enhance the image resolution. To examine the fabricated metallic nanostructures in real time, a compact and highly resolved optical inspection system was designed for practical industrial use. Structured illumination microscopy yields multiple images with various linear illumination patterns, which can be used to reconstruct resolution-enhanced images. Images of nanosized posts and complex structures reflected in the structured illumination were reconstructed into images with improved resolution. A comparison with wide-field images demonstrates that the optical inspection system exhibits high performance and is available as a real-time nanostructure inspection platform. Because it does not require special environmental conditions and enables multiple systems to be covered in arrays, the developed system is expected to provide real-time and noninvasive inspections during the production of large-area nanostructured components.

Scene Classification in the Environmental Art Design by Using the Lightweight Deep Learning Model under the Background of Big Data

On the basis of scene visual understanding technology, the research aims to further improve the classification efficiency and classification accuracy of art design scenes. The lightweight deep learning (DL) model based on big data is used as the main method to achieve real-time detection and recognition of multiple targets and classification of the multilabel scene. This research introduces the related foundations of the DL network and the lightweight object detection involved. The data for a multilabel scene classifier are constructed and the design of the convolutional neural network (CNN) model is described. On public datasets, the effectiveness of the lightweight object detection algorithm is verified to ensure its feasibility in the classification of actual scenes. The simulation results indicate that compared with the YOLOv3-Tiny model, the improved IRDA-YOLOv3 model reduces the number of parameters by 56.2%, the amount of computation by 46.3%, and the forward computation time of the network by 0.2 ms. It means that the IRDA-YOLOv3 network obtained after the improvement can realize the lightweight of the network. In the scene classification of complex traffic roads, the classification model of the multilabel scene can predict all kinds of semantic information of a single image and the classification accuracy for the four scenes is more than 90%. In summary, the discussed classification method based on the lightweight DL model is suitable for complex practical scenes. The constructed lightweight network improves the representational ability of the network and has certain research value for scene classification problems.

Through the Eyes of Creators: Observing Artificial Molecular Motors

Inspired by molecular motors in biology, there has been significant progress in building artificial molecular motors, using a number of quite distinct approaches. As the constructs become more sophisticated, there is also an increasing need to directly observe the motion of artificial motors at the nanoscale and to characterize their performance. Here, we review the most used methods that tackle those tasks. We aim to help experimentalists with an overview of the available tools used for different types of synthetic motors and to choose the method most suited for the size of a motor and the desired measurements, such as the generated force or distances in the moving system. Furthermore, for many envisioned applications of synthetic motors, it will be a requirement to guide and control directed motions. We therefore also provide a perspective on how motors can be observed on structures that allow for directional guidance, such as nanowires and microchannels. Thus, this Review facilitates the future research on synthetic molecular motors, where observations at a single-motor level and a detailed characterization of motion will promote applications.

Blood Cell Analysis: From Traditional Methods to Super-Resolution Microscopy

Blood cell analysis is essential for the diagnosis and identification of hematological malignancies. The use of digital microscopy systems has been extended in clinical laboratories. Super-resolution microscopy (SRM) has attracted wide attention in the medical field due to its nanoscale spatial resolution and high sensitivity. It is considered to be a potential method of blood cell analysis that may have more advantages than traditional approaches such as conventional optical microscopy and hematology analyzers in certain examination projects. In this review, we firstly summarize several common blood cell analysis technologies in the clinic, and analyze the advantages and disadvantages of these technologies. Then, we focus on the basic principles and characteristics of three representative SRM techniques, as well as the latest advances in these techniques for blood cell analysis. Finally, we discuss the developmental trend and possible research directions of SRM, and provide some discussions on further development of technologies for blood cell analysis.

Real-time reconstruction using electro-optics modulator-based structured illumination microscopy

Structured illumination microscopy (SIM), a super-resolution technology, has a wide range of applications in life sciences. In this study, we present an electro-optic high-speed phase-shift super-resolution microscopy imaging system including 2D SIM, total internal reflection fluorescence-SIM, and 3D SIM modes. This system uses galvanometers and an electro-optic modulator to flexibly and quickly control the phase and direction of structured illumination patterns. Moreover, its design consists of precise timing for improved acquisition speed and software architecture for real-time reconstruction. The highest acquisition rate achieved was 151 frames/s, while the highest real-time super-resolution reconstruction frame rate achieved was over 25 frames/s.

Answering some questions about structured illumination microscopy

Structured illumination microscopy (SIM) provides images of fluorescent objects at an enhanced resolution greater than that of conventional epifluorescence wide-field microscopy. Initially demonstrated in 1999 to enhance the lateral resolution twofold, it has since been extended to enhance axial resolution twofold (2008), applied to live-cell imaging (2009) and combined with myriad other techniques, including interferometric detection (2008), confocal microscopy (2010) and light sheet illumination (2012). Despite these impressive developments, SIM remains, perhaps, the most poorly understood 'super-resolution' method. In this article, we provide answers to the 13 questions regarding SIM proposed by Prakash et al. along with answers to a further three questions. After providing a general overview of the technique and its developments, we explain why SIM as normally used is still diffraction-limited. We then highlight the necessity for a non-polynomial, and not just nonlinear, response to the illuminating light in order to make SIM a true, diffraction-unlimited, super-resolution technique. In addition, we present a derivation of a real-space SIM reconstruction approach that can be used to process conventional SIM and image scanning microscopy (ISM) data and extended to process data with quasi-arbitrary illumination patterns. Finally, we provide a simple bibliometric analysis of SIM development over the past two decades and provide a short outlook on potential future work. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 2)'.

Microscope-Cockpit: Python-based bespoke microscopy for bio-medical science

We have developed “Microscope-Cockpit” (Cockpit), a highly adaptable open source user-friendly Python-based Graphical User Interface (GUI) environment for precision control of both simple and elaborate bespoke microscope systems. The user environment allows next-generation near instantaneous navigation of the entire slide landscape for efficient selection of specimens of interest and automated acquisition without the use of eyepieces. Cockpit uses “Python-Microscope” (Microscope) for high-performance coordinated control of a wide range of hardware devices using open source software. Microscope also controls complex hardware devices such as deformable mirrors for aberration correction and spatial light modulators for structured illumination via abstracted device models. We demonstrate the advantages of the Cockpit platform using several bespoke microscopes, including a simple widefield system and a complex system with adaptive optics and structured illumination. A key strength of Cockpit is its use of Python, which means that any microscope built with Cockpit is ready for future customisation by simply adding new libraries, for example machine learning algorithms to enable automated microscopy decision making while imaging.

Dual color DMD-SIM by temperature-controlled laser wavelength matching

Structured illumination microscopy (SIM) is a fast and gentle super-resolution fluorescence imaging technique, featuring live-cell compatible excitation light levels and high imaging speeds. To achieve SIM, spatial modulation of the fluorescence excitation light is employed. This is typically achieved by interfering coherent laser beams in the sample plane, which are often created by spatial light modulators (SLMs). Digital micromirror devices (DMDs) are a form of SLMs with certain advantages, such as high speed, low cost and wide availability, which present certain hurdles in their implementation, mainly the blazed grating effect caused by the jagged surface structure of the tilted mirrors. Recent works have studied this effect through modelling, simulations and experiments, and laid out possible implementations of multi-color SIM imaging based on DMDs. Here, we present an implementation of a dual-color DMD based SIM microscope using temperature-controlled wavelength matching. By carefully controlling the output wavelength of a diode laser by temperature, we can tune two laser wavelengths in such a way that no opto-mechanical realignment of the SIM setup is necessary when switching between both wavelengths. This reduces system complexity and increases imaging speed. With measurements on nano-bead reference samples, as well as the actin skeleton and membrane of fixed U2OS cells, we demonstrate the capabilities of the setup.

Super-Resolution Fluorescence Microscopy Methods for Assessing Mouse Biology

Super-resolution (diffraction unlimited) microscopy was developed 15 years ago; the developers were awarded the Nobel Prize in Chemistry in recognition of their work in 2014. Super-resolution microscopy is increasingly being applied to diverse scientific fields, from single molecules to cell organelles, viruses, bacteria, plants, and animals, especially the mammalian model organism Mus musculus. In this review, we explain how super-resolution microscopy, along with fluorescence microscopy from which it grew, has aided the renaissance of the light microscope. We cover experiment planning and specimen preparation and explain structured illumination microscopy, super-resolution radial fluctuations, stimulated emission depletion microscopy, single-molecule localization microscopy, and super-resolution imaging by pixel reassignment. The final section of this review discusses the strengths and weaknesses of each super-resolution technique and how to choose the best approach for your research. © 2021 The Authors. Current Protocols published by Wiley Periodicals LLC.

Drosophila Models Rediscovered with Super-Resolution Microscopy

With the advent of super-resolution microscopy, we gained a powerful toolbox to bridge the gap between the cellular- and molecular-level analysis of living organisms. Although nanoscopy is broadly applicable, classical model organisms, such as fruit flies, worms and mice, remained the leading subjects because combining the strength of sophisticated genetics, biochemistry and electrophysiology with the unparalleled resolution provided by super-resolution imaging appears as one of the most efficient approaches to understanding the basic cell biological questions and the molecular complexity of life. Here, we summarize the major nanoscopic techniques and illustrate how these approaches were used in Drosophila model systems to revisit a series of well-known cell biological phenomena. These investigations clearly demonstrate that instead of simply achieving an improvement in image quality, nanoscopy goes far beyond with its immense potential to discover novel structural and mechanistic aspects. With the examples of synaptic active zones, centrosomes and sarcomeres, we will explain the instrumental role of super-resolution imaging pioneered in Drosophila in understanding fundamental subcellular constituents.

Structured illumination microscopy with noise-controlled image reconstructions

Super-resolution structured illumination microscopy (SIM) has become a widely used method for biological imaging. Standard reconstruction algorithms, however, are prone to generate noise-specific artifacts that limit their applicability for lower signal-to-noise data. Here we present a physically realistic noise model that explains the structured noise artifact, which we then use to motivate new complementary reconstruction approaches. True-Wiener-filtered SIM optimizes contrast given the available signal-to-noise ratio, and flat-noise SIM fully overcomes the structured noise artifact while maintaining resolving power. Both methods eliminate ad hoc user-adjustable reconstruction parameters in favor of physical parameters, enhancing objectivity. The new reconstructions point to a trade-off between contrast and a natural noise appearance. This trade-off can be partly overcome by further notch filtering but at the expense of a decrease in signal-to-noise ratio. The benefits of the proposed approaches are demonstrated on focal adhesion and tubulin samples in two and three dimensions, and on nanofabricated fluorescent test patterns.

Self-contained and modular structured illumination microscope

We present a modular implementation of structured illumination microscopy (SIM) that is fast, largely self-contained and that can be added onto existing fluorescence microscopes. Our strategy, which we call HIT-SIM, can theoretically deliver well over 50 super-resolved images per second and is readily compatible with existing acquisition software packages. We provide a full technical package consisting of schematics, a list of components and an alignment scheme that provides detailed specifications and assembly instructions. We illustrate the performance of the instrument by imaging optically large samples containing sequence-specifically stained DNA fragments.

GPU-accelerated real-time reconstruction in Python of three-dimensional datasets from structured illumination microscopy with hexagonal patterns

We present a structured illumination microscopy system that projects a hexagonal pattern by the interference among three coherent beams, suitable for implementation in a light-sheet geometry. Seven images acquired as the illumination pattern is shifted laterally can be processed to produce a super-resolved image that surpasses the diffraction-limited resolution by a factor of over 2 in an exemplar light-sheet arrangement. Three methods of processing data are discussed depending on whether the raw images are available in groups of seven, individually in a stream or as a larger batch representing a three-dimensional stack. We show that imaging axially moving samples can introduce artefacts, visible as fine structures in the processed images. However, these artefacts are easily removed by a filtering operation carried out as part of the batch processing algorithm for three-dimensional stacks. The reconstruction algorithms implemented in Python include specific optimizations for calculation on a graphics processing unit and we demonstrate its operation on experimental data of static objects and on simulated data of moving objects. We show that the software can process over 239 input raw frames per second at 512 × 512 pixels, generating over 34 super-resolved frames per second at 1024 × 1024 pixels. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 1)'.

mmSIM: an open toolbox for accessible structured illumination microscopy

Since the first practical super-resolution structured illumination fluorescence microscopes (SIM) were demonstrated more than two decades ago, the method has become increasingly popular for a wide range of bioimaging applications. The high cost and relative inflexibility of commercial systems, coupled with the conceptual simplicity of the approach and the desire to exploit and customize existing hardware, have led to the development of a large number of home-built systems. Several detailed hardware designs are available in the scientific literature, complemented by open-source software tools for SIM image validation and reconstruction. However, there remains a lack of simple open-source software to control these systems and manage the synchronization between hardware components, which is critical for effective SIM imaging. This article describes a new suite of software tools based on the popular Micro-Manager package, which enable the keen microscopist to develop and run a SIM system. We use the software to control two custom-built, high-speed, spatial light modulator-based SIM systems, evaluating their performance by imaging a range of fluorescent samples. By simplifying the process of SIM hardware development, we aim to support wider adoption of the technique. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 1)'.

Super-resolution fluorescence microscopy by line-scanning with an unmodified two-photon microscope

Fluorescence-based microscopy as one of the standard tools in biomedical research benefits more and more from super-resolution methods, which offer enhanced spatial resolution allowing insights into new biological processes. A typical drawback of using these methods is the need for new, complex optical set-ups. This becomes even more significant when using two-photon fluorescence excitation, which offers deep tissue imaging and excellent z-sectioning. We show that the generation of striped-illumination patterns in two-photon laser scanning microscopy can readily be exploited for achieving optical super-resolution and contrast enhancement using open-source image reconstruction software. The special appeal of this approach is that even in the case of a commercial two-photon laser scanning microscope no optomechanical modifications are required to achieve this modality. Modifying the scanning software with a custom-written macro to address the scanning mirrors in combination with rapid intensity switching by an electro-optic modulator is sufficient to accomplish the acquisition of two-photon striped-illumination patterns on an sCMOS camera. We demonstrate and analyse the resulting resolution improvement by applying different recently published image resolution evaluation procedures to the reconstructed filtered widefield and super-resolved images. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 1)'.

Simulating digital micromirror devices for patterning coherent excitation light in structured illumination microscopy

Digital micromirror devices (DMDs) are spatial light modulators that employ the electro-mechanical movement of miniaturized mirrors to steer and thus modulate the light reflected off a mirror array. Their wide availability, low cost and high speed make them a popular choice both in consumer electronics such as video projectors, and scientific applications such as microscopy. High-end fluorescence microscopy systems typically employ laser light sources, which by their nature provide coherent excitation light. In super-resolution microscopy applications that use light modulation, most notably structured illumination microscopy (SIM), the coherent nature of the excitation light becomes a requirement to achieve optimal interference pattern contrast. The universal combination of DMDs and coherent light sources, especially when working with multiple different wavelengths, is unfortunately not straight forward. The substructure of the tilted micromirror array gives rise to a blazed grating, which has to be understood and which must be taken into account when designing a DMD-based illumination system. Here, we present a set of simulation frameworks that explore the use of DMDs in conjunction with coherent light sources, motivated by their application in SIM, but which are generalizable to other light patterning applications. This framework provides all the tools to explore and compute DMD-based diffraction effects and to simulate possible system alignment configurations computationally, which simplifies the system design process and provides guidance for setting up DMD-based microscopes. This article is part of the Theo Murphy meeting 'Super-resolution structured illumination microscopy (part 1)'.

Multicolor structured illumination microscopy and quantitative control of polychromatic light with a digital micromirror device

Linear structured illumination microscopy (SIM) is a super-resolution microscopy technique that does not impose photophysics requirements on fluorescent samples. Multicolor SIM implementations typically rely on liquid crystal on silicon (LCoS) spatial light modulators (SLM's) for patterning the excitation light, but digital micromirror devices (DMD's) are a promising alternative, owing to their lower cost and higher speed. However, existing coherent DMD SIM implementations use only a single wavelength of light, limited by the lack of efficient approaches for solving the blazed grating effect for polychromatic light. We develop the requisite quantitative tools, including a closed form solution of the blaze and diffraction conditions, forward models of DMD diffraction and pattern projection, and a model of DMD aberrations. Based on these advances, we constructed a three-color DMD microscope, quantified the effect of aberrations from the DMD, developed a high-resolution optical transfer function measurement technique, and demonstrated SIM on fixed and live cells. This opens the door to applying DMD's in polychromatic applications previously restricted to LCoS SLM's.

Multicolor structured illumination microscopy and quantitative control of polychromatic light with a digital micromirror device

Linear structured illumination microscopy (SIM) is a super-resolution microscopy technique that does not impose photophysics requirements on fluorescent samples. Multicolor SIM implementations typically rely on liquid crystal on silicon (LCoS) spatial light modulators (SLM's) for patterning the excitation light, but digital micromirror devices (DMD's) are a promising alternative, owing to their lower cost and higher speed. However, existing coherent DMD SIM implementations use only a single wavelength of light, limited by the lack of efficient approaches for solving the blazed grating effect for polychromatic light. We develop the requisite quantitative tools, including a closed form solution of the blaze and diffraction conditions, forward models of DMD diffraction and pattern projection, and a model of DMD aberrations. Based on these advances, we constructed a three-color DMD microscope, quantified the effect of aberrations from the DMD, developed a high-resolution optical transfer function measurement technique, and demonstrated SIM on fixed and live cells. This opens the door to applying DMD's in polychromatic applications previously restricted to LCoS SLM's.

Multi-color structured illumination microscopy for live cell imaging based on the enhanced image recombination transform algorithm

Structured illumination microscopy (SIM) has attracted considerable interest in super-resolution, live-cell imaging because of its low light dose and high imaging speed. Obtaining a high-quality reconstruction image in SIM depends on the precise determination of the parameters of the fringe illumination pattern. The image recombination transform (IRT) algorithm is superior to other algorithms in obtaining the precise initial phase without any approximation, which is promising to provide a considerable solution to address the difficulty of initial phase estimation at low-modulation-depth conditions. However, the IRT algorithm only considers a phase shift of π∕2, which limits its applications in general scenarios. In this letter, we present a general form of IRT algorithm suitable for arbitrary phase shifts, providing a powerful tool for parameter estimation in low signal-to-noise cases. To demonstrate the effectiveness of the enhanced IRT algorithm, we constructed a multicolor, structured illumination microscope and studied at super-resolution, the cargo traffic in HRPE cells, and monitored the movement of mitochondrial structures and microtubules in COS-7 cells. The custom SIM system using the enhanced IRT algorithm allows multicolor capability and a low excitation intensity fluorescence imaging less than 1 W/cm². High-quality super-resolution images are obtained, which demonstrates the utility of this approach in imaging in the life sciences.

A survey on applications of deep learning in microscopy image analysis

Advanced microscopy enables us to acquire quantities of time-lapse images to visualize the dynamic characteristics of tissues, cells or molecules. Microscopy images typically vary in signal-to-noise ratios and include a wealth of information which require multiple parameters and time-consuming iterative algorithms for processing. Precise analysis and statistical quantification are often needed for the understanding of the biological mechanisms underlying these dynamic image sequences, which has become a big challenge in the field. As deep learning technologies develop quickly, they have been applied in bioimage processing more and more frequently. Novel deep learning models based on convolution neural networks have been developed and illustrated to achieve inspiring outcomes. This review article introduces the applications of deep learning algorithms in microscopy image analysis, which include image classification, region segmentation, object tracking and super-resolution reconstruction. We also discuss the drawbacks of existing deep learning-based methods, especially on the challenges of training datasets acquisition and evaluation, and propose the potential solutions. Furthermore, the latest development of augmented intelligent microscopy that based on deep learning technology may lead to revolution in biomedical research.