Open-source image reconstruction of super-resolution structured illumination microscopy data in ImageJ

Selective-plane-activation structured illumination microscopy

The background light from out-of-focus planes hinders resolution enhancement in structured illumination microscopy when observing volumetric samples. Here we used selective plane illumination and reversibly photoswitchable fluorescent proteins to realize structured illumination within the focal plane and eliminate the out-of-focus background. Theoretical investigation of the imaging properties and experimental demonstrations show that selective plane activation is beneficial for imaging dense microstructures in cells and cell spheroids.

Facile Conversion and Optimization of Structured Illumination Image Reconstruction Code into the GPU Environment

Superresolution, structured illumination microscopy (SIM) is an ideal modality for imaging live cells due to its relatively high speed and low photon-induced damage to the cells. The rate-limiting step in observing a superresolution image in SIM is often the reconstruction speed of the algorithm used to form a single image from as many as nine raw images. Reconstruction algorithms impose a significant computing burden due to an intricate workflow and a large number of often complex calculations to produce the final image. Further adding to the computing burden is that the code, even within the MATLAB environment, can be inefficiently written by microscopists who are noncomputer science researchers. In addition, they do not take into consideration the processing power of the graphics processing unit (GPU) of the computer. To address these issues, we present simple but efficient approaches to first revise MATLAB code, followed by conversion to GPU-optimized code. When combined with cost-effective, high-performance GPU-enabled computers, a 4- to 500-fold improvement in algorithm execution speed is observed as shown for the image denoising Hessian-SIM algorithm. Importantly, the improved algorithm produces images identical in quality to the original.

Evaluation of Swin Transformer and knowledge transfer for denoising of super-resolution structured illumination microscopy data

BACKGROUND

Convolutional neural network (CNN)-based methods have shown excellent performance in denoising and reconstruction of super-resolved structured illumination microscopy (SR-SIM) data. Therefore, CNN-based architectures have been the focus of existing studies. However, Swin Transformer, an alternative and recently proposed deep learning-based image restoration architecture, has not been fully investigated for denoising SR-SIM images. Furthermore, it has not been fully explored how well transfer learning strategies work for denoising SR-SIM images with different noise characteristics and recorded cell structures for these different types of deep learning-based methods. Currently, the scarcity of publicly available SR-SIM datasets limits the exploration of the performance and generalization capabilities of deep learning methods.

RESULTS

In this work, we present SwinT-fairSIM, a novel method based on the Swin Transformer for restoring SR-SIM images with a low signal-to-noise ratio. The experimental results show that SwinT-fairSIM outperforms previous CNN-based denoising methods. Furthermore, as a second contribution, two types of transfer learning-namely, direct transfer and fine-tuning-were benchmarked in combination with SwinT-fairSIM and CNN-based methods for denoising SR-SIM data. Direct transfer did not prove to be a viable strategy, but fine-tuning produced results comparable to conventional training from scratch while saving computational time and potentially reducing the amount of training data required. As a third contribution, we publish four datasets of raw SIM images and already reconstructed SR-SIM images. These datasets cover two different types of cell structures, tubulin filaments and vesicle structures. Different noise levels are available for the tubulin filaments.

CONCLUSION

The SwinT-fairSIM method is well suited for denoising SR-SIM images. By fine-tuning, already trained models can be easily adapted to different noise characteristics and cell structures. Furthermore, the provided datasets are structured in a way that the research community can readily use them for research on denoising, super-resolution, and transfer learning strategies.

In Situ Hybridization of circRNAs in Cells and Tissues through BaseScope™ Strategy

Imaging-based approaches are powerful strategies that nowadays have been largely used to gain insight into the function of different types of macromolecules. As for RNA, it is becoming clear how important is its intracellular localization for the control of proper cell differentiation and development and how its perturbation can be linked to several pathological states. This aspect is even more important if one thinks of highly polarized cells such as neurons.In this chapter, we describe in detail an innovative RNA-FISH approach for the detection of circular RNAs (circRNAs), a recently discovered class of noncoding RNAs, which display different subcellular localizations and whose functions still largely remain to be elucidated. The detection of these molecules represents a great challenge, above all because they share most of their sequence with the corresponding linear counterparts, from which they differ only for the back-splicing junction (BSJ) originating from the circularization reaction. This implies the use of RNA-FISH probes capable of specifically binding the BSJ and avoiding the detection of the linear counterpart. This requirement imposes the design of probes on a very small region, which implies the risk of obtaining a low and undetectable signal. The BaseScope™ Assay RNA-FISH technology overpasses this problem since it is based on branched-DNA probes. With this approach it is possible to target a specific region of the RNA, even small such as a splicing junction, and at the same time to obtain a strong and well detectable signal. All this is possible thanks to subsequent series of probes that, starting from the first hybridization to the BSJ, build a branched tree of probes that greatly amplifies the signal. Here we provide a detailed step-by-step protocol of BaseScope™ RNA-FISH on circRNAs coupled with immunofluorescence, both in cells and tissues, and we address difficulties which may arise when using this methodology that depend on cell type, specific permeabilization, image acquisition, and post-acquisition analyses.

Easily scalable multi-color DMD-based structured illumination microscopy

Structured illumination microscopy (SIM) achieves super-resolution imaging using a series of phase-shifted sinusoidal illumination patterns to down-modulate high spatial-frequency information of samples. Digital micromirror devices (DMDs) have been increasingly used to generate SIM illumination patterns due to their high speed and moderate cost. However, a DMD micromirror array's blazed grating structure causes strong angular dispersion for different wavelengths of light, thus severely hampering its application in multicolor imaging. We developed a multi-color DMD-SIM setup that employs a diffraction grating to compensate the DMD's dispersion and demonstrate super-resolution SIM imaging of both fluorescent beads and live cells samples with four color channels. This simple but effective approach can be readily scaled to more color channels, thereby greatly expanding the application of SIM in the study of complex multi-component structures and dynamics in soft matter systems.

Tandem aberration correction optics (TACO) in wide-field structured illumination microscopy

Structured illumination microscopy (SIM) is a powerful super-resolution imaging technique that uses patterned illumination to down-modulate high spatial-frequency information of samples. However, the presence of spatially-dependent aberrations can severely disrupt the illumination pattern, limiting the quality of SIM imaging. Conventional adaptive optics (AO) techniques that employ wavefront correctors at the pupil plane are not capable of effectively correcting these spatially-dependent aberrations. We introduce the Tandem Aberration Correction Optics (TACO) approach that combines both pupil AO and conjugate AO for aberration correction in SIM. TACO incorporates a deformable mirror (DM) for pupil AO in the detection path to correct for global aberrations, while a spatial light modulator (SLM) is placed at the plane conjugate to the aberration source near the sample plane, termed conjugate AO, to compensate spatially-varying aberrations in the illumination path. Our numerical simulations and experimental results show that the TACO approach can recover the illumination pattern close to an ideal condition, even when severely misshaped by aberrations, resulting in high-quality super-resolution SIM reconstruction. The TACO approach resolves a critical traditional shortcoming of aberration correction for structured illumination. This advance significantly expands the application of SIM imaging in the study of complex, particularly biological, samples and should be effective in other wide-field microscopies.

Grazing incidence to total internal reflection fluorescence structured illumination microscopy enabled by a prism telescope

In super-resolution structured illumination microscopy (SR-SIM) the separation between opposing laser spots in the back focal plane of the objective lens affects the pattern periodicity, and, thus, the resulting spatial resolution. Here, we introduce a novel hexagonal prism telescope which allows us to seamlessly change the separation between parallel laser beams for 3 pairs of beams, simultaneously. Each end of the prism telescope is composed of 6 Littrow prisms, which are custom-ground so they can be grouped together in the form of a tight hexagon. By changing the distance between the hexagons, the beam separation can be adjusted. This allows us to easily control the position of opposing laser spots in the back focal plane and seamlessly adjust the spatial frequency of the resulting interference pattern. This also enables the seamless transition from 2D-SIM to total internal reflection fluorescence (TIRF) excitation using objective lenses with a high numerical aperture. In linear SR-SIM the highest spatial resolution can be achieved for extreme TIRF angles. The prism telescope allows us to investigate how the spatial resolution and contrast depend on the angle of incidence near, at, and beyond the critical angle. We demonstrate this by imaging the cytoskeleton and plasma membrane of liver sinusoidal endothelial cells, which have a characteristic morphology consisting of thousands of small, transcellular pores that can only be observed by super-resolution microscopy.

Optimal sparsity allows reliable system-aware restoration of fluorescence microscopy images

Fluorescence microscopy is one of the most indispensable and informative driving forces for biological research, but the extent of observable biological phenomena is essentially determined by the content and quality of the acquired images. To address the different noise sources that can degrade these images, we introduce an algorithm for multiscale image restoration through optimally sparse representation (MIRO). MIRO is a deterministic framework that models the acquisition process and uses pixelwise noise correction to improve image quality. Our study demonstrates that this approach yields a remarkable restoration of the fluorescence signal for a wide range of microscopy systems, regardless of the detector used (e.g., electron-multiplying charge-coupled device, scientific complementary metal-oxide semiconductor, or photomultiplier tube). MIRO improves current imaging capabilities, enabling fast, low-light optical microscopy, accurate image analysis, and robust machine intelligence when integrated with deep neural networks. This expands the range of biological knowledge that can be obtained from fluorescence microscopy.

High-speed TIRF and 2D super-resolution structured illumination microscopy with a large field of view based on fiber optic components

Super-resolved structured illumination microscopy (SR-SIM) is among the most flexible, fast, and least perturbing fluorescence microscopy techniques capable of surpassing the optical diffraction limit. Current custom-built instruments are easily able to deliver two-fold resolution enhancement at video-rate frame rates, but the cost of the instruments is still relatively high, and the physical size of the instruments based on the implementation of their optics is still rather large. Here, we present our latest results towards realizing a new generation of compact, cost-efficient, and high-speed SR-SIM instruments. Tight integration of the fiber-based structured illumination microscope capable of multi-color 2D- and TIRF-SIM imaging, allows us to demonstrate SR-SIM with a field of view of up to 150 × 150 µm2 and imaging rates of up to 44 Hz while maintaining highest spatiotemporal resolution of less than 100 nm. We discuss the overall integration of optics, electronics, and software that allowed us to achieve this, and then present the fiberSIM imaging capabilities by visualizing the intracellular structure of rat liver sinusoidal endothelial cells, in particular by resolving the structure of their trans-cellular nanopores called fenestrations.

Open-3DSIM: an open-source three-dimensional structured illumination microscopy reconstruction platform

Open-3DSIM is an open-source reconstruction platform for three-dimensional structured illumination microscopy. We demonstrate its superior performance for artifact suppression and high-fidelity reconstruction relative to other algorithms on various specimens and over a range of signal-to-noise levels. Open-3DSIM also offers the capacity to extract dipole orientation, paving a new avenue for interpreting subcellular structures in six dimensions (xyzθλt). The platform is available as MATLAB code, a Fiji plugin and an Exe application to maximize user-friendliness.

Superresolution structured illumination microscopy reconstruction algorithms: a review

Structured illumination microscopy (SIM) has become the standard for next-generation wide-field microscopy, offering ultrahigh imaging speed, superresolution, a large field-of-view, and long-term imaging. Over the past decade, SIM hardware and software have flourished, leading to successful applications in various biological questions. However, unlocking the full potential of SIM system hardware requires the development of advanced reconstruction algorithms. Here, we introduce the basic theory of two SIM algorithms, namely, optical sectioning SIM (OS-SIM) and superresolution SIM (SR-SIM), and summarize their implementation modalities. We then provide a brief overview of existing OS-SIM processing algorithms and review the development of SR-SIM reconstruction algorithms, focusing primarily on 2D-SIM, 3D-SIM, and blind-SIM. To showcase the state-of-the-art development of SIM systems and assist users in selecting a commercial SIM system for a specific application, we compare the features of representative off-the-shelf SIM systems. Finally, we provide perspectives on the potential future developments of SIM.

Structured illumination phase and fluorescence microscopy for bioimaging

This study presents a dual-modality microscopic imaging approach that combines quantitative phase microscopy and fluorescence microscopy based on structured illumination (SI) to provide structural and functional information for the same sample. As the first imaging modality, structured illumination digital holographic microscopy (SI-DHM) is implemented along the transmission beam path. SI-DHM acts as a label-free, noninvasive approach and provides high-contrast and quantitative phase images utilizing the refractive index contrast of the inner structures of samples against the background. As the second imaging modality, structured illumination (fluorescence) microscopy (SIM) is constructed along the reflection beam path. SIM utilizes fluorescent labeling and provides super-resolution images for specific functional structures of samples. We first experimentally demonstrated phase imaging of SI-DHM on rice leaves and fluorescence (SIM) imaging on mouse kidney sections. Then, we demonstrated dual-modality imaging of biological samples, using DHM to acquire the overall cell morphology and SIM to obtain specific functional structures. These results prove that the proposed technique is of great importance in biomedical studies, such as providing insight into cell physiology by visualizing and quantifying subcellular structures.

Quantitative structured illumination microscopy via a physical model-based background filtering algorithm reveals actin dynamics

Despite the prevalence of superresolution (SR) microscopy, quantitative live-cell SR imaging that maintains the completeness of delicate structures and the linearity of fluorescence signals remains an uncharted territory. Structured illumination microscopy (SIM) is the ideal tool for live-cell SR imaging. However, it suffers from an out-of-focus background that leads to reconstruction artifacts. Previous post hoc background suppression methods are prone to human bias, fail at densely labeled structures, and are nonlinear. Here, we propose a physical model-based Background Filtering method for living cell SR imaging combined with the 2D-SIM reconstruction procedure (BF-SIM). BF-SIM helps preserve intricate and weak structures down to sub-70 nm resolution while maintaining signal linearity, which allows for the discovery of dynamic actin structures that, to the best of our knowledge, have not been previously monitored.

High-speed spatially re-modulated structured illumination microscopy

Structured illumination microscopy (SIM) allows non-invasive visualization of nanoscale subcellular structures. However, image acquisition and reconstruction become the bottleneck to further improve the imaging speed. Here, we propose a method to accelerate SIM imaging by combining the spatial re-modulation principle with Fourier domain filtering and using measured illumination patterns. This approach enables high-speed, high-quality imaging of dense subcellular structures using a conventional nine-frame SIM modality without phase estimation of the patterns. In addition, seven-frame SIM reconstruction and additional hardware acceleration further improve the imaging speed using our method. Furthermore, our method is also applicable to other spatially uncorrelated illumination patterns, such as distorted sinusoidal, multifocal, and speckle patterns.

Rapid, artifact-reduced, image reconstruction for super-resolution structured illumination microscopy

Super-resolution structured illumination microscopy (SR-SIM) is finding increasing application in biomedical research due to its superior ability to visualize subcellular dynamics in living cells. However, during image reconstruction artifacts can be introduced and when coupled with time-consuming postprocessing procedures, limits this technique from becoming a routine imaging tool for biologists. To address these issues, an accelerated, artifact-reduced reconstruction algorithm termed joint space frequency reconstruction-based artifact reduction algorithm (JSFR-AR-SIM) was developed by integrating a high-speed reconstruction framework with a high-fidelity optimization approach designed to suppress the sidelobe artifact. Consequently, JSFR-AR-SIM produces high-quality, super-resolution images with minimal artifacts, and the reconstruction speed is increased. We anticipate this algorithm to facilitate SR-SIM becoming a routine tool in biomedical laboratories.

Quantitative Single-Molecule Localization Microscopy

Super-resolution fluorescence microscopy allows the investigation of cellular structures at nanoscale resolution using light. Current developments in super-resolution microscopy have focused on reliable quantification of the underlying biological data. In this review, we first describe the basic principles of super-resolution microscopy techniques such as stimulated emission depletion (STED) microscopy and single-molecule localization microscopy (SMLM), and then give a broad overview of methodological developments to quantify super-resolution data, particularly those geared toward SMLM data. We cover commonly used techniques such as spatial point pattern analysis, colocalization, and protein copy number quantification but also describe more advanced techniques such as structural modeling, single-particle tracking, and biosensing. Finally, we provide an outlook on exciting new research directions to which quantitative super-resolution microscopy might be applied.

Feasibility of Coacervate-Like Nanostructure for Instant Drug Nanoformulation

Despite the enormous advancements in nanomedicine research, a limited number of nanoformulations are available on the market, and few have been translated to clinics. An easily scalable, sustainable, and cost-effective manufacturing strategy and long-term stability for storage are crucial for successful translation. Here, we report a system and method to instantly formulate NF achieved with a nanoscale polyelectrolyte coacervate-like system, consisting of anionic pseudopeptide poly(l-lysine isophthalamide) derivatives, polyethylenimine, and doxorubicin (Dox) via simple "mix-and-go" addition of precursor solutions in seconds. The coacervate-like nanosystem shows enhanced intracellular delivery of Dox to patient-derived multidrug-resistant (MDR) cells in 3D tumor spheroids. The results demonstrate the feasibility of an instant drug formulation using a coacervate-like nanosystem. We envisage that this technique can be widely utilized in the nanomedicine field to bypass the special requirement of large-scale production and elongated shelf life of nanomaterials.

ERnet: a tool for the semantic segmentation and quantitative analysis of endoplasmic reticulum topology

The ability to quantify structural changes of the endoplasmic reticulum (ER) is crucial for understanding the structure and function of this organelle. However, the rapid movement and complex topology of ER networks make this challenging. Here, we construct a state-of-the-art semantic segmentation method that we call ERnet for the automatic classification of sheet and tubular ER domains inside individual cells. Data are skeletonized and represented by connectivity graphs, enabling precise and efficient quantification of network connectivity. ERnet generates metrics on topology and integrity of ER structures and quantifies structural change in response to genetic or metabolic manipulation. We validate ERnet using data obtained by various ER-imaging methods from different cell types as well as ground truth images of synthetic ER structures. ERnet can be deployed in an automatic high-throughput and unbiased fashion and identifies subtle changes in ER phenotypes that may inform on disease progression and response to therapy.

Reconstruction algorithm using 2N+1 raw images for structured illumination microscopy

This paper presents a structured illumination microscopy (SIM) reconstruction algorithm that allows the reconstruction of super-resolved images with 2N + 1 raw intensity images, with N being the number of structured illumination directions used. The intensity images are recorded after using a 2D grating for the projection fringe and a spatial light modulator to select two orthogonal fringe orientations and perform phase shifting. Super-resolution images can be reconstructed from the five intensity images, enhancing the imaging speed and reducing the photobleaching by 17%, compared to conventional two-direction and three-step phase-shifting SIM. We believe the proposed technique will be further developed and widely applied in many fields.

Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes

The goal when imaging bioprocesses with optical microscopy is to acquire the most spatiotemporal information with the least invasiveness. Deep neural networks have substantially improved optical microscopy, including image super-resolution and restoration, but still have substantial potential for artifacts. In this study, we developed rationalized deep learning (rDL) for structured illumination microscopy and lattice light sheet microscopy (LLSM) by incorporating prior knowledge of illumination patterns and, thereby, rationally guiding the network to denoise raw images. Here we demonstrate that rDL structured illumination microscopy eliminates spectral bias-induced resolution degradation and reduces model uncertainty by five-fold, improving the super-resolution information by more than ten-fold over other computational approaches. Moreover, rDL applied to LLSM enables self-supervised training by using the spatial or temporal continuity of noisy data itself, yielding results similar to those of supervised methods. We demonstrate the utility of rDL by imaging the rapid kinetics of motile cilia, nucleolar protein condensation during light-sensitive mitosis and long-term interactions between membranous and membrane-less organelles.

Untrained, physics-informed neural networks for structured illumination microscopy

Structured illumination microscopy (SIM) is a popular super-resolution imaging technique that can achieve resolution improvements of 2× and greater depending on the illumination patterns used. Traditionally, images are reconstructed using the linear SIM reconstruction algorithm. However, this algorithm has hand-tuned parameters which can often lead to artifacts, and it cannot be used with more complex illumination patterns. Recently, deep neural networks have been used for SIM reconstruction, yet they require training sets that are difficult to capture experimentally. We demonstrate that we can combine a deep neural network with the forward model of the structured illumination process to reconstruct sub-diffraction images without training data. The resulting physics-informed neural network (PINN) can be optimized on a single set of diffraction-limited sub-images and thus does not require any training set. We show, with simulated and experimental data, that this PINN can be applied to a wide variety of SIM illumination methods by simply changing the known illumination patterns used in the loss function and can achieve resolution improvements that match theoretical expectations.

Parameter estimation of the structured illumination pattern based on principal component analysis (PCA): PCA-SIM

Principal component analysis (PCA), a common dimensionality reduction method, is introduced into SIM to identify the frequency vectors and pattern phases of the illumination pattern with precise subpixel accuracy, fast speed, and noise-robustness, which is promising for real-time and long-term live-cell imaging.

Spatial proteomics of vesicular trafficking: coupling mass spectrometry and imaging approaches in membrane biology

In plants, membrane compartmentalization requires vesicle trafficking for communication among distinct organelles. Membrane proteins involved in vesicle trafficking are highly dynamic and can respond rapidly to changes in the environment and to cellular signals. Capturing their localization and dynamics is thus essential for understanding the mechanisms underlying vesicular trafficking pathways. Quantitative mass spectrometry and imaging approaches allow a system-wide dissection of the vesicular proteome, the characterization of ligand-receptor pairs and the determination of secretory, endocytic, recycling and vacuolar trafficking pathways. In this review, we highlight major proteomics and imaging methods employed to determine the location, distribution and abundance of proteins within given trafficking routes. We focus in particular on methodologies for the elucidation of vesicle protein dynamics and interactions and their connections to downstream signalling outputs. Finally, we assess their biological applications in exploring different cellular and subcellular processes.

Optimal Transport for Unsupervised Denoising Learning

Recently, much progress has been made in unsupervised denoising learning. However, existing methods more or less rely on some assumptions on the signal and/or degradation model, which limits their practical performance. How to construct an optimal criterion for unsupervised denoising learning without any prior knowledge on the degradation model is still an open question. Toward answering this question, this work proposes a criterion for unsupervised denoising learning based on the optimal transport theory. This criterion has favorable properties, e.g., approximately maximal preservation of the information of the signal, whilst achieving perceptual reconstruction. Furthermore, though a relaxed unconstrained formulation is used in practical implementation, we prove that the relaxed formulation in theory has the same solution as the original constrained formulation. Experiments on synthetic and real-world data, including realistic photographic, microscopy, depth, and raw depth images, demonstrate that the proposed method even compares favorably with supervised methods, e.g., approaching the PSNR of supervised methods while having better perceptual quality. Particularly, for spatially correlated noise and realistic microscopy images, the proposed method not only achieves better perceptual quality but also has higher PSNR than supervised methods. Besides, it shows remarkable superiority in harsh practical conditions with complex noise, e.g., raw depth images. Code is available at https://github.com/wangweiSJTU/OTUR.

Transmission structured illumination microscopy with tunable frequency illumination using tilt mirror assembly

We present experimental demonstration of tilt-mirror assisted transmission structured illumination microscopy (tSIM) that offers a large field of view super resolution imaging. An assembly of custom-designed tilt-mirrors are employed as the illumination module where the sample is excited with the interference of two beams reflected from the opposite pair of mirror facets. Tunable frequency structured patterns are generated by changing the mirror-tilt angle and the hexagonal-symmetric arrangement is considered for the isotropic resolution in three orientations. Utilizing high numerical aperture (NA) objective in standard SIM provides super-resolution compromising with the field-of-view (FOV). Employing low NA (20X/0.4) objective lens detection, we experimentally demonstrate [Formula: see text] (0.56 mm[Formula: see text]0.35 mm) size single FOV image with [Formula: see text]1.7- and [Formula: see text]2.4-fold resolution improvement (exploiting various illumination by tuning tilt-mirrors) over the diffraction limit. The results are verified both for the fluorescent beads as well as biological samples. The tSIM geometry decouples the illumination and the collection light paths consequently enabling free change of the imaging objective lens without influencing the spatial frequency of the illumination pattern that are defined by the tilt-mirrors. The large and scalable FOV supported by tSIM will find usage for applications where scanning large areas are necessary as in pathology and applications where images must be correlated both in space and time.

Machine learning assisted interferometric structured illumination microscopy for dynamic biological imaging

Structured Illumination Microscopy, SIM, is one of the most powerful optical imaging methods available to visualize biological environments at subcellular resolution. Its limitations stem from a difficulty of imaging in multiple color channels at once, which reduces imaging speed. Furthermore, there is substantial experimental complexity in setting up SIM systems, preventing a widespread adoption. Here, we present Machine-learning Assisted, Interferometric Structured Illumination Microscopy, MAI-SIM, as an easy-to-implement method for live cell super-resolution imaging at high speed and in multiple colors. The instrument is based on an interferometer design in which illumination patterns are generated, rotated, and stepped in phase through movement of a single galvanometric mirror element. The design is robust, flexible, and works for all wavelengths. We complement the unique properties of the microscope with an open source machine-learning toolbox that permits real-time reconstructions to be performed, providing instant visualization of super-resolved images from live biological samples.

Miniaturized structured illumination microscopy using two 3-axis MEMS micromirrors

We present the development and performance characterisation of a novel structured illumination microscope (SIM) in which the grating pattern is generated using two optical beams controlled via 2 micro-electro-mechanical system (MEMS) three-axis scanning micromirrors. The implementation of MEMS micromirrors to accurately and repeatably control angular, radial and phase positioning delivers flexible control of the fluorescence excitation illumination, with achromatic beam delivery through the same optical path, reduced spatial footprint and cost-efficient integration being further benefits. Our SIM architecture enables the direct implementation of multi-color imaging in a compact and adaptable package. The two-dimensional SIM system approach is enabled by a pair of 2 mm aperture electrostatically actuated three-axis micromirrors having static angular tilt motion along the x- and y-axes and static piston motion along the z-axis. This allows precise angular, radial and phase positioning of two optical beams, generating a fully controllable spatial interference pattern at the focal plane by adjusting the positions of the beam in the back-aperture of a microscope objective. This MEMS-SIM system was applied to fluorescent bead samples and cell specimens, and was able to obtain a variable lateral resolution improvement between 1.3 and 1.8 times the diffraction limited resolution.

Electron-beam patterned calibration structures for structured illumination microscopy

Super-resolution fluorescence microscopy can be achieved by image reconstruction after spatially patterned illumination or sequential photo-switching and read-out. Reconstruction algorithms and microscope performance are typically tested using simulated image data, due to a lack of strategies to pattern complex fluorescent patterns with nanoscale dimension control. Here, we report direct electron-beam patterning of fluorescence nanopatterns as calibration standards for super-resolution fluorescence. Patterned regions are identified with both electron microscopy and fluorescence labelling of choice, allowing precise correlation of predefined pattern dimensions, a posteriori obtained electron images, and reconstructed super-resolution images.

Cost-effective high-speed, three-dimensional live-cell imaging of HIV-1 transfer at the T cell virological synapse

The availability of cost-effective, highly portable, and easy to use high-resolution live-cell imaging systems could present a significant technological break-through in challenging environments, such as high-level biosafety laboratories or sites where new viral outbreaks are suspected. We describe and demonstrate a cost-effective high-speed fluorescence microscope enabling the live tracking of virus particles across virological synapses that form between infected and uninfected T cells. The dynamics of HIV-1 proteins studied at the cellular level and the formation of virological synapses in living T cells reveals mechanisms by which cell-cell interactions facilitate infection between immune cells. Dual-color 3D fluorescence deconvolution microscopy of HIV-1 particles at frames rates of 100 frames per second allows us to follow the transfer of HIV-1 particles across the T cell virological synapse between living T cells. We also confirm the successful transfer of virus by imaging T cell samples fixed at specific time points during cell-cell virus transfer by super-resolution structured illumination microscopy.

Resolution doubling in light-sheet microscopy via oblique plane structured illumination

Structured illumination microscopy (SIM) doubles the spatial resolution of a fluorescence microscope without requiring high laser powers or specialized fluorophores. However, the excitation of out-of-focus fluorescence can accelerate photobleaching and phototoxicity. In contrast, light-sheet fluorescence microscopy (LSFM) largely avoids exciting out-of-focus fluorescence, thereby enabling volumetric imaging with low photobleaching and intrinsic optical sectioning. Combining SIM with LSFM would enable gentle three-dimensional (3D) imaging at doubled resolution. However, multiple orientations of the illumination pattern, which are needed for isotropic resolution doubling in SIM, are challenging to implement in a light-sheet format. Here we show that multidirectional structured illumination can be implemented in oblique plane microscopy, an LSFM technique that uses a single objective for excitation and detection, in a straightforward manner. We demonstrate isotropic lateral resolution below 150 nm, combined with lower phototoxicity compared to traditional SIM systems and volumetric acquisition speed exceeding 1 Hz.

Resolution-enhanced optical inspection system to examine metallic nanostructures using structured illumination

We developed a structured illumination-based optical inspection system to inspect metallic nanostructures in real time. To address this, we used post-image-processing techniques to enhance the image resolution. To examine the fabricated metallic nanostructures in real time, a compact and highly resolved optical inspection system was designed for practical industrial use. Structured illumination microscopy yields multiple images with various linear illumination patterns, which can be used to reconstruct resolution-enhanced images. Images of nanosized posts and complex structures reflected in the structured illumination were reconstructed into images with improved resolution. A comparison with wide-field images demonstrates that the optical inspection system exhibits high performance and is available as a real-time nanostructure inspection platform. Because it does not require special environmental conditions and enables multiple systems to be covered in arrays, the developed system is expected to provide real-time and noninvasive inspections during the production of large-area nanostructured components.

Large-field lattice structured illumination microscopy

In this paper, we present large-field, five-step lattice structured illumination microscopy (Lattice SIM). This method utilizes a 2D grating for lattice projection and a spatial light modulator (SLM) for phase shifting. Five phase-shifted intensity images are recorded to reconstruct a super-resolution image, enhancing the imaging speed and reducing the photo-bleaching both by 17%, compared to conventional two-direction and three-shift SIM. Furthermore, lattice SIM has a three-fold spatial bandwidth product (SBP) enhancement compared to SLM/DMD-based SIM, of which the fringe number is limited by the SLM/DMD pixel number. We believe that the proposed technique will be further developed and widely applied in many fields.

A Hitchhiker's Guide through the Bio-image Analysis Software Universe

Modern research in the life sciences is unthinkable without computational methods for extracting, quantifying and visualizing information derived from microscopy imaging data of biological samples. In the past decade, we observed a dramatic increase in available software packages for these purposes. As it is increasingly difficult to keep track of the number of available image analysis platforms, tool collections, components and emerging technologies, we provide a conservative overview of software that we use in daily routine and give insights into emerging new tools. We give guidance on which aspects to consider when choosing the platform that best suits the user's needs, including aspects such as image data type, skills of the team, infrastructure and community at the institute and availability of time and budget.

Near-native state imaging by cryo-soft-X-ray tomography reveals remodelling of multiple cellular organelles during HSV-1 infection

Herpes simplex virus-1 (HSV-1) is a large, enveloped DNA virus and its assembly in the cell is a complex multi-step process during which viral particles interact with numerous cellular compartments such as the nucleus and organelles of the secretory pathway. Transmission electron microscopy and fluorescence microscopy are commonly used to study HSV-1 infection. However, 2D imaging limits our understanding of the 3D geometric changes to cellular compartments that accompany infection and sample processing can introduce morphological artefacts that complicate interpretation. In this study, we used soft X-ray tomography to observe differences in whole-cell architecture between HSV-1 infected and uninfected cells. To protect the near-native structure of cellular compartments we used a non-disruptive sample preparation technique involving rapid cryopreservation, and a fluorescent reporter virus was used to facilitate correlation of structural changes with the stage of infection in individual cells. We observed viral capsids and assembly intermediates interacting with nuclear and cytoplasmic membranes. Additionally, we observed differences in the morphology of specific organelles between uninfected and infected cells. The local concentration of cytoplasmic vesicles at the juxtanuclear compartment increased and their mean width decreased as infection proceeded, and lipid droplets transiently increased in size. Furthermore, mitochondria in infected cells were elongated and highly branched, suggesting that HSV-1 infection alters the dynamics of mitochondrial fission/fusion. Our results demonstrate that high-resolution 3D images of cellular compartments can be captured in a near-native state using soft X-ray tomography and have revealed that infection causes striking changes to the morphology of intracellular organelles.

Large-field structured illumination microscopy based on 2D grating and a spatial light modulator

Structured illumination microscopy (SIM) has been widely used in biological research due to its merits of fast imaging speed, minimal invasiveness, super-resolution, and optical sectioning imaging capability. However, the conventional SIM that uses a spatial light modulator (SLM) for fringe projection often has a limited imaging field of view. Herein, we report a large-field SIM technique that combines a 2D grating for fringe pattern projection and an SLM for selecting fringe orientation and performing phase shifting digitally. The proposed SIM technique breaks the bottleneck of fringe number limited by the digital projection devices, while maintaining the advantage of high-speed (digital) phase shifting of conventional SIM. The method avoids the pixilation and dispersion effects of the SLMs. Finally, a 1.8-fold resolution enhancement in a large field of 690 × 517 µm² under a 20×/NA0.75 objective is experimentally demonstrated. The proposed technique can be widely applied to biology, chemistry, and industry.

Modifications of the Dental Hard Tissues in the Cervical Area of Occlusally Overloaded Teeth Identified Using Optical Coherence Tomography

Background and objectives: Occlusal overloads produce a series of manifestations in teeth, especially attrition and non-carious cervical lesions (NCCL). Optical Coherence Tomography (OCT) can highlight and evaluate tooth lesions. The aim of this study was to examine the changes of dental hard tissues in the cervical area because of occlusal overload by macroscopic examination and using in vitro Swept Source OCT examination. Materials and Methods: The study was performed on 21 extracted teeth with occlusal trauma. After macroscopic and OCT examination, the 2D OCT images were transformed into 3D images using ImageJ software. Statistical analysis of macroscopic and OCT images was performed with Statistical Package for Social Sciences. Results: On 21 teeth, 88 cervical lesions (cracks) were identified. Upper premolars with an occlusal Smith and Knight tooth wear score of 2 had the most NCCL. Statistical analysis revealed significant differences in the median widths of cervical lesions between teeth with score 1 and score 3. Additionally, we obtained statistically significant differences in median widths between the buccal and oral surfaces. Conclusions: These cracks can be considered precursors of NCCL. NCCL can be located on dental surfaces in the cervical area other than the buccal one. A 3D reconstruction of OCT images emphasized that cracks are located especially at enamel level, evolving towards the enamel-dentin junction, with multiple ramifications.

UCsim2: two-dimensionally structured illumination microscopy using UC2

State-of-the-art microscopy techniques enable the imaging of sub-diffraction barrier biological structures at the price of high costs or a lack of transparency. We try to reduce some of these barriers by presenting a super-resolution upgrade to our recently presented open-source optical toolbox UC2. Our new injection moulded parts allow larger builds with higher precision. The 4× lower manufacturing tolerance compared to three-dimensional printing makes assemblies more reproducible. By adding consumer-grade available open-source hardware such as digital mirror devices and laser projectors, we demonstrate a compact three-dimensional multimodal setup that combines image scanning microscopy and structured illumination microscopy. We demonstrate a gain in resolution and optical sectioning using the two different modes compared to the widefield limit by imaging Alexa Fluor ® 647- and Silicon Rhodamine-stained HeLa cells. We compare different objective lenses and by sharing the designs and manuals of our setup, we make super-resolution imaging available to everyone. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 2)'.

Answering some questions about structured illumination microscopy

Structured illumination microscopy (SIM) provides images of fluorescent objects at an enhanced resolution greater than that of conventional epifluorescence wide-field microscopy. Initially demonstrated in 1999 to enhance the lateral resolution twofold, it has since been extended to enhance axial resolution twofold (2008), applied to live-cell imaging (2009) and combined with myriad other techniques, including interferometric detection (2008), confocal microscopy (2010) and light sheet illumination (2012). Despite these impressive developments, SIM remains, perhaps, the most poorly understood 'super-resolution' method. In this article, we provide answers to the 13 questions regarding SIM proposed by Prakash et al. along with answers to a further three questions. After providing a general overview of the technique and its developments, we explain why SIM as normally used is still diffraction-limited. We then highlight the necessity for a non-polynomial, and not just nonlinear, response to the illuminating light in order to make SIM a true, diffraction-unlimited, super-resolution technique. In addition, we present a derivation of a real-space SIM reconstruction approach that can be used to process conventional SIM and image scanning microscopy (ISM) data and extended to process data with quasi-arbitrary illumination patterns. Finally, we provide a simple bibliometric analysis of SIM development over the past two decades and provide a short outlook on potential future work. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 2)'.

SIM reconstruction framework for high-speed multi-dimensional super-resolution imaging

Structured illumination microscopy (SIM) holds great promise for live cell imaging applications due to its potential to obtain multidimensional information such as intensity, spectrum and polarization (I, λ , p) at high spatial-temporal resolution, enabling the observation of more complex dynamic interactions between subcellular structures. However, the reconstruction results of polarized samples are prone to artifacts because all current SIM reconstruction frameworks use incomplete imaging models which neglect polarization modulation. Such polarization-related artifacts are especially prevalent for SIM reconstruction using a reduced number of raw images (RSIM) and severely undermine the ability of SIM to capture multi-dimensional information. Here, we report a new SIM reconstruction framework (PRSIM) that can recover multi-dimensional information (I, λ, p) using a reduced number of raw images. PRSIM adopts a complete imaging model that is versatile for normal and polarized samples and uses a frequency-domain iterative reconstruction algorithm for artifact-free super-resolution (SR) reconstruction. It can simultaneously obtain the SR spatial structure and polarization orientation of polarized samples using 6 raw SIM images and can perform SR reconstruction using 4 SIM images for normal samples. In addition, PRSIM has less spatial computational complexity and achieves reconstruction speeds tens of times higher than that of the state-of-the-art non-iterative RSIM, making it more suitable for large field-of-view imaging. Thus, PRSIM is expected to facilitate the development of SIM into an ultra-high-speed and multi-dimensional SR imaging tool.

Applications of Optical Coherence Tomography in the Diagnosis of Enamel Defects

Developmental defects of enamel (DDEs) are deviations from the normal appearance in terms of the quantity and quality of tooth enamel. They may be genetic or acquired. The most important DDEs are hypomineralization and hypoplasia. The aim of this study was to produce “in vivo” DDE in Wistar rats by administering amoxicillin to pregnant females and to highlight these lesions after sacrifice of the pups by macroscopic and microscopic examination and optical coherence tomography (OCT). Amoxicillin (100 mg/kg) was administered to two pregnant Wistar female rats for the production of DDEs. When the pups were 2 months old, they were sacrificed, and their jaws were harvested together with their teeth. The jaws were examined macroscopically, microscopically, and by OCT. Following the macroscopic and microscopic examination, it was established that four pups had a total of 42 DDE lesions. At the OCT examination, the hypomineralization was characterized by an intense, inhomogeneous OCT signal, and the hypoplasia was characterized by the absence of the signal. Administration of amoxicillin to pregnant females of Wistar rats resulted in DDEs in their offspring. The OCT examination confirmed the presence of these lesions in the teeth of rat pups.

Multiplexed structured illumination super-resolution imaging with lifetime-engineered upconversion nanoparticles

The emerging optical multiplexing within nanoscale shows super-capacity in encoding information by using lifetime fingerprints from luminescent nanoparticles. However, the optical diffraction limit compromises the decoding accuracy and throughput of the nanoparticles during conventional widefield imaging. This, in turn, challenges the quality of nanoparticles to afford the modulated excitation condition and further retain the multiplexed optical fingerprints for super-resolution multiplexing. Here we report a tailor-made multiplexed super-resolution imaging method using the lifetime-engineered upconversion nanoparticles. We demonstrate that the nanoparticles are bright, uniform, and stable under structured illumination, which supports a lateral resolution of 185 nm, less than 1/4th of the excitation wavelength. We further develop a deep learning algorithm to coordinate with super-resolution images for more accurate decoding compared to a numeric algorithm. We demonstrate a three-channel super-resolution imaging based optical multiplexing with decoding accuracies above 93% for each channel and larger than 60% accuracy for potential seven-channel multiplexing. The improved resolution provides high throughput by resolving the particles within the diffraction-limited spots, which enables higher multiplexing capacity in space. This lifetime multiplexing super-resolution method opens a new horizon for handling the growing amount of information content, disease source, and security risk in modern society.

Frequency-spatial domain joint optimization for improving super-resolution images of nonlinear structured illumination microscopy

Introducing nonlinear fluorophore excitation into structured illumination microscopy (SIM) can further extend its spatial resolution without theoretical limitation. However, it is a great challenge to recover the weak higher-order harmonic signal and reconstruct high-fidelity super-resolution (SR) images. Here, we proposed a joint optimization strategy in both the frequency and spatial domains to reconstruct high-quality nonlinear SIM (NL-SIM) images. We demonstrate that our method can reconstruct SR images with fewer artifacts and higher fidelity on the BioSR dataset with patterned-activation NL-SIM. This method could robustly overcome one of the long-lived obstacles on NL-SIM imaging, thereby promoting its wide application in biology.

Improved resolution in 3D structured illumination microscopy using 3D model-based restoration with positivity-constraint

The performance of structured illumination microscopy (SIM) systems depends on the computational method used to process the raw data. In this paper, we present a regularized three-dimensional (3D) model-based (MB) restoration method with positivity constraint (PC) for 3D processing of data from 3D-SIM (or 3-beam interference SIM), in which the structured illumination pattern varies laterally and axially. The proposed 3D-MBPC method introduces positivity in the solution through the reconstruction of an auxiliary function using a conjugate-gradient method that minimizes the mean squared error between the data and the 3D imaging model. The 3D-MBPC method provides axial super resolution, which is not the same as improved optical sectioning demonstrated with model-based approaches based on the 2D-SIM (or 2-beam interference SIM) imaging model, for either 2D or 3D processing of a single plane from a 3D-SIM dataset. Results obtained with our 3D-MBPC method show improved 3D resolution over what is achieved by the standard generalized Wiener filter method, the first known method that performs 3D processing of 3D-SIM data. Noisy simulation results quantify the achieved 3D resolution, which is shown to match theoretical predictions. Experimental verification of the 3D-MBPC method with biological data demonstrates successful application to data volumes of different sizes.

Dual color DMD-SIM by temperature-controlled laser wavelength matching

Structured illumination microscopy (SIM) is a fast and gentle super-resolution fluorescence imaging technique, featuring live-cell compatible excitation light levels and high imaging speeds. To achieve SIM, spatial modulation of the fluorescence excitation light is employed. This is typically achieved by interfering coherent laser beams in the sample plane, which are often created by spatial light modulators (SLMs). Digital micromirror devices (DMDs) are a form of SLMs with certain advantages, such as high speed, low cost and wide availability, which present certain hurdles in their implementation, mainly the blazed grating effect caused by the jagged surface structure of the tilted mirrors. Recent works have studied this effect through modelling, simulations and experiments, and laid out possible implementations of multi-color SIM imaging based on DMDs. Here, we present an implementation of a dual-color DMD based SIM microscope using temperature-controlled wavelength matching. By carefully controlling the output wavelength of a diode laser by temperature, we can tune two laser wavelengths in such a way that no opto-mechanical realignment of the SIM setup is necessary when switching between both wavelengths. This reduces system complexity and increases imaging speed. With measurements on nano-bead reference samples, as well as the actin skeleton and membrane of fixed U2OS cells, we demonstrate the capabilities of the setup.

Fast 3D super-resolution imaging using a digital micromirror device and binary holography

SIGNIFICANCE

High-speed three-dimensional (3D) super-resolution microscopy is a unique tool to investigate various biological phenomena; yet the technology is not broadly adopted due to its high cost and complex system design.

AIM

We present a compact, low-cost, and high-speed 3D structured illumination microscopy (SIM) based on a digital micromirror device and binary holography to visualize fast biological events with super-resolution.

APPROACH

The 3D SIM uses a digital micromirror device to generate three laser foci with individually controllable positions, phases, and amplitudes via binary holography at the back aperture of objective lens to form optimal 3D structured patterns. Fifteen raw images are sequentially recorded and processed by the 3D SIM algorithm to reconstruct a super-resolved image.

RESULTS

Super-resolution 3D imaging at a speed of 26.7 frames per second is achieved with a lateral and axial resolution of 155 and 487 nm, which corresponds to a 1.65- and 1.63-times resolution enhancement, respectively, comparing with standard deconvolution microscopy.

CONCLUSIONS

The 3D SIM realizes fast super-resolution imaging with optimal 3D structured illumination, which may find important applications in biophotonics.

Super-Resolution Fluorescence Microscopy Methods for Assessing Mouse Biology

Super-resolution (diffraction unlimited) microscopy was developed 15 years ago; the developers were awarded the Nobel Prize in Chemistry in recognition of their work in 2014. Super-resolution microscopy is increasingly being applied to diverse scientific fields, from single molecules to cell organelles, viruses, bacteria, plants, and animals, especially the mammalian model organism Mus musculus. In this review, we explain how super-resolution microscopy, along with fluorescence microscopy from which it grew, has aided the renaissance of the light microscope. We cover experiment planning and specimen preparation and explain structured illumination microscopy, super-resolution radial fluctuations, stimulated emission depletion microscopy, single-molecule localization microscopy, and super-resolution imaging by pixel reassignment. The final section of this review discusses the strengths and weaknesses of each super-resolution technique and how to choose the best approach for your research. © 2021 The Authors. Current Protocols published by Wiley Periodicals LLC.

Turn-key mapping of cell receptor force orientation and magnitude using a commercial structured illumination microscope

Many cellular processes, including cell division, development, and cell migration require spatially and temporally coordinated forces transduced by cell-surface receptors. Nucleic acid-based molecular tension probes allow one to visualize the piconewton (pN) forces applied by these receptors. Building on this technology, we recently developed molecular force microscopy (MFM) which uses fluorescence polarization to map receptor force orientation with diffraction-limited resolution (~250 nm). Here, we show that structured illumination microscopy (SIM), a super-resolution technique, can be used to perform super-resolution MFM. Using SIM-MFM, we generate the highest resolution maps of both the magnitude and orientation of the pN traction forces applied by cells. We apply SIM-MFM to map platelet and fibroblast integrin forces, as well as T cell receptor forces. Using SIM-MFM, we show that platelet traction force alignment occurs on a longer timescale than adhesion. Importantly, SIM-MFM can be implemented on any standard SIM microscope without hardware modifications.

Microelectrode Arrays for Simultaneous Electrophysiology and Advanced Optical Microscopy

Advanced optical imaging techniques address important biological questions in neuroscience, where structures such as synapses are below the resolution limit of a conventional microscope. At the same time, microelectrode arrays (MEAs) are indispensable in understanding the language of neurons. Here, the authors show transparent MEAs capable of recording action potentials from neurons and compatible with advanced microscopy. The electrodes are made of the conducting polymer poly(3,4-ethylenedioxythiophene) doped with polystyrene sulfonate (PEDOT:PSS) and are patterned by optical lithography, ensuring scalable fabrication with good control over device parameters. A thickness of 380 nm ensures low enough impedance and >75% transparency throughout the visible part of the spectrum making them suitable for artefact-free recording in the presence of laser illumination. Using primary neuronal cells, the arrays record single units from multiple nearby sources with a signal-to-noise ratio of 7.7 (17.7 dB). Additionally, it is possible to perform calcium (Ca²⁺) imaging, a measure of neuronal activity, using the novel transparent electrodes. Different biomarkers are imaged through the electrodes using conventional and super-resolution microscopy (SRM), showing no qualitative differences compared to glass substrates. These transparent MEAs pave the way for harnessing the synergy between the superior temporal resolution of electrophysiology and the selectivity and high spatial resolution of optical imaging.

Structured illumination microscopy with noise-controlled image reconstructions

Super-resolution structured illumination microscopy (SIM) has become a widely used method for biological imaging. Standard reconstruction algorithms, however, are prone to generate noise-specific artifacts that limit their applicability for lower signal-to-noise data. Here we present a physically realistic noise model that explains the structured noise artifact, which we then use to motivate new complementary reconstruction approaches. True-Wiener-filtered SIM optimizes contrast given the available signal-to-noise ratio, and flat-noise SIM fully overcomes the structured noise artifact while maintaining resolving power. Both methods eliminate ad hoc user-adjustable reconstruction parameters in favor of physical parameters, enhancing objectivity. The new reconstructions point to a trade-off between contrast and a natural noise appearance. This trade-off can be partly overcome by further notch filtering but at the expense of a decrease in signal-to-noise ratio. The benefits of the proposed approaches are demonstrated on focal adhesion and tubulin samples in two and three dimensions, and on nanofabricated fluorescent test patterns.

Self-contained and modular structured illumination microscope

We present a modular implementation of structured illumination microscopy (SIM) that is fast, largely self-contained and that can be added onto existing fluorescence microscopes. Our strategy, which we call HIT-SIM, can theoretically deliver well over 50 super-resolved images per second and is readily compatible with existing acquisition software packages. We provide a full technical package consisting of schematics, a list of components and an alignment scheme that provides detailed specifications and assembly instructions. We illustrate the performance of the instrument by imaging optically large samples containing sequence-specifically stained DNA fragments.