Phase optimisation for structured illumination microscopy

Surpassing light inhomogeneities in structured-illumination microscopy with FlexSIM

Super-resolution structured-illumination microscopy (SIM) is a powerful technique that allows one to surpass the diffraction limit by up to a factor two. Yet, its practical use is hampered by its sensitivity to imaging conditions which makes it prone to reconstruction artefacts. In this work, we present FlexSIM, a flexible SIM reconstruction method capable to handle highly challenging data. Specifically, we demonstrate the ability of FlexSIM to deal with the distortion of patterns, the high level of noise encountered in live imaging, as well as out-of-focus fluorescence. Moreover, we show that FlexSIM achieves state-of-the-art performance over a variety of open SIM datasets.

Ultra-high spatio-temporal resolution imaging with parallel acquisition-readout structured illumination microscopy (PAR-SIM)

Structured illumination microscopy (SIM) has emerged as a promising super-resolution fluorescence imaging technique, offering diverse configurations and computational strategies to mitigate phototoxicity during real-time imaging of biological specimens. Traditional efforts to enhance system frame rates have concentrated on processing algorithms, like rolling reconstruction or reduced frame reconstruction, or on investments in costly sCMOS cameras with accelerated row readout rates. In this article, we introduce an approach to elevate SIM frame rates and region of interest (ROI) coverage at the hardware level, without necessitating an upsurge in camera expenses or intricate algorithms. Here, parallel acquisition-readout SIM (PAR-SIM) achieves the highest imaging speed for fluorescence imaging at currently available detector sensitivity. By using the full frame-width of the detector through synchronizing the pattern generation and image exposure-readout process, we have achieved a fundamentally stupendous information spatial-temporal flux of 132.9 MPixels · s-1, 9.6-fold that of the latest techniques, with the lowest SNR of -2.11 dB and 100 nm resolution. PAR-SIM demonstrates its proficiency in successfully reconstructing diverse cellular organelles in dual excitations, even under conditions of low signal due to ultra-short exposure times. Notably, mitochondrial dynamic tubulation and ongoing membrane fusion processes have been captured in live COS-7 cell, recorded with PAR-SIM at an impressive 408 Hz. We posit that this novel parallel exposure-readout mode not only augments SIM pattern modulation for superior frame rates but also holds the potential to benefit other complex imaging systems with a strategic controlling approach.

Selective-plane-activation structured illumination microscopy

The background light from out-of-focus planes hinders resolution enhancement in structured illumination microscopy when observing volumetric samples. Here we used selective plane illumination and reversibly photoswitchable fluorescent proteins to realize structured illumination within the focal plane and eliminate the out-of-focus background. Theoretical investigation of the imaging properties and experimental demonstrations show that selective plane activation is beneficial for imaging dense microstructures in cells and cell spheroids.

Projective oblique plane structured illumination microscopy

Structured illumination microscopy (SIM) can double the spatial resolution of a fluorescence microscope and video rate live cell imaging in a two-dimensional format has been demonstrated. However, rapid implementations of 2D SIM typically only cover a narrow slice of the sample immediately at the coverslip, with most of the cellular volume out of reach. Here we implement oblique plane structured illumination microscopy (OPSIM) in a projection format to rapidly image an entire cell in a 2D SIM framework. As no mechanical scanning of the sample or objective is involved, this technique has the potential for rapid projection imaging with doubled resolution. We characterize the spatial resolution with fluorescent nanospheres, compare projection and 3D imaging using OPSIM and image mitochondria and ER dynamics across an entire cell at up to 2.7 Hz. To our knowledge, this represents the fastest whole cell SIM imaging to date.

Superresolution structured illumination microscopy reconstruction algorithms: a review

Structured illumination microscopy (SIM) has become the standard for next-generation wide-field microscopy, offering ultrahigh imaging speed, superresolution, a large field-of-view, and long-term imaging. Over the past decade, SIM hardware and software have flourished, leading to successful applications in various biological questions. However, unlocking the full potential of SIM system hardware requires the development of advanced reconstruction algorithms. Here, we introduce the basic theory of two SIM algorithms, namely, optical sectioning SIM (OS-SIM) and superresolution SIM (SR-SIM), and summarize their implementation modalities. We then provide a brief overview of existing OS-SIM processing algorithms and review the development of SR-SIM reconstruction algorithms, focusing primarily on 2D-SIM, 3D-SIM, and blind-SIM. To showcase the state-of-the-art development of SIM systems and assist users in selecting a commercial SIM system for a specific application, we compare the features of representative off-the-shelf SIM systems. Finally, we provide perspectives on the potential future developments of SIM.

Quantitative structured illumination microscopy via a physical model-based background filtering algorithm reveals actin dynamics

Despite the prevalence of superresolution (SR) microscopy, quantitative live-cell SR imaging that maintains the completeness of delicate structures and the linearity of fluorescence signals remains an uncharted territory. Structured illumination microscopy (SIM) is the ideal tool for live-cell SR imaging. However, it suffers from an out-of-focus background that leads to reconstruction artifacts. Previous post hoc background suppression methods are prone to human bias, fail at densely labeled structures, and are nonlinear. Here, we propose a physical model-based Background Filtering method for living cell SR imaging combined with the 2D-SIM reconstruction procedure (BF-SIM). BF-SIM helps preserve intricate and weak structures down to sub-70 nm resolution while maintaining signal linearity, which allows for the discovery of dynamic actin structures that, to the best of our knowledge, have not been previously monitored.

Reconstruction algorithm using 2N+1 raw images for structured illumination microscopy

This paper presents a structured illumination microscopy (SIM) reconstruction algorithm that allows the reconstruction of super-resolved images with 2N + 1 raw intensity images, with N being the number of structured illumination directions used. The intensity images are recorded after using a 2D grating for the projection fringe and a spatial light modulator to select two orthogonal fringe orientations and perform phase shifting. Super-resolution images can be reconstructed from the five intensity images, enhancing the imaging speed and reducing the photobleaching by 17%, compared to conventional two-direction and three-step phase-shifting SIM. We believe the proposed technique will be further developed and widely applied in many fields.

Structured Illumination Microscopy of Mitochondrial in Mouse Hepatocytes with an Improved Image Reconstruction Algorithm

In this paper, a structured illumination microscopy (SIM) image reconstruction algorithm combined with notch function (N-SIM) is proposed. This method suppresses the defocus signal in the imaging process by processing the low-frequency signal of the image. The existing super-resolution image reconstruction algorithm produces streak artifacts caused by defocus signal. The experimental results show that the algorithm proposed in our study can well suppress the streak artifacts caused by defocused signals during the imaging process without losing the effective information of the image. The image reconstruction algorithm is used to analyze the mouse hepatocytes, and the image processing tool developed by MATLAB is applied to identify, detect and count the reconstructed images of mitochondria and lipid droplets, respectively. It is found that the mitochondrial activity in oxidative stress induced growth inhibitor 1 (OSGIN1) overexpressed mouse hepatocytes is higher than that in normal cells, and the interaction with lipid droplets is more obvious. This paper provides a reliable subcellular observation platform, which is very meaningful for biomedical work.

Parameter estimation of the structured illumination pattern based on principal component analysis (PCA): PCA-SIM

Principal component analysis (PCA), a common dimensionality reduction method, is introduced into SIM to identify the frequency vectors and pattern phases of the illumination pattern with precise subpixel accuracy, fast speed, and noise-robustness, which is promising for real-time and long-term live-cell imaging.

Machine learning assisted interferometric structured illumination microscopy for dynamic biological imaging

Structured Illumination Microscopy, SIM, is one of the most powerful optical imaging methods available to visualize biological environments at subcellular resolution. Its limitations stem from a difficulty of imaging in multiple color channels at once, which reduces imaging speed. Furthermore, there is substantial experimental complexity in setting up SIM systems, preventing a widespread adoption. Here, we present Machine-learning Assisted, Interferometric Structured Illumination Microscopy, MAI-SIM, as an easy-to-implement method for live cell super-resolution imaging at high speed and in multiple colors. The instrument is based on an interferometer design in which illumination patterns are generated, rotated, and stepped in phase through movement of a single galvanometric mirror element. The design is robust, flexible, and works for all wavelengths. We complement the unique properties of the microscope with an open source machine-learning toolbox that permits real-time reconstructions to be performed, providing instant visualization of super-resolved images from live biological samples.

Electron-beam patterned calibration structures for structured illumination microscopy

Super-resolution fluorescence microscopy can be achieved by image reconstruction after spatially patterned illumination or sequential photo-switching and read-out. Reconstruction algorithms and microscope performance are typically tested using simulated image data, due to a lack of strategies to pattern complex fluorescent patterns with nanoscale dimension control. Here, we report direct electron-beam patterning of fluorescence nanopatterns as calibration standards for super-resolution fluorescence. Patterned regions are identified with both electron microscopy and fluorescence labelling of choice, allowing precise correlation of predefined pattern dimensions, a posteriori obtained electron images, and reconstructed super-resolution images.

Resolution doubling in light-sheet microscopy via oblique plane structured illumination

Structured illumination microscopy (SIM) doubles the spatial resolution of a fluorescence microscope without requiring high laser powers or specialized fluorophores. However, the excitation of out-of-focus fluorescence can accelerate photobleaching and phototoxicity. In contrast, light-sheet fluorescence microscopy (LSFM) largely avoids exciting out-of-focus fluorescence, thereby enabling volumetric imaging with low photobleaching and intrinsic optical sectioning. Combining SIM with LSFM would enable gentle three-dimensional (3D) imaging at doubled resolution. However, multiple orientations of the illumination pattern, which are needed for isotropic resolution doubling in SIM, are challenging to implement in a light-sheet format. Here we show that multidirectional structured illumination can be implemented in oblique plane microscopy, an LSFM technique that uses a single objective for excitation and detection, in a straightforward manner. We demonstrate isotropic lateral resolution below 150 nm, combined with lower phototoxicity compared to traditional SIM systems and volumetric acquisition speed exceeding 1 Hz.

An LED-Based structured illumination microscope using a digital micromirror device and GPU accelerated image reconstruction

When combined with computational approaches, fluorescence imaging becomes one of the most powerful tools in biomedical research. It is possible to achieve resolution figures beyond the diffraction limit, and improve the performance and flexibility of high-resolution imaging systems with techniques such as structured illumination microscopy (SIM) reconstruction. In this study, the hardware and software implementation of an LED-based super-resolution imaging system using SIM employing GPU accelerated parallel image reconstruction is presented. The sample is illuminated with two-dimensional sinusoidal patterns with various orientations and lateral phase shifts generated using a digital micromirror device (DMD). SIM reconstruction is carried out in frequency space using parallel CUDA kernel functions. Furthermore, a general purpose toolbox for the parallel image reconstruction algorithm and an infrastructure that allows all users to perform parallel operations on images without developing any CUDA kernel code is presented. The developed image reconstruction algorithm was run separately on a CPU and a GPU. Two different SIM reconstruction algorithms have been developed for the CPU as mono-thread CPU algorithm and multi-thread OpenMP CPU algorithm. SIM reconstruction of 1024 × 1024 px images was achieved in 1.49 s using GPU computation, indicating an enhancement by ∼28 and ∼20 in computation time when compared with mono-thread CPU computation and multi-thread OpenMP CPU computation, respectively.

Large-field lattice structured illumination microscopy

In this paper, we present large-field, five-step lattice structured illumination microscopy (Lattice SIM). This method utilizes a 2D grating for lattice projection and a spatial light modulator (SLM) for phase shifting. Five phase-shifted intensity images are recorded to reconstruct a super-resolution image, enhancing the imaging speed and reducing the photo-bleaching both by 17%, compared to conventional two-direction and three-shift SIM. Furthermore, lattice SIM has a three-fold spatial bandwidth product (SBP) enhancement compared to SLM/DMD-based SIM, of which the fringe number is limited by the SLM/DMD pixel number. We believe that the proposed technique will be further developed and widely applied in many fields.

Polarized illumination coded structured illumination microscopy (picoSIM): experimental results

The need for acquiring at least three images to reconstruct an optical section of a sample limits the acquisition rate in structured illumination microscopy (SIM) for optical sectioning. In polarized illumination coded structured illumination microscopy (picoSIM) the three individual light patterns are encoded in a single polarized illumination light distribution, enabling the acquisition of the complete SIM data in a single exposure. Here, we describe our experimental set-up and show experimental results acquired with sequential and single-shot picoSIM. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 2)'.

SIM reconstruction framework for high-speed multi-dimensional super-resolution imaging

Structured illumination microscopy (SIM) holds great promise for live cell imaging applications due to its potential to obtain multidimensional information such as intensity, spectrum and polarization (I, λ , p) at high spatial-temporal resolution, enabling the observation of more complex dynamic interactions between subcellular structures. However, the reconstruction results of polarized samples are prone to artifacts because all current SIM reconstruction frameworks use incomplete imaging models which neglect polarization modulation. Such polarization-related artifacts are especially prevalent for SIM reconstruction using a reduced number of raw images (RSIM) and severely undermine the ability of SIM to capture multi-dimensional information. Here, we report a new SIM reconstruction framework (PRSIM) that can recover multi-dimensional information (I, λ, p) using a reduced number of raw images. PRSIM adopts a complete imaging model that is versatile for normal and polarized samples and uses a frequency-domain iterative reconstruction algorithm for artifact-free super-resolution (SR) reconstruction. It can simultaneously obtain the SR spatial structure and polarization orientation of polarized samples using 6 raw SIM images and can perform SR reconstruction using 4 SIM images for normal samples. In addition, PRSIM has less spatial computational complexity and achieves reconstruction speeds tens of times higher than that of the state-of-the-art non-iterative RSIM, making it more suitable for large field-of-view imaging. Thus, PRSIM is expected to facilitate the development of SIM into an ultra-high-speed and multi-dimensional SR imaging tool.

A role for BDNF- and NMDAR-induced lysosomal recruitment of mTORC1 in the regulation of neuronal mTORC1 activity

Memory and long term potentiation require de novo protein synthesis. A key regulator of this process is mTORC1, a complex comprising the mTOR kinase. Growth factors activate mTORC1 via a pathway involving PI3-kinase, Akt, the TSC complex and the GTPase Rheb. In non-neuronal cells, translocation of mTORC1 to late endocytic compartments (LEs), where Rheb is enriched, is triggered by amino acids. However, the regulation of mTORC1 in neurons remains unclear. In mouse hippocampal neurons, we observed that BDNF and treatments activating NMDA receptors trigger a robust increase in mTORC1 activity. NMDA receptors activation induced a significant recruitment of mTOR onto lysosomes even in the absence of external amino acids, whereas mTORC1 was evenly distributed in neurons under resting conditions. NMDA receptor-induced mTOR translocation to LEs was partly dependent on the BDNF receptor TrkB, suggesting that BDNF contributes to the effect of NMDA receptors on mTORC1 translocation. In addition, the combination of Rheb overexpression and artificial mTORC1 targeting to LEs by means of a modified component of mTORC1 fused with a LE-targeting motif strongly activated mTOR. To gain spatial and temporal control over mTOR localization, we designed an optogenetic module based on light-sensitive dimerizers able to recruit mTOR on LEs. In cells expressing this optogenetic tool, mTOR was translocated to LEs upon photoactivation. In the absence of growth factor, this was not sufficient to activate mTORC1. In contrast, mTORC1 was potently activated by a combination of BDNF and photoactivation. The data demonstrate that two important triggers of synaptic plasticity, BDNF and NMDA receptors, synergistically power the two arms of the mTORC1 activation mechanism, i.e., mTORC1 translocation to LEs and Rheb activation. Moreover, they unmask a functional link between NMDA receptors and mTORC1 that could underlie the changes in the synaptic proteome associated with long-lasting changes in synaptic strength.

Structured illumination microscopy with noise-controlled image reconstructions

Super-resolution structured illumination microscopy (SIM) has become a widely used method for biological imaging. Standard reconstruction algorithms, however, are prone to generate noise-specific artifacts that limit their applicability for lower signal-to-noise data. Here we present a physically realistic noise model that explains the structured noise artifact, which we then use to motivate new complementary reconstruction approaches. True-Wiener-filtered SIM optimizes contrast given the available signal-to-noise ratio, and flat-noise SIM fully overcomes the structured noise artifact while maintaining resolving power. Both methods eliminate ad hoc user-adjustable reconstruction parameters in favor of physical parameters, enhancing objectivity. The new reconstructions point to a trade-off between contrast and a natural noise appearance. This trade-off can be partly overcome by further notch filtering but at the expense of a decrease in signal-to-noise ratio. The benefits of the proposed approaches are demonstrated on focal adhesion and tubulin samples in two and three dimensions, and on nanofabricated fluorescent test patterns.

GPU-accelerated real-time reconstruction in Python of three-dimensional datasets from structured illumination microscopy with hexagonal patterns

We present a structured illumination microscopy system that projects a hexagonal pattern by the interference among three coherent beams, suitable for implementation in a light-sheet geometry. Seven images acquired as the illumination pattern is shifted laterally can be processed to produce a super-resolved image that surpasses the diffraction-limited resolution by a factor of over 2 in an exemplar light-sheet arrangement. Three methods of processing data are discussed depending on whether the raw images are available in groups of seven, individually in a stream or as a larger batch representing a three-dimensional stack. We show that imaging axially moving samples can introduce artefacts, visible as fine structures in the processed images. However, these artefacts are easily removed by a filtering operation carried out as part of the batch processing algorithm for three-dimensional stacks. The reconstruction algorithms implemented in Python include specific optimizations for calculation on a graphics processing unit and we demonstrate its operation on experimental data of static objects and on simulated data of moving objects. We show that the software can process over 239 input raw frames per second at 512 × 512 pixels, generating over 34 super-resolved frames per second at 1024 × 1024 pixels. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 1)'.

Super-resolved live-cell imaging using random illumination microscopy

Current super-resolution microscopy (SRM) methods suffer from an intrinsic complexity that might curtail their routine use in cell biology. We describe here random illumination microscopy (RIM) for live-cell imaging at super-resolutions matching that of 3D structured illumination microscopy, in a robust fashion. Based on speckled illumination and statistical image reconstruction, easy to implement and user-friendly, RIM is unaffected by optical aberrations on the excitation side, linear to brightness, and compatible with multicolor live-cell imaging over extended periods of time. We illustrate the potential of RIM on diverse biological applications, from the mobility of proliferating cell nuclear antigen (PCNA) in U2OS cells and kinetochore dynamics in mitotic S. pombe cells to the 3D motion of myosin minifilaments deep inside Drosophila tissues. RIM's inherent simplicity and extended biological applicability, particularly for imaging at increased depths, could help make SRM accessible to biology laboratories.

Chromatin investigation in the nucleus using a phasor approach to structured illumination microscopy

Chromatin in the nucleus is organized in functional sites at variable level of compaction. Structured illumination microscopy (SIM) can be used to generate three-dimensional super-resolution (SR) imaging of chromatin by changing in phase and in orientation a periodic line illumination pattern. The spatial frequency domain is the natural choice to process SIM raw data and to reconstruct an SR image. Using an alternative approach, we demonstrate that the additional spatial information encoded in the knowledge of the position of the illumination pattern can be efficiently decoded using a generalized version of separation of photon by lifetime tuning (SPLIT) that does not require lifetime measurements. In the resulting SPLIT-SIM, the SR image is obtained by isolating a fraction of the intensity corresponding to the center of the diffraction-limited point spread function. This extends the use of the SPLIT approach from stimulated emission depletion microscopy to SIM. The SPLIT-SIM algorithm is based only on phasor analysis and does not require deconvolution. We show that SPLIT-SIM can be used to generate SR images of chromatin organizational motifs with tunable resolution and can be a valuable tool for the imaging of functional sites in the nucleus.

ML-SIM: universal reconstruction of structured illumination microscopy images using transfer learning

Structured illumination microscopy (SIM) has become an important technique for optical super-resolution imaging because it allows a doubling of image resolution at speeds compatible with live-cell imaging. However, the reconstruction of SIM images is often slow, prone to artefacts, and requires multiple parameter adjustments to reflect different hardware or experimental conditions. Here, we introduce a versatile reconstruction method, ML-SIM, which makes use of transfer learning to obtain a parameter-free model that generalises beyond the task of reconstructing data recorded by a specific imaging system for a specific sample type. We demonstrate the generality of the model and the high quality of the obtained reconstructions by application of ML-SIM on raw data obtained for multiple sample types acquired on distinct SIM microscopes. ML-SIM is an end-to-end deep residual neural network that is trained on an auxiliary domain consisting of simulated images, but is transferable to the target task of reconstructing experimental SIM images. By generating the training data to reflect challenging imaging conditions encountered in real systems, ML-SIM becomes robust to noise and irregularities in the illumination patterns of the raw SIM input frames. Since ML-SIM does not require the acquisition of experimental training data, the method can be efficiently adapted to any specific experimental SIM implementation. We compare the reconstruction quality enabled by ML-SIM with current state-of-the-art SIM reconstruction methods and demonstrate advantages in terms of generality and robustness to noise for both simulated and experimental inputs, thus making ML-SIM a useful alternative to traditional methods for challenging imaging conditions. Additionally, reconstruction of a SIM stack is accomplished in less than 200 ms on a modern graphics processing unit, enabling future applications for real-time imaging. Source code and ready-to-use software for the method are available at http://ML-SIM.github.io.

Highly compact and cost-effective 2-beam super-resolution structured illumination microscope based on all-fiber optic components

Current super-resolution structured illumination microscopes (SR-SIM) utilize relatively expensive electro-optic components and free-space optics, resulting in large setups. Moreover, high power laser sources are required to compensate for the losses associated with generating the illumination pattern by diffractive optics. Here, we present a highly compact and flexible 2D SR-SIM microscope based on all-fiber optic components (fiberSIM). Fiber-splitters deliver the laser light to the sample resulting in the interference illumination pattern. A microelectromechanical systems (MEMS) based fiber switch performs rapid pattern rotation. The pattern phase shift is achieved by the spatial displacement of one arm of the fiber interferometer using a piezoelectric crystal. Compared with existing methods, fiberSIM is highly compact and significantly reduces the SR-SIM component cost while achieving comparable results, thus providing a route to making SR-SIM technology accessible to even more laboratories in the life sciences.

High-fidelity structured illumination microscopy by point-spread-function engineering

Structured illumination microscopy (SIM) has become a widely used tool for insight into biomedical challenges due to its rapid, long-term, and super-resolution (SR) imaging. However, artifacts that often appear in SIM images have long brought into question its fidelity, and might cause misinterpretation of biological structures. We present HiFi-SIM, a high-fidelity SIM reconstruction algorithm, by engineering the effective point spread function (PSF) into an ideal form. HiFi-SIM can effectively reduce commonly seen artifacts without loss of fine structures and improve the axial sectioning for samples with strong background. In particular, HiFi-SIM is not sensitive to the commonly used PSF and reconstruction parameters; hence, it lowers the requirements for dedicated PSF calibration and complicated parameter adjustment, thus promoting SIM as a daily imaging tool.

Enhanced reconstruction of structured illumination microscopy on a polarized specimen

Structured illumination microscopy (SIM) requires polarization control to guarantee the high-contrast illumination pattern. However, this modulated polarization will induce artifacts in SIM when imaging fluorescent dipoles. Here we proposed the polarization weighted recombination of frequency components to reconstruct SIM data with suppressed artifacts and better resolving power. Both the simulation results and experimental data demonstrate that our algorithm can obtain isotropic resolution on dipoles and resolve a clearer structure in high-density sections compared to the conventional algorithm. Our work reinforces the SIM theory and paves the avenue for the application of SIM on a polarized specimen.

Fast TIRF-SIM imaging of dynamic, low-fluorescent biological samples

Fluorescence microscopy is the standard imaging technique to investigate the structures and dynamics of living cells. However, increasing the spatial resolution comes at the cost of temporal resolution and vice versa. In addition, the number of images that can be taken in sufficiently high quality is limited by fluorescence bleaching. Hence, super-resolved imaging at several Hertz of low fluorescent biological samples is still a big challenge and, especially in structured illumination microscopy (SIM), is often visible as imaging artifacts. In this paper, we present a TIRF-SIM system based on scan-mirrors and a Michelson interferometer, which generates images at 110 nm spatial resolution and up to 8 Hz temporal resolution. High resolution becomes possible by optimizing the illumination interference contrast, even for low fluorescent, moving samples. We provide a framework and guidelines on how the modulation contrast, which depends on laser coherence, polarization, beam displacement or sample movements, can be mapped over the entire field of view. In addition, we characterize the influence of the signal-to-noise ratio and the Wiener filtering on the quality of reconstructed SIM images, both in real and frequency space. Our results are supported by theoretical descriptions containing the parameters leading to image artifacts. This study aims to help microscopists to better understand and adjust optical parameters for structured illumination, thereby leading to more trustworthy measurements and analyses of biological dynamics.

Fast reconstruction algorithm for structured illumination microscopy

Structured illumination microscopy (SIM) is a powerful technique for providing super-resolution imaging, but its reconstruction algorithm, i.e., linear reconstruction structured illumination microscopy (LRSIM) algorithm in the Fourier domain, limits the imaging speed due to its computational effort. Here, we present a novel reconstruction algorithm that can directly process SIM data in the spatial domain. Compared to LRSIM, this approach uses the same number of frames to achieve a comparable resolution but with a much faster processing speed. Our algorithm was verified on both simulated and experimental data using sinusoidal pattern illumination. Moreover, this algorithm is also applicable for speckle pattern illumination.

Jointly super-resolved and optically sectioned Bayesian reconstruction method for structured illumination microscopy

Structured Illumination Microscopy (SIM) is an imaging technique for achieving both super-resolution (SR) and optical sectioning (OS) in wide-field microscopy. It consists in illuminating the sample with periodic patterns at different orientations and positions. The resulting images are then processed to reconstruct the observed object with SR and/or OS. In this work, we present BOSSA-SIM, a general-purpose SIM reconstruction method, applicable to moving objects such as encountered in in vivo retinal imaging, that enables SR and OS jointly in a fully unsupervised Bayesian framework. By modeling a 2-layer object composed of an in-focus layer and a defocused layer, we show that BOSSA-SIM is able to jointly reconstruct them so as to get a super-resolved and optically sectioned in-focus layer. The achieved performance, assessed quantitatively by simulations for several noise levels, compares favorably with a state-of-the-art method. Finally, we validate our method on open-access experimental microscopy data.

Video-rate multi-color structured illumination microscopy with simultaneous real-time reconstruction

Super-resolved structured illumination microscopy (SR-SIM) is among the fastest fluorescence microscopy techniques capable of surpassing the optical diffraction limit. Current custom-build instruments are able to deliver two-fold resolution enhancement with high acquisition speed. SR-SIM is usually a two-step process, with raw-data acquisition and subsequent, time-consuming post-processing for image reconstruction. In contrast, wide-field and (multi-spot) confocal techniques produce high-resolution images instantly. Such immediacy is also possible with SR-SIM, by tight integration of a video-rate capable SIM with fast reconstruction software. Here we present instant SR-SIM by VIGOR (Video-rate Immediate GPU-accelerated Open-Source Reconstruction). We demonstrate multi-color SR-SIM at video frame-rates, with less than 250 ms delay between measurement and reconstructed image display. This is achieved by modifying and extending high-speed SR-SIM image acquisition with a new, GPU-enhanced, network-enabled image-reconstruction software. We demonstrate high-speed surveying of biological samples in multiple colors and live imaging of moving mitochondria as an example of intracellular dynamics.

Sequential acquisition and image reconstruction in super-resolved structured illumination microscopy (SR-SIM) is time-consuming. Here the authors optimise both acquisition and reconstruction software to achieve multicolour SR-SIM at video frame-rates with reconstructed images displaying with only milliseconds delay during the experiment.

Fast and robust phase-shift estimation in two-dimensional structured illumination microscopy

A method of determining unknown phase-shifts between elementary images in two-dimensional Structured Illumination Microscopy (2D-SIM) is presented. The proposed method is based on the comparison of the peak intensity of spectral components. These components correspond to the inherent structured illumination spectral content and the residual component that appears from wrongly estimated phase-shifts. The estimation of the phase-shifts is carried out by finding the absolute maximum of a function defined as the normalized peak intensity difference in the Fourier domain. This task is performed by an optimization method providing a fast estimation of the phase-shift. The algorithm stability and robustness are tested for various levels of noise and contrasts of the structured illumination pattern. Furthermore, the proposed approach reduces the number of computations compared to other existing techniques. The method is supported by the theoretical calculations and validated by means of simulated and experimental results.

Dynamic super-resolution structured illumination imaging in the living brain

Cells in the brain act as components of extended networks. Therefore, to understand neurobiological processes in a physiological context, it is essential to study them in vivo. Super-resolution microscopy has spatial resolution beyond the diffraction limit, thus promising to provide structural and functional insights that are not accessible with conventional microscopy. However, to apply it to in vivo brain imaging, we must address the challenges of 3D imaging in an optically heterogeneous tissue that is constantly in motion. We optimized image acquisition and reconstruction to combat sample motion and applied adaptive optics to correcting sample-induced optical aberrations in super-resolution structured illumination microscopy (SIM) in vivo. We imaged the brains of live zebrafish larvae and mice and observed the dynamics of dendrites and dendritic spines at nanoscale resolution.

Quantitative Analysis of Nuclear Lamins Imaged by Super-Resolution Light Microscopy

The nuclear lamina consists of a dense fibrous meshwork of nuclear lamins, Type V intermediate filaments, and is ~14 nm thick according to recent cryo-electron tomography studies. Recent advances in light microscopy have extended the resolution to a scale allowing for the fine structure of the lamina to be imaged in the context of the whole nucleus. We review quantitative approaches to analyze the imaging data of the nuclear lamina as acquired by structured illumination microscopy (SIM) and single molecule localization microscopy (SMLM), as well as the requisite cell preparation techniques. In particular, we discuss the application of steerable filters and graph-based methods to segment the structure of the four mammalian lamin isoforms (A, C, B1, and B2) and extract quantitative information.

Optoacoustic mesoscopy for biomedicine

Fuelled by innovation, optical microscopy plays a critical role in the life sciences and medicine, from basic discovery to clinical diagnostics. However, optical microscopy is limited by typical penetration depths of a few hundred micrometres for in vivo interrogations in the visible spectrum. Optoacoustic microscopy complements optical microscopy by imaging the absorption of light, but it is similarly limited by penetration depth. In this Review, we summarize progress in the development and applicability of optoacoustic mesoscopy (OPAM); that is, optoacoustic imaging with acoustic resolution and wide-bandwidth ultrasound detection. OPAM extends the capabilities of optical imaging beyond the depths accessible to optical and optoacoustic microscopy, and thus enables new applications. We explain the operational principles of OPAM, its placement as a bridge between optoacoustic microscopy and optoacoustic macroscopy, and its performance in the label-free visualization of tissue pathophysiology, such as inflammation, oxygenation, vascularization and angiogenesis. We also review emerging applications of OPAM in clinical and biological imaging.

Tunable structured illumination light sheet microscopy for background rejection and imaging depth in minimally processed tissues

We demonstrate improved optical sectioning in light sheet fluorescence microscopy using tunable structured illumination (SI) frequencies to optimize image quality in scattering specimens. The SI patterns are generated coherently using a one-dimensional spatial light modulator for maximum pattern contrast, and the pattern spatial frequency is adjustable up to half the incoherent cutoff frequency of our detection objective. At this frequency, we demonstrate background reductions of 2 orders of magnitude.

Structured illumination imaging without grating rotation based on mirror operation on 1D Fourier spectrum

Structured illumination microscopy (SIM) is a rapidly developing a super-resolution optical microscopy technique. With SIM, the grating is needed in order to rotate several angles for illuminating the sample in different directions. Multiple rotations reduce the imaging speed and grating rotation angle errors damage the image recovery quality. We introduce mirror transformation on one-dimension (1D) Fourier spectrum to SIM for resolving the problems of low imaging speed and severe impact on image reconstruction quality by grating rotation angle errors. When mirror operation and SIM are combined, the grating is placed at an orientation for obtaining three shadow images. The three shadow images are acquired by CCD at three different phase shift for a direction of grating. Thus, the SIM imaging speed is faster and the effect on image reconstruction quality by grating rotation angle errors is greatly reduced.

Imaging neural events in zebrafish larvae with linear structured illumination light sheet fluorescence microscopy

Light sheet fluorescence microscopy (LSFM) is a powerful tool for investigating model organisms including zebrafish. However, due to scattering and refractive index variations within the sample, the resulting image often suffers from low contrast. Structured illumination (SI) has been combined with scanned LSFM to remove out-of-focus and scattered light using square-law detection. Here, we demonstrate that the combination of LSFM with linear reconstruction SI can further increase resolution and contrast in the vertical and axial directions compared to the widely adopted root-mean square reconstruction method while using the same input images. We apply this approach to imaging neural activity in 7-day postfertilization zebrafish larvae. We imaged two-dimensional sections of the zebrafish central nervous system in two colors at an effective frame rate of 7 frames per second.

Inverse matrix based phase estimation algorithm for structured illumination microscopy

The fast imaging speed and low-intensity requirement of structured illumination microscopy (SIM) have made it one of the most widely used imaging tools in live cell imaging. In order to obtain a high fidelity reconstructed image, a precise estimation of the phase of the illumination pattern is required, especially in those structured illumination based techniques that rely on high-order harmonics to improve the resolution. This can be achieved in one of two fundamental ways. The first is to build a high-end control system capable of shifting a sinusoidal pattern with high precision, while the second is to apply estimation algorithms to determine how patterns shift during post-processing. The latter method is preferred in low-cost super-resolution imaging systems; however, existing algorithms are either time-consuming or fail due to noise and a low modulation depth. In this paper, we introduce additional matrixes into the phase estimation algorithm and propose an inverse matrix based phase estimation method with which analytical solutions of the phases can be determined without iteration. The proposed algorithm was validated via simulation and experiments using a home-made total internal reflection fluorescent SIM system (TIRF-SIM). When tested, the method obtained the true phase even when the modulation depth was low. The source code is now available for download by researchers and others.

Single particle trajectories reveal active endoplasmic reticulum luminal flow

The Endoplasmic Reticulum (ER), a network of membranous sheets and pipes, supports functions encompassing biogenesis of secretory proteins and delivery of functional solutes throughout the cell1,2. Molecular mobility through the ER network enables these functionalities, but diffusion alone is not sufficient to explain luminal transport across supramicron distances. Understanding the ER structure-function relationship is critical in light of mutations in ER morphology regulating proteins that give rise to neurodegenerative disorders3,4. Here, super-resolution microscopy and analysis of single particle trajectories of ER luminal proteins revealed that the topological organization of the ER correlates with distinct trafficking modes of its luminal content: with a dominant diffusive component in tubular junctions and a fast flow component in tubules. Particle trajectory orientations resolved over time revealed an alternating current of the ER contents, whilst fast ER super-resolution identified energy-dependent tubule contraction events at specific points as a plausible mechanism for generating active ER luminal flow. The discovery of active flow in the ER has implications for timely ER content distribution throughout the cell, particularly important for cells with extensive ER-containing projections such as neurons.

A frequency domain SIM reconstruction algorithm using reduced number of images

Conventional two-dimensional structured illumination microscopy (SIM) requires 9 raw images to reconstruct a super-resolved image. In order to increase the frame rate of 2DSIM, attempts are being made to reduce the number of raw SIM images. However, all the proposed SIM reconstruction algorithms (SIM-RA) capable of reconstructing super-resolution (SR) image with a reduced number of raw SIM images operate in the spatial domain. Here, we present a frequency domain SIM-RA based on ordinary least squares technique, which enables reconstruction of SR image using 4 raw SIM images. Unlike the spatial domain RA, which produces the SR image through iterative convergence, the presented RA provides a single step solution. It also reveals the fundamental limitation of least number of raw images required for resolution doubling in SIM.

Widefield and total internal reflection fluorescent structured illumination microscopy with scanning galvo mirrors

We present an alternative approach to realize structured illumination microscopy (SIM), which is capable for live cell imaging. The prototype utilizes two sets of scanning galvo mirrors, a polarization converter and a piezo-platform to generate a fast shifted, s-polarization interfered and periodic variable illumination patterns. By changing the angle of the scanning galvanometer, we can change the position of the spots at the pupil plane of the objective lens arbitrarily, making it easy to switch between widefield and total internal reflection fluorescent-SIM mode and adapting the penetration depth in the sample. Also, a twofold resolution improvement is achieved in our experiments. The prototype offers more flexibility of pattern period and illumination orientation changing than previous systems.

Image restoration approach to address reduced modulation contrast in structured illumination microscopy

The performance of structured illumination microscopy (SIM) is hampered in many biological applications due to the inability to modulate the light when imaging deep into the sample. This is in part because sample-induced aberration reduces the modulation contrast of the structured pattern. In this paper, we present an image restoration approach suitable for processing raw incoherent-grid-projection SIM data with a low fringe contrast. Restoration results from simulated and experimental ApoTome SIM data show results with improved signal-to-noise ratio (SNR) and optical sectioning compared to the results obtained from existing methods, such as 2D demodulation and 3D SIM deconvolution. Our proposed method provides satisfactory results (quantified by the achieved SNR and normalized mean square error) even when the modulation contrast of the illumination pattern is as low as 7%.

Accurate estimation of the illumination pattern's orientation and wavelength in sinusoidal structured illumination microscopy

Structured illumination microscopy is able to improve the spatial resolution of wide-field fluorescence imaging by applying sinusoidal stripe pattern illumination to the sample. The corresponding computational image reconstruction requires precise knowledge of the pattern's parameters, which are its phase (ϕ) and wave vector (p). Here, a computationally inexpensive method for estimation of p from the raw data is proposed and illustrated with simulations. The method estimates p through a selective discrete Fourier transform at tunable subpixel precision. This results in an accurate p estimation for all the illumination patterns and subsequently improves the superresolution image recovery by a factor of 10 around sharp edges as compared to an integer pixel approach. The technique as presented here is of major interest to the large variety of custom-build systems that are used. The feasibility of the presented method is proven in comparison with published data.

Structured illumination microscopy with interleaved reconstruction (SIMILR)

Structured illumination microscopy (SIM) is the commonly used super-resolution (SR) technique for imaging subcellular dynamics. However, due to its need for multiple illumination patterns, the frame rate is just a fraction of that of conventional microscopy and is thus too slow for fast dynamic studies. A new SR image reconstruction method that maximizes the use of each subframe of the acquisition series is proposed for improving the super-resolved frame rate by N times for N illumination directions. The method requires no changes in raw data and is appropriate for many versions of SIM setup, including those implementing fast illumination pattern generation mechanism based on spatial light modulator or digital micromirror device. The performance of the proposed method is demonstrated through imaging the highly dynamic endoplasmic reticulum where continuous rapid growths or shape changes of tiny structures are observed.

High-resolution light-scattering imaging with two-dimensional hexagonal illumination patterns: system implementation and image reconstruction formulations

Structured illumination microscopy (SIM) was recently adapted to coherent imaging, named structured oblique-illumination microscopy (SOIM), to improve the contrast and resolution of a light-scattering image. Herein, we present high-resolution laterally isotropic SOIM imaging with 2D hexagonal illuminations. The SOIM is implemented in a SIM fluorescence system based on a spatial-light modulator (SLM). We design an SLM pattern to generate diffraction beams at 0° and ± 60.3° simultaneously to form a 2D hexagonal illumination, and undertake calculations to obtain optimal SLM shifts at 19 phases to yield a reconstructed image correctly. Beams of linear and circular polarizations are used to show the effect of polarization on the resolution improvement. We derive the distributions of the electric field of the resultant hexagonal patterns and work out the formulations of the corresponding coherent-scattering imaging for image reconstruction. The reconstructed images of gold nanoparticles (100 nm) confirm the two-fold improvement of resolution and reveal the effect of polarization on resolving adjacent nanoparticles. To demonstrate biological applications, we present the cellular structures of a label-free fixed HeLa cell with improved contrast and resolution. This work enables one to perform high-resolution dual-mode - fluorescence and light-scattering - imaging in a system, and is expected to broaden the applications of SOIM.

Pattern-illuminated Fourier ptychography microscopy with a pattern-estimation algorithm

In this paper we proposed a new method that combines random pattern illumination, the pattern-estimation algorithm, and the Fourier ptychography (FP) algorithm to recover a super-resolution image. We shifted one multispot pattern to different positions to capture images, and estimated these illumination patterns using a gradient descent algorithm that shares the same root with blind structured illumination microscopy (SIM). Based on the captured images and estimated patterns, the FP algorithm is then applied to recover a super-resolution image. Our method, termed as pattern-estimated Fourier ptychography (PEFP) microscopy, does not need the prior information about the scanning position, and is thus insensitive to rotational errors and shift errors. The performance of PEFP has been demonstrated both in simulations and experiments, and PEFP achieves better resolution than the pattern-illuminated FP method when shift errors appear in our simulations. Moreover, PEFP shows strong resistance towards aberrations and works fine when there is noise in the captured image. Compared with a newly proposed blind-SIM method, PEFP also shows better resolution enhancement both in our simulations and experiments. Our method also provides the possibility to extend the application of pattern-illuminated FP to any illumination pattern because we estimated every illumination pattern separately, as blind-SIM does.

Resolution enhancement of the microscopic imaging by unknown sinusoidal structured illumination with iterative algorithm

Microscopy by sinusoidal structured illumination is a conventional method to improve resolution, which largely depends on accurate knowledge of the illumination pattern. Two steps are included in the reconstruction process of our proposed technique. The first step solves the parameters of the structured illumination in the spatial domain. Besides the phase-shifting amounts, the period, the modulation factor, and the background intensity of the pattern are extracted from three segmented raw images by the iterative algorithm. The second step is retrieval and synthesis of the low- and high-frequency information of the object in the Fourier space with obtained data. Since the unknown object information is not involved in the pattern parameters' solving process, it is possible to figure out the problem with higher precision and less requirements. We test the performance of this method in the experiments. The resolution is improved with the designed carrier frequency of the illumination pattern.

Joint Reconstruction Strategy for Structured Illumination Microscopy With Unknown Illuminations

The blind structured illumination microscopy strategy proposed by Mudry et al. is fully re-founded in this paper, unveiling the central role of the sparsity of the illumination patterns in the mechanism that drives super-resolution in the method. A numerical analysis shows that the resolving power of the method can be further enhanced with optimized one-photon or two-photon speckle illuminations. A much improved numerical implementation is provided for the reconstruction problem under the image positivity constraint. This algorithm rests on a new preconditioned proximal iteration faster than existing solutions, paving the way to 3D and real-time 2D reconstruction.