Three-dimensional focal stack imaging in scanning transmission X-ray microscopy with an improved reconstruction algorithm

Automated, high-throughput quantification of EGFP-expressing neutrophils in zebrafish by machine learning and a highly-parallelized microscope

Normal development of the immune system is essential for overall health and disease resistance. Bony fish, such as the zebrafish (Danio rerio), possess all the major immune cell lineages as mammals and can be employed to model human host response to immune challenge. Zebrafish neutrophils, for example, are present in the transparent larvae as early as 48 hours post fertilization and have been examined in numerous infection and immunotoxicology reports. One significant advantage of the zebrafish model is the ability to affordably generate high numbers of individual larvae that can be arrayed in multi-well plates for high throughput genetic and chemical exposure screens. However, traditional workflows for imaging individual larvae have been limited to low-throughput studies using traditional microscopes and manual analyses. Using a newly developed, parallelized microscope, the Multi-Camera Array Microscope (MCAM™), we have optimized a rapid, high-resolution algorithmic method to count fluorescently labeled cells in zebrafish larvae in vivo. Using transgenic zebrafish larvae, in which neutrophils express EGFP, we captured 18 gigapixels of images across a full 96-well plate, in 75 seconds, and processed the resulting datastream, counting individual fluorescent neutrophils in all individual larvae in 5 minutes. This automation is facilitated by a machine learning segmentation algorithm that defines the most in-focus view of each larva in each well after which pixel intensity thresholding and blob detection are employed to locate and count fluorescent cells. We validated this method by comparing algorithmic neutrophil counts to manual counts in larvae subjected to changes in neutrophil numbers, demonstrating the utility of this approach for high-throughput genetic and chemical screens where a change in neutrophil number is an endpoint metric. Using the MCAM™ we have been able to, within minutes, acquire both enough data to create an automated algorithm and execute a biological experiment with statistical significance. Finally, we present this open-source software package which allows the user to train and evaluate a custom machine learning segmentation model and use it to localize zebrafish and analyze cell counts within the segmented region of interest. This software can be modified as needed for studies involving other zebrafish cell lineages using different transgenic reporter lines and can also be adapted for studies using other amenable model species.

High dimensional optical data - varifocal multiview imaging, compression and evaluation

Varifocal multiview (VFMV) is an emerging high-dimensional optical data in computational imaging and displays. It describes scenes in angular, spatial, and focal dimensions, whose complex imaging conditions involve dense viewpoints, high spatial resolutions, and variable focal planes, resulting in difficulties in data compression. In this paper, we propose an efficient VFMV compression scheme based on view mountain-shape rearrangement (VMSR) and all-directional prediction structure (ADPS). The VMSR rearranges the irregular VFMV to form a new regular VFMV with mountain-shape focusing distributions. This special rearrangement features prominently in enhancing inter-view correlations by smoothing focusing status changes and moderating view displacements. Then, the ADPS efficiently compresses the rearranged VFMV by exploiting the enhanced correlations. It conducts row-wise hierarchy divisions and creates prediction dependencies among views. The closest adjacent views from all directions serve as reference frames to improve the prediction efficiency. Extensive experiments demonstrate the proposed scheme outperforms comparison schemes by quantitative, qualitative, complexity, and forgery protection evaluations. As high as 3.17 dB gains of peak signal-to-noise ratio (PSNR) and 61.1% bitrate savings can be obtained, achieving the state-of-the-art compression performance. VFMV is also validated could serve as a novel secure imaging format protecting optical data against the forgery of large models.

Automated, high-throughput quantification of EGFP-expressing neutrophils in zebrafish by machine learning and a highly-parallelized microscope

Normal development of the immune system is essential for overall health and disease resistance. Bony fish, such as the zebrafish (Danio rerio), possess all the major immune cell lineages as mammals and can be employed to model human host response to immune challenge. Zebrafish neutrophils, for example, are present in the transparent larvae as early as 48 hours post fertilization and have been examined in numerous infection and immunotoxicology reports. One significant advantage of the zebrafish model is the ability to affordably generate high numbers of individual larvae that can be arrayed in multi-well plates for high throughput genetic and chemical exposure screens. However, traditional workflows for imaging individual larvae have been limited to low-throughput studies using traditional microscopes and manual analyses. Using a newly developed, parallelized microscope, the Multi-Camera Array Microscope (MCAM™), we have optimized a rapid, high-resolution algorithmic method to count fluorescently labeled cells in zebrafish larvae in vivo. Using transgenic zebrafish larvae, in which neutrophils express EGFP, we captured 18 gigapixels of images across a full 96-well plate, in 75 seconds, and processed the resulting datastream, counting individual fluorescent neutrophils in all individual larvae in 5 minutes. This automation is facilitated by a machine learning segmentation algorithm that defines the most in-focus view of each larva in each well after which pixel intensity thresholding and blob detection are employed to locate and count fluorescent cells. We validated this method by comparing algorithmic neutrophil counts to manual counts in larvae subjected to changes in neutrophil numbers, demonstrating the utility of this approach for high-throughput genetic and chemical screens where a change in neutrophil number is an endpoint metric. Using the MCAM™ we have been able to, within minutes, acquire both enough data to create an automated algorithm and execute a biological experiment with statistical significance. Finally, we present this open-source software package which allows the user to train and evaluate a custom machine learning segmentation model and use it to localize zebrafish and analyze cell counts within the segmented region of interest. This software can be modified as needed for studies involving other zebrafish cell lineages using different transgenic reporter lines and can also be adapted for studies using other amenable model species.

End-to-end varifocal multiview images coding framework from data acquisition end to vision application end

The emerging data, varifocal multiview (VFMV) has an exciting prospect in immersive multimedia. However, the distinctive data redundancy of VFMV derived from dense arrangements and blurriness differences among views causes difficulty in data compression. In this paper, we propose an end-to-end coding scheme for VFMV images, which provides a new paradigm for VFMV compression from data acquisition (source) end to vision application end. VFMV acquisition is first conducted in three ways at the source end, including conventional imaging, plenoptic refocusing, and 3D creation. The acquired VFMV has irregular focusing distributions due to varying focal planes, which decreases the similarity among adjacent views. To improve the similarity and the consequent coding efficiency, we rearrange the irregular focusing distributions in descending order and accordingly reorder the horizontal views. Then, the reordered VFMV images are scanned and concatenated as video sequences. We propose 4-directional prediction (4DP) to compress the reordered VFMV video sequences. Four most similar adjacent views from the left, upper left, upper and upper right directions serve as reference frames to improve the prediction efficiency. Finally, the compressed VFMV is transmitted and decoded at the application end, benefiting potential vision applications. Extensive experiments demonstrate that the proposed coding scheme is superior to the comparison scheme in objective quality, subjective quality and computational complexity. Experiments on new view synthesis show that VFMV can achieve extended depth of field than conventional multiview at the application end. Validation experiments show the effectiveness of view reordering, the advantage over typical MV-HEVC, and the flexibility on other data types, respectively.

Electrically addressed focal stack plenoptic camera based on a liquid-crystal microlens array for all-in-focus imaging

Focal stack cameras are capable of capturing a stack of images focused at different spatial distance, which can be further integrated to present a depth of field (DoF) effect beyond the range restriction of conventional camera's optics. To date, all of the proposed focal stack cameras are essentially 2D imaging architecture to shape 2D focal stacks with several selected focal lengths corresponding to limited objective distance range. In this paper, a new type of electrically addressed focal stack plenoptic camera (EAFSPC) based on a functional liquid-crystal microlens array for all-in-focus imaging is proposed. As a 3D focal stack camera, a sequence of raw light-field images can be rapidly manipulated through rapidly shaping a 3D focal stack. The electrically addressed focal stack strategy relies on the electric tuning of the focal length of the liquid-crystal microlens array by efficiently selecting or adjusting or jumping the signal voltage applied over the microlenses. An algorithm based on the Laplacian operator is utilized to composite the electrically addressed focal stack leading to raw light-field images with an extended DoF and then the all-in-focus refocused images. The proposed strategy does not require any macroscopic movement of the optical apparatus, so as to thoroughly avoid the registration of different image sequence. Experiments demonstrate that the DoF of the refocused images can be significantly extended into the entire tomography depth of the EAFSPC, which means a significant step for an all-in-focus imaging based on the electrically controlled 3D focal stack. Moreover, the proposed approach also establishes a high correlation between the voltage signal and the depth of in-focus plane, so as to construct a technical basis for a new type of 3D light-field imaging with an obvious intelligent feature.

Virtual depth-scan multi-slice ptychography for improved three-dimensional imaging

Multi-slice ptychography (MSP) is a fast three-dimensional ptychography technology developed on the basis of conventional ptychography. With this method, three-dimensional imaging can be achieved without rotating the sample. The prototype multi-slice algorithm can only reconstruct three-dimensional samples with a limited number of slices, which greatly limits the depth range and resolution of sample imaging. Here we reported a virtual depth-scan scheme of MSP in which a thick sample is scanned virtually in the depth direction across its whole thickness range within the reconstruction process, thereby eliminating the restriction on slice number and potentially improving the depth resolution of MSP. This new approach also improves the flexibility of multi-slice ptychography. Both the simulation and experimental results validate the feasibility of our new approach.

Three-dimensional fast elemental mapping by soft X-ray dual-energy focal stacks imaging

The three-dimensional (3D) dual-energy focal stacks (FS) imaging method has been developed to quickly obtain the spatial distribution of an element of interest in a sample; it is a combination of the 3D FS imaging method and two-dimensional (2D) dual-energy contrast imaging based on scanning transmission soft X-ray microscopy (STXM). A simulation was firstly performed to verify the feasibility of the 3D elemental reconstruction method. Then, a sample of composite nanofibers, polystyrene doped with ferric acetylacetonate [Fe(acac)3], was further investigated to quickly reveal the spatial distribution of Fe(acac)3 in the sample. Furthermore, the data acquisition time was less than that for STXM nanotomography under similar resolution conditions and did not require any complicated sample preparation. The novel approach of 3D dual-energy FS imaging, which allows fast 3D elemental mapping, is expected to provide invaluable information for biomedicine and materials science.

Staircase array of inclined refractive multi-lenses for large field of view pixel super-resolution scanning transmission hard X-ray microscopy

Owing to the development of X-ray focusing optics during the past decades, synchrotron-based X-ray microscopy techniques allow the study of specimens with unprecedented spatial resolution, down to 10 nm, using soft and medium X-ray photon energies, though at the expense of the field of view (FOV). One of the approaches to increase the FOV to square millimetres is raster-scanning of the specimen using a single nanoprobe; however, this results in a long data acquisition time. This work employs an array of inclined biconcave parabolic refractive multi-lenses (RMLs), fabricated by deep X-ray lithography and electroplating to generate a large number of long X-ray foci. Since the FOV is limited by the pattern height if a single RML is used by impinging X-rays parallel to the substrate, many RMLs at regular intervals in the orthogonal direction were fabricated by tilted exposure. By inclining the substrate correspondingly to the tilted exposure, 378000 X-ray line foci were generated with a length in the centimetre range and constant intervals in the sub-micrometre range. The capability of this new X-ray focusing device was first confirmed using ray-tracing simulations and then using synchrotron radiation at BL20B2 of SPring-8, Japan. Taking account of the fact that the refractive lens is effective for focusing high-energy X-rays, the experiment was performed with 35 keV X-rays. Next, by scanning a specimen through the line foci, this device was used to perform large FOV pixel super-resolution scanning transmission hard X-ray microscopy (PSR-STHXM) with a 780 ± 40 nm spatial resolution within an FOV of 1.64 cm × 1.64 cm (limited by the detector area) and a total scanning time of 4 min. Biomedical implant abutments fabricated via selective laser melting using Ti-6Al-4V medical alloy were measured by PSR-STHXM, suggesting its unique potential for studying extended and thick specimens. Although the super-resolution function was realized in one dimension in this study, it can be expanded to two dimensions by aligning a pair of presented devices orthogonally.

A bidirectional scanning method for scanning transmission X-ray microscopy

Scanning mode is a key factor for the comprehensive performance, including imaging efficiency, of scanning transmission X-ray microscopy (STXM). Herein is presented a bidirectional scanning method designed for STXM with an S-shaped moving track. In this method, artificially designed ramp waves are generated by a piezo-stage controller to control the two-dimensional scanning of the sample. The sample position information is measured using laser interferometric sensors and sent to a field-programmable gate array (FPGA) board which also acquires the X-ray signals simultaneously from the detector. Since the data recorded by the FPGA contain the real position of each scanned point, the influence of the backlash caused by the back-turning movement on the STXM image can be eliminated. By employing an adapted post-processing program, a re-meshed high-resolution STXM image can be obtained. This S-track bidirectional scanning method in fly-scan mode has been implemented on the STXM endstation at the Shanghai Synchrotron Radiation Facility (SSRF), and successfully resolved the ∼30 nm interval between the innermost strips of a Siemens star. This work removes the limitation on bidirectional scanning caused by motor backlash and vibration, and significantly improves the efficiency of STXM experiments.

Block-wise focal stack image representation for end-to-end applications

In optical imaging systems, the depth of field (DoF) is generally constricted due to the nature of optical lens. The limited DoF produces partially focused images of the scene. Focal stack images (FoSIs) are a sequence of images that focused on serial depths of a scene. FoSIs are capable of extending DoF of optical systems and provide practical solutions for computational photography, macroscopic and microscopic imaging, interactive and immersive media. However, high volumes of data remains one of the biggest obstacles to the development of end-to-end applications. In order to solve this challenge, we propose a block-wise Gaussian based representation model for FoSIs and utilize this model to solve the problem of coding, reconstruction and rendering for end-to-end applications. Experimental results demonstrate the high efficiency of proposed representation model and the superior performance of proposed schemes.

Additive Nano-Lithography with Focused Soft X-rays: Basics, Challenges, and Opportunities

Focused soft X-ray beam induced deposition (FXBID) is a novel technique for direct-write nanofabrication of metallic nanostructures from metal organic precursor gases. It combines the established concepts of focused electron beam induced processing (FEBIP) and X-ray lithography (XRL). The present setup is based on a scanning transmission X-ray microscope (STXM) equipped with a gas flow cell to provide metal organic precursor molecules towards the intended deposition zone. Fundamentals of X-ray microscopy instrumentation and X-ray radiation chemistry relevant for FXBID development are presented in a comprehensive form. Recently published proof-of-concept studies on initial experiments on FXBID nanolithography are reviewed for an overview on current progress and proposed advances of nanofabrication performance. Potential applications and advantages of FXBID are discussed with respect to competing electron/ion based techniques.