High-Magnification Object Tracking with Ultra-Fast View Adjustment and Continuous Autofocus Based on Dynamic-Range Focal Sweep

Active vision systems (AVSs) have been widely used to obtain high-resolution images of objects of interest. However, tracking small objects in high-magnification scenes is challenging due to shallow depth of field (DoF) and narrow field of view (FoV). To address this, we introduce a novel high-speed AVS with a continuous autofocus (C-AF) approach based on dynamic-range focal sweep and a high-frame-rate (HFR) frame-by-frame tracking pipeline. Our AVS leverages an ultra-fast pan-tilt mechanism based on a Galvano mirror, enabling high-frequency view direction adjustment. Specifically, the proposed C-AF approach uses a 500 fps high-speed camera and a focus-tunable liquid lens operating at a sine wave, providing a 50 Hz focal sweep around the object's optimal focus. During each focal sweep, 10 images with varying focuses are captured, and the one with the highest focus value is selected, resulting in a stable output of well-focused images at 50 fps. Simultaneously, the object's depth is measured using the depth-from-focus (DFF) technique, allowing dynamic adjustment of the focal sweep range. Importantly, because the remaining images are only slightly less focused, all 500 fps images can be utilized for object tracking. The proposed tracking pipeline combines deep-learning-based object detection, K-means color clustering, and HFR tracking based on color filtering, achieving 500 fps frame-by-frame tracking. Experimental results demonstrate the effectiveness of the proposed C-AF approach and the advanced capabilities of the high-speed AVS for magnified object tracking.

Simple, non-mechanical and automatic calibration approach for axial-scanning microscopy with an electrically tunable lens

We describe a simple and robust calibration approach for axial-scanning microscopy that realizes axial focus shifts with an electrically tunable lens (ETL). We demonstrate the calibration approach based on a microscope with an ETL placed close to the rear stop of the objective lens. By introducing a target-consisted of repeating lines at one known frequency and placed at a ~45° angle to the imaging path, the calibration method captures multiple images at different ETL currents and calibrates the dependence of the axial focus shift on the ETL current by evaluating the sharpness of the captured images. It calibrates the dependence of the magnification of the microscope on the ETL current by measuring the period of the repeating lines in the captured images. The experimental results show that different from the conventional calibration procedure, the proposed scheme does not involve any mechanical scanning and can simultaneously calibrate the dependence of the axial focus shift and the magnification on the ETL current. This might facilitate imaging studies that require the measurement of fine structures in a 3D volume. We also show the calibration procedure can be used to estimate the radius of a conner-arc sample, fabricated using laser micromachining. We believe that this easy-to-use calibration approach may facilitate use of ETLs for a variety of imaging platforms. It may also provide new insights for the development of novel 3D surface measurement methods. RESEARCH HIGHLIGHTS: The proposed calibration scheme does not involve any mechanical scanning and can simultaneously calibrate the dependence of the axial focus shift and the magnification on the electrically tunable lens (ETL) current. It might facilitate imaging studies that require the measurement of fine structures in a 3D volume, and the use of ETLs for a variety of imaging platforms. It may also provide new insights for the development of novel 3D surface measurement methods.

High-Speed and Low-Latency 3D Fluorescence Imaging for Robotic Microscope

In this study, we propose a high-speed and low-latency 3D fluorescence imaging method for robotic microscopes. The prototype system consists of a focus-tunable lens called a TAG lens, which operates at several hundred kHz, an image intensifier (I.I.) that enhances faint light such as fluorescence, and a high-speed vision system that can transfer acquired images to the host PC in 500 Hz. The proposed method can acquire images at arbitrary focal lengths at frame rates on the order of 1 kHz by synchronizing the focal-length fluctuation of the TAG lens and the exposure timing of the I.I., whose duration is a few hundred nanoseconds. The low-latency we aim for in this paper is on the order of a few milliseconds. A prototype system was developed to validate the proposed method. High-speed 3D tracking of the Brownian motion of a fluorescent bead of 0.5 μm diameter was demonstrated to verify the feedback performance of the proposed low-latency 3D fluorescence imaging method.

Advanced tools and methods for single-cell surgery

Highly precise micromanipulation tools that can manipulate and interrogate cell organelles and components must be developed to support the rapid development of new cell-based medical therapies, thereby facilitating in-depth understanding of cell dynamics, cell component functions, and disease mechanisms. This paper presents a literature review on micro/nanomanipulation tools and their control methods for single-cell surgery. Micromanipulation methods specifically based on laser, microneedle, and untethered micro/nanotools are presented in detail. The limitations of these techniques are also discussed. The biological significance and clinical applications of single-cell surgery are also addressed in this paper.

Reversible single cell trapping of Paramecium caudatum to correlate swimming behavior and membrane state

Single cell measurements with living specimen like, for example, the ciliated protozoan Paramecium caudatum can be a challenging task. We present here a microfluidic trapping mechanism for measurements with these micro-organisms that can be used, e.g., for optical measurements to correlate cellular functions with the phase state of the lipid membrane. Here, we reversibly trap single cells in small compartments. Furthermore, we track and analyze the swimming behavior of single cells over several minutes. Before and after reversible trapping the swimming speed is comparable, suggesting that trapping does not have a large effect on cell behavior. Last, we demonstrate the feasibility of membrane order measurements on living cells using the fluorescent dye 6-lauryl-2-dimethylaminonaphthalene (Laurdan).

2D vs 3D tracking in bacterial motility analysis

<abstract> <p>Digital holographic microscopy provides the ability to observe throughout a large volume without refocusing. This capability enables simultaneous observations of large numbers of microorganisms swimming in an essentially unconstrained fashion. However, computational tools for tracking large 4D datasets remain lacking. In this paper, we examine the errors introduced by tracking bacterial motion as 2D projections vs. 3D volumes under different circumstances: bacteria free in liquid media and bacteria near a glass surface. We find that while XYZ speeds are generally equal to or larger than XY speeds, they are still within empirical uncertainties. Additionally, when studying dynamic surface behavior, the Z coordinate cannot be neglected.</p> </abstract>

1000-volume/s high-speed volumetric display for high-speed HMD

In this paper, we propose a high-speed volumetric display principle that can solve two problems faced by three-dimensional displays using the parallax stereo principle (namely, the vergence-accommodation conflict and display latency) and we report evaluation results. The proposed display method can update a set of images at different depths at 1000 Hz and is consistent with accommodation. The method selects the depth position in microseconds by combining a high-speed variable-focus lens that vibrates at about 69 kHz and sub-microsecond control of illumination light using an LED. By turning on the LED for only a few hundred nanoseconds when the refractive power of the lens is at a certain value, an image can be presented with this specific refractive power. The optical system is combined with a DMD to form an image at each depth. 3D information consisting of multiple planes in the depth direction can be presented at a high refresh rate by switching the images and changing the refractive power at high speed. A proof-of-concept system was developed to show the validity of the proposed display principle. The system successfully displayed 3D information consisting of six binary images at an update rate of 1000 volume/s.

Quasi-simultaneous multi-focus imaging using a lock-in pixel image sensor and TAG lens

In this paper, a quasi-simultaneous multi-focus imaging technique named simulfocus imaging is reported. This technique was developed for measuring an entire object distributed in the depth direction beyond the depth of field (DOF) with high resolution in a single shot. Simulfocus imaging can acquire multiple focal planes in one shot by synchronizing a tunable acoustic gradient index (TAG) lens and a lock-in pixel image sensor. The TAG lens is a tunable-focus lens whose focal position can be changed at a high speed of several tens to several hundreds of kilohertz. The lock-in pixel image sensor is a special image sensor that can execute multiple exposures at an arbitrary timing during a single shooting. The sensor includes a number of photoelectron storage units in each pixel, and the units where the photoelectrons generated by each exposure are stored can be freely selected. Since an image can be acquired for a single storage unit, and the lock-in pixel image sensor has a number of storage units, the lock-in pixel image sensor can acquire multiple images in one shot. By assigning a specific exposure timing to each unit and synchronizing the exposure timing with the focus fluctuation of the TAG lens, it is possible to simultaneously acquire images in different focal planes. To evaluate the system, we conducted experiments to show the effectiveness of simulfocus imaging in microscope and telescope configurations. From the experimental results, it was confirmed that simulfocus was effective in both configurations.