Calibration method for projector-camera-based telecentric fringe projection profilometry system

Multi-view fringe projection profilometry for surfaces with intricate structures and high dynamic range

Fringe projection profilometry plays an important role for quality control in production line. However, it is facing challenges in the measurement of objects with intricate structures and high dynamic range that involved in precision manufacturing and semiconductor packaging. In this paper, a multi-view fringe projection profilometry system, which deploys a vertical telecentric projector and four oblique tilt-shift cameras, is presented to address the "blind spots" caused by shadowing, occlusion and local specular reflection. A flexible and accurate system calibration method is proposed, in which the corrected pinhole imaging model is used to calibrate the telecentric projection, and the unified calibration is performed by bundle adjustment. Experimental results show that the 3D repeated measurement error and standard deviation are no more than 10 μm within a measurable volume of 70 × 40 × 20 mm3. Furthermore, a group of experiments prove that the developed system can achieve complete and accurate 3D measurement for high dynamic range surfaces with complex structures.

Adaptive focus stacking for large depth-of-field microscopic structured-light 3D imaging

This paper presents an adaptive focus stacking method for large depth-of-field (DOF) 3D microscopic structured-light imaging systems. Conventional focus stacking methods typically capture images under a series of pre-defined focus settings without considering the attributes of the measured object. Therefore, it is inefficient since some of the focus settings might be redundant. To address this problem, we first employ the focal sweep technique to reconstruct an initial rough 3D shape of the measured objects. Then, we leverage the initial 3D data to determine effective focus settings that focus the camera on the valid areas of the measured objects. Finally, we reconstruct a high-quality 3D point cloud using fringe images obtained from these effective focus settings by focus stacking. Experimental results demonstrate the success of the proposed method.

Flexible and accurate system calibration method in microscopic fringe projection profilometry

The three-dimensional (3D) measurement task of complex microstructures holds paramount significance in the domains of precision manufacturing and inspection. The calibration of the 3D system heavily determines the final reconstruction accuracy. The widely adopted system calibration method is phase-height mapping (PHM) and stereo vision (SV) based. The former can be applied directly to the calculation without considering the imaging model of the system, but it relies on highly precise and expensive translation stages or standard blocks. The latter's accuracy cannot be guaranteed because it is difficult to accurately calibrate the projector. In this paper, we establish an optically coupled microscopic fringe projection profilometry system that consists of a Scheimpflug pinhole projector and a super-low distortion bi-telecentric camera. We introduce a simplified 3D system calibration approach that combines phase modulation transfer and ray propagation. Our method enables the simultaneous calibration of the system, including the calibration of the projector, camera, and the phase to a 3D coordinates relationship, using only a 2D target. The calibrated projector's external parameters are used to obtain the target's complete poses, and then the direct mapping coefficients of the phase to the 3D coordinates can be obtained through the optical geometry structure and phase labels. Comparable experiments verify the feasibility of the proposed method.

Accurate and flexible calibration method for a 3D microscopic structured light system with telecentric imaging and Scheimpflug projection

3D imaging and metrology of complex micro-structures is a critical task for precision manufacturing and inspection. In this paper, an accurate and flexible calibration method for 3D microscopic structured light system with telecentric imaging and Scheimpflug projector is proposed. Firstly, a fringe projection 3D microscopy (FP-3DM) system consisting of a telecentric camera and a Scheimpflug projector is developed, which can take full advantage of the depth of field (DOF) and increase the measurement depth range. Secondly, an accurate and flexible joint calibration method is proposed to calibrate the developed system, which utilizes the established pinhole imaging model and Scheimpflug distortion model to calibrate telecentric imaging, and fully considers the correction and error optimization of the Scheimpflug projection model. Meanwhile, the optimized local homography is calculated to obtain more accurate sub-pixel correspondence between the camera and the projector, and the perspective-n-point (PnP) method make the 3D coordinate estimation of the feature point more accurate. Finally, a prototype and a dedicated calibration program are developed to realize high-resolution and high-precision 3D imaging. The experimental results demonstrate that the re-projection error is less than 1µm, and the 3D repeated measurement error based on feature fitting is less than 4µm, within the calibrated volume of 10(H)mm × 50(W)mm × 40(D)mm.

Training generative adversarial networks for optical property mapping using synthetic image data

We demonstrate the training of a generative adversarial network (GAN) for the prediction of optical property maps (scattering and absorption) using spatial frequency domain imaging (SFDI) image data sets that are generated synthetically with a free open-source 3D modelling and rendering software, Blender. The flexibility of Blender is exploited to simulate 5 models with real-life relevance to clinical SFDI of diseased tissue: flat samples containing a single material, flat samples containing 2 materials, flat samples containing 3 materials, flat samples with spheroidal tumours and cylindrical samples with spheroidal tumours. The last case is particularly relevant as it represents wide-field imaging inside a tubular organ e.g. the gastro-intestinal tract. In all 5 scenarios we show the GAN provides an accurate reconstruction of the optical properties from single SFDI images with a mean normalised error ranging from 1.0-1.2% for absorption and 1.1%-1.2% for scattering, resulting in visually improved contrast for tumour spheroid structures. This compares favourably with the ∼10% absorption error and ∼10% scattering error achieved using GANs on experimental SFDI data. Next, we perform a bi-directional cross-validation of our synthetically-trained GAN, retrained with 90% synthetic and 10% experimental data to encourage domain transfer, with a GAN trained fully on experimental data and observe visually accurate results with an error of 6.3%-10.3% for absorption and 6.6%-11.9% for scattering. Our synthetically trained GAN is therefore highly relevant to real experimental samples but provides the significant added benefits of large training datasets, perfect ground-truths and the ability to test realistic imaging geometries, e.g. inside cylinders, for which no conventional single-shot demodulation algorithms exist. In the future, we expect that the application of techniques such as domain adaptation or training on hybrid real-synthetic datasets will create a powerful tool for fast, accurate production of optical property maps for real clinical imaging systems.

A High-Accuracy Calibration Method for a Telecentric Structured Light System

We propose a method for accurately calibrating a telecentric structured light system consisting of a camera attached to a bilateral telecentric lens and a pin-hole projector. The proposed method can be split into two parts: axial calibration and transverse calibration. The first part is used for building the relationship between phase and depth by means of a planar plate with ring markers on its surface at several different positions in the measuring volume. The second part is used for establishing the relationship between transverse coordinates and pixel positions with the depth offered by a translation stage and the extracted ring centers. Compared with existing methods that require projector calibration, the proposed method can avoid a propagation of the correspondence error between the camera imaging plane and projector imaging plane, thus increasing calibration accuracy. The calibrated telecentric structured light system is further used for three-dimensional (3D) reconstructions of a planar, a ruled surface, and complex surfaces. Experimental results demonstrate that the proposed system calibration method can be used for accurate 3D measurement.

Metrology for Measuring Bumps in a Protection Layer Based on Phase Shifting Fringe Projection

We developed a telecentric 3D measurement system based on the fringe projection technique to measure phases corresponding to the 3D shape of bumps on a surface. A measurement algorithm based on an area scan camera with a flexibly programmable region of interest (ROI) was applied to reconstruct the 3D model. Only the pixel information within the ROI is stored, which reduces the data volume and processing and accelerates the area scan acquisition. Experiments showed that the fringe projection system provided good linearity between the calibration and optical data with excellent accuracy. A reflectometry sensor was implemented to provide a correction offset to the bump height in a protection layer in a subsequent high-speed inspection.

Relative pose estimation of uncooperative spacecraft using 2D-3D line correspondences

The relative pose estimation of uncooperative spacecraft is an essential technology in space missions. However, lacking the prior knowledge of uncooperative targets brings great challenges to accurately measure their pose, which hinders their application in space. To overcome these problems, an accurate and robust pose estimation approach for uncooperative spacecraft is proposed based on a time-of-flight camera coupled with a gray-scale camera. We reconstruct the target model using our hierarchical registration strategy by the ToF camera and extract 2D lines from image sequences captured by the gray-scale camera. We attempt to establish 2D-3D line correspondences for pose estimation, which effectively associates the 2D image information with the reconstructed 3D model. Experimental results manifest that, when dealing with a completely unknown model, our method computes its relative position and attitude with significantly high accuracy.

Improved computer-generated moiré profilometry with flat image calibration

An improved computer-generated moiré profilometry (CGMP) with flat image calibration is proposed. In CGMP, the purification of the AC component plays a decisive role. While a composite grating modulated with both the sinusoidal grating and its background light substitutes for the sinusoidal grating itself, the sinusoidal deformed pattern and flat image can be demodulated from the captured pattern. It is found that the sinusoidal deformed pattern and flat image may deviate, which is caused by ambient light. So flat image calibration is conducted to obtain a purer AC component that can effectively suppress the influence of ambient light and ensure the measurement accuracy, even if spectrum aliasing exists. Experimental results show the feasibility and validity of the proposed method.

3D surface reconstruction of small height object based on thin structured light scanning

Aiming at the problem of 3D surface reconstruction of small height objects, a method based on the scanning principle with thin structured light is proposed. The laser fringe is produced by a laser light source. It scans the surface of small height object under the control of a precise motion control system, and is integrated with a stereo light microscope and two cameras to form a complete structured light profilometry system. This method is very suitable for 3D surface reconstruction of small height objects. In order to deal with the phenomenon of "cracking" and local uneven brightness in microscope fringe image, the fringe image sequence is captured under different camera exposure parameters, and the quality of fringe image is improved by image fusion. The location of pixels on the center line of laser stripes is detected by the Loess local second-order fitting method, and the positions of these pixels are smoothed and predicted by using the Lowess local linear fitting method. The deformation of the laser stripe center curve is calculated by constructing a baseline, and the 3D surface reconstruction of the object with small height is realized in the image space. Taking the small characters on the surface of Chinese coins and human hair as test samples, their 3D surfaces are constructed by using the method proposed in this paper. The reconstructed 3D surface shapes are highly consistent with the real 3D surface shapes of objects.

Distorted pinhole camera modeling and calibration

Camera modeling and calibration are essential tasks in modern optics. Conventionally, the pinhole model is adopted with a further extension for lens distortion. However, pinhole and distortion models are mutually dependent; thus, the standard approach induces systematic camera calibration errors. This research presents a unifying distorted pinhole camera model that includes a telecentric, distortion-free pinhole, and radial lens distortion as particular cases. An iterative calibration method based on the derived distorted pinhole model is proposed, and experimental evaluation by calibrating a camera with high radial distortion is performed. The calibration results are compared with the standard and fisheye models using a well-known commercial camera calibrator software. The proposed method outperforms the standard model and achieves accuracy comparable to the fisheye model, and the proposed approach is a versatile and accurate tool for diverse optical metrology applications.

Quantitative 3D Reconstruction from Scanning Electron Microscope Images Based on Affine Camera Models

Scanning electron microscopes (SEMs) are versatile imaging devices for the micro- and nanoscale that find application in various disciplines such as the characterization of biological, mineral or mechanical specimen. Even though the specimen’s two-dimensional (2D) properties are provided by the acquired images, detailed morphological characterizations require knowledge about the three-dimensional (3D) surface structure. To overcome this limitation, a reconstruction routine is presented that allows the quantitative depth reconstruction from SEM image sequences. Based on the SEM’s imaging properties that can be well described by an affine camera, the proposed algorithms rely on the use of affine epipolar geometry, self-calibration via factorization and triangulation from dense correspondences. To yield the highest robustness and accuracy, different sub-models of the affine camera are applied to the SEM images and the obtained results are directly compared to confocal laser scanning microscope (CLSM) measurements to identify the ideal parametrization and underlying algorithms. To solve the rectification problem for stereo-pair images of an affine camera so that dense matching algorithms can be applied, existing approaches are adapted and extended to further enhance the yielded results. The evaluations of this study allow to specify the applicability of the affine camera models to SEM images and what accuracies can be expected for reconstruction routines based on self-calibration and dense matching algorithms.

Single-shot coherent power-spectrum imaging of objects hidden by opaque scattering media

We report coherent imaging of objects behind opaque scattering media with only one piece of the power spectrum pattern. We solve the unique solution and improve algorithm speed for the inverse problem. Based on the proposed scattering-disturbance model, with only one piece of the Fourier transform power spectrum pattern under coherent illumination, we successfully reconstruct clear images of the objects fully hidden by an opaque diffuser. The experimental results demonstrate the feasibility of the reconstruction method and the scattering-disturbance model. Our method makes it possible to carry out snapshot coherent imaging of the objects obscured by scattering media, which extends the methodology of x-ray crystallography to visible-light scattering imaging for underwater and living biomedical imaging.

Microscopic Three-Dimensional Measurement Based on Telecentric Stereo and Speckle Projection Methods

Three-dimensional (3D) measurement of microstructures has become increasingly important, and many microscopic measurement methods have been developed. For the dimension in several millimeters together with the accuracy at sub-pixel or sub-micron level, there is almost no effective measurement method now. Here we present a method combining the microscopic stereo measurement with the digital speckle projection. A microscopy experimental setup mainly composed of two telecentric cameras and an industrial projection module is established and a telecentric binocular stereo reconstruction procedure is carried out. The measurement accuracy has firstly been verified by performing 3D measurements of grid arrays at different locations and cylinder arrays with different height differences. Then two Mitutoyo step masters have been used for further verification. The experimental results show that the proposed method can obtain 3D information of the microstructure with a sub-pixel and even sub-micron measuring accuracy in millimeter scale.