Accurate calibration method for camera and projector in fringe patterns measurement system

Flexible and accurate system calibration method in microscopic fringe projection profilometry

The three-dimensional (3D) measurement task of complex microstructures holds paramount significance in the domains of precision manufacturing and inspection. The calibration of the 3D system heavily determines the final reconstruction accuracy. The widely adopted system calibration method is phase-height mapping (PHM) and stereo vision (SV) based. The former can be applied directly to the calculation without considering the imaging model of the system, but it relies on highly precise and expensive translation stages or standard blocks. The latter's accuracy cannot be guaranteed because it is difficult to accurately calibrate the projector. In this paper, we establish an optically coupled microscopic fringe projection profilometry system that consists of a Scheimpflug pinhole projector and a super-low distortion bi-telecentric camera. We introduce a simplified 3D system calibration approach that combines phase modulation transfer and ray propagation. Our method enables the simultaneous calibration of the system, including the calibration of the projector, camera, and the phase to a 3D coordinates relationship, using only a 2D target. The calibrated projector's external parameters are used to obtain the target's complete poses, and then the direct mapping coefficients of the phase to the 3D coordinates can be obtained through the optical geometry structure and phase labels. Comparable experiments verify the feasibility of the proposed method.

Unsupervised-learning-based calibration method in microscopic fringe projection profilometry

Microscopic fringe projection profilometry (MFPP) technology is widely used in 3D measurement. The measurement precision performed by the MFPP system is closely related to the calibration accuracy. However, owing to the shallow depth of field, calibration in MFPP is frequently influenced by low-quality target images, which would generate inaccurate features and calibration parameter estimates. To alleviate the problem, this paper proposes an unsupervised-learning-based calibration robust to defocus and noise, which could effectively enhance the image quality and increase calibration accuracy. In this method, first, an unsupervised image deblurring network (UIDNet) is developed to recover a sharp target image from the deteriorated one. Free from capturing strictly paired images by a specific vision system or generating the dataset by simulation, the unsupervised deep learning framework can learn more accurate features from the multi-quality target dataset of convenient image acquisition. Second, multi-perceptual loss and Fourier frequency loss are introduced into the UIDNet to improve the training performance. Third, a robust calibration compensation strategy based on 2D discrete Fourier transform is also developed to evaluate the image quality and improve the detection accuracy of the reference feature centers for fine calibration. The relevant experiments demonstrate that the proposed calibration method can achieve superior performance in terms of calibration accuracy and measurement precision.

High-flexibility and high-accuracy phase delay calibration method for MEMS-based fringe projection systems

Microelectromechanical system (MEMS) mirror based laser beam scanning (LBS) projectors for fringe projection profilometry (FPP) are becoming increasingly popular attributing to their small size and low cost. However, the initial phase of the scanning MEMS mirror employed in an LBS projector may vary over time, resulting in unstable and distorted fringe patterns. The distorted fringe patterns will largely decrease the accuracy of the three-dimensional (3D) topographic reconstruction. In this paper, an efficient phase delay calibration method based on a unique fringe projection sequence and a corresponding image processing algorithm is proposed. The proposed method can compensate the phase uncertainty and variation with no need to add any extra components. One LBS projector has been constructed using a uniaxial electrostatic MEMS mirror that has a mirror size of 2.5 mm × 2.5 mm and a scanning field of view of 60 ^∘ at its resonance of 1523 Hz. 3D reconstruction experiments are conducted to study how the 3D reconstruction results are affected by the phase delay. The standard deviation of a sphere reconstruction is improved from 2.05 mm to 0.20 mm after the positive phase delay deviation of 5 μs is compensated using this new calibration method.

Methodology for Designing an Optimal Test Stand for Camera Thermal Drift Measurements and Its Stability Verification

The effects of temperature changes on cameras are realized by observing the drifts of characteristic points in the image plane. Compensation for these effects is crucial to maintain the precision of cameras applied in machine vision systems and those expected to work in environments with varying factors, including temperature changes. Generally, mathematical compensation models are built by measuring the changes in the intrinsic and extrinsic parameters under the temperature effect; however, due to the assumptions of certain factors based on the conditions of the test stand used for the measurements, errors can become apparent. In this paper, test stands for thermal image drift measurements used in other works are assessed, and a methodology to design a test stand, which can measure thermal image drifts while eliminating other external influences on the camera, is proposed. A test stand was built accordingly, and thermal image drift measurements were performed along with a measurement to verify that the test stand did eliminate external influences on the camera. The experiment was performed for various temperatures from 5 °C to 45 5 °C, and as a result, the thermal image drift measured with the designed test stand showed its maximum error of 16% during its most rapid temperature change from 25 °C to 5 °C.

5D-fusion imaging for surface shape, polarization, and hyperspectral measurement

We present a five-dimensional (5D) imager that is capable of simultaneous detection of the surface shape, spectral characteristics, and polarization states of macroscopic objects, and straightforwardly fuse collected data into a 5D data set. A polarized module that uses a polarized camera obtains polarized images, while a 3D hyperspectral module reconstructs the target as a 3D point cloud using a fringe projection technique. A liquid-crystal tunable filter is placed in front of the camera of this module to acquire spectral data that can be assigned to corresponding point clouds directly. The two modules are coupled by a dual-path configuration that allows the polarization information to be merged into a comprehensive point cloud with spectral information, generating a new 5D model. The 5D imager shows excellent performance, with a spectral resolution of 10 nm, depth accuracy of 30.7 µm, and imaging time of 8 s. Sample experiments on a toy car with micro scratch defects and a yellowing plant are presented to demonstrate the capabilities of the 5D imager and its potential for use in a broad range of applications, such as industrial manufacturing inspection, plant health monitoring, and biological analysis.

High-efficiency 3D reconstruction with a uniaxial MEMS-based fringe projection profilometry

Micro-Electro-Mechanical System (MEMS) scanning is increasingly popular in 3D surface measurement with the merits of the compact structure and high frame-rate. In this paper, we achieve real-time fringe structured 3D reconstruction by using a uniaxial MEMS-based projector. To overcome the limitations on uniaxial MEMS-based projector of lensless structure and unidirectional fringe projection, a novel isophase plane model is proposed, in which the laser line from MEMS-based projector is regarded as an isophase plane. Our model directly establishes the mapping relationship between phase and spatial 3D coordinates through the intersection point of camera back-projection light ray and isophase plane. Furthermore, a flexible calibration strategy to obtain 3D mapping coefficients is introduced with a specially designed planar target. Experiments demonstrated that our method can achieve high-accuracy and real-time 3D reconstruction.

Iterative calibration method for measurement system having lens distortions in fringe projection profilometry

In fringe projection profilometry, system calibration is crucial for guaranteeing the measurement accuracies. Its difficulty lies in calibrating projector parameters, especially when the projector lens has distortions, since the projector, unlike a camera, cannot capture images, leading to an obstacle to knowing the correspondences between its pixels and object points. For solving this issue, this paper, exploiting the fact that the fringe phases on a plane board theoretically have a distribution of rational function, proposes an iterative calibration method based on phase measuring. Projecting fringes onto the calibration board and fitting the measured phases with a rational function allow us to determine projector pixels corresponding to the featured points on the calibration board. Using these correspondences, the projector parameters are easy to estimate. Noting that the projector lens distortions may deform the fitted phase map thus inducing errors in the estimates of the projector parameters, this paper suggests an iterative strategy to overcome this problem. By implementing the phase fitting and the parameter estimating alternately, the intrinsic and extrinsic parameters of the projector, as well as its lens distortion coefficients, are determined accurately. For compensating for the effects of the lens distortions on measurement, this paper gives two solutions. The pre-compensation actively curves the fringes in computer when generating them; whereas when using the post-compensation, the lens distortion correction is performed in the data processing stage. Both methods are experimentally verified to be effective in improving the measurement accuracies.

Temperature Compensation Method for Digital Cameras in 2D and 3D Measurement Applications

This paper presents the results of several studies concerning the effect of temperature on digital cameras. Experiments were performed using three different camera models. The presented results conclusively demonstrate that the typical camera design does not adequately take into account the effect of temperature variation on the device’s performance. In this regard, a modified camera design is proposed that exhibits a highly predictable behavior under varying ambient temperature and facilitates thermal compensation. A novel temperature compensation method is also proposed. This compensation model can be applied in almost every existing camera application, as it is compatible with every camera calibration model. A two-dimensional (2D) and three-dimensional (3D) application of the proposed compensation model is also described. The results of the application of the proposed compensation approach are presented herein.

Depth-driven variable-frequency sinusoidal fringe pattern for accuracy improvement in fringe projection profilometry

Sinusoidal fringe pattern is widely used in optical profilometry; however, the traditional constant-frequency sinusoidal fringe pattern reduces 3D measurement accuracy in the defocus region. To this end, this paper presents a variable-frequency sinusoidal fringe pattern method that is optimized by the measurement depth. The proposed method improves the pixel matching accuracy and thus increases measurement accuracy. This paper theoretically determines the optimal frequency by analyzing the pixel matching error caused by intense noise in a captured image; presents the online frequency optimization along abscissa and ordinate axes in the sinusoidal fringe patterns; and details the encoding and decoding to use variable-frequency fringe patterns for 3D profilometry. Simulations and experiments demonstrate that our proposed method can improve the 3D measurement accuracy and increase measurement robustness.

Three-dimensional shape measurement using a structured light system with dual projectors

This paper introduces a structured light system with two projectors and one camera for three-dimensional (3D) shape measurement to alleviate problems created by a single projector such as the shadow problem. In particular, we developed (1) a system calibration framework that can accurately calibrate each such camera-projector system; (2) a residual error correction method based on the system error function; and (3) a data fusion method utilizing the angle between the projection direction and surface normal. Experimental results demonstrate that the proposed dual-projector structured light system improves the measurement accuracy besides extending the measurement range of a single projector system.

Homography estimation by two PClines Hough transforms and a square-radial checkerboard pattern

The homography matrix is crucial for many optical metrology applications, such as three-dimensional surface imaging by structured-light projection. In this paper, a straightforward homography estimation method using the image of three particular points on the reference plane is proposed. For this, a new square-radial checkerboard pattern designed to generate three sets of concurrent lines is proposed. The lines and points of interest are detected using two PClines Hough transforms. Relevant concepts such as parallel coordinates and the used Hough transforms are explained. The usefulness of our proposal is verified experimentally.

Out-of-Focus Projector Calibration Method with Distortion Correction on the Projection Plane in the Structured Light Three-Dimensional Measurement System

The three-dimensional measurement system with a binary defocusing technique is widely applied in diverse fields. The measurement accuracy is mainly determined by out-of-focus projector calibration accuracy. In this paper, a high-precision out-of-focus projector calibration method that is based on distortion correction on the projection plane and nonlinear optimization algorithm is proposed. To this end, the paper experimentally presents the principle that the projector has noticeable distortions outside its focus plane. In terms of this principle, the proposed method uses a high-order radial and tangential lens distortion representation on the projection plane to correct the calibration residuals caused by projection distortion. The final accuracy parameters of out-of-focus projector were obtained using a nonlinear optimization algorithm with good initial values, which were provided by coarsely calibrating the parameters of the out-of-focus projector on the focal and projection planes. Finally, the experimental results demonstrated that the proposed method can accuracy calibrate an out-of-focus projector, regardless of the amount of defocusing.

Self-recalibration of a robot-assisted structured-light-based measurement system

The structured-light-based measurement method is widely employed in numerous fields. However, for industrial inspection, to achieve complete scanning of a work piece and overcome occlusion, the measurement system needs to be moved to different viewpoints. Moreover, frequent reconfiguration of the measurement system may be needed based on the size of the measured object, making the self-recalibration of extrinsic parameters indispensable. To this end, this paper proposes an automatic self-recalibration and reconstruction method, wherein a robot arm is employed to move the measurement system for complete scanning; the self-recalibration is achieved using fundamental matrix calculations and point cloud registration without the need for an accurate calibration gauge. Experimental results demonstrate the feasibility and accuracy of our method.

Calibration method for a large-scale structured light measurement system

The structured light method is an effective non-contact measurement approach. The calibration greatly affects the measurement precision of structured light systems. To construct a large-scale structured light system with high accuracy, a large-scale and precise calibration gauge is always required, which leads to an increased cost. To this end, in this paper, a calibration method with a planar mirror is proposed to reduce the calibration gauge size and cost. An out-of-focus camera calibration method is also proposed to overcome the defocusing problem caused by the shortened distance during the calibration procedure. The experimental results verify the accuracy of the proposed calibration method.

Position and orientation measurement adopting camera calibrated by projection geometry of Plücker matrices of three-dimensional lines

A position and orientation measurement method is investigated by adopting a camera calibrated by the projection geometry of the skew-symmetric Plücker matrices of 3D lines. The relationship between the Plücker matrices of the dual 3D lines and the 2D projective lines is provided in two vertical world coordinate planes. The transform matrix is generated from the projections of the 3D lines. The differences between the coordinates of the reprojective lines and the coordinates of extracted lines are employed to verify the calibration validity. Moreover, the differences between the standard movement distance of the target and the measurement distance are also presented to compare the calibration accuracy of the 3D line to 2D line method and the point-based method. Furthermore, we also explore the noise immunity of the two methods by adding Gaussian noises. Finally, an example to measure the position and orientation of a cart is performed as an application case of this method. The results are tabled for the reproduction by the readers. The results demonstrate that the line to line method contributes higher calibration accuracy and better noise immunity. The position and orientation measurement adopting the line to line method is valid for the future applications.

Phase-3D mapping method developed from back-projection stereovision model for fringe projection profilometry

Two major methods for 3D reconstruction in fringe projection profilometry, phase-height mapping and stereovision, have their respective problems: the former has low-flexibility in practical application due to system restrictions and the latter requires time-consuming homogenous points searching. Given these limitations, we propose a phase-3D mapping method developed from back-projection stereovision model to achieve flexible and high-efficient 3D reconstruction for fringe projection profilometry. We showed that all dimensional coordinates (X, Y, and Z), but not just the height coordinate (Z), of a measured point can be mapped from phase through corresponding rational functions directly and independently. To determine the phase-3D mapping coefficients, we designed a flexible two-step calibration strategy. The first step, ray reprojection calibration, is to determine the stereovision system parameters; the second step, sampling-mapping calibration, is to fit the mapping coefficients using the calibrated stereovision system parameters. Experimental results demonstrated that the proposed method was suitable for flexible and high-efficient 3D reconstruction that eliminates practical restrictions and dispenses with the time-consuming homogenous point searching.

A method to calibrate a camera using perpendicularity of 2D lines in the target observations

Camera calibration based on point features leads the main trends in vision-based measurement systems for both fundamental researches and potential applications. However, the calibration results tend to be affected by the precision of the feature point extraction in the camera images. As the point features are noise sensitive, line features are more appropriate to provide a stable calibration due to the noise immunity of line features. We propose a calibration method using the perpendicularity of the lines on a 2D target. The objective function of the camera internal parameters is theoretically constructed by the reverse projections of the image lines on a 2D target in the world coordinate system. We experimentally explore the performances of the perpendicularity method and compare them with the point feature methods at different distances. By the perpendicularity and the noise immunity of the lines, our work achieves a relatively higher calibration precision.