 Original Article
 Open Access
 Published:
ThreeDimensional Reconstruction of Welding Pool Surface by Binocular Vision
Chinese Journal of Mechanical Engineering volume 34, Article number: 47 (2021)
Abstract
Current research of binocular vision systems mainly need to resolve the camera’s intrinsic parameters before the reconstruction of threedimensional (3D) objects. The classical Zhang’ calibration is hardly to calculate all errors caused by perspective distortion and lens distortion. Also, the imagematching algorithm of the binocular vision system still needs to be improved to accelerate the reconstruction speed of welding pool surfaces. In this paper, a preset coordinate system was utilized for camera calibration instead of Zhang’ calibration. The binocular vision system was modified to capture images of welding pool surfaces by suppressing the strong arc interference during gas metal arc welding. Combining and improving the algorithms of speeded up robust features, binary robust invariant scalable keypoints, and KAZE, the feature information of points (i.e., RGB values, pixel coordinates) was extracted as the feature vector of the welding pool surface. Based on the characteristics of the welding images, a mismatchelimination algorithm was developed to increase the accuracy of imagematching algorithms. The world coordinates of matching feature points were calculated to reconstruct the 3D shape of the welding pool surface. The effectiveness and accuracy of the reconstruction of welding pool surfaces were verified by experimental results. This research proposes the development of binocular vision algorithms that can reconstruct the surface of welding pools accurately to realize intelligent welding control systems in the future.
Introduction
Gas metal arc welding (GMAW) is widely applied in modern manufacturing industries. To improve the weld quality of manual GMAW, welders correct either the welding parameters or the position of welding gun based on information from the welding pool surface acquired by sight and their expertise [1]. This means that the welding pool surface contains important visual information that allows skilled welders to control GMAW process. To decrease the workload and occupational risks for welders, much research has been done in the last few decades to transfer this inspection work to intelligent control systems [2].
Caprio [3] utilized the oscillation of the welding pool to estimate its penetration during the Laser Powder Bed Fusion process. Thomas [4] reconstructed the thermal images of cold metal transfer process to monitor the porosity defects and improper weld beads. Vasilev [5] utilized an ultrasonic thickness measurement system to control the welding current and welding speed. These methods have mostly focused on improving the welding process with information from onedimensional or twodimensional welding pool surface data. However, it is clear that threedimensional (3D) information of the welding pool surface can better reflect the weld quality.
The 3D reconstruction method for welding pool can be categorized into the 3D structured light method [6], shape from shading method [7], and binocular stereo vision method [8]. To avoid arc interference during measurement, Zhang et al. [9] proposed a novelstructured laser vision system for reconstructing 3D welding pool surface. In their experiments, a structured laser was projected onto the welding pool surface and was specularly reflected onto a preset imaging plane. The images of the laser points on the plate were captured by camera. Finally, the geometry of the welding pool surface was reconstructed by the Edgepoint algorithm and Onepoint algorithm [10]. However, the deviation of this method can reach a maximum of 1.22 mm. Chen [11] proposed a new image processing framework that could extract key characteristics of 3D welding pool geometry through its twodimensional passive vision images. It was found that the support vector machine classification model was accurate enough to obtain the width and height of the welding pool. The camera’s exposure time had to be carefully adjusted for different welding currents, as it remarkably affected the image quality of the welding pool and the accuracy of the results. Zhong et al. [12] put forward measures to increase the effectiveness and accuracy of shape from shading (SFS) method. An improved algorithm was used to reconstruct the weld surface of aluminum alloy for gas tungsten arc welding (GTAW). However, the SFS method is difficult to be used in GMAW process as its spatters, arc interference, and welding stability is much worse than that in GTAW. An accurate and antijamming 3D reconstruction method for GMAW is still needed for the suppression of strong electromagnetic interference, high heat radiation interference, weld fumes, and other conditions.
The binocular stereo vision method is more accurate and efficient in acquiring reconstruction results. Mnich [13] attempted to use this method to reconstruct a GMAW pool, yet results showed that the binocular vision system needed to be improved, as extrabright areas on pool surface could not be reconstructed. Richardson [14] used a particle image velocimeter (PIV) to track the movement of oxide particles to determine the velocity field on welding pool and then transferred the above data into a 3D fluid velocity by the stereo vision method. The calibration process was the main factor that influenced measurement precision during 3D reconstruction; the average distance error between the tested corner points was 0.14 mm. Liang [15] established a biprism stereo vision system to characterize weld penetration based on the geometry of the welding pool surface under pulsed GMAW. A twostep stereo matching algorithm was proposed to reconstruct the 3D shape of the welding pool surface. However, some regions on the surface were discontinuous, and the accuracy of this algorithm was not discussed. Xiong [16] also developed a biprism stereo vision system to reconstruct welding pool geometry in GMAW. A globalbased iterative matching algorithm and triangle measuring method were optimized. The results were validated by a reconstructed standard cylinder with clear checkboard, yet the maximum height error was 4.15%, showing that the accuracy and usability of a globalbased iterative matching algorithm for GMAW process without checkboard still needs to be verified and improved.
In this paper, an improved binocular vision system was developed to observe the welding pool surface. An automatic 3D reconstruction method was proposed to suppress arc interference on welding pool information. The mathematical models including detection, description and matching of feature points were established to effectively and robustly calculate the world coordinates of welding pool surface against different welding conditions. This study was foundation of online intelligent control of welding pool behavior for GMAW process.
Experimental System
The binocular vision system is given in Figure 1. Two Basler acA2000165uc cameras (color) were used, and their CMOS sizes were both 2/3". The focal length of the camera lens was 35 mm with a 650nm cutoff filter and 7# welding goggle. The aperture was F/5.6. Two cameras were connected with a synchronizer trigger to generate two synchronous images. The frequency of sampling was 200 frames/s with a resolution of 600×500 pixels and exposure time of 40 μs.
The weld workpiece was fixed on a moveable workbench. Beadonplate welding tests were carried out on Q235 mild steel plates with dimensions of 250.0 mm × 70.0 mm × 5.0 mm. The welding wire material was H08Mn2Si with a diameter of 1.2 mm. The distance between the welding wire tip and workpiece surface was 18.0 mm. The chemical compositions of the welding wire and base metal are presented in Table 1. The direct current electrode negative mode was used, and the distance from the wire tip to the workpiece was 18.0 mm. Other welding parameters are listed in Table 2. The torch and cameras were stationary during the welding process. The workpiece and workbench were moved at a preset welding speed.
Calibration System
In order to effectively avoid errors caused by perspective distortion and lens distortion (including radial distortion, centrifugal distortion, and prism distortion), a preset coordinate system for camera calibration [17] was used to establish the corresponding relationship between the world coordinates and the image coordinates via a calibration target with a point matrix at different positions. For example, Figure 1 illustrates how two cameras were fixed on a tripod. Figure 2 shows how the target paper was moved along the y direction from y=0.0 mm to y=20.0 mm with a moving step length of 0.5 mm. The target paper was captured by the two cameras at each position. The red dot on the target paper represents the coordinate origin of the point matrix. Finally, O_{xyz} and O_{x0y0z0} were the world coordinate system and pixel coordinate system, respectively.
The world coordinates of black dots in each target image were calculated by their distance from the red dot. That is, when the black dot P_{b} was located in the a row to the left of the red dot, its world coordinate was x_{d}= a×1 mm = a mm; when the black dot was located in the b row above the red dot, its world coordinate was z_{d} = b×1 mm = b mm, as shown in Figure 3. It can be seen that the world coordinates of the black dots are [x_{d}, y_{d}, z_{d}]=[a, y_{m}, b]. When the world coordinates of all grid points on the graph were assigned, the world coordinate system of the calibration system was successfully constructed.
It is assumed that a point P_{a} was located in the area of calibration that was captured by two cameras, as shown in Figure 4.
The world coordinates of P_{a} could be reconstructed from the following procedure:
(1) The image coordinates of P_{a} were P_{1}(X_{1}, Y_{1}) in camera 1 and P_{2}(X_{2}, Y_{2}) in camera 2. The four nearest neighbors of P_{1} in the captured image were A(x_{A}, y_{A}), B(x_{B}, y_{B}), C(x_{C}, y_{C}), and D(x_{D}, y_{D}). Their world coordinates, which were calibrated prior, were A′(x_{A'}, y_{p}, z_{A'}), B′(x_{B'}, y_{p}, z_{B'}), C′(x_{C'}, y_{p}, z_{C'}), and D′(x_{D'}, y_{p}, z_{D'}). P′(x′_{yp}, y_{p}, z′_{yp}) could be calculated by the four nearest neighbors of P_{1} as follows:
Similarly, all the corresponding world coordinates of P_{2} at 41 planes can be obtained as P′′(x′′_{yp}, y_{p}, z′′_{yp})(y_{p}=0.5i, i = –20, –19, …, 19, 20).
(2) The distance D_{yp} between P′(x′_{yp}, y_{p}, z′_{yp}) and P′′(x′′_{yp}, y_{p}, z′′_{yp}) at all the yplanes was calculated as follows:
When P_{a} was in a certain yplane (y = 0.5i), P′ and P′′ overlapped, and D_{yp} was 0. Then x′_{yp} = x′′_{yp} and z′_{yp} = z′′_{yp}; the world coordinates of P_{a} were (x'_{yp}, y_{p}, z'_{yp}). Otherwise, P_{a} was located between two adjacent plane matrices where the sum of D_{yp1} and D_{yp2} represented the minimum. If y_{p1}=0.5i and y_{p2}=0.5(i+1), the world coordinates were as follows:
where P′_{yp1} and P′_{yp2} were points captured by camera 1, and their world coordinates are (x′_{yp1}, y_{p1}, z′_{yp1}) and (x′_{yp2}, y_{p2}, z′_{yp2}), respectively. Similarly, P′′_{yp1} and P′′_{yp2} were points captured by camera 1, and their world coordinates were (x′′_{yp1}, y_{p1}, z′′_{yp1}) and (x′′_{yp2}, y_{p2}, z′′_{yp2}), respectively. Thus, the world coordinates of point P_{a} were successfully achieved. Verification experiments showed that the maximum relative error between the recovered and measured results was less than 0.6% [17].
Prepreprocessing of Welding Pool Image
The size of the welding pool under the arc is small, and there are various noises in the process of image acquisition and quantification. In addition, the existence of a strong arc will also affect the extracting and matching processes of feature points, which decreases the accuracy of 3D reconstruction. Therefore, image processing algorithms, which mainly include graylevel transformations, image filtering, and edge detection, are needed to preprocess the welding pool images [18]. Figure 5 shows the original images of the welding pool under the arc. Figure 6 shows the processed images, which can be used for featurepoint detection.
SURFBRISKKAZE Feature Point Matching Algorithm
Finding the matching feature points accurately and efficiently is a high priority for welding pool surface reconstruction. The algorithms need to adapt to various requirements, such as the small scale of the welding pool, affine transformation of camera angles, noise interference of spatter and arc electromagnetism, severe illumination of arc, and nonlinear image distortion.
SURF Feature Point Detection
For some welding pool images captured by cameras 1 and 2, box filters [19] were used to approximately replace the Gaussian second order derivatives. The different box filters were selected to establish the scale space for featurepoint detection. The feature points were extracted by a Hessian matrix and a threshold on each scale space layer. For an image I(x, y), a Hessian matrix with scale \(\sigma\) at point (x, y) was defined as follows:
where L_{xx} is the convolution of the Gaussian second order partial derivative (Gaussian filter), and the input image I(x, y) at point (x, y), the meaning of L_{xy}, L_{yy}, and L_{yx} are the same as L_{xx}. The Gaussians is defined as
To facilitate the calculation, the convolution D_{xx}, D_{xy}, and D_{yy} of the box filters and the input image was used to replace L_{xy}, L_{yy}, and L_{yx} to construct the fast Hessian matrix. Box filters in size 9×9 were substituted by the Gaussian filter when parameter λ was 1.2. The relationship between D_{xx}, D_{yy} and L_{xx}, L_{yy} is as follows:
where ∙_{F} is the Frobenius norm, and ω is used to balance the relative weights in the expression for the Hessian’s determinant. In practical application, a value of 0.9 was adopted to obtain an approximate Hessian matrix determinant:
A threshold was needed to estimate the feature points detected by the Hessian matrix. When the value of Hessian’s determinant for the test point was greater than this threshold, a nonmaximum suppression in a 3×3×3 neighborhood was applied. When the value of a text point was greater than that of 26 neighbor points, it was selected as an optimal feature point. When the maxima detected by Hessian matrix was less than the threshold, this maximum was excluded, so the speed of maxima detection was accelerated. Figure 7 shows the feature points extracted by the speeded up robust features (SURF) algorithm. It can be seen that the SURF feature points (red dots) were evenly distributed in the welding pool, having the advantage of affine invariance; however, it cannot fully reflect the edge profile.
BRISK Feature Point Detection
In order to obtain the characteristic of scale invariance on the edges of the welding pool, the scale space was composed of four inner layers c_{i} and four middle layers d_{i}(i=0,1,2,3) in the frame structure of binary robust invariant scalable keypoint (BRISK) feature detection [20]. Each inner layer image was obtained by 0.5times down sampling of the previous inner layer image, where the original image corresponded to the c_{0} layer. Each intermediate layer d_{i} was located between two adjacent inner layers c_{i} and c_{i+1}. The first intermediate layer d_{0} was obtained by 1.5times down sampling of the original image c_{0}, and the remaining intermediate layers were obtained by 0.5times down sampling of the previous intermediate layer. It was assumed that δ represented the scale, where δ(c_{i})=2^{i} and δ(d_{i})=1.5×2^{i}.
After the scale space of BRISK was constructed, the feature points were extracted on each scale. The extreme points detected in the spatial domain and the scale domain were regarded as feature points, so the BRISK corner points had scale invariance for the welding pool edges. BRISK feature point detection was determined by the following equation:
where I(R) was the gray value of the central pixel R, I(x) was the gray value of the pixel points surrounding P, and ε was the threshold, which was 0.00001 in this study. If N was greater than ε, the candidate point was the feature point. BRISK feature points on the welding pool images are shown in Figure 8. It can be seen that the BRISK feature points reflected the edge profile well. However, the resolution on each layer of scale space calculated by the SURF and BRISK algorithms was blurry, which affected the accuracy of feature points. It also affected the number of feature points detected in the blurry area, created by welding pool oscillation.
KAZE Feature Point Detection
The additive operator splitting algorithm and the variable conductance diffusion method were used to construct nonlinear scale space in this study. The logarithmic steps were used in nonlinear scale space to generate the O group and S group [21]. Different groups and layers were identified by group index o and layer index s, respectively. The corresponding relationship between group, layer and scale parameter σ was expressed as follows:
where σ_{0} was the initial scale parameter, which flattens the original image to reduce the noise induced by the arc’s magnetic field. The relationship between time parameter t and scale parameter σ_{i} is:
The nonlinear scale space is expressed as:
where L is the luminance of the image and A_{i} is a matrix that encodes the image conductivities for each dimension.
Combining Eqs. (11) and (13), the KAZE feature points on two images captured by cameras 1 and 2 are shown in Figure 9. The results show that this method was effective to extract feature points from smooth surface with small brightness differences, especially for the whitedottedline area.
SURFBRISKKAZE Feature Description and Matching
In order to match the feature points in the SURF, BRLSK, and KAZE methods, a descriptor was determined for the extracted points. The details of the descriptor, which is a 64dimensional vector, has been discussed in reference [19].
The 64dimensional vector obtained by the above method only included grayscale information of the welding pool images; the color information of images were not considered, which might decrease the matching accuracy of welding pool images.
The information of the original welding pool images (color) was much richer than the preprocessed welding pool images (gray) in Section 4. The signal interference was also included in the original images. To utilize the color information of the original images effectively, only the information of feature points in Sections 5.1‒5.3 and their eight neighborhoods were used to improve the above 64dimensional vector. The steps were as follows:
(1) It was assumed that r, g, b were the values of color information for the feature point (x, y). The value n_{c} was the total number of the feature points (x, y) and their neighborhood pixels. Here, r, g, b∈[0, 255], and n_{c} was set at 9 in this study.
(2) Calculating the mean values μ_{r}, μ_{g}, and μ_{b} of the feature point (x, y) and its eight neighborhoods. The value E was the 3D vector composed by μ_{r}, μ_{g}, and μ_{b}. The equations were detailed as follows:
(3) Calculating the variances δ_{r}, δ_{g}, and δ_{b} of the feature point (x, y) and its eight neighborhoods as follows. F was the 3D vector composed by δ_{r}, δ_{g}, and δ_{b}.
(4) Normalizing vector E and F, respectively.
(5) The normalized mean value vector E_{n} and variance vector S_{n} were combined to form a sixdimensional RGB color classification descriptor vector V_{c}.
(6) By substituting Eq. (17) into V_{s}=(i_{1}, i_{2},…, i_{64}), the descriptor V was expressed as follows:
The matching process was carried out as follows. A feature point P_{1} (image of camera 1) was compared with a feature point P_{2} (image of camera 2) by calculating the Euclidean distance between their descriptor V. The Euclidean distance is:
where V′=(i′_{1}, i′_{2},…, δ′_{g}/255, δ′_{b}/255) is the descriptor of P_{1} and V″=(I″_{1}, I″_{2}, …, δ″_{g}/255, δ″_{b}/255) is the descriptor of P_{2}. A matching pair was detected if the ratio of their distance to the distance of the second nearest neighbor was less than 0.6.
Figure 10 shows the matching pairs obtained by the SURFBRISKKAZE algorithms whose descriptor was 64dimensional. Figure 11 shows the matching pairs obtained by the improved 70dimensional descriptor V. Though the number of mismatching pairs was decreased because of the use of V, some mismatching pairs still remained, and mismatch elimination was needed.
Improved RANSAC Algorithm
There were some mismatching pairs (gross errors) in the above matching pairs. A random sample consensus algorithm was used to eliminate the mismatches in this work. The traditional RANSAC algorithm can complete the estimation for the model parameters through data iteration and result validation. It can also reduce the number of gross errors to increase the matching accuracy [22]. However, the traditional RANSAC algorithm also has shortcomings. For example, the iteration time of the traditional RANSAC algorithm depends on the experimental data and the valid data rate. When the number of valid data is small, the number of iterations of the algorithm will increase exponentially because of a large number of mismatching pairs, which greatly increases the running time of the algorithm. Meanwhile, the initial model parameters are calculated based on a subset of the experimental data. When the valid data rate of the subset is not high, the initial model will be extremely unreasonable, and the verification of this unreasonable model will consume a lot of time, which seriously affects the overall efficiency of the algorithm. The traditional RANSAC algorithm was improved as follows.
Data Preprocessing
The data preprocessing model was defined as M(P, C). P was the matching pairs (P_{1}, P_{2}) obtained by Section 5.4, which was written as:
C was the criterion for eliminating mismatching point pairs of P, and it was expressed as:
where k_{i} is the ratio calculated by pixel coordinates, and k is the median of k_{i}. Δk is the threshold value, which was 0.9 in this study. If d_{i}* is the square root calculated by pixel coordinates, d_{i} is the top 80% of d_{i}* value from small to large. d is the median of d_{i}, and Δd is the threshold value, which was 13 in this study. The k_{i} and d_{i} were expressed as:
where x_{i} and y_{i} are the pixel coordinates of P_{1}[i] captured by camera 1, and x_{j} and y_{j} are the pixel coordinates of P_{2}[i] captured by camera 2.
To increase the proportion of valid data in the data set of P, all the matching pairs were first calculated based on Eqs. (22) and (23). Any matching pair that did not satisfy Eq. (21) was deleted from the data set of P, and others were selected as a new data set of Q. The Q contained enough matching pairs, which can be used to estimate the truest homography matrix parameter model as much as possible. The proportion of valid data in the Q became larger, which greatly reduces the number of iterations to calculate the maximum valid data. As a result, the efficiency of the algorithm was improved. The above work completed the prepurification of the raw data for the RANSAC algorithm. Figure 12 shows the matching pairs preprocessed by the data preprocessing model. Compared to Figure 11, the number of mismatching pairs was decreased.
Pretest Model
In the traditional RANSAC algorithm, the subset (S) is randomly extracted from experimental data (P) of Section 5.4, and the corresponding initial model is estimated by the subset. The initialization model is tested by all the remaining matching points (C_{P}S), which are not belong to the above subset. This circulative progress needs much verification time because many initialization models are not reasonable, especially when the proportion of mismatching pairs is high.
To increase efficiency of the RANSAC algorithm, a pretesting model was proposed to estimate the initial models before the traditional testing of RANSAC algorithm. The steps of the pretesting model were as follws.
(1) n pairs of matching points were randomly selected from the new data set (Q) of Section 6.1.
(2) The improved initial model T was estimated by m pairs of matching points, which was written as follows, where m ⊆ n:
where, t_{11}, t_{12},…, t_{33} are the parameters of T, (x_{1}, y_{1}) is the pixel coordinate of P_{1}[i], and (x_{2}, y_{2}) is the pixel coordinate of P_{2}[j]. m is the number of matching pairs needed to solve the Eq. (25), which was 4 in this study.
(3) The remaining nm pairs were used to verify the reasonability of T by the following equation:
where E_{s} is test error calculated by the matching pairs of C_{n}m. The threshold t_{p} was 0.6 in this study.
(4) If the E_{s} values of all nm pairs were less than t_{p}, the initial model T was started to be tested by the remaining matching points (C_{Q}n) of Section 6.1 as the traditional RANSAC algorithm; otherwise, the initialization model T was discarded directly, and n pairs of matching points were reselected for the next cycle of iterative estimation.
The unreasonable initial models were discarded quickly by the above calculation, which reduced the detection time to verify the initial model. A flow chart of the improved RANSAC algorithm is shown in Figure 13. The real detection time of traditional and improved RANSAC algorithms are discussed in Section 6.3. Figure 14 is the matching point pairs of the welding pool under the arc processed by the improved RANSAC algorithm.
Estimation of Commutating Time
To obtain at least one subset that could pass through the verification of the traditional RANSAC algorithm under a certain confidence p (0.95‒0.99), the minimum sampling times k must meet the following requirement:
where ω is the proportion of the mismatching pairs in P, and m_{t} is the minimum sample number for all subsets to calculate the initial model parameters of verification which was 4 in this study.
Supposing that the number of P is n_{P}, it takes t_{s} seconds to conduct a random sampling from the data set, and it takes t_{c} seconds to calculate the initial model. t_{j} is the time to test the initial model by one couple of remaining matching pairs. The total time to test the initial model is (n_{P}−4)t_{j}. Therefore, the calculation time T_{t} of traditional RANSAC algorithm with k random sampling times is:
In this study, the average k was 500, t_{s} was 2.21×10^{−4} s, t_{c} was 5.45×10^{−5} s, and t_{j} was 3.67×10^{−5} s. The computer processor used was an Intel(R) Core(TM) i79750H CPU @ 2.60 GHz with a memory of 8 GB.
For the improved RANSAC algorithm, the minimum sampling times k′ must meet Eq. (27) as well, where m_{t} is n mentioned in Section 6.2. Eq. (28) is modified as:
where T′ is the calculation time of the improved RANSAC algorithm, n_{Q} is the number of Q, and ω_{Q} is the proportion of the mismatching pairs in Q.
The time reduced by the improved RANSAC algorithm is as follows:
In this study, k′ was 1500. Though the total time of t_{s} and t_{c} was increased if k′ was larger than k, it was an order of magnitude smaller than the time for testing the initial model. Therefore, the improved RANSAC algorithm reduced a series of unnecessary operations and improved the efficiency of the algorithm. While its calculation accuracy was not smaller than that of the traditional RANSAC algorithm, the values of k′, n, and m were not optimal yet and can be optimized in the future.
Compared to the traditional RANSAC algorithm, the calculating efficiency in the improved RANSAC algorithm was increased by at least 160% in this study, as shown in Table 3, which showed a large potential to reduce the calculating time.
Reconstruction Results and Validation
The GMAW experiments were carried out to capture pairs of the welding pool images under the arc. The parameters of the welding process are shown in Table 2. The featurematching points on each pair of welding pool images were extracted according to Sections 5 and 6. The world coordinates of the welding pool surface were reconstructed according to Section 3. The results for 240 A and 33 V are shown in Figure 15.
Surface Reconstruction
The way to transfer the point cloud into the 3D reconstructed surface was through the LOWESS regression smoothing filtering algorithm, which was a local scatter weighted estimation algorithm depending on the adjacent points [23]. The algorithm added a sliding window on the basis of the least square method. If the sliding window width was large, there would be more scattered points in the covered window, resulting in a smooth surface of welding pool and loss of large original data information. On the contrary, if the sliding window width was small, the welding pool surface was rough, which also increased the accuracy of reconstructed surface.
In this experiment, the window width was selected as 6 pixels based on the comparative tests, which provided high accuracy for the 3D reconstructed data and produced a smoothly reconstructed surface without trimming. The point with the smallest z value in the point cloud was selected as the center of the welding pool. It is assumed that the world coordinate of the point was (x_{0}, y_{0}, z_{0}), and the final surface reconstruction was the symmetric treatment of the smoothly reconstructed surface in the plane where y=y_{0}. The final surface reconstruction of Figure 15 is shown in Figure 16. It can be seen that the reconstructed welding pool surface was consistent with the actual welding pool surface in GMAW.
Validation
In order to verify the validity of the above feature point extracting and matching algorithms for welding pool images pairs during GMAW, a verification experiment was carried out. The diameter of silicon nitride tracer particle was a little smaller than that of blind hole, so the tracer particle could be embedded tightly in the workpiece as shown in the Figure 17. These tracer particles would float on the surface of the welding pool when the solid metal near the blind hole was melted. This phenomenon was captured by the above binocular vision system, as shown in Figure 18. The feature points at the junction of the tracer particle and the welding pool can be selected manually, as shown in Figure 18 (yellow dots). It was clear that the yellow dots were the correct feature matching points. Their world coordinate values, calculated directly by Section 3, can be used to verify the accuracy of their neighbor feature matching points (red dots in Figure 18) extracted by the above improved algorithms. Here, the heights of the yellow dots and red dots were assumed to be similar as they were close to each other on the smooth surface of the welding pool, though their world coordinates in the x and y directions were different.
Table 4 provides a comparison between the heights of the yellow dots and red dots. The maximum absolute error in the z direction was less than 0.07 mm, and the maximum relative error was smaller than 6.0%. All the results indicated that the proposed algorithms mentioned above can reconstruct the 3D surface of welding pool for GMAW with high efficiency.
Conclusions

(1)
A new method was proposed to determine the feature matching of welding pool captured by a binocular stereo vision system. It included improved SURFBRISKKAZE algorithms, improved RANSAC algorithms, and surface reconstruction algorithms, which realized the 3D reconstruction of the welding pool surface under the GMAW arc.

(2)
The feature points descriptor was improved by considering the color information of images, which increased the matching accuracy of welding pool images. A data preprocessing and pretest model were added to the RANSAC algorithm to improve the calculating efficiency of the algorithm.

(3)
The experimental results exhibited high accuracy and efficiency of the new method in reconstruction of welding pool surface. For the reconstructed data, the maximum relative error was smaller than 6.0%.

(4)
The quantitative relationship between welding pool surface and weld quality should be established based on the experience of a skilled welder, and the parallel computing method should be improved by the CUDA to realize the realtime 3D reconstruction of welding pool surface in the future.
References
 [1]
Y Liu, Y Zhang. Control of 3D welding pool surface. Control Engineering Practice, 2013, 21(11): 14691480.
 [2]
B C Wang, S J Hu, L Sun, et al. Intelligent welding system technologies: Stateoftheart review and perspectives. Journal of Manufacturing Systems, 2020, 56: 373391.
 [3]
L Caprio, A G Demir, B Previtali. Observing molten pool surface oscillations during keyhole processing in laser powder bed fusion as a novel method to estimate the penetration depth. Additive Manufacturing, 2020, 36: 101470.
 [4]
K R Thomas, S Unnikrishnakurup, P V Nithin, et al. Online monitoring of cold metal transfer (CMT) process using infrared thermography. Quantitative Infrared Thermography Journal, 2017, 14(1): 6878.
 [5]
M Vasilev, C MacLeod, Y Javadi, et al. Feed forward control of welding process parameters through online ultrasonic thickness measurement. Journal of Manufacturing Processes, 2021, 64: 576584.
 [6]
R Zong, J Chen, C S Wu, et al. Undercutting formation mechanism in gas metal arc welding. Welding Journal, 2016, 95: 174s184s.
 [7]
L Yang, E Li, T Long, et al. A welding quality detection method for arc welding robot based on 3D reconstruction with SFS algorithm. International Journal of Advanced Manufacturing Technology, 2018, 94: 12091220.
 [8]
J Chen, Z L Feng. Strain and distortion monitoring during arc welding by 3D digital image correlation. Science and Technology of Welding and Joining, 2018, 23(6): 536542.
 [9]
W J Zhang, J Xiao, Y M Zhang. A mobile sensing system for realtime 3D welding pool surface measurement in manual GTAW. Measurement Science and Technology, 2016, 27: 045102.
 [10]
J K Huang, W Pan, J S Chen, et al. The transient behaviours of free surface in a fully penetrated welding pool in gas tungsten arc welding. Journal of Manufacturing Process, 2018, 36: 405416.
 [11]
Z Chen, J Chen, Z Feng. 3D welding pool surface geometry measurement with adaptive passive vision images. Welding Journal, 2019, 98: 379s386s.
 [12]
J Zhong, C Yang, Y Xu, et al. Research on reconstruction of welding pool surface based on shape from shading during robot aluminum alloy pulse GTAW. Advances in Intelligent Systems and Computing, 2015, 363: 525538.
 [13]
C Mnich, F Al Bayat, C Debrunner, et al. In situ welding pool measurement using stereovision. JapanUSA Symposium on Flexible Automation, Denver, USA, 2004.
 [14]
Z Liang, H Chang, Q Wang, et al. 3D reconstruction of welding pool surface in pulsed GMAW by passive biprism stereo vision. IEEE Robotics and Automation Letters, 2019, 4(3): 30913097.
 [15]
J Xiong, Y Liu, Z Yin. Passive vision measurement for robust reconstruction of molten pool in wire and arc additive manufacturing. Measurement, 2020, 153:107407.
 [16]
L Wang, J Chen, X H Fan, et al. Influence of fluid flow on humping bead during highspeed GMAW. Welding Journal, 2019: 315327.
 [17]
R Mehrotra, S M Zhan. Computational approach to zerocrossingbased twodimensional edge detection. Graphical Models and Image Processing, 1996, 58(1): 117.
 [18]
T Janumala, K B Ramesh. Development of an algorithm for vertebrae identification using speeded up robust features (SURF) technique in scoliosis xray images. Image Processing and Capsule Networks(ICIPCN 2020), Bangkok, Thailand, May 67, 2020: 5462.
 [19]
A Khare, B R Mounika, M Khare. Keyframe extraction using binary robust invariant scalable keypoint features. Twelfth International Conference on Machine Vision (ICMV 2019), Amsterdam, Netherlands, Nov 1618, 2019: UNSP 1143308.
 [20]
B Ramkumar, R Laber, H Bojinov, et al. GPU acceleration of the KAZE image feature extraction algorithm. Journal of RealTime Image Processing, 2020, 17(5): 11691182.
 [21]
H C Shih, C H Ma, C L Lu. An efficient fragment reconstruction using RANSAC algorithm. 2019 IEEE 8th Global Conference on Consumer Electronics, Osaka, Japan, Oct. 1518, 2019: 529530.
 [22]
W S Cleveland. Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association, 1979, 74(368): 829836.
Acknowledgements
Not applicable.
Funding
Supported by National Natural Science Foundation of China (Grant No. 51775313), Major Program of Shandong Province Natural Science Foundation (Grant No. ZR2018ZC1760), and Young Scholars Program of Shandong University (Grant No. 2017WLJH24).
Author information
Affiliations
Contributions
JC was in charge of the whole research; ZG and JC wrote the manuscript; CW assisted with sampling and laboratory analyses. All authors read and approved the final manuscript.
Authors’ information
Zunan Gu, born in 1995, is currently a master candidate at Key Laboratory for LiquidSolid Structural Evolution & Processing of Materials, Ministry of Education, Institute of Materials Joining, Shandong University, China.
Ji Chen, born in 1982, is currently a professor at Key Laboratory for LiquidSolid Structural Evolution & Processing of Materials, Ministry of Education, Institute of Materials Joining, Shandong University, China. He received his PhD degree from Shandong University, China, in 2009. His research interests include intelligent control in welding processes and numerical simulation.
Chuansong Wu, born in 1959, is currently a professor at Key Laboratory for LiquidSolid Structural Evolution & Processing of Materials, Ministry of Education, Institute of Materials Joining, Shandong University, China. He received his PhD degree from Harbin Institute of Technology, China, in 1988. His main research interests include intelligent control in welding processes, solidstate welding, and numerical simulation.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Gu, Z., Chen, J. & Wu, C. ThreeDimensional Reconstruction of Welding Pool Surface by Binocular Vision. Chin. J. Mech. Eng. 34, 47 (2021). https://doi.org/10.1186/s10033021005672
Received:
Revised:
Accepted:
Published:
Keywords
 Surface reconstruction
 Welding pool
 Binocular imaging
 Gas metal arc welding
 Feature points matching