Regression Method for Rail Fastener Tightness Based on Center-Line Projection Distance Feature and Neural Network

Abstract

In the railway system, fasteners have the functions of damping, maintaining the track distance, and adjusting the track level. Therefore, routine maintenance and inspection of fasteners are important to ensure the safe operation of track lines. Currently, assessment methods for fastener tightness include manual observation, acoustic wave detection, and image detection. There are limitations such as low accuracy and efficiency, easy interference and misjudgment, and a lack of accurate, stable, and fast detection methods. Aiming at the small deformation characteristics and large elastic change of fasteners from full loosening to full tightening, this study proposes high-precision surface-structured light technology for fastener detection and fastener deformation feature extraction based on the center-line projection distance and a fastener tightness regression method based on neural networks. First, the method uses a 3D camera to obtain a fastener point cloud and then segments the elastic rod area based on the iterative closest point algorithm registration. Principal component analysis is used to calculate the normal vector of the segmented elastic rod surface and extract the point on the centerline of the elastic rod. The point is projected onto the upper surface of the bolt to calculate the projection distance. Subsequently, the mapping relationship between the projection distance sequence and fastener tightness is established, and the influence of each parameter on the fastener tightness prediction is analyzed. Finally, by setting up a fastener detection scene in the track experimental base, collecting data, and completing the algorithm verification, the results showed that the deviation between the fastener tightness regression value obtained after the algorithm processing and the actual measured value RMSE was 0.2196 mm, which significantly improved the effect compared with other tightness detection methods, and realized an effective fastener tightness regression.

1 Introduction

Railway fasteners connect rails and sleepers and play a role in maintaining rails at a predetermined position to prevent their horizontal and vertical movement, provide elasticity to the overall track structure, reduce vibration, and delay the accumulation of residual rail deformation [1]. High-speed railways in China have developed rapidly in recent years. Presently, China's high-speed railway operation mileage has exceeded 41000 km, ranking first in the world, followed by the heavy daily operation and maintenance of the railway facilities. The railway fastenings are mostly connected by bolts in the center. In busy railway operations, loose or tight bolts are becoming increasingly prominent, particularly in high-speed and heavy-haul lines with large vibrations and impacts. These problems lead to the loss and damage of fastenings [2].

The safety state of railway fasteners should be tested regularly for each railway working section. The standard manual fastener maintenance operation is generally determined according to different fastener types and the scope of the elastic tongue from the seam. Daily operation uses a "three-point contact method" to determine whether a fastener is qualified [3]. However, the degree of contact between the tongue and rail clamping surface is not controllable, especially when electric or internal combustion engine wrenches are used, because excessive pressure causes the elastic rod to collapse, lose elasticity, and damage the screw sleeve on the sleeper. Moreover, there are operational standards that require tightening wrenches to reach the specified torque range [4]; however, considering the complex conditions at the operation site, it is difficult to control the torque of the wrenches to a stable value. Few controllable torque machines are suitable for large-scale high-speed rail fastener operations; therefore, it is difficult to obtain the actual buckle pressure [5].

Both domestic and foreign scholars have conducted extensive research on fastener detection algorithms. Yang et al. [6] used the LDA algorithm to set the weight of the matrix and calculated the distance between the matrix weight of the target fastener and template fastener image to determine the safety state of the target fastener. However, this study was not comprehensive and only provided a method for detecting whether the fastener was lost. This did not solve the problem of detecting fasteners in a damaged state. Liu et al. [7] set various empirical values for different types of fasteners and proposed relevant positioning and segmentation algorithms based on the characteristics of the regular arrangements of railway fasteners. This method is simple and direct; however, the algorithm has poor robustness. Li et al. [8] combined two traditional visual detection methods, studied and realized the edge detection of the rail surface, and determined the actual position of the fastener according to an empirical value. Feng et al. [9] used an LDA feature extraction algorithm to locate the target fastener position and obtain the Haar features of the target image. The classification and detection of the safety state of the fastener were completed based on the degree of similarity of the feature result comparison. Few studies have focused on detecting and evaluating the tightness of fasteners because of the limited spatial information available.

Compared with 2D images, 3D point clouds have an irreplaceable advantage in terms of the depth information. The most significant difference between 3D and 2D cameras is that the former can obtain depth information at the scale of the real world, whereas the latter can only obtain two-dimensional plane image information at the pixel scale, which plays a significant role in the three-dimensional tightness detection of the fastener when the elastic deformation is small. Some researchers have studied methods for realizing fastener detection based on structured light. Mao et al. [3] proposed a method for extracting the centerline of a complex cylindrical surface and a method for calculating the gap between elastic rods to evaluate the tightness of fasteners. Aytekin et al. [10] established a real-time system using a structural light camera to detect missing fastening nuts. The similarities between the histogram pixels and various detection methods were analyzed. By constructing a cumulative height function and prior knowledge, Wang et al. [2] proposed a method to determine the bolt-tightening state threshold based on an online updating threshold database and compared the height difference with the threshold database to realize bolt-tightening state detection. Han et al. [11] proposed a basic multisource visual data-detection method. By combining two-dimensional strength information and three-dimensional depth information generated by line-structured light projection, nut or bolt position positioning. An accurate perception of the height information can be realized in the dynamic running environment of railways, and a dynamic template-matching algorithm can be used to compare and detect the tightness of fasteners.

Currently, there is little research on tightness tests and evaluation methods for fasteners. Owing to the influences of construction, impact, vibration, and other factors, bolts and nuts inevitably become loose or too tight. Detection methods based on traditional two-dimensional cameras cannot easily obtain information in the depth direction; therefore, it is difficult to realize tightness detection. According to a field investigation [12], the fastener pressure is positively correlated with the number of tightening laps of the bolts, whereas the fastener pressure requirement is closely related to the environment of the track, radius of the bend, and site construction requirements of the track elevation. For example, in a bend where the inclination angle of the track must be adjusted using fasteners, the buckling pressure inside the railway is greater than that outside. Based on the influence of seasons and temperatures, the fasteners on the entire seamless rail line must be operated regularly. The buckle pressure should be reduced quantitatively to release the stress inside the entire rail to reduce the occurrence of rail expansion or breakage and ensure the safety of railway operation. Additionally, the buckle pressure must be moderate: if it is too large, it will reduce the elasticity of the cushion and crush the elastic rod resulting in damage and if it is too small, it will make the fastener slide along the shoulder of the concrete pillow causing shoulder damage and gauge distance expansion. In summary, the tightness evaluation standard of fasteners should not be simply determined as loose or tight according to the threshold value but should map the tightness state of the fasteners according to the relationship between the size and deformation of the buckle pressure. Therefore, identifying the tightness state of fasteners reasonably, quickly, and accurately is the key objective of this study. Figure 1 shows the structural diagram of this study, which describes the identification and prediction processes of fastener tightness.

Figure 1

Paper structure frame diagram

Based on the railway fastener maintenance robot platform, this study proposes a method for calculating the projection distance after dimensional-reduction processing of the centerline of fasteners using feed-forward and backward neural network regression. First, the fastener point cloud is obtained from a structural light camera (such as the RVC X mini camera), and then the region of the fasteners is extracted by registration segmentation and other operations. The normal vector of the vertical surface of the fastener point cloud was obtained using the PCA method to extract the centerline of the fastener, which was then projected onto the upper surface of the fastener bolts to obtain the projection distance and sort it for the regression feature extraction. The trained network was used to identify and predict unknown fastener states. The accuracy and effectiveness of the algorithm were proven through data collection and analysis, and the effects of various parameters on the regression prediction performance of the algorithm were analyzed.

The remainder of the paper is organized as follows: Section 2 discusses the work and provides notation and problem definitions. In Section 3, an algorithm for the centerline projection distance (CPD) is proposed and verified. Section 4 presents the experimental results and analysis. Section 5 summarizes and discusses the issues related to future research.

2 Problem Definition

The focus of this study was to obtain a fastener topography point cloud using a 3D camera and to study the mapping relationship between the fastener topography point cloud and tightness of the fastener based on a pattern recognition algorithm. According to the research, the overall stress of the fastener is roughly linearly correlated with the deformation of some positions within a certain range; therefore, the tightness of the fastener can be identified through deformation. The algorithm in this study focuses on the feature extraction of the fastener point cloud and makes the following agreement considering the feasibility of the fastener point cloud feature and algorithm deployment.

• (1) Because there are many types of fasteners, this study mainly focuses on WJ-8 fasteners, which are widely used in high-speed railways.

• (2) Owing to the corrosion of the bolts in complex environments in some sections, the material properties change, and the buckling pressure is lost. This study evaluates fasteners in good condition, and the algorithm is not temporarily applicable to failed fasteners.

• (3) The performance of the algorithm is significantly affected when unknown objects invade the fastener system. However, this is less likely to occur in ballast-less track systems.

The variables in this study are defined as Table 1.

Table 1 Definition of variables

The predicted fastener tightness determined in this study is represented by the tightening angle of the bolt, and its state is related to the number of points on the centerline p, number of neural network nodes n, number of adjacent points k in the PCA method, and other factors. The definition is ${\varvec{\theta}} \in \{ {\varvec{p}},{\varvec{n}},{\varvec{k}}\}$.

P is the point cloud of the originally collected fasteners. After extracting the point cloud of the fastener, the following parameters were used to predict its tightness state:

$${\varvec{L}}_{ * } \user2{ = }\varsigma \left( { {{\varvec{P}}_{\user2{*}} \user2{,\theta }_{\user2{*}} |\user2{\theta ,P,L}} } \right),$$

(1)

where p, k, n and L are from the training dataset, and L_* and P_* are the unknown parameters of the fastener to be tested. The goal of this study was to establish a relationship between the point cloud characteristics of a fastener's geometric deformation appearance and its tightness, $\varsigma ( \cdot )$. The algorithm that was used is the extracted CPD method.

3 Center-Line Projection Distance Algorithm

3.1 Framework of Algorithm

The overall evaluation process for identifying the tightness of the fasteners is shown in Figure 2. First, the original three-dimensional point cloud of the fastener was obtained using a structural light camera, and the ICP algorithm was used to locate the fastener bolt area and determine the XYZ coordinate direction on the fastener. The X-axis is parallel to the rail along the positive direction of the track repair vehicle. The point cloud area where the fastener and bolt were located was significantly higher than that in the other areas of the background, including the iron bottom plate, concrete pillow, and insulation block, making it easier for the algorithm to segment the elastic rod area. After the accurate registration and segmentation of the fasteners using the standard point cloud template of the bolts, iron bottom plate, and insulating block, the normal vector perpendicular to the surface of the elastic rod was extracted using the PCA method to extract the centerline of the elastic rod. Subsequently, the point on the centerline was projected onto the upper surface of the bolt, and the resulting projection distance was calculated. After the projection distance was ranked, it was input into the neural network for regression prediction to determine the geometric features of the fasteners. To establish a better mapping relationship between the projection distance and bolt turns of the fastener, the influence of the number of points on the fastener centerline (p), number of nodes in the regression network (n), and number of points near k in the PCA method were analyzed. Finally, the CPD method based on the above parameters was used to identify and predict the tightness of the fasteners. Figure 3 shows that WJ-8 fasteners are laid on the high-speed rail system.

Figure 2

Fastener tightness state test flowchart

Figure 3

Conditions of WJ-8 fastener

3.2 Segmentation and Extraction of Fastener Region Based on ICP Algorithm

Currently, commonly used point-cloud segmentation algorithms are based on model fitting, regional growth, and clustering features. Researchers have conducted bolt detection and segmentation using a point cloud containing a bolt structure. Researchers have used the Hough algorithm to detect bolt holes in the point clouds of steel structures to achieve rapid and stable detection. Others have used the region-of-interest method to extract point-cloud features and perform fast segmentation based on the region-growing method. Its advantages are that it can segment connected regions with the same features and provide good boundary information and segmentation results. However, iteration-based methods are time-consuming and computationally intensive. Noise and uneven grayscales may lead to voids and over segmentation.

Because there is no clear feature point for the fastener, setting the initial position of the seed point is difficult. Because the bolt and circular gasket blocked the shooting of the 3D camera, there was a discontinuous fastener point cloud after the segmentation. Therefore, the algorithm in this study transforms the idea of using a continuous fastener point cloud to extract features, focusing on the overall geometric shape features of the cloud to quickly perform segmentation operations. According to China’s railway standard, the distance between the bolt midpoint for adjacent fasteners is approximately 0.6 m, hence, the distance between the rail car running can be set accordingly to determine the approximate initial position of the fasteners to be tested. The conical area of the structural light acquisition point cloud shown in Figure 4 contains bolts, gaskets, fasteners, an iron bottom plate, insulating gaskets, and a concrete base. These parts were arranged sequentially from top to bottom.

Figure 4

Main targets within the range of structural light cameras

This study proposes an iterative segmentation method based on an iterative closest point (ICP) registration algorithm that can locate and segment fasteners more accurately and stably. The process of point cloud registration involves inputting two point clouds, $P_{s}$ and $P_{t}$, and outputting the transformation matrix T to make the coincidence degree of the two point clouds as high as possible. The transformation matrix T can be rigid or flexible; this study applied it to a rigid transformation [13]. This process is divided into rough and accurate registrations. Rough registration is based on the high accuracy of the ICP algorithm, which provides a good initial transformation value for accurate registration. Otherwise, it would be difficult for the registration results to converge to the global optimal value in the subsequent iterative calculations. Because the track detection vehicle moves with reference to a fixed distance on the track and the attitude changes between the fasteners under the same track are not significant, it can provide a good prerequisite for rough registration. After passing the rough registration, the two point clouds are in a roughly overlapping state. The transformation matrix is optimized until the convergence is less than the set threshold or the maximum number of iterations, and a transformation with a smaller and better registration error is obtained. Methods based on the feature points, main direction matching, or improved search point pairs help accelerate the algorithm convergence and improve the registration accuracy.

The discrete point cloud obtained by the 3D camera is defined as

$$P_{s} = \{ p_{1} ,p_{2} ,...,p_{n} \} ,\,\,p_{n} \in\mathbb{R}^ {3} .$$

The target point cloud is defined as

$$P_{t} = \{ p_{1} ,p_{2} ,...,p_{i} \} ,\,\,\,p_{i} \in\mathbb{R}^{3} .$$

The registration problem of the point cloud can be described as

$${\varvec{R}}^{\user2{*}} ,{\varvec{t}}^{\user2{*}} = \mathop {\arg \min }\limits_{{\user2{R,t}}} \{ \frac{1}{{\left| {P_{s} } \right|}}\sum\limits_{i = 1}^{{\left| {P_{s} } \right|}} {\left\| {p_{t}^{i} - \left( {{\varvec{R}} \cdot p_{s}^{i} + {\varvec{t}}} \right)} \right\|^{2} \} } .$$

(2)

According to the above formula, the ICP problem can be divided into two steps: finding the nearest point and solving the optimal transformation. Through constant iterations, optimal varying parameters ${\varvec{R}}_{{\varvec{k}}}$ and ${\varvec{t}}_{{\varvec{k}}}$ are obtained each time. The registration iteration stops if the registration error is less than or equal to the threshold, the maximum number of iterations is reached, or the rate of $\Delta R$ and $\Delta t$ change is less than the threshold; the registration error function is expressed as

$$F({\varvec{R}},{\varvec{t}}) = \sum\limits_{i = 1}^{N} {\left\| {P_{t} - \left( {{\varvec{R}} \cdot P_{s} + {\varvec{t}}} \right)} \right\|^{2} } ,$$

(3)

where R is a 3×3 rotation transformation matrix and t is a 3×1 translation transformation vector. The ICP algorithm does not require deliberate segmentation and extraction of features from a point cloud, making it suitable for faster point clouds with fewer features. In the case of good initial values, the precision and convergence are better. Because of the relationship between the camera layout and position of the fastener on the rail, it has a strong regularity. Except for missing and damaged fasteners, the fasteners to be registered do not show significant changes in their position and attitude.

The track-detection vehicle moved to the top of the fasteners to be evaluated at fixed intervals, as shown in Figure 5, and the original point cloud was captured by the 3D camera. After preprocessing, such as noise reduction and downsampling, the prepared standard fastener bolt point-cloud template was used for registration. The coordinates of the bolt center point after registration were calculated, and the bolt area was divided using the AABB bounding box method, as shown in Figure 6. The iron bottom plate and insulation block area were divided in the same manner to complete the accurate division of the fastener elastic rod. The above steps were repeated until all the fastener elastic rods were extracted. The algorithm can successfully separate the point cloud between the fastener and other objects in the background because the fastener is in contact with the bolt and the contact surface is a plane. When the camera is shooting downward between the fasteners and iron base plate and the point cloud part of the insulating block, there is a gap of approximately 6.5 mm in the radius of the elastic rod.

Figure 5

Structural light camera layout

Figure 6

Fastener bolt separated from surrounding box

3.3 PCA Method Based on k-Nearest Neighbor for Extracting Fastener Center-Line

The tightness feature of the elastic rod mapping the centerline was extracted based on the registration and segmentation results of the fastener. The tightness of the fastener is determined by the elastic force of the elastic rod and buckling pressure of the bolt. Tightening the bolt exerted a downward pressure on the elastic rod. There is a point contact between the bar and rail, and the rail is fixed by the buckle pressure to prevent transverse and longitudinal movements. Because of the complex cylindrical shape of the elastic rod, it is difficult to find feature points or feature surfaces on its body; therefore, it is difficult to apply the traditional feature-matching algorithm to the fastener. Changes in the elastic range were observed while pressing the fasteners. According to a survey, when the elastic deformation of the fasteners is fully pressed, the range of variation is approximately 9–13 mm, which indicates that the geometric characteristics of the fasteners can be described as a whole. To facilitate the feature extraction of the fasteners, it is common to extract the centerline of a complex structure and convert the cylinder feature, which is difficult to describe, into a line feature for description.

The point cloud skeleton is a morphological expression of a three-dimensional model that aims to reduce the complexity of the data dimensions and calculation. Point cloud skeleton extraction has been systematically studied by many scholars. A representative method is the L1 median skeleton extraction method, which uses the L1 theory to map a three-dimensional point cloud and takes the median value of a series of points instead of the average value for iterative contraction to obtain skeleton points [14]. Some researchers have also proposed a point-cloud model skeleton extraction method based on the ROSA, which processes the information extraction skeleton of the central axis of the local area of the point cloud [15]. Point cloud skeleton extraction is based on the Laplace operator contraction, which gradually shrinks iteratively from the periphery to the interior [16]. After summarizing and referring to the studies of other scholars and institutions, and conducting experiments and comparative analysis, we finally determined our research method [17,18,19,20,21]. In this study, the point cloud extraction of the centerline of the elastic rod was performed based on the commonly used PCA method. As a common data processing method, the main principle is to process the data according to the matrix transformation method to make it linearly independent in all dimensions and obtain the important feature components of the data and other information.

The WJ-8 fastener was composed of 60Si2Mn or 55Si2Mn hot-rolled round spring steel with a diameter of 13 mm. To solve the extraction problem of the elastic rod skeleton, the method adopted in this study obtains the normal vector of discrete points on the fastener surface using the PCA method, which must be perpendicular to the fastener surface and point to the corresponding point on the centerline of the fastener. The point on the centerline was located on the central axis of the round steel strip.

According to the point search method using the k-nearest neighbor (KNN), the point set is expressed as

$$P = \{ p_{i} |p_{i} = (x_{i} ,y_{i} ,z_{i} )^{{\text{T}}} \} ,i = 0,1,2,3,...,k.$$

If we use $\overline{p} = {{\sum\limits_{i = 0}^{k} {p_{i} } } \mathord{\left/ {\vphantom {{\sum\limits_{i = 0}^{k} {p_{i} } } {(k + 1}}} \right. \kern-0pt} {(k + 1}})$ for the mean of the point cloud coordinates, the covariance matrix can be expressed as the SVD decomposition of the matrix for eigenvalues and eigenvectors.

$${\mathbf{cov\_}}{\varvec{P}} = \sum\limits_{i = 0}^{k} {(p_{i} - \overline{p})(p_{i} - \overline{p})^{{\text{T}}} } .$$

(4)

The decomposition process of matrix H is as

$${\varvec{H}} = \tilde{P}\tilde{P}^{{\text{T}}} = {\varvec{U}}_{r} \sum {^{2} } \varvec{U}_{r}^{{\text{T}}} .$$

(5)

The first pivot vector is the first column of the matrix ${\varvec{U}}_{r}$, the second vector is the second column, etc. In this case, the eigenvalues of the matrix ${\mathbf{cov\_}}{\varvec{P}}$ and corresponding eigenvectors are respectively $\lambda_{1} ,\lambda_{2} ,\lambda_{3} ,{\varvec{\alpha}}_{{\mathbf{1}}} ,{\varvec{\alpha}}_{{\mathbf{2}}} ,{\varvec{\alpha}}_{{\mathbf{3}}}$ and satisfied with $\lambda_{1} > \lambda_{2} > \lambda_{3}$. The smallest eigenvalue $\lambda_{3}$, corresponding to the eigenvector $\alpha_{3}$ is the normal vector of the section at this point.

The KNN algorithm was used to calculate the k points closest to each point on the elastic rod surface. The value of k affected the extraction accuracy of the normal vector on the elastic strip surface of the fastener. Because the surface of an elastic bar is a continuously changing curved surface, the curvature changes at different points are significantly different. When the curvature of the surface at a point changes significantly, the value of k should be increased appropriately to obtain a more reasonable estimate of the normal vector; however, it should not be increased excessively so that the normal vector can be perpendicular to the local plane composed of k+1 points.

Assuming that there are p points on the point cloud of the elastic rod surface, p samples are formed in the model. The value of p is determined by the sampling proportion of the original point cloud. The larger the p value, the more accurately the curvature of the local plane and the estimation of the normal vector can be fitted. However, the amount of computation will also increase. Therefore, an appropriate p value must be selected to ensure the accuracy and computational requirements of the algorithm.

The preliminary normal line extraction effect of the point cloud based on principal component analysis is shown in Figure 7. Six adjacent points near the point were fitted into a local plane to estimate the normal vector for each point. In this case, the calculated normal vector is ambiguous; that is, only the straight line where the normal vector is located is obtained, but the direction of the line is not determined as the final direction of the normal vector. It can be observed that the normal vector arrangement of the point cloud is relatively scattered, and the direction points are random, which cannot meet the requirement that the normal vector should be perpendicular to the surface of the projectile bar and toward the outside, and the centerline of the elastic rod cannot be extracted. Assuming that the point cloud is sufficiently dense and that the sampling plane is smooth, the normal vector of two adjacent points $\vec{\user2{n}}_{{\varvec{i}}} , \vec{\varvec{n}}_{{\varvec{j}}}$ is assumed to be close to parallel. If the inner product of the two results is negative $\vec{\varvec{n}}_{i} \, \cdot \vec{\varvec{n}}_{j} < 0$, then the normal vector at a certain point must be flipped. Therefore, the normal vector orientation is first set to a certain point in the point cloud. The current point normal vector is set to $\vec{\user2{n}}_{{\varvec{i}}}$ and then traverses all other points and $\vec{\user2{n}}_{{\varvec{j}}}$ is the next point to traverse. If the inner product is negative, $\vec{\user2{n}}_{{\varvec{j}}}$ flips; otherwise, it remains the same.

Figure 7

Original PCA normal vector extraction effect

The point-cloud normal vector extraction algorithm can satisfy the requirements after the directional integration of the normal vectors, as shown in Figure 8. After extraction, the points on the elastic rod point cloud move the radius of the elastic rod by 6.5 mm in reverse, according to the normal vector, to obtain the original data of the points on the centerline. The outlier data were removed, and the results are shown in Figure 9.

Figure 8

Redirect normal vector extraction effect

Figure 9

Extraction of centerline of elastic rod

3.4 Projection Distance and Neural Network Regression Calculated by Fitting Plane

The RANSAC algorithm can estimate the parameters of the mathematical model from a set of observation data containing ‘‘external points’’ through an iterative way, which is often used to determine the plane, sphere, cylinder, etc., whereas the M-estimator Sample Consensus (MSAC) algorithm is a variant of the RANSAC algorithm to fit the plane with higher accuracy. The MSAC algorithm was used to fit the upper surface of the bolt, and the fitting results are shown in Figures 10 and 11.

Figure 10

Bolt upper surface fitting

Figure 11

Projection onto the upper surface plane of the bolt

The general equation for a three-dimensional plane in space can be expressed as

$$Ax + By + Cz + D = 0.$$

(6)

The set and index of the plane point cloud are obtained and solve the equation of the plane according to the known points. Let the coordinates of the three-dimensional space points not on the plane be $(x_{0} ,y_{0} ,z_{0} )$, then the projection coordinate on the plane is set to $(x_{p} ,y_{p} ,z_{p} )$. The line from the projection point to the current point is perpendicular to the plane $U^{2} = A^{2} + B^{2} + C^{2}$. According to the vertical constraint, we obtain

$$\left\{ \begin{gathered} x_{p} = {\raise0.7ex\hbox{${((B^{2} + C^{2} )x_{0} - A(By_{0} + Cz_{0} + D))}$} \!\mathord{\left/ {\vphantom {{((B^{2} + C^{2} )x_{0} - A(By_{0} + Cz_{0} + D))} {U^{2} }}}\right.\kern-0pt} \!\lower0.7ex\hbox{${U^{2} }$}}, \hfill \\ y_{p} = {\raise0.7ex\hbox{${((A^{2} + C^{2} )y_{0} - B(Ax_{0} + Cz_{0} + D))}$} \!\mathord{\left/ {\vphantom {{((A^{2} + C^{2} )y_{0} - B(Ax_{0} + Cz_{0} + D))} {U^{2} }}}\right.\kern-0pt} \!\lower0.7ex\hbox{${U^{2} }$}}, \hfill \\ z_{p} = {\raise0.7ex\hbox{${((A^{2} + B^{2} )z_{0} - C(Ax_{0} + By_{0} + D))}$} \!\mathord{\left/ {\vphantom {{((A^{2} + B^{2} )z_{0} - C(Ax_{0} + By_{0} + D))} {U^{2} }}}\right.\kern-0pt} \!\lower0.7ex\hbox{${U^{2} }$}}. \hfill \\ \end{gathered} \right.$$

(7)

After obtaining the plane equation for the upper surface of the fastener bolt, the original image of the fastener was rotated such that the upper surface of the bolt was parallel to the horizontal plane. The normal vector of the horizontal plane was set to $n_{0}$(0, 0, 1) and the direction was upward. The normal vector of the upper surface of the bolt is expressed as ${\varvec{n}}_{{\mathbf{1}}}$(A, B, and C). Using the vector cross-product method, we determine the angle $\varphi$ between the normal vectors and the corresponding axis $\varpi$ of rotation. The Rodriguez method is then used to solve the rotation matrix R, and the coordinates of the corresponding points in the point cloud are multiplied by the rotation matrix R to obtain the rotating point cloud. The rotation effect is illustrated in Figure 12, where the upper surface of the bolt is parallel to the horizontal ground.

Figure 12

Rotated fastener point-cloud image

Under the equation of the plane in a known space, the projected distance D from point to plane can be expressed as

$$D = {\raise0.7ex\hbox{${(Ax_{0} + By_{0} + Cz_{0} + D)}$} \!\mathord{\left/ {\vphantom {{(Ax_{0} + By_{0} + Cz_{0} + D)} {\sqrt {U^{2} } }}}\right.\kern-0pt} \!\lower0.7ex\hbox{${\sqrt {U^{2} } }$}}.$$

(8)

The centerline point of the fastener was projected onto the plane of the upper surface of the bolt. Based on the principles of the point cloud disorder and displacement invariance, the projection distance was calculated and arranged in ascending order to generate the feed-forward and backward propagation neural network training samples with p features, as shown in Figure 13.

Figure 13

Point-cloud feature sequencing after projection

Preliminary data on the fastener characteristics were obtained based on the previous section. These original data were input into a neural network for feature extraction to reconstruct the mapping relationship between the dataset and tightness features of the fasteners. In an actual situation, the buckle pressure of the fastener is positively correlated with the shape variable of the elastic rod; therefore, this study adopts a two-layer feed-forward network with a sigmoid activation function of hidden neurons and linear output neurons. Linear regression outputs a continuous value, making it suitable for regression problems. Linear regression assumes that the relationship between the output and input is linear.

$$\hat{y} = x_{1} \omega_{1} + x_{2} \omega_{2} + ... + x_{n} \omega_{n} + b,$$

(9)

where $\omega_{n}$ is the weight and b is the deviation of both scalars. The model output $\hat{y}$ is the prediction and estimation of the real output y using linear regression. Here, we set the model output y as the tightening angle of the fastener bolts. Label q of the training set has eight states, representing the corresponding fastener tightening angle. Considering the requirements of the actual tightening condition of fasteners and the accuracy of field construction, the interval of the fastening angles was set as 45°, which was used as the basis for labeling each sample in the training set. In the model training, $\varepsilon$ is an error. The function used to measure the error is called the loss function, and the smaller its value, the smaller the error. The evaluation index $i$ for the sample error is expressed as follows:

$$\varepsilon^{(i)} (\omega_{1} ,\omega_{2} ,...,\omega_{n} ,b) = \tfrac{1}{2}(\hat{y}^{(i)} - y^{(i)} )^{2} .$$

(10)

After training, the total loss of all the training samples was minimized, and the model parameters were obtained:

$$\omega_{1}^{*} ,\omega_{2}^{*} ,b^{*} = \mathop {\arg \min }\limits_{{\omega_{1}^{{}} ,\omega_{2}^{{}} ,b^{{}} }} \varepsilon (\omega_{1}^{{}} ,\omega_{2}^{{}} ,b).$$

(11)

The training algorithm adopted by the network was the L-M algorithm. The small-batch stochastic gradient descent is widely used in deep learning to optimize the algorithms. This process randomly selects the initial values of a group of model parameters and performs several iterations to reduce the loss function value. In each iteration, a small batch $\beta$ consisting of a fixed number of training data samples was randomly and evenly sampled, and the gradient of its average loss with respect to the model parameters was calculated. Finally, the product of this result and a predetermined positive number were used to reduce the model parameters in this iteration. In the process of training the linear regression model, each parameter was iterated as follows:

$$\left\{ \begin{gathered} \omega_{1} - \frac{\eta }{|\beta |}\sum\limits_{i \in \beta } {\frac{{\partial \varepsilon^{(i)} (\omega_{1} ,\omega_{2} ,...,\omega_{n} ,b)}}{{\partial \omega_{1} }}} \mathop{\longrightarrow}\limits^{{}}\omega_{1} , \hfill \\ \omega_{2} - \frac{\eta }{|\beta |}\sum\limits_{i \in \beta } {\frac{{\partial \varepsilon^{(i)} (\omega_{1} ,\omega_{2} ,...,\omega_{n} ,b)}}{{\partial \omega_{2} }}} \mathop{\longrightarrow}\limits^{{}}\omega_{2} , \hfill \\ b - \frac{\eta }{|\beta |}\sum\limits_{i \in \beta } {\frac{{\partial \varepsilon^{(i)} (\omega_{1} ,\omega_{2} ,...,\omega_{n} ,b)}}{\partial b}} \mathop{\longrightarrow}\limits^{{}}b. \hfill \\ \end{gathered} \right.$$

(12)

In the above formula, $|\beta |$ represents the number of samples in each small batch, and $\eta$ is the learning rate of the network. Appropriate results can be obtained by repeatedly adjusting the hyperparameter values. The effect of the verification model training can be evaluated using independent verification sets.

3.5 Flowchart of CPD Algorithm

To express the CPD algorithm proposed in this study more systematically, we designed a flowchart of the algorithm, as shown in Figure 14. The main steps of the algorithm are as follows.

• (1) Step1: Obtain the original fastener point aggregation.

Figure 14

Flowchart of CPD algorithm

$P_{s} = \{ p_{1} ,p_{2} ,...,p_{n} \}$, $p_{n} \in\mathbb{R}^{3}$, and data preprocessing

• (2) Step2: Use the ICP registration algorithm to locate and segment the elastic rod area of the fastener to obtain the point set of the elastic rod surface.

• (3) Step3: Use the KNN algorithm to calculate the neighborhood points of each point on the elastic rod $P = \{ p_{i} |p_{i} = (x_{i} ,y_{i} ,z_{i} )^\text{T} \} ,i = 0,1,2,3,...,k$, use the PCA algorithm to estimate the normal vector of the elastic rod surface and extract the centerline of the elastic rod surface after reverse translation.

• (4) Step4: Fit the plane of the upper surface of the fastener bolt, project the point on the centerline onto the plane, output the projection distance, introduce it into the neural network for regression, and save the well-trained model $M_{i}$.

• (5) Step5: Put the test sample ${\varvec{T}}_{{\varvec{i}}}$ into the well-trained model to predict the unknown fastener state and obtain the result $S_{i}$.

4 Experiments and Result Analysis

4.1 Fastener Inspection and Maintenance Robot

For the WJ-8 fastener widely used on ballastless tracks, field tests were conducted in a track railway test base to evaluate the performance of the proposed fastener safety state detection algorithm. In this section, the collection of the fastener point cloud data is established for the railway fastener maintenance robot, which is composed of the vehicle body, electric torque feedback bolt wrench, manipulator slider module, high-precision structural light camera, material conveying module, and industrial computer, as shown in Figure 15.

Figure 15

Rail fastener inspection and maintenance robot

The 3D structured light camera was installed at a fixed position on the vehicle, and its accuracy can reach 0.012–0.040 mm at a calibration depth of 600 mm. After calibrating the camera parameters, the movement of the vehicle was controlled to sample the data of the test fastenings. The collected fastener data are stored sequentially. This section describes the extraction of the geometric features from the fastener point clouds data to generate training sets, test sets, and other operations.

By comparing the results of the tightness detection method of the fastener in this study with those of other fastener detection methods and using the manually calibrated dataset as the reference standard, the identification and prediction results of the tightness state of the fastener were obtained, and the error was compared with the standard dataset to evaluate the accuracy of the algorithm. Finally, the influence of each parameter on the prediction accuracy of the neural network was discussed.

4.2 Collection Process of Fastener Point Cloud Data Set

The experiment was conducted in the railway base of the Research Institute. The railway in the test site was ballastless track, and the fasteners were WJ-8 type fasteners. Firstly, the fastener was manually adjusted to the test state using the calibration plate for marking. The bolt angle was defined as 0° when the fastener was fully loose, that is, no fastening pressure was generated on the elastic rod. When the fastener was fully tightened, that is, when the middle of the elastic rod part just touches the insulation block, a bolt angle of 270° can be measured. Because the torque value of the bolt wrench should be kept within $\pm 5\%$ of the set value, the seven states that were sampled were based on a benchmark of 45° intervals from 0 to 270°. Seven states were obtained $N_{q} (1,2,3,4,5,6,7)*45^\circ$ by tightening each fastener from 0 to 270°, and then another seven states were obtained by loosening the fastener from 270° to 0°. The data of each state were sampled five times. Through cyclic sampling, the data of 50 groups of different fasteners are collected on the track, as illustrated in Table 2. Choosing 45° as the rotation reference is better controlled on the calibration plate, and the range of the fastener changes is more uniform. The orbital test base is in the outdoor scene. To ensure the sampling accuracy of the structured light camera, the setting state and marking of the fastener are completed manually. In the experiment, standard angle calibration equipment is used to ensure the accuracy in the manual tightening and loosening of the fastener.

Table 2 Data sets of different tightness of fastener bolts

The original data were split, and each fastener had 70 states collected separately. In total, 3500 sample data points were used. Among the 50 groups of fasteners, forty-five groups were randomly selected as the training data and 5 groups as the testing data. After obtaining the original image of the fastener, the original data were finely segmented to obtain an area with only the elastic rod, as shown in Figure 16. The CPD method was then used to extract the normal vector of the fastener surface points, and the points on the centerline were extracted. The points were projected onto the upper surface of the fastener bolts, and the projection distance (PD) was used to sort them. Finally, the points were introduced into the neural network for regression and geometric feature extraction.

Figure 16

Elastic bar region after feature extraction

4.3 Weight of Experimental Parameter in the Tightness State

In the process of predicting the tightness of fasteners using the CPD algorithm, the main parameters affecting the prediction performance of the network include the number of points on the centerline p and the number of k nearest neighbor points in the PCA method. The number of neural network nodes n has a less obvious effect on the results. In the process of predicting the state of unknown fasteners, different p and k values have different influences on the predicted results. The predicted results are compared with the original results, and the root mean square error was calculated as an evaluation index to select the optimal p and k. Table 3 presents the results of the study. After repeated experiments, better results were obtained at p = 4000 and k = 16. For p = 4000, it can be ensured that the calculated amount of training data is appropriate, and the accuracy requirements are reasonable. For k = 16, good symmetry can be guaranteed in the neighborhood near the data point, and the calculation accuracy is higher. The RMSE in the experimental results is a dimensionless value between the state quantities 1 and 7, which is then converted into the overall variation of the elastic strip according to the bolt rotation angle and pitch value in mm.

Table 3 Parameter setting and experimental results

According to the parameters p = 4000 and k = 16, the experimental results were further analyzed to verify the generalization performance of the algorithm. The results are shown in Figure 17.

Figure 17

Conclusion of RMSE results

From the comparison of the above experimental results, it can be observed that the error in the training effect of the network fluctuated between 0.2100 and 0.3500. In the eighth training group, a good error convergence was attained to achieve a good training effect. As shown in Figure 18, the best and worst predicted values of the ten groups of the training results were selected for comparison. After the data analysis, it can be concluded that the deviation of the final neural network training and verification results was caused by the interference of human factors in the data collection process or unstable factors in the algorithm operation process. Subsequent work will focus on strengthening the analysis and correction of data disturbance factors and improving the accuracy of the fastener segmentation and centerline extraction algorithms to better fit the neural network model of the fastener tightness detection and optimize the parameter adjustment.

Figure 18

Comparison of results

4.4 Compare with Other Algorithms

For the point cloud data of the fastener, we simulated and reproduced other tightness-detection methods for experimental comparison. The difference in the Euclidean distance between the processing results and original data was used to represent the final effect of the algorithm, as illustrated in Table 4.

Table 4 Comparison of other registration methods

Method 1 is the centerline projection point distance method and Method 2 is the original point cloud ICP registration method. The RMSE error generated after registration is obtained by the direct registration operation of the original point cloud, which describes the degree of deviation between the target point cloud and original point cloud, which is when the fastener generates geometric deformation under the action of the buckle pressure. The experimental results were compared with those of the original registration in a fully loose state. Method 3 was the centerline clearance measurement method. The extracted centerline was further processed to obtain a point set on the symmetric plane of the middle line of the elastic rod. The vertical distance between the point set and insulating block was calculated as the basis for determining the unseam value of the elastic rod. Method 4 was the ICP registration method for the centerline. The extracted centerlines were registered together, the center-line template in the fully loose state was created for registration with the other states, and the experimental results were compared. The experimental results showed that the proposed CPD algorithm can accurately identify and predict the tightness of fasteners.

5 Conclusions

Detecting the tightness of fasteners is an important and challenging task in high-speed railways. Based on high-precision plane-structured light technology, this study proposed a fastener tightness detection method based on the centerline projection distance feature and neural network regression. The process of algorithm implementation was as follows.

• (1) By obtaining the three-dimensional point cloud of fasteners, accurate registration and segmentation operations were performed based on the ICP algorithm.

• (2) The PCA algorithm was used to estimate the normal vector of the elastic rod surface to extract the centerline of the fastener in the plane formed by each point and its domain point, project the points on the centerline to the upper surface of the fastener bolt and calculate the projection distance.

• (3) A linear neural network was used for regression to establish the mapping relationship between the projection distance and geometric tightness state of the fastener. The obtained network model with good performance was used to predict the tightness of unknown fasteners.

In the experiment, an angle-measuring instrument was used to manually calibrate the tightness of 50 groups of fasteners in different states, and the data were collected and marked as the calibration benchmark. From the RMSE value compared with the final verification set and the real data, it can be observed that the deviation between the method and manual calibration result is less than 0.2196 mm, which proves that the method can effectively predict the tightness state of unknown fasteners.

Additionally, it was compared with other algorithms to verify its performance.

Simultaneously, by analyzing the influence of different p and k parameters, it was deduced that the proposed method can replace manual maintenance in the future automatic field of fastener maintenance and improve the operation and maintenance efficiency of railway fastener maintenance.

The subsequent algorithm was deployed on the railway maintenance robot, and the designed torque wrench was applied to precisely tighten or loosen the bolts of the fastener to meet the safety operation standards of the rail transit.