《1. Introduction》

1. Introduction

The assembly of large components plays an extremely important role in the manufacturing of major equipment, such as aerospace vehicles, ships, and automobiles [1], and the workload often exceeds 50% of the entire manufacturing process [2,3]. Moreover, the extremely complex assembly relationships among aircraft components and large sizes of components increase the difficulty of accurate assembly control for key components, and periodic accuracy checks of the key components are necessary. Due to the large number of panel structures that are employed during aircraft assembly and directly affect the assembly accuracy of the whole aircraft, the detection of key points (KPTs) and key profiles (KPFs) for assembly components is extremely important.

In the process of modern manufacturing [4,5], laser trackers, industrial photogrammetry, proximity sensors, and other highprecision measurement equipment are used alone or in combination to acquire assembly process data and establish digital systems to guide the assembly process. Due to the different measurement abilities of high-precision equipment, the application of different devices varies. A laser tracker [6–8] with high precision is suitable for the periodic static inspection of the KPTs and KPFs of assembly tooling and components. However, due to the use of optical methods for point-by-point measurement, the measurement capability in a compact space or online detection is limited to some extent. Industrial photogrammetry [9–11], with the advantages of high efficiency and high precision, is widely used in many fields, such as processing, manufacturing, testing, and online health monitoring. However, due to the complexity of the measurement environment, the limitation of the field of view limits the measurement of KPTs and KPFs in compact or occluded areas. Proximity sensors [12–14] are widely used in high-precision position detection in various fields due to their advantages, such as a compact volume, low weight, high precision, and fast response. However, a single proximity sensor, which obtains one-dimensional (1D) information, cannot meet three-dimensional (3D) measurement requirements. Thus, proximity sensors are often used in combination with a high-precision manipulator, such as a coordinate measurement machine (CMM), for geometric information detection.

In the assembly process of an aircraft, high precision, high efficiency, and low loss are required in measurements of KPTs and KPFs. Thus, the motivation of this work is to propose a hybrid 3D coordinate measurement method based on the combination of proximity sensors and industrial photogrammetry and develop a portable noncontact profile scanning system to simultaneously consider the precision, efficiency, convenience, and cost performance of the online measurement of KPTs and KPFs for aircraft components in the assembly process.

The remainder of this paper is organized as follows. Section 2 reviews the previous work related to this paper. Section 3 describes the hybrid 3D coordinate measurement model. Section 4 details the profile-driven 3D automated scanning strategy. In Section 5, measurement experiments are presented. Finally, the paper is concluded in Section 6.

《2. Related works》

2. Related works

Multiequipment joint measurement is an effective way to meet the multiple requirements (such as high precision and efficiency) of in situ profile inspection. Scholars in various fields have performed many studies of profile scanning methods, including establishing 3D coordinate measurement models and methods based on multiple equipment types and automated scanning strategies.

《2.1. Establishment of the 3D coordinate measurement model》

2.1. Establishment of the 3D coordinate measurement model

Research on the establishment of 3D coordinate measurement models has focused on vision-based, proximity sensor-based, laser tracker-based, and robot-based methods. The intrinsic measurement characteristics of the equipment and spatial geometric constraints are frequently adopted to establish 3D measurement models.

Lembono et al. [15] proposed a 3D measurement system and a calibration method based on a robot and a laser rangefinder (LRF). The experimental results indicated that the method and system reduced the mean planar error from 0.53 mm to approximately 0.23 mm; additionally, this approach was inexpensive and convenient. To measure a six-degree-of-freedom (6-DOF) pose from a long distance at the submillimeter level of accuracy, Kim et al. [16] proposed a novel noncontact measurement system based on a camera and laser sensors. The performance at 30 m was validated, with accuracies of 4 mm and 0.5° and precisions of 0.7 mm and 0.3°. Wu and Ren [17] proposed a hand–eye calibration method using 3D position data to obtain the 3D kinematic base frame of a robot by optimizing the coordinate conversion error, and the unknown parameters of the 3D base frame were determined. An et al. [18] developed a new omnidirectional 3D laser ranging system that consisted of an LRF, a camera, and a rotating platform. The system was calibrated with an average error of 0.9875 pixels, and tests were conducted in indoor and outdoor scenes; the results indicated that the laser ranging system achieved good performance. Combined with low-cost 1D laser sensors and a vision camera, Kim et al. [19] developed a three-beam detector for sensing 6-DOF motion at a remote distance of up to 30 m. Error analysis showed that the accuracy was within 3 mm for translation and 1° for rotation.

《2.2. 3D scanning method and strategy》

2.2. 3D scanning method and strategy

Compared with traditional large-scale profile inspection in a point-by-point manner in the aircraft assembly process, automated scanning and synchronous multipoint measurement can be applied with high efficiency to accurately acquire 3D point data. Most current studies of scanning methods and strategies focus on the combination of CMMs and tips, robot and proximity sensors, robots and cameras, and cameras and lasers.

Zhang and Tang [20] presented a novel method based on a CMM and a tip for automatically performing five-axis inspections of arbitrary free-form surfaces. The proposed method considered both the surface geometrical information and the kinematic capacity to maximize the inspection efficiency and reduce the total inspection time by a factor of as much as seven. Huang et al. [21] presented an automatic robotic ultrasound system for 3D imaging, and adopted a depth camera and a robot to capture the point clouds of surfaces. The scan range and scan path were automatically determined according to the 3D contours of the surface to be scanned. The experimental results indicated that the proposed system yielded good performance in 3D reconstruction. Macleod et al. [22] proposed an automated ultrasonic remote sensing system for the thickness mapping of large-scale areas. A scan test for a 2 m² carbon steel sample with a 10 mm nominal thickness was performed in a timeframe of 15 min with a minimum thickness mapping error of 0.21 mm, which directly led to increased task efficiency and reduced inspection time for large structural assets. Palomer et al. [23] presented an underwater laser scanner based on a laser line projector and a camera, and the calibration error was under 1 mm in the measurement range of 0.5–1.2 m. Tests were performed, and the average errors were 0.44 and 0.98 mm with standard deviations of 0.35 and 0.72 mm in air and under water, respectively.

《2.3. Discussion》

2.3. Discussion

Most research currently focuses on two kinds of profile scanning methods: contact scanning and noncontact scanning. Contact scanning methods, such as those that use laser trackers and CMMs, usually employ tips to sweep the profiles of parts and obtain the 3D information for profiles. However, for the parts with high surface quality requirements, the tips may scratch the surface, and for profiles with notable curvatures, the tips may excessively touch or be separated from the measured profile during the scanning process, which often requires manual intervention. Noncontact methods, such as industrial photogrammetry and laser scanning, mainly involve profile scanning through optical means. However, for some highly reflective surfaces, overexposure may easily occur in images or light strips, thus reducing the scanning accuracy.

Therefore, taking advantage of both contact and noncontact methods, this paper proposes a hybrid measurement method based on proximity sensors as tips and industrial photogrammetry to achieve noncontact measurement while avoiding the accuracy degradation caused by high-reflective interference. Moreover, a profile-driven scanning strategy is proposed to avoid tips scratching the measured surface and exceeding the measurement range when measuring profiles with compound curvatures, such as aircraft tailplanes, wings, and envelopes; therefore, the proposed method facilitates an efficient automatic scanning process.

《3. Hybrid 3D coordinate measurement model》

3. Hybrid 3D coordinate measurement model

Industry photogrammetry, with the advantages of high efficiency and high precision, is capable of rapidly detecting largescale geometry. Additionally, proximity sensors can provide good performance in distance detection, even in compact spaces, due to their compact volume, fast response, and high sensitivity. Therefore, for the detection of KPTs and KPFs in both large-scale and compact spaces during the aircraft assembly process, the combination of industrial photogrammetry and proximity sensors is necessary. By combining the advantages of photogrammetry for largescale geometric measurement and proximity sensors in compact spaces for high-precision distance measurement, KPTs and KPFs distributed among aircraft components can be measured at various scales.

The principle of the measurement model based on photogrammetry and proximity sensors is shown in Fig. 1.

《Fig. 1》

Fig. 1. Principle of the 3D coordinate measurement model. WCS: world coordinate system; LCS: local coordinate system;  : the probe base point (PBP) of the proximity sensor in the LCS;  : the unit displacement vector (UDV) of the proximity sensor in the LCS;  : the measured point of the ith measurement in the LCS;  : the measured point of the ith measurement in the WCS; 

The world coordinate system (WCS) is first established based on the camera. Then, the visual reference points (VRPs) are measured by photogrammetry in the WCS. Because the VRPs are rigidly connected to the proximity sensor, the structural relationships between the proximity sensor and the VRPs can be constant. By obtaining the 3D coordinates of the VRPs and the measured distances to the proximity sensor, the 3D coordinates of the measured points can be determined. The steps in establishing the 3D measurement model are described in detail below.

During the ith measurement, as shown in Fig. 1, the 3D coordinates of the VRPs in the WCS and the measured distance of each proximity sensor are obtained. Thus, the relationship between the measurement information and the measurement point in the WCS can be described as:

where h is the functional relationship between  and  is the measured point of the ith measurement in the WCS;  is the lth visual reference point of the ith measurement in the WCS;  is the measured distance of the proximity sensor of the ith measurement; i is the number of the measurements, and is the number of the visual reference points.

Then,  is established as the origin of the local coordinate system (LCS), and the plane composed of   is the X–Y plane of the LCS. In addition, suppose that  is on the positive X-axis of the LCS and that  is in the direction of the positive Y-axis; thus, the LCS is uniquely established. The coordinates of the VRPs in the LCS () can be expressed as:

where θ is the angle between  and .

Thus, the control vectors   for the measurement model in the LCS can be expressed as: : the lth visual reference point of the ith measurement in the WCS; : the measured distance of the proximity sensor of the ith measurement; i: the number of the measurements; l: the number of the visual reference points.

where n is the number of the control vectors.

In the LCS, the 3D coordinates of a measured point of the ith measurement ( ) relate to the probe base point (PBP) and the unit displacement vector (UDV) of the proximity sensor.

where is the PBP of the proximity sensor in the LCS and is the UDV of the proximity sensor in the LCS.

 and  can be expressed as a function of the control vectors .

where   are the parameters of the 3D measurement model.

By substituting Eqs. (9) and (10) into Eq. (8), we can obtain

Through the coordinate transformation of the VRPs in the WCS and LCS, the rotation matrix R_i and the transformation matrix T_i  can be calculated, and the following equation is satisfied.

Thus, the expression of a measured point  can be derived.

To calculate the model parameters   , a calibration plane with VRPs is adopted. VRPs on the calibration plane are measured to calculate the calibration plane equation  in the WCS. Then, the constraints between the measured points  and the calibration plane equation can be obtained.

At this point, the following optimization problem is constructed.

Finally, the optimal solution  is obtained. By substituting the optimal solution into Eq. (13), the measurement model is established.

《4. Profile-driven 3D automated scanning strategy》

4. Profile-driven 3D automated scanning strategy

In the process of aircraft assembly condition monitoring, the 3D coordinate measurement model established above can be applied in two cases: one is the case in which the proximity sensors are fixed at specific positions for 3D coordinate measurement, and the other is the case in which the proximity sensors are hand-held for profile scanning. However, in the second case, to improve the efficiency and accuracy of scanning, additional proximity sensors can be employed in single measurements to acquire more information than that provided by a single sensor. Moreover, to obtain rapid, stable, and automated measurements, a set of scanning strategies needs to be designed. A scanning strategy consists of three parts: the profile reconstruction method for a single measurement, the profile updating method during scanning, and the profile-driven automated scanning strategy.

《4.1. Profile reconstruction method for a single measurement》

4.1. Profile reconstruction method for a single measurement

In a single measurement, multiple proximity sensors can be used to acquire information and improve the efficiency and accuracy of the scanning process.

As shown in Fig. 2, with the measured points  the measured profile  can be easily reconstructed as:

where  is the reconstruction profile based on the ith single measurement;  is the jth measured point of the ith measurement in the WCS;  is the functional relationship between  and .

For the convenience of specifying the scanning position for the automated scanning process, the center and the vector of the single measurement are then defined.

where is the scanning center and m is the number of measurement points in the single measurement.

where  is the scanning vector at point  and  is the functional relationship between  and .

Thus, the profile  ,  the scanning center ,  and the scanning vector  are calculated for updating in the (i + 1)th measurement step.

《Fig. 2》

Fig. 2. Profile reconstruction principle based on a single measurement. : the reconstruction profile based on the ith single measurement. : the jth measured point of the ith measurement in the WCS; j: the number of the measurement points in a single measurement.

《4.2. Profile updating method based on multiple measurements》

4.2. Profile updating method based on multiple measurements

In the single measurement approach, the measured profile is reconstructed as described above. However, regardless of the reconstruction range or reconstruction accuracy, the results of the single measurement cannot meet certain requirements. Thus, the reconstructed profile can be updated based on multiple measurements.

As shown in Fig. 3, the measured profile is reconstructed based on the reconstruction method for a single measurement. For the first measurement, the profile reconstructed with the profile updating method, defined as , is the same as the profile :

where F₁ is the functional relationship between  and .

In the second measurement, the measured profile is reconstructed based on the reconstruction method for a single measurement, meanwhile, the profile is updated based on the first two measurements and reconstructed as :

where F₂ is the functional relationship between  , and .

Thus, for the ith measurement, the profile can be reconstructed as  based on the multiple measurements from  to :

where F_i is the functional relationship between .

In profile updating, the scanning vector  of the profile  at the scanning center  is also updated: 

where G_i is the functional relationship between and .

《Fig. 3》

Fig. 3. Profile reconstruction principle based on multiple measurements. : the reconstruction profile based on multiple measurements from the to .

《4.3. Profile-driven automated scanning strategy》

4.3. Profile-driven automated scanning strategy

To achieve automatic scanning, a planning strategy for the scanning path is essential, and it should consider the trend of the scan curve, definition of the curve break point and definition of the scanning end point.

As shown in Fig. 4, a gyratory path is adopted to scan from outside to inside. According to Eq. (18), suppose that is the Kth scanning center on the Jth side in the Ith scanning loop in the scanning process.

《Fig. 4》

Fig. 4. Automated scanning strategy. B: the number of boundary measuring lines; L, M, N, H: the numbers of scanning centers on each boundary measuring line.

First, the boundary measurement line is scanned manually. The number of the boundary measuring lines is B, and the number of scanning centers on each line is recorded as L, M,N, ..., H. Thus, the 3D coordinates of the scanning centers on the boundary measuring lines () are achieved.

Then, the profile-driven automated scanning process starts from , which is predicted as :

Thus, the trend of the scan curve can be expressed as the general form of Eq. (24).

In addition, the scanning vector can be expressed as:

where  is the reconstructed profile based on multiple measurements prior to . Through the real-time updating of the measured profile, the scanning center and vector of the next measurement point can be predicted, thus guaranteeing the validity of the measurement point without scratching the surface or exceeding the measurement range.

The break point between and the Jth side and the next side will occur at the position of the th scanning center. A side with fewer than three scanning centers will not be scanned in the current loop or the next loop.

Finally, the scanning process ends when there are fewer than three sides in one loop. The pseudocode is shown in Fig. 5.

《Fig. 5》

Fig. 5. Pseudocode for automated scanning.

《5. Experiments》

5. Experiments

In this study, coordinate scanning experiments are performed for a tailplane panel based on the proposed portable noncontact scanning system. The scanning system is first established.

(1) The proximity sensors (KD2306-4SB, Kaman, USA) are adopted to acquire the distances from the sensors to the tailplane panel; the sensors have a high resolution of 0.4 ，μm, high response of 50 kHz, and compact volume of Φ22.2 mm × 6.35 mm. Three sensors are employed for simultaneous measurement, thus ensuring efficiency of the scanning process and the compactness of the structure volume. Moreover, a KD2306-4SB adopts the eddy current ranging principle, which is not affected by profile reflection.

(2) A scan head is designed to hold the proximity sensors, and it can be installed on a robot or held manually. The VRPs are stuck to the scan head; thus, the position relationship between the VRPs and the sensors is constant. Moreover, low-reflectivity material should be employed onto the scan head to ensure that the VRPs obtain high-precision measurements.

(3) High-accuracy cameras (MPS/M20, Chenway, China) are adopted to measure the 3D coordinates of the VRPs with an accuracy of 8 μm+8 μm·m^–1 and a high response of 20 Hz. Additionally, optical tool points (OTPs) are designed to establish the WCS.

(4) An industrial robot (KR10 R1420, KUKA, Germany) is also employed to hold and move the scan head and facilitate the automated scanning process. The robot moving speed along the six axes exceeds 200 degrees per second, and the working radius can reach 1420 mm.

The overall layout of the system is shown in Fig. 6.

《Fig. 6》

Fig. 6. Layout of the portable noncontact scanning system.

《5.1. Establishment of the 3D coordinate measurement model》

5.1. Establishment of the 3D coordinate measurement model

The scan head is first assembled with three proximity sensors and three VRPs. To establish the 3D coordinate measurement model, the structural relationships between the proximity sensors and the VRPs should be obtained following calibration. Thus, a calibration plane (processed with a flatness error of less than 3 μm) is employed. The calibration process is shown in Fig. 7.

《Fig. 7》

Fig. 7. Establishment of the 3D model and the calibration of the model parameters.

The VRPs on the calibration plane are first measured by the cameras to determine the plane function. Then, the scan head is manually held to measure the calibration plane; moreover, the distances from the proximity sensors to the calibration plane and the 3D coordinates of the VRPs on the scan head can be acquired. Through multiple measurements, the data sets of the structural relationship between VRPs and the sensors will be obtained. Finally, the model parameters  in Eq. (13) can be determined. The calibration results are shown in Table 1.

《Table 1 》

Table 1 Calibration results for the model parameters.

By substituting the values of  and  into Eq. (11), the 3D coordinates of the measurement points in the LCS can be calculated. Then, according to the measured 3D coordinates of  in the WCS and  in the LCS, the rotation matrix and the transformation matrix can be obtained through Eq. (12). Thus, the 3D coordinate measurement model can be finally established in the WCS through Eq. (13).

《5.2. Profile reconstruction for a tailplane panel》

5.2. Profile reconstruction for a tailplane panel

Modern methods of profile reconstruction are mainly based on point-by-point measurements with a laser tracker and photogrammetry with cameras. Laser trackers provide high accuracy, and photogrammetry is highly efficient. Our goal is to propose a scanning method and system that considers not only accuracy and efficiency but also cost performance and automation.

The strategy of automated scanning is detailed in Section 3 and Section 4, and the composition of the proposed system is shown in Fig. 6. When the automated scanning process begins, as many points as possible are scanned to increase the profile reconstruction accuracy. Notably, the scan head can measure three points simultaneously within a 50 mm diameter, and the final scan spacing is designed to be 100 mm in consideration of both the accuracy and efficiency. The scanning results for a tailplane panel (approximately 1760 mm × 460 mm) are shown in Fig. 8.

《Fig. 8》

Fig. 8. Profile reconstruction for a tailplane panel based on the proposed method and automated scanning system.

《5.3. Accuracy and practicality analysis》

5.3. Accuracy and practicality analysis

Comparison tests were conducted for the laser tracker and photogrammetry methods. Corresponding to the production site, a target ball was combined with the laser tracker to perform manual point-by-point measurements of the profile; moreover, measurements points were added to the profile for photogrammetric measurements. Finally, OTPs were employed to transform the measurement results for the laser tracker and photogrammetric method into a format suitable for the WCS. The measurement system and measurement results for a tailplane panel are shown in Fig. 9.

Figs. 8 and 9 show that the scan spacing can be very small for the laser tracker and the method proposed in this paper. However, the scan spacing will be relatively bigger for the photogrammetric method, because each measurement point for photogrammetric method needs a more paste space; as a result, the reconstruction accuracy will be slightly limited.

《Fig. 9》

Fig. 9. Profile reconstruction for a tailplane panel based on (a) a laser tracker and (b) photogrammetry.

Then, accuracy tests based on the laser tracker are conducted. The errors for the profile reconstructed with the method proposed in this paper and the photogrammetric method are shown in Fig. 10.

《Fig. 10》

Fig. 10. Accuracy tests. (a) Errors for the profile reconstructed based on the proposed method and (b) errors for the profile reconstructed based on the photogrammetric method.

Comparison tests were conducted in terms of accuracy, efficiency and automation level. Thus, the number of steps in the scanning task, the quantity of measurements points, the duration of the scanning process and the accuracy of the reconstructed profile were recorded. Since the laser tracker was determined to be the most accurate measurement instrument in a wide range of spaces, it was regarded as the benchmark for accuracy in this paper. A Leica AT960 with a precision of 15 μm+6 μm·m^–1 was used, and in the measurement range of 4 m, the accuracy reached 0.039 mm. A comparison of indexes is shown in Table 2.

Table 2 shows that the measurement based on the laser tracker is the most time-consuming but provides the highest accuracy. The laser tracker is the benchmark for accuracy, and as many measurement points as possible are measured to obtain high accuracy for the reconstructed surface. Notably, a total of 456 points are manually measured in approximately 985 s, and the average measurement time for each point is 2.16 s. Compared with the measurement based on the laser tracker, a total of 40 points are measured in 341 s (8.53 s for each point) by photogrammetry, and 349 points are scanned in 208 s (0.60 s for each point) with the method proposed in the paper. In terms of accuracy, it is easier to measure more points with the proposed method, which can achieve a high accuracy by reconstructing the profile with abundant measurement information; additionally, errors are within 0.003–0.378 mm, and the mean error is 0.121 mm. Thus, in general, the proposed method and system exhibit good performance based on both efficiency and accuracy; therefore, this method is convenient and can rapidly perform automated noncontact detection for large-scale surfaces in the process of aircraft inspection.

《Table 2》

Table 2 Accuracy and practicability comparison.

《6. Conclusions》

6. Conclusions

This paper is motivated by the requirements of high accuracy, high efficiency, automation, and low cost for large-scale profile scanning. Although there are many profile scanning methods and systems, they are mainly based on laser trackers, coordinate measurement devices and photogrammetry and generally do not meet the above requirements. Compared with the existing measurement methods, the proposed automated hybrid scanning system and method have a different inspection mode, which tremendously increases the inspection efficiency by simultaneously considering multipoint measurements and profile-driven automated scanning results in real-time path planning. The automated noncontact profile scanning system and method we proposed in the paper are novel in that by combining the advantages of multiple devices, solving the hybrid 3D measurement model and determining the best scanning route according to the curved surface, they provide considerable advantages in improving accuracy and efficiency. Thus, the method is particularly suitable for industrial parts that are large and relatively smooth, such as aircraft or automobile panels, envelopes and blades. Finally, the proposed method is tested in an in situ profile scanning process for a tailplane panel (1760 mm × 460 mm). Benefitting from convenient operation and automation, the scanning process can be completed in 208 s. Finally, an accuracy analysis was conducted, and the errors of the automated 3D profile scanning results were less than 0.378 mm, with an average of 0.121 mm; therefore, the proposed method provides strong data support for large aircraft inspection after assembly.

《Acknowledgments》

Acknowledgments

This work was supported in part by project of National Key R&D Program of China (2018YFA0703304), National Natural Science Foundation of China (U1808217), Youth Program of National Natural Science Foundation of China (51905077), and Liaoning Revitalization Talents Program (XLYC1807086).

《Compliance with ethics guidelines》

Compliance with ethics guidelines

Bing Liang, Wei Liu, Kun Liu, Mengde Zhou, Yang Zhang, and Zhenyuan Jia declare that they have no conflict of interest or financial conflicts to disclose.