By imitating the self-sensing and self-diagnosis abilities of humans, structural health monitoring (SHM) deals with the real-time sensing, identification, and assessment of the safety and performance evolution of structures. In general, an SHM system consists of various sensors, data-acquisition devices, a data transmission system, a database for data management, data analysis and modeling, condition assessment and performance prediction, alarm devices, a visualization user interface, and a software and operating system. SHM systems have been widely implemented in the fields of aerospace, civil engineering, and mechanical engineering [1–9].

Data have recently become crucial in society, as their availability and effectiveness create value. Data are a core part of the field of SHM; the field dealing with data is termed data science and engineering [10].

Data science and engineering includes algorithms and applications of data acquisition, abnormal data diagnosis and reconstruction, data transmission and lost-data recovery, data management, data mining, and data modeling.

Until a few decades ago, data were commonly sampled following the Shannon–Nyquist sampling theorem. However, this sampling strategy leads to mass data [11]. In 2005 and 2006, a compressive sampling (CS) technique was proposed [12,13], which breaks through the limitations of the traditional sampling theorem. If the data signal is sparse in a certain domain, CS randomly collects much less data, rather than sampling the data with two times the highest frequency of the signal. This signal-sampling technique is able to dramatically reduce data volume. Bao et al. [14] have investigated the application of CS in data acquisition in SHM, with the aim of reducing data volume. Peckens and Lynch [15] have proposed a bio-inspired CS technique to acquire data for SHM. O’Connor et al. [16] have implemented CS theory into a wireless sensor node in order to perform random data sampling for power consumption reduction. In CS theory, data sparsity is very important; however, many signals are only approximately sparse in reality. Huang et al. [17] have proposed a robust Bayesian compressive sensing method to reconstruct approximately sparse signals in SHM. To further increase the data-reconstruction accuracy of CS in SHM, Bao et al. [18] have proposed a group sparse optimization algorithm that considers the group sparsity of the structure vibration signal (the measured vibration data at different locations of a structure has a very similar sparse structure in the frequency domain) for CS data reconstruction; this algorithm will be discussed in Section 2. Using the idea of random data sampling and the data reconstruction of CS theory, Bao et al. [19] and Zou et al. [20] have proposed CS-based data-loss recovery methods for wireless sensors and sensor networks. The CS method has also been utilized for system identification tasks, such as structural modal identification, structural damage identification, and load identification [21–30]. For structural modal identification, modal parameters are directly identified from the compressed measurements [21,22]; however, for structural damage identification and load identification, the spatial sparsity of structural damage and load distributions is utilized to solve optimization problems involved in the identification [23–30].

Because SHM systems are operated in harsh and noisy environments, abnormal data are inevitable. Monitored data with outliers, trends, saturation, and missing data have frequently been observed; this presents a barrier for the automatic warning of SHM systems because it becomes difficult to distinguish which abnormal data are caused by out-of-order SHM systems or by damage to instrumented structures. Outlier detection approaches have been extensively investigated [31–35], whereas the research on abnormal data detection is still insufficient [36–38]. In actual application, the large variation in extracted features from massive SHM data causes conventional anomaly detection techniques to perform poorly. Even with expert intervention, the parameter tuning associated with multiple data preprocessing methods is still a challenge, and makes the procedure expensive and inefficient. In addition, anomaly detection techniques that are focused on a single type of anomaly frequently mis-detect other types of anomaly. More intelligent approaches are needed to deal with the massive data collected by multiple sensors from actual SHM systems. Deep learning (DL)-based approaches may be potential alternatives to diagnose various kinds of abnormal data through learning from big data containing abnormal data. Imputation algorithms for addressing missing data or anomaly data have also been extensively investigated [39–45].

Damage detection is a classic issue in SHM. A number of modal-based damage-detection approaches have been proposed [46]. However, practical experience has indicated that modal-based damage-detection approaches are insensitive to minor local damage and are readily influenced by temperature. Computer vision (CV) is a critical technology in artificial intelligence (AI); it uses computers to process visual information and enables a computer to see and interpret the real world as a human does. Specific problems in CV include measurement, detection, and recognition of objects, features, or actions, and the use of image or video data in computational processes. Damage or change on the surface of a structure can be automatically identified by means of a CV technique (e.g., when taking pictures of a structure, image-processing algorithms or DL algorithms can be employed to automatically identify any damage or change in the picture). Many recent studies on structural damage detection have been performed using the CV technique, in which the pictures can be obtained by unmanned aerial vehicles, robots, or wearable devices [47].

Structural responses vary with the external loads and structure models; this makes it difficult to distinguish whether the variation in structural response is caused by the loads or by the structural parameters themselves, because loads are very difficult to monitor exactly. Complex correlations between responses can be obtained through machine learning (ML) algorithms or DL algorithms; as the correlation of responses is free from loads, it can be used as an indicator of structural performance. A condition assessment can be further performed based on variation in the indicators [48,49].

The structure of this paper is arranged as follows: Section 2 introduces the CS-based data-acquisition algorithm; Section 3 discusses the DL-based anomaly data diagnosis approach; and Section 4 presents the CV-based crack identification algorithm. Condition assessment approaches for bridges using ML algorithms are proposed, followed by the conclusions.

The Shannon–Nyquist sampling theorem is a criterion that must be obeyed in traditional data acquisition; it requires a signal to be sampled at least two times at the highest frequency in the signal. Thus, the data recorded by an SHM system will be huge if this theorem is used. Compressive sensing, also known as CS, which was first proposed by Donoho [12] and Candès [13], provides a new sampling theory for signals with sparse features in a certain domain. In CS theory, the signal is randomly collected. The sample size is much smaller than the sample of recorded signals obtained when following the Shannon–Nyquist sampling theorem. The original signal can then be reconstructed exactly with sparse optimization algorithms.

In CS theory, a signal x∈Rn can be sensed using a linear measurement as follows:

where Φ is a measurement matrix or sampling operator in an m × n matrix and e is the measurement noise.

As Φ is an m × n matrix with m << n, the problem of recovering the signal x is ill-posed. However, in CS theory, if signal x is sparse (i.e., the signal has a sparse representation in some basis Ψ, where x  = αΨ) and the Φ matrix satisfies the restrictive isometry property, then the coefficient α can be reconstructed by the l₁ (norm of α∼1) optimization problem:

where ε is the bound on the level of the measurement error, e2≤ε; the matrix Θ is Θ = ΦΨ; α̂ is the optimal solution of the coefficient;α∼ represents all possible solutions; and the definition of l₁ norm of α∼1 is α∼1=∑i=1na∼i, where a∼i is the element of α∼.

The sparsity of the signal is the basic premise in CS theory. Fortunately, most of the vibration signals of civil infrastructure have this characteristic (i.e., the vibration is dominated by a few modes only). Therefore, the vibration data of most structures are usually or almost always sparse in the frequency domain or time-frequency domain. Bao et al. [14] found the sparsity of structural vibration signal and first used the CS for acceleration data acquisition of a bridge SHM system.

Furthermore, the vibration data measured by sensors placed at different locations on a structure have almost the same sparse structure in the frequency domain. Based on the group sparsity of the structural vibration data, a group sparse optimization algorithm based on CS for wireless sensors has been proposed [18]. Supposing that there are K sensors implemented in the structure, the signal matrix U can be expressed as follows, considering the measurement noise:

where ε∈RM×K is the Gaussian noise matrix, U is the collected data matrix with M × K, Ψ is a Fourier matrix, and X is a Fourier coefficients matrix. The Fourier coefficients matrix X can then be recovered by solving the following optimization problem:

where PΩ:RM×K→RM×K is the zero padding operator and μ is the penalty weight. Once the optimal solution X_rec has been obtained, the recovered signal matrix can be obtained by the following:

The augmented Lagrange multiplier algorithm is proposed to solve the sparse optimization problem given in Eq. (4) [18]. Fig. 1 shows an example of data-reconstruction results from 10% and 20% measured samples from an actual long-span bridge. Using the reconstructed data, the first two mode shapes can be well identified, as shown in Fig. 2, from the reconstruction signal with 10% samples.

Practical experience with a large number of SHM systems has shown that SHM data always contain multiple types of anomalies caused by sensor faults, system malfunctions, and environmental effects. The most common data anomalies in SHM are missing data, minors, outliers, squares, trends, and drifts. These anomalies will seriously shield the real information contained in the data, resulting in a risk of warning misjudgment. Therefore, data preprocessing or data cleaning is important for SHM.

Inspired by actual manual inspection, Bao et al. [50] have proposed a CV- and DL-based anomaly detection method that first visualizes and converts a raw time series into image data, and then trains a deep neural network (DNN) for anomaly classification. The framework of the proposed approach is shown in Fig. 3. This method includes two main steps: ① data conversion by data visualization, in which the time series signals are piecewise represented in the image vector space; and ② DNN training via techniques known as stacked auto encoders and greedy layer-wise training. The trained DNN can then be used to detect potential anomalies in large amounts of SHM data.

Acceleration data from a long-span cable-stayed bridge was employed to illustrate the performance of the proposed method. Fig. 4 shows the data anomaly distribution identification results over a year. The results in Fig. 4 indicate that the multi-pattern anomalies of the data can be automatically detected with high accuracy, resulting in a global accuracy of 87.0%.

CV carries the advantages of less incorporation of expensive professional instruments and sensors, and independence from subjective human experience. By virtue of DL, which automatically trains end-to-end models and generates high-level features of input images, DL-based CV can overcome the limitations of conventional CV, such as the requirement for predesigned filter-based detectors, assumptions about crack geometry, and robustness for complicated real-world images.

Restricted Boltzmann machines (RBMs) and deep convolutional neural networks (CNNs) are two representative DL architectures that have been investigated for image-based crack identification tasks. Xu et al. [51] have established a crack identification framework using RBM for cracks on steel structure surfaces. The proposed RBM model is stacked with an input layer, three hidden layers, and an output layer, as shown in Fig. 5.

RBMs are generative learning modules that can be stacked to form deep networks. An RBM is energy function based and is generally defined as follows:

where x is the input variable of interest, P(x) is the probability distribution with regard to the energy function E(x), and Z is the partition function by traversing all possible inputs. Under some circumstances, the input x can be categorized into the visible and hidden parts (denoted by v and h, respectively). In a Boltzmann machine, the energy function is defined with a second-order polynomial as follows:

where b_i and c_j are the offsets and biases associated with the visible vector v and the hidden vector h, respectively, and W_ij, U_ij, and V_ij are the corresponding weights related to the unit pairs (v_i, h_j), (v_i, v_j), and (h_i, h_j). An RBM is defined with its binary units—that is, vi,hj∈{0,1}. In addition, h_j is independent of hi(i≠j) conditional on v and v_i is independent of vj(i≠j) conditional on h. Therefore, U_ij = 0 and V_ij = 0. The energy function of RBM can then be written as follows:

The original images, including complex background information, were taken by a consumer-grade camera (Nikon D7000, resolution: 3264 × 4928) inside the steel box girder of the bridge. Some of the cracks in the original images are very small, with a width of only a few pixels. Complex disturbance information is also present, such as the edges of steel plates and U-ribs, handwriting by inspectors, electric wires, and corrosion areas. The greyscale 3264 × 4928 pixel images were reshaped into an image with the dimensions of 3264 × 3264 pixels, and then cut into sub-images with the dimensions of 24 × 24 pixels. In total, a complete sub-image set of 240 448 elements was built up. The output label [1 0]^T was assigned to crack elements and [0 1]^T was assigned to background elements. The input to the deep network was a 576 × 1 vector reshaped from the corresponding sub-image of 24 × 24 pixels and normalized by 255 in the grayscale. A contrastive divergence learning algorithm [52] was applied to all the RBM layers in sequence from bottom to top in order to update the weights, biases, and offsets of the stacked network based on the sigmoid function, as follows:

where α and η are the momentum and learning rate hyper-parameters. Fig. 6(a) indicates that the reconstruction error ratio decreased to 4.8% after 1000 iterations. Fig. 6(b) shows that the crack elements could generally be differentiated from the disturbance edges, despite the existence of complicated background information.

Inspired by the richer CNN [53], Xu et al. [54] further proposed a modified fusion CNN (FCNN) to handle the multi-level convolutional features for crack identification, as shown in Fig. 7. Unlike the conventional input-hidden-output structure of chain-like CNN, bypass stages are added in FCNN in order to handle the multi-level features. Softmax loss function with a regularization term to penalize large weights was adopted for classification as follows:

where 1{y(i)=j} returns 1 if the prediction is correct; otherwise, it returns 0. λ is the weight decay parameter and m represents the mini-batch size. W_j and b_j denote the weights and biases in the previous layer, respectively. C represents the total number of categories, and c is the index number ranging from 1 to C; x(i) represents the ith input of the classification layer; WjT and bj represent the weights and biases acting on the ith input x(i); WcT and bc represent the indexes of weights and biases in the internal summation recycle in case of confusion. The recognition error is element-based and represents the proportion of the predictions missing the ground-truth labels for the input sub-images (i.e., represents whether the prediction of an input element is correct or not). If the predicted label of an input sub-image is different from the ground-truth target label, the number of incorrect predictions is increased by 1 and the recognition error is changed accordingly. Fig. 7(b) shows the training and validation errors during the training process with minimum values of 3.62% and 4.06%, respectively.

Fig. 8 shows the crack identification results of original images taken with different focal distances, in order to validate the performance of the trained network on multiscale images. The figure shows that crack elements are differentiated from the background and from handwriting, even though the latter has crack-like features and acts as a major disturbance for crack identification. The mean value and standard deviation of precision are 88.8% and 6.7%, respectively. The mean value and standard deviation of precision are the statistical results computed over test images under different conditions. The precision is computed in an element-based manner, in terms of whether an input sub-image is correctly categorized as crack, handwriting, or background.

ML-based condition assessment involves the establishment of a statistical representation of a structure by means of monitored data and assessing whether the condition is normal or not based on changes in the probability density function. For this reason, these methods are also known as pattern-recognition methods.

This paper provides a brief review of the state of the art of data science and engineering in the field of SHM. Conclusions and future trends are provided below.

A CS-based data-acquisition algorithm is able to randomly sample dynamic signals and reduce the volume of the dynamic signal (e.g., the acceleration, dynamic strain, displacement, etc.) because the dynamic signal in civil structures is sparse in the frequency domain and in the frequency-time domain.

ML, DL, and CV techniques provide efficient algorithms to automatically diagnose anomaly data and perform crack identification and condition assessment using big data from monitoring; they can be extensively applied in SHM.

The concepts of the “automatic AI scientist” and the “automatic AI engineer” are gaining considerable scholarly interest in the field of AI, as they can learn and create theorems, theories, and designs. AI, virtual realization or augmented realization, wearable devices, crowd smart-sensing technology, and their combinations will make it possible to collect more data and information at a low cost, and will lead to novel theories on structural health diagnosis and prognosis by overcoming many challenging issues in traditional damage detection, model updating, safety evaluation, and reliability analysis. These technologies will help us identify a new intrinsic evolution in the long-term performance of full-scale structures in real operational surroundings and under real loads.