Deep Learning with Vision-based Technologies for Structural Damage Detection and Health Monitoring

There are three main research conducted in this paper, including using deep learning methods with vision-based technologies on Structural Damage Detection (SDD), Structural Health Monitoring (SHM) and progressive collapse study. During the learning and improvement process, many goals of automation in SDD and SHM have been achieved, although there will be a large room for further improvement and development on these studies. In progressive collapse study, remote sensing technologies and data fusion are applied on a field experiment of a real building at the Central Campus of the Ohio State University. The major contributions of this paper are shown as follows:A few comprehensive experimental studies for automated SDD in extreme events using deep learning methods for processing 2D images. In the first study, a 152-layer Residual Network (ResNet) is utilized to identify multiple classes in eight SDD tasks, which include identification of scene levels, damage levels, material types, etc. The proposed ResNet achieved high accuracy for each task while the positions of the damage are not identifiable. In the second study, the existing ResNet and a segmentation network (U-Net) are combined into a new pipeline, cascaded networks, for categorizing and locating structural damage. The results show that the accuracy of damage detection is significantly improved compared to only using a segmentation network. In the third and fourth studies, end-to-end networks are developed and tested as a new solution to directly detect cracks and spalling in the image collections of recent large earthquakes. One of the proposed networks can achieve an accuracy above $67.6\%$ for all tested images at various scales and resolutions, and shows its robustness for these human-free detection tasks.Studies are conducted with a pipeline to automatically track and measure displacements and vibrations of structures or structural components in laboratory and field experiments. This novel framework that uses computer vision and deep learning methods to mimic human vision system for the dynamic performance assessment of in-service infrastructure with various camera placements. On one hand, the static deformations of cylinders and small-scale reinforced concrete beams in the laboratory tests are captured and measured by the proposed framework at first. Then two shaking table tests in the lab are utilized to assess the dynamic performance of the simulated structures. On the other hand, several bridges, including pedestrian, railway, and traffic bridges, are tested for their dynamic performance in field experiments with different camera placements: remote, structure-mounted, and drone-mounted cameras. To remove systematic motions of cameras and to capture the fundamental frequency of these tested structures, two techniques, displacement subtraction and frequency subtraction, are applied. To better understanding of practical applications, critical parameters for camera settings and data processing techniques, such as video frame-rates, window size and locations, and sampling rates on visual data are studied. It shows that not only the vibrations and the frequencies of the simulated structures (i.e., in lab tests) or the in-service structures (i.e., in filed experiments), but also the static deformation of structures or structural components, can be tracked and measured accurately by our proposed framework.The performance of a building in progressive collapse study is monitored and analyzed with the methods developed in previous studies. On one hand, we applied the proposed methods to detect structural damage on this reinforced concrete (RC) structure that was tested, including visual data from cell phones, inside and outside cameras, and drones. The experiments indicate that these solutions for automatic detection of structural damage using deep learning methods are feasible and promising. On the other hand, the proposed methods to measure the deformations and vibrations are utilized to process visual data from outside and inside cameras, and drones. Data from different sources are fused for capturing the performance of the structural elements under the scenarios of the sudden loss of the columns and slabs. The data fusion technique is useful to investigate the characteristics of this framed structure, especially when the traditional sensors such as strain gauges and Linear Variable Displacement Transducers (LVDTs) didn't work in the experiments. Since visual data acquisition and preparation, and techniques of data processing have been inclusively researched for real applications with deep learning, many experimental studies on SDD and SHM are carried out and promising results are obtained in this paper. In addition, the vibration-based technologies or traditional sensors are used as the reference. Our goal is to meet the needs for automatic detection of structural damage and accurate measurements for the performance of the structures or structural elements, thus, the efficiency and effectiveness of these frameworks are tested and analyzed. In summary, these research in this paper indicate that vision-based technologies with deep learning can be applied well on structural engineering domain, and facilitate structural engineers' job by providing reliable data and credential results. The methodologies developed in this paper can fill the gap of research and engineering applications in the future.