Innovative approaches for infrastructure monitoring through photogrammetry and deep learning

Nowadays, civil infrastructures are getting older and older all over the world. For this reason, innovative and more efficient monitoring systems are required to determine the health and safety level of these infrastructures. In the last decades, different techniques have been investigated in the field of material testing and infrastructure monitoring. Material testing is usually performed in the laboratory to understand the response of building materials. The principle of material testing is based on the mechanical loading of a material up to a certain level of deformation. During testing, the applied force and material deformation are measured; the aim is to retrieve the rheology of the material which is responsible for its behaviour. During testing, standard pointwise sensors such as strain gauges and Linear Variable Differential Transducers (LVDT) and image based methods (e.g. Digital Image Correlation - DIC) have been widely adopted to measure material deformation and crack propagation. In particular, DIC can measure displacements, strains and cracks by comparing the positions of points in images acquired using a camera with a fixed position. On the other hand, in the field of infrastructure monitoring, cracks are early indicators of damage; crack detection and measurement represent, indeed, key parameters in evaluating the safety and durability of structural components. Therefore, cracks should be detected as soon as possible and monitored over time to assess the condition of the infrastructure and identify the necessary countermeasures. Traditionally, cracks have been routinely measured during visual infrastructure inspections all over the world. The procedure consists of visually estimating the crack width using graduated cards provided with a set of thickness lines. This method is time-consuming, expensive and prone to human errors. Furthermore, the inspections are typically conducted at night since the infrastructure needs to be closed during inspections. While this minimizes the impact of infrastructure downtime, it provides limited time to conduct the inspections in what often are long infrastructure systems. As a result, the efficiency and thoroughness of such inspections have been questioned as the risk of missing cracks is high. To address the limitations of visual inspections, contact and non-contact techniques have been extensively investigated. However, contact sensors provide only local measurements whereas DIC requires a fixed camera to acquire the images, which becomes a challenge for long-term monitoring outside the controlled laboratory conditions. Nowadays, cameras can be placed on vehicles or drones which enable an automatic procedure to collect imagery from infrastructures. An important task for the inspections is to determine whether or not the length or width of the cracks has propagated since the last inspection. However, due to the limitations of standard DIC, crack propagation cannot be measured from these types of images since the position of the camera between the inspections differs. Therefore, the information from the images acquired using mobile mapping systems can only be adopted to generate a digital representation of the infrastructure (the so-called digital twin) and manually compare the characteristics of the cracks between different inspections. Due to the large amount of collected data, this approach is still time-consuming, inefficient and affected by human errors. The aim of this thesis is to investigate image-based techniques for accurate measurements during laboratory tests and infrastructure monitoring. Specifically, this thesis investigates standard and innovative DIC-based approaches to measure displacements, strains and crack propagation at the laboratory scale and innovative image-based techniques (based on the combination of photogrammetry and deep learning) for infrastructure monitoring. For laboratory measurement and material characterization, the DIC technique is investigated. The developed local 2D DIC software, named Py2DIC, is tested comparing its performances with those of other open-source and commercial software applications (Ncorr, DICe and Vic-2D) and the strain gauge sensor. Three different datasets acquired during laboratory tests using a camera with a fixed position are analysed. For software comparisons, the displacement and strain fields are compared globally and locally, whereas, for the strain gauge, the DIC results are retrieved around the area covered by the sensor and then averaged. The results highlight an agreement of a few microns between Py2DIC, the other software packages and the strain gauge. The analyses confirm the potential of the implemented software in measuring displacements, strains and cracks. To overcome the DIC limitations, a new methodology is also presented in this thesis. The approach can measure the propagation of cracks in images captured using cameras with a not fixed position. The innovative model, named Deformation from Motion (DfM), exploits the homography transformation between image planes and template matching algorithms. The model, completely implemented in Python and based on OpenCV libraries, is validated using synthetic data and images acquired with moving cameras during a laboratory test. The results are compared with the standard DIC technique and the LVDT sensor glued on the sample surface. The comparison shows an agreement of a few hundredths of millimetres between DfM and the LVDT. The methodology, tested in the laboratory, can compute the crack propagation using a time series of images; it is, therefore, suitable for processing images captured with mobile mapping systems. For infrastructure monitoring, the TACK (Tunnel and bridge Automatic CracK monitoring using deep learning and photogrammetry) project is presented and discussed. The project, funded by the Swedish Innovation Agency and the Swedish Transport Administration, aims at investigating a new methodology for efficient and accurate infrastructure monitoring. The methodology is based on the LiDAR and close-range photogrammetric techniques for 3D infrastructure modelling, deep learning approaches (Convolutional Neural Networks - CNNs) for crack detection and the DfM model for crack measurement. In this thesis, the use of LiDAR data acquired in a tunnel and images of a bridge are investigated. Specifically, the images are processed using Pix4Dmapper to investigate a low-cost approach (i.e. close-range photogrammetry) for 3D modelling of bridges. Furthermore, the potentialities of CNNs are investigated using different datasets of cracks. Preliminary results obtained using a U-Net based architecture (Crack-SegNet) trained with a small dataset show Intersection over Union (IoU) of 51%, Precision of 67%, Recall of 71%, F1 of 68% and Accuracy of 99% on the test set. The results also demonstrate the possibility of detecting cracks in unseen images acquired in a tunnel. Unfortunately, other objects with a similar shape can be erroneously detected as cracks. For this reason, a new processing pipeline based on the U-Net segmentation model combined with different backbones is developed and trained using a larger dataset. The trained model achieves better results on the testing set and the unseen tunnel dataset. The result assessment using U-Net combined with the VGG (Visual Geometry Group) 19 backbone highlights IoU of 77%, Precision of 88%, Recall of 91%, F1 of 87% and Accuracy of 99% on the test set. The trained model can also better distinguish cracks from similarly-looking objects. The performed experiments demonstrate the possibility to adopt deep learning architectures to detect cracks in images of tunnels increasing, in this way, the efficiency of infrastructure monitoring.