Search

In modern ocean exploration, forward-looking sonar (FLS) provides real-time 2D imaging of the seabed ahead, but its detection range is relatively limited. Conversely, side-scan sonar (SSS) enables large-scale imaging of the seabed during movement but struggles to effectively image areas directly beneath the sensor. Integrating FLS and SSS offers a promising solution by leveraging their complementary strengths to achieve comprehensive seabed mapping. However, no prior research has explored this fusion approach. This paper presents a novel method for FLS and SSS fusion mapping. Firstly, a novel sonar image enhancement method based on equalization is proposed, enabling simultaneous enhancement and grayscale unification of two sonar images. Additionally, an effective area extraction approach for FLS images, grounded on the approximate erosion method, is introduced to produce high-quality FLS mapping. Furthermore, by examining the data distribution in FLS and SSS mappings, the standard deviation of these datasets is utilized to refine the grayscale distribution of FLS mapping, thereby enhancing the grayscale distribution similarity between the two mapping results. Finally, FLS map data are seamlessly integrated into the gaps of the SSS map, resulting in a fused, comprehensive seabed representation. Large-scale experiments demonstrate that the proposed method effectively combines the strengths of FLS and SSS, producing complete and detailed seabed topography maps. Simultaneously, numerous ablation experiments are conducted to evaluate the impact of various parameters on fusion mapping, providing guidelines for selecting the optimal parameters. This fusion approach, thus, holds significant practical value for ocean exploration and seabed mapping applications. [ABSTRACT FROM AUTHOR]

The risk of gas leakage due to geological flaws in offshore carbon capture, utilization, and storage, as well as leakage from underwater oil or gas pipelines, highlights the need for underwater gas leakage monitoring technology. Remotely operated vehicles (ROVs) and autonomous underwater vehicles (AUVs) are equipped with high-resolution imaging sonar systems that have broad application potential in underwater gas and target detection tasks. However, some bubble clusters are relatively weak scatterers, so detecting and distinguishing them against the seabed reverberation in forward-looking sonar images are challenging. This study uses the dual-tree complex wavelet transform to extract the image features of multibeam forward-looking sonar. Underwater gas leakages with different flows are classified by combining deep learning theory. A pool experiment is designed to simulate gas leakage, where sonar images are obtained for further processing. Results demonstrate that this method can detect and classify underwater gas leakage streams with high classification accuracy. This performance indicates that the method can detect gas leakage from multibeam forward-looking sonar images and has the potential to predict gas leakage flow. [ABSTRACT FROM AUTHOR]

Given the propagation characteristics of sound waves and the complexity of the underwater environment, denoising forward-looking sonar image data presents a formidable challenge. Existing studies often add noise to sonar images and then explore methods for its removal. This approach neglects the inherent complex noise in sonar images, resulting in inaccurate evaluations of traditional denoising methods and poor learning of noise characteristics by deep learning models. To address the lack of high-quality data for FLS denoising model training, we propose a simulation algorithm for forward-looking sonar data based on RGBD data. By utilizing rendering techniques and noise simulation algorithms, high-quality noise-free and noisy sonar data can be rapidly generated from existing RGBD data. Based on these data, we optimize the loss function and training process of the FLS denoising model, achieving significant improvements in noise removal and feature preservation compared to other methods. Finally, this paper performs both qualitative and quantitative analyses of the algorithm's performance using real and simulated sonar data. Compared to the latest FLS denoising models based on traditional methods and deep learning techniques, our method demonstrates significant advantages in denoising capability. All inference results for the Marine Debris Dataset (MDD) have been made open source, facilitating subsequent research and comparison. [ABSTRACT FROM AUTHOR]

In modern ocean exploration, forward-looking sonar (FLS) provides real-time 2D imaging of the seabed ahead, but its detection range is relatively limited. Conversely, side-scan sonar (SSS) enables large-scale imaging of the seabed during movement but struggles to effectively image areas directly beneath the sensor. Integrating FLS and SSS offers a promising solution by leveraging their complementary strengths to achieve comprehensive seabed mapping. However, no prior research has explored this fusion approach. This paper presents a novel method for FLS and SSS fusion mapping. Firstly, a novel sonar image enhancement method based on equalization is proposed, enabling simultaneous enhancement and grayscale unification of two sonar images. Additionally, an effective area extraction approach for FLS images, grounded on the approximate erosion method, is introduced to produce high-quality FLS mapping. Furthermore, by examining the data distribution in FLS and SSS mappings, the standard deviation of these datasets is utilized to refine the grayscale distribution of FLS mapping, thereby enhancing the grayscale distribution similarity between the two mapping results. Finally, FLS map data are seamlessly integrated into the gaps of the SSS map, resulting in a fused, comprehensive seabed representation. Large-scale experiments demonstrate that the proposed method effectively combines the strengths of FLS and SSS, producing complete and detailed seabed topography maps. Simultaneously, numerous ablation experiments are conducted to evaluate the impact of various parameters on fusion mapping, providing guidelines for selecting the optimal parameters. This fusion approach, thus, holds significant practical value for ocean exploration and seabed mapping applications.

【Objective】Cage biometrics and statistics are one of the key reference factors for marine pasture farming management. Aiming at the interference of reverberation noise and complex background, this paper constructs fish detection data sets under different lighting conditions, and uses forward-looking sonar imaging technology to propose a fish recognition lightweight model based on YOLOV5-MobilenetV3 and sonar images (LAPR-Net) to realize fish recognition in water cages in turbid or dark scenes.【Method】Taking tilapia as the research object, based on the frame structure of the YOLOV5 model, the backbone network module ado pts the lightweight Mob ileNetV3 bneck block, using the linear bottleneck inverse residual structure and depth separable convolution extract the features of fish in sonar images, applying the attention mechanism SE-Net to obtain multi-scale semantic features of sonar images and enhance the correlation between features; the neck network adopts the path aggregation network structure to perform multi-scale fusion of target features, to enhance the feature fusion ability; the prediction part adopts the maximum local search based on the non-maximum suppression method, removes the redundant detection frame, screens the detection frame with the highest confidence, and finally outputs and displays the detection result of the fish, including the position, category and detection probability of detecting an object.【Result】Four other mainstream detection models were selected for comparative experiments, including YOLOV3-ting (Darknet53), YOLOV5 (CSPdarknet53), YOLOV5 (Repvgg), and YOLOV5s (Transformer). It proposes the model parameter quantityof 3 545 453, FLOPs of 6.3 G, and the mAP of 0.957, and the average inference speed of each picture of the model is 0.08868 s. Compared with the YOLOV5 model, the mAP of the improved model has increased by 9.7%.【Conclusion】The proposed network improves the speed of training and recognition, reduces the requirements for hardware equipment, and provides a reference for the detection model of cage cultured fish in marine pastures.

Some structures in the harbour environment need to be inspected regularly. However, these scenarios present a major challenge for the accurate estimation of a vehicle's position and subsequent recognition of similar images. In these scenarios, visibility can be poor, making place recognition a difficult task as the visual appearance of a local feature can be compromised. Under these operating conditions, imaging sonars are a promising solution. The quality of the captured images is affected by some factors but they do not suffer from haze, which is an advantage. Therefore, a purely acoustic approach for unsupervised recognition of similar images based on forward-looking sonar (FLS) data is proposed to solve the perception problems in harbour facilities. To simplify the variation of environment parameters and sensor configurations, and given the need for online data for these applications, a harbour scenario was recreated using the Stonefish simulator. Therefore, experiments were conducted with preconfigured user trajectories to simulate inspections in the vicinity of structures. The place recognition approach performs better than the results obtained from optical images. The proposed method provides a good compromise in terms of distinctiveness, achieving 87.5% recall considering appropriate constraints and assumptions for this task given its impact on navigation success. That is, it is based on a similarity threshold of 0.3 and 12 consistent features to consider only effective loops. The behaviour of FLS is the same regardless of the environment conditions and thus this work opens new horizons for the use of these sensors as a great aid for underwater perception, namely, to avoid degradation of navigation performance in muddy conditions. [ABSTRACT FROM AUTHOR]

Forward-looking sonar (FLS) images present various challenges in interpretation, recognition, and segmentation due to limitations like low resolution, speckle noise, and low contrast, making them more complex than optical images. Existing methods often focus solely on denoising or enhancement, neglecting the potential benefits of utilizing multi-scale features to create an integrated image processing approach. This paper introduces the Laplacian pyramid-based multi-scale denoising and enhancement (LPMsDE) method tailored for FLS images. The proposed method begins by presenting a novel multiplicative speckle noise model, grounded in the Gaussian distribution, specifically designed for FLS images. Next, the Laplacian pyramid decomposition is utilized to estimate noise variance, with an modified adaptive local filter. Lastly, a combination of the Laplacian pyramid framework, the enhanced adaptive local filter, and Contrast-Limited Histogram Equalization (CLHE) is employed to denoise and enhance images at different resolution levels. Through comprehensive experiments conducted on both simulated and real sonar images, the effectiveness of the LPMsDE method is demonstrated. It surpasses other denoising and enhancement techniques, as evidenced by superior scores in Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), Contrast-to-Noise Ratio (CNR), Equivalent Number of Looks (ENL), Natural Image Quality Evaluator (NIQE), and Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE).

Copyright of Guangdong Agricultural Sciences is the property of South China Agricultural University, Guangdong Academy of Agricultural Sciences and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Precise localization of unmanned underwater vehicle (UUV) on multi-beam forward-looking sonar (MFLS) is a key technology in underwater robotic exploration. However, large appearance change and weak features of targets in MFLS image, pose a serious challenge on target detection. This paper proposes an efficient UUV detection model with an enhanced training scheme and convolutional block attention mechanism. The proposed training scheme is composed of dual-path gradient information optimization and adaptive loss adjustment, which improves the network's detection capabilities and the underwater environment adaptability. The convolutional block attention module (CBAM) is developed for extracting channel and spatially weighted feature maps, and providing a mechanism to adaptive feature refinement. We construct a comprehensive sonar detection dataset from field experiments. By comparing existing state-of-the-art detection algorithms, our proposed UUVDNet outperforms Faster R-CNN, Cascade R-CNN, CenterNet, SSD, YOLOv8n, DPFIN, and MBSNN in terms of mAP 50 – 95 by 39.5%, 35.7%, 8.4%, 14.0%, 2.5%, 13.6%, and 21.8%, respectively. Furthermore, our inference model size is the most compact among all detection models, with a size of 5.1M. Final ablation experiments effectively showcase the proposed components' efficiency. [Display omitted] • A novel detection framework called UUVDNet considering enhanced training scheme and attention mechanism is proposed for underwater FLS image detection task, which improves the detection precision and speed. • Enhanced training with dual-path optimization and DCIoU loss boosts underwater detection. • The convolutional block attention module is proposed to refine the spatial and channel feature maps of deep networks. • A new multibeam forward-looking sonar target detection dataset has been constructed, encompassing a rich variety of detection environments and UUV target quantities. [ABSTRACT FROM AUTHOR]

Forward-looking sonar is a commonly used underwater detection device at present, but the detection accuracy is poor due to the complex underwater environment, small target highlight area and fuzzy feature details. Therefore, this paper proposes a forward sonar image target detection model based on You Only Look Once Version 5 (YOLOv5) network using transfer learning method. First, the YOLOv5 network is pretrained with COCO data set. Then the pre-training model is fine-tuned according to the training set of forward-looking sonar images. Before fine-tuning, the traditional k-means clustering is improved. The intersection over union ( $IoU$ ) value is used as the distance function to cluster the labeling information of the training set of the forward-looking sonar image. The results of clustering serve as the initial anchor frame of the training network. This operation greatly improves the detection speed. Second, due to the characteristics of weak echo intensity and small target area of forward-looking sonar image, an improved feature extraction method of CoordConv was proposed to give corresponding coordinate information to high-level features which improves the accuracy of network detection regression. Finally, the fine-tuned network is used to detect the target in the forward-looking sonar image. The experimental results show that the improved model based on YOLOv5 network is superior to the original YOLOv5 network and other popular deep neural networks for target detection in the forward-looking sonar image, which has a reference significance for underwater target detection. The CoordConv-YOLOv5 network based on transfer learning proposed in this paper shows the best performance in both detection accuracy and detection speed. Detection accuracy mAP@0.5:0.95 can reach 56.95%, and detection speed can reach 9ms.

Some structures in the harbour environment need to be inspected regularly. However, these scenarios present a major challenge for the accurate estimation of a vehicle’s position and subsequent recognition of similar images. In these scenarios, visibility can be poor, making place recognition a difficult task as the visual appearance of a local feature can be compromised. Under these operating conditions, imaging sonars are a promising solution. The quality of the captured images is affected by some factors but they do not suffer from haze, which is an advantage. Therefore, a purely acoustic approach for unsupervised recognition of similar images based on forward-looking sonar (FLS) data is proposed to solve the perception problems in harbour facilities. To simplify the variation of environment parameters and sensor configurations, and given the need for online data for these applications, a harbour scenario was recreated using the Stonefish simulator. Therefore, experiments were conducted with preconfigured user trajectories to simulate inspections in the vicinity of structures. The place recognition approach performs better than the results obtained from optical images. The proposed method provides a good compromise in terms of distinctiveness, achieving 87.5% recall considering appropriate constraints and assumptions for this task given its impact on navigation success. That is, it is based on a similarity threshold of 0.3 and 12 consistent features to consider only effective loops. The behaviour of FLS is the same regardless of the environment conditions and thus this work opens new horizons for the use of these sensors as a great aid for underwater perception, namely, to avoid degradation of navigation performance in muddy conditions.

Forward-looking sonar is a technique widely used for underwater detection. However, most sonar images have underwater noise and low resolution due to their acoustic properties. In recent years, the semantic segmentation model U-Net has shown excellent segmentation performance, and it has great potential in forward-looking sonar image segmentation. However, forward-looking sonar images are affected by noise, which prevents the existing U-Net model from segmenting small objects effectively. Therefore, this study presents a forward-looking sonar semantic segmentation model called Feature Pyramid U-Net with Attention (FPUA). This model uses residual blocks to improve the training depth of the network. To improve the segmentation accuracy of the network for small objects, a feature pyramid module combined with an attention structure is introduced. This improves the model's ability to learn deep semantic and shallow detail information. First, the proposed model is compared against other deep learning models and on two datasets, of which one was collected in a tank environment and the other was collected in a real marine environment. To further test the validity of the model, a real forward-looking sonar system was devised and employed in the lake trials. The results show that the proposed model performs better than the other models for small-object and few-sample classes and that it is competitive in semantic segmentation of forward-looking sonar images. [ABSTRACT FROM AUTHOR]

Imaging and mapping sonars such as forward-looking sonars (FLS) and side-scan sonars (SSS) are sensors frequently used onboard autonomous underwater vehicles. To acquire information from around the vehicle, it is desirable for these sonar systems to insonify a large area; thus, the sonar transmit beampattern should have a wide field of view. In this work, we study the problem of the optimization of wide transmission beampatterns. We consider the conventional phased-array beampattern design problem where all array elements transmit an identical waveform. The complex weight vector is adjusted to create the desired beampattern shape. In our experiments, we consider wide transmission beampatterns (≥20 ∘ ) with uniform output power. In this paper, we introduce a new iterative-convex optimization method for narrowband linear phased arrays and compare it to existing approaches for convex and concave–convex optimization. In the iterative-convex method, the phase of the weight parameters is allowed to be complex as in disciplined convex–concave programming (DCCP). Comparing the iterative-convex optimization method and DCCP to the standard convex optimization, we see that the former methods archive optimized beampatterns closer to the desired beampatterns. Furthermore, for the same number of iterations, the proposed iterative-convex method achieves optimized beampatterns, which are closer to the desired beampattern than the beampatterns achieved by optimization with DCCP. [ABSTRACT FROM AUTHOR]

Forward-looking sonar (FLS) image registration is a key step in many underwater applications such as underwater target detection, ocean observation, and mapping. However, low resolution, low signal-to-noise ratio, and the complex nonlinear transformation relationship between FLS images from two different viewpoints have brought great challenges to register them. In order to better cope with this challenge, we propose a global perspective and local flow registration (GPLFR) method for FLS images. GPLFR consists of two networks, i.e., a regression correction network (RCNet) and a deformable network (IRRDNet) with the iterative refinement of the residual. For a given pair of FLS images, RCNet is used to estimate the global transformation parameters to achieve global registration, and then, IRRDNet is used to estimate the deformation field or flow field to realize local alignment. The experimental results on real FLS image and 2D face expression image registration tasks demonstrate the effectiveness and robustness of the proposed method. [ABSTRACT FROM AUTHOR]

Autonomous Underwater Vehicles (AUVs) are used by the scientific community for various applications, from collecting well‐distributed high‐quality data to mapping the seafloor or exploring unknown areas. Nonpredictable environmental conditions and sensor acquisitions make the design of AUV surveys challenging even for expert operators. Multiple attempts are required, and the collected data quality is not guaranteed: The AUV passively stores the sensors' acquisitions that are then analyzed offline after its recovery. In Forward‐Looking SONAR (FLS) seabed inspections, the vehicle follows lawnmower paths designed by an expert operator that considers the sensor characteristics. The performance of FLSs is affected by several environmental conditions and possible protruding objects. This paper presents a probabilistic framework for FLS‐based seabed inspections that endow the AUV with the ability to autonomously conducting the survey and ensure adequate coverage of the target area. A three‐dimensional probabilistic occupancy mapping system for FLS reconstructions to update the covered area map was developed. The map is used by the Coverage Path Planning (CPP) algorithm to evaluate the visibility of the viewpoints that are generated as nodes of a random tree. The Next‐Best Viewpoint (NBV) is selected as the first node in the branch expected to collect more data, and the path to reach the NBV is computed using the rapidly exploring random tree (RRT*) algorithm. The sensor‐driven coverage approach is used in a receding‐horizon manner. The proposed Receding‐Horizon Coverage Approach was validated with simulations and real prerecorded data. Finally, the framework was used online during an experimental campaign where several FLS seabed inspections were performed. [ABSTRACT FROM AUTHOR]

In order to meet the needs of complex seafloor scenes target detection, an image registration method based on sensitive region is proposed which considering the fuzzy boundary, poor contrast and low overall gray value of forward-looking sonar image. In this method, gradient enhancement algorithm is used to enhance the boundary of the target region in forward-looking sonar(FLS) image, and morphological operation method is used to remove a large number of noise points in the image background. And then, the coordinates and the sensitive region which contains a part of obvious grayscale changes is located in the floating image. The combined method of particle swarm optimization(PSO) algorithm and mutual information(MI) algorithm are used for initial registration, and the re-registration method of mutual information is used to optimize the registration error and improve the registration accuracy. The experimental results show that this method has the advantages of accurate location of the sensitive region, low computation time and high registration accuracy.

Autonomous Underwater Vehicle (AUV) is an automated navigation device, which can independently complete the underwater investigation and detection without supervision. However, the marine environment is getting increasingly severe, and the garbage in the ocean and fishnets cast by fishermen have a great negative influence on the work of AUV. To solve the problems of low precision and slow speed in the state-of-the-art network detection of weak and small targets in underwater sonar images, Yolo5 was improved by building the multi-branch shuttle neural network The dataset is collected by multi-beam forward-looking sonar Gemini720i, and includes “fishnet” and two representative types of knitted and plastic garbage, i.e., “cloth” and “plasticbag”. The original dataset is enhanced and balanced, and the effect of dataset distribution on the model performance is studied. Utilizing pre-training, the effects of Yolo5 family, i.e., Yolo5s, Yolo5m, Yolo5l and Yolo5x are compared using the balanced dataset, to explore the impacts on detecting weak and small targets of the deeper and wider networks of this family. In addition, the combinations of Yolo5s with lightweight networks, i.e., MobileNet3 and ShuffleNet2 are experimented, which further illustrates the effectiveness of the proposed network, and can satisfy the high accuracy and real-time requirements of AUV. [ABSTRACT FROM AUTHOR]

Three-dimensional reconstruction is a crucial technique for mapping and object-search tasks, but it is challenging in sonar imaging because of the nature of acoustics. In underwater sensing, many advanced studies have introduced approaches that have included feature-based methods and multiple imaging at different locations. However, most existing methods are prone to environmental conditions, and they are not adequate for continuous data acquisition on moving autonomous underwater vehicles (AUVs). This paper proposes a sensor fusion method for 3D reconstruction using acoustic sonar data with two sonar devices that provide complementary features. The forward-looking multibeam sonar (FLS) is an imaging sonar capable of short-range scanning with a high horizontal resolution, and the profiling sonar (PS) is capable of middle-range scanning with high reliability in vertical information. Using both sonars, which have different data acquisition planes and times, we propose a probabilistic sensor fusion method. First, we extract the region of interest from the background and develop a sonar measurement model. Thereafter, we utilize the likelihood field generated by the PS and estimate the elevation ambiguity using importance sampling. We also present the evaluation of our method in a ray-tracing-based sonar simulation environment and the generation of the pointclouds. The experimental results indicate that the proposed method can provide a better accuracy than that of the conventional method. Because of the improved accuracy of the generated pointclouds, this method can be expanded for pointcloud-based mapping and classification methods. [ABSTRACT FROM AUTHOR]

Due to the exceptional detection capabilities, the forward-looking sonar could be adopted in simultaneous localization and mapping (SLAM) for autonomous underwater vehicle (AUVs). This paper primarily investigates the application of the factor graph optimization SLAM algorithm based on feature maps in AUV. It achieves this by combining the smallest of constant false alarm rate (SO-CFAR) and adaptive threshold (ADT) to filter noise from the forward-looking sonar and extract feature point clouds. Furthermore, a weighted iterative closest point (WICP) algorithm is employed for feature point registration, which is extracted from the sonar image. The experimental result based on field data demonstrates that the proposed method, with an 8.52% improvement in root mean square error (RMSE) compared with dead reckoning (DR)., Competing Interests: Declaration of conflicting interestsThe authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Nowadays, high-frequency forward-looking sonar is an effective device to obtain the main information of underwater objects. Detection and segmentation of underwater objects are also one of the key topics of current research. Deep learning has shown excellent performance in image features extracting and has been extensively used in image object detection and instance segmentation. With the network depth increasing, training accuracy gets saturated and training parameters also increase rapidly. In this paper, a series of residual blocks are used to build a 32-layer feature extraction network and take place of the Resnet50/101 in Mask RCNN, which reduces the training parameters of the network while guaranteeing the detection performance. The parameters of the proposed network are 29% less than Resnet50 and 50.2% less than Resnet101, which is of great significance for future hardware implementation. In addition, Adagrad optimizer is introduced into this research to improve the detection performance of sonar images. Finally, the object detection results of 500 test sonar images show that the mAP is 96.97% that is only 0.18% less than Resnet50 (97.15%) but more than Resnet101 (95.15%). [ABSTRACT FROM AUTHOR]

In this article, a novel dense underwater 3-D mapping paradigm based on pose graph simultaneous localization and mapping (SLAM) using an acoustic camera mounted on a rotator is proposed. The demands of underwater tasks, such as unmanned construction using robots, are growing rapidly. In recent years, the acoustic camera, which is a state-of-the-art forward-looking imaging sonar, has been gradually applied in underwater exploration. However, distinctive imaging principles make it difficult to gain an intuitive perception of an underwater environment. In this study, an acoustic camera with a rotator was used for dense 3-D mapping of the underwater environment. The proposed method first applies a 3-D occupancy mapping framework based on the acoustic camera rotating around the acoustic axis using a rotator at a stationary position to generate 3-D local maps. Then, scan matching of adjacent local maps is implemented to calculate odometry without involving internal sensors, and an approximate dense global map is built in real time. Finally, based on a graph optimization scheme, offline refinement is performed to generate a final dense global map. Our experimental results demonstrate that our 3-D mapping framework for an acoustic camera can achieve dense 3-D mapping of underwater environments robustly and accurately. [ABSTRACT FROM AUTHOR]

Robust and accurate estimation of position and attitude of a UUV (Unmanned Underwater Vehicle) from sonar scans is essential for simultaneous localization and mapping (SLAM). Both dead-reckoning based on the inertial navigation system and the motion parameter estimation based on the registration of the ultrasound scan sequence can contribute to the performance of the system. However, the rapidly-growing accumulated error tends to counteract the precise localization of the vehicle. In this paper, a method for loop-closure detection is proposed that adjusts the accumulated error for the underwater acoustic SLAM when the vehicle scans the underwater environment using an Mechanical Scanning Imaging Sonar (MSIS). Firstly, a similarity matrix for pairs of scans is constructed to represent the loop-closing tracks. In the registration step, two novel features, namely the intensity projection histograms and a polar gradient matrix, are extracted to calculate the translational and rotational parameters respectively. Secondly, the probability hypothesis density (PHD) filter is used to extract the possible loop-closure constraints from the similarity matrix, removing the random noise brought by accidental correlation and refining the concurrent loop-closing tracks resulted from long-range scanning. Lastly, the loop-closure constraints from the refined loop-closing tracks are fed into the GraphSLAM system to adjust the pose of each scan by constraint optimization. Experiments on the MSIS sonar images collected in structured and unstructured underwater environments validate the effectiveness of the proposed loop-closure detection method.

Underwater automatic target recognition (ATR) is a challenging task for marine robots due to the complex environment. The existing recognition methods basically use hand-crafted features and classifiers to recognize targets, which are difficult to achieve ideal recognition accuracy. In this paper, we proposed a novel method to realize accurate multiclass underwater ATR by using forward-looking sonar-Echoscope and deep convolutional neural networks (DCNNs). A complete recognition process from data preprocessing to network training and image recognition was realized. Firstly, we established a real, measured Echoscope sonar image dataset. Inspired by the human visual attention mechanism, the suspected target region was extracted via the graph-based manifold ranking method in image preprocessing. Secondly, an end-to-end DCNNs model, named EchoNet, was designed for Echoscope sonar image feature extraction and recognition. Finally, a network training method based on transfer learning was developed to solve the problem of insufficient training data, and mini-batch gradient descent was used for network optimization. Experimental results demonstrated that our method can implement efficiently, and the recognition accuracy on a nine-class underwater ATR task reached 97.3%, outperforming traditional feature-based methods. The proposed method is expected to be a potential novel technology for the intelligent perception of autonomous underwater vehicles.

In this paper, we propose a novel method to generate realistic acoustic datasets for forward-looking sonars. A forward-looking sonar, which is also known as an acoustic camera, outperforms other imaging sensor when applied in underwater tasks as it can provide more accurate and detailed information about the environment, even in dark or turbid water. However, the difficulty and high cost of acquiring acoustic images in real experiments encourage researchers to consider the generation of simulated acoustic image datasets. In particular, deep learning-based methods demonstrated high performance in computer vision tasks, such as in object detection. However, a large dataset is necessary in most cases. In the proposed method, we first build a novel user-friendly acoustic image simulator based on 3D modeling software. Then, the CycleGAN is applied to generate realistic acoustic images based on the generated dataset from the simulator. The experimental results demonstrate that our method can generate effective and realistic acoustic datasets with relatively simple operations. [ABSTRACT FROM AUTHOR]

Forward-Looking Sonar (FLS) is one of the most effective devices for underwater exploration which provides high-resolution images that can be used for several tasks in marine research, oceanographic, and deep-sea exploration. The limitation of current underwater acoustic channels does not allow transmitting these images in real-time, therefore image compression is required. Since acoustic images are characterized by speckle noise, an important challenge, in this area, is how to perform the compression while preserving relevant information. In this paper, a novel lossy forward-looking acoustic image compression method based on the combination between keypoint clustering and Morphological Skeleton (MS) is proposed. Keypoints are extracted by using A-KAZE feature extractor, while Density-Based Spatial Clustering of Application with Noise (DBSCAN) is used to find keypoint clusters representing a region-of-interest (ROI). Then, MS is executed to compact the ROI. The rest of the image is down-sampled and quantized through K-Means Clustering and represented via colour indexing. Finally, the information is compressed by using Brotli data compression. The experimental results on real FLS images demonstrate that our method achieves good outcomes in terms of quality metrics and compression ratio. [ABSTRACT FROM AUTHOR]

The paper addresses path planning and path-following problems in an unknown complex environment for an underactuated autonomous underwater vehicle (AUV). The AUV is required to follow a given reference path represented as a sequence of smoothly joined lines and arcs, bypassing obstacles encountered on the path. A two-level control system is proposed with an upper level for event-driven path planning and a lower level for path-following. A discrete event system is designed to identify situations that require planning a new path. An improved waypoint guidance algorithm and a Dubins curves based algorithm are proposed to build paths that allow the AUV to avoid collision with obstacles and to return to the reference path respectively. Both algorithms generate paths that meet the minimum turning radius constraint. A robust parameter-varying controller is designed using sublinear vector Lyapunov functions to solve the path-following problem. The performance of the developed event-based control system is demonstrated in three different simulation scenarios: with a sharp-edged obstacle, with a U-shaped obstacle, and with densely scattered obstacles. The proposed scheme does not require significant computing resources and allows for easy implementation on board. [ABSTRACT FROM AUTHOR]

The deep-sea mining vehicle is one of the key equipment for deep-sea mining system, which needs to complete tasks such as traveling on the seabed, ore crushing, ore collection, and ore transportation. During the operation of deep-sea mining vehicles, real-time mapping of the surrounding terrain environment is a necessary guarantee for safety. Forward-looking sonars can perform real-time imaging of underwater environments without being affected by water turbidity, which has been gradually used in underwater environment exploration. In this study, a novel seabed terrain mapping algorithm based on occupancy-elevation grid map for deep-sea mining vehicles, which uses a wide aperture forward-looking imaging sonar is proposed. This paper proposes an inverse sensor model for forward-looking sonar, which can map pixels in the image to the occupied probability and elevation information corresponding to the real space, and use it to generate the occupancy-elevation grid map. Then, a local occupancy-elevation grid map can be constructed as a submap with the inverse sensor model and position information from sonar odometer or other sensors. And the poses of different local submaps are optimized based on factor graph to generate the global occupancy-elevation grid map. The proposed mapping algorithm is validated by both simulation experiments and real data from forward-looking sonar during sea trials. • Proposing an inverse sensor model of the forward-looking sonar to estimate the occupied probability and elevation. • Using 2-D probabilistic space carving to reduce the redundant impact of uncertainty on obstacle areas in sonar images. • Both the simulation and the dataset from sea trials proved that the proposed method can generate a reliable occupancy-elevation grid map successfully. [ABSTRACT FROM AUTHOR]

Forward-looking sonar (FLS) has been gaining attractions in the realm of near-bottom, close-range underwater inspections, owing to its high-resolution and rapid framerate capabilities. Although automatic target recognition (ATR) algorithms have been tentatively employed for object detection tasks, the necessity for human oversight remains vital, particularly in sensitive areas. A comprehensive FLS mosaic encapsulating all relevant information is highly desired to aid experts in managing an extensive array of perception data. Yet, prior research has often presumed that FLS operates under an ideal system configuration, assuming optimal sonar imaging setups and the availability of precise positioning data. Without these assumptions, intra-frame and inter-frame artifacts would emerge, deteriorating the quality of the resultant mosaic by obscuring essential information. In this study, we propose an innovative blending method specifically for FLS mosaicing aiming preserve important information. We first develop a long–short time sliding window (LST-SW) to rectify the local statistics of raw sonar images. These statistics then serve to create a global variance map (GVM). This step helps to emphasize the useful information contained in images in the blending phase by classifying the informative and featureless pixels, thereby enhancing the quality of final mosaic. This approach is substantiated through data collected in real-world scenarios. The results show that our method can preserve more details in FLS mosaics for human inspection purposes in practice. • Analyze the intra- and inter-frame artifacts of forward-looking sonar. • Develop a method to ameliorate the influence from these artifacts on the mosaic. • The method is validated through extensive field experiments. [ABSTRACT FROM AUTHOR]

Underwater fishing nets represent a danger faced by autonomous underwater vehicles (AUVs). To avoid irreparable damage to the AUV caused by fishing nets, the AUV needs to be able to identify and locate them autonomously and avoid them in advance. Whether the AUV can avoid fishing nets successfully depends on the accuracy and efficiency of detection. In this paper, we propose an object detection multiple receptive field network (MRF-Net), which is used to recognize and locate fishing nets using forward-looking sonar (FLS) images. The proposed architecture is a center-point-based detector, which uses a novel encoder-decoder structure to extract features and predict the center points and bounding box size. In addition, to reduce the interference of reverberation and speckle noises in the FLS image, we used a series of preprocessing operations to reduce the noises. We trained and tested the network with data collected in the sea using a Gemini 720i multi-beam forward-looking sonar and compared it with state-of-the-art networks for object detection. In order to further prove that our detector can be applied to the actual detection task, we also carried out the experiment of detecting and avoiding fishing nets in real-time in the sea with the embedded single board computer (SBC) module and the NVIDIA Jetson AGX Xavier embedded system of the AUV platform in our lab. The experimental results show that in terms of computational complexity, inference time, and prediction accuracy, MRF-Net is better than state-of-the-art networks. In addition, our fishing net avoidance experiment results indicate that the detection results of MRF-Net can support the accurate operation of the later obstacle avoidance algorithm.

This paper proposes a 3D autonomous collision avoidance method based on convolutional gated recurrent units to improve the autonomy of Unmanned Underwater Vehicle (UUV). The state equations of the UUV autonomous collision avoidance system are constructed by studying its mechanism and integrating dynamic/static obstacle recognition, dynamic obstacle motion prediction, collision risk assessment, and collision avoidance. Then a multi-input single-output neural network architecture that integrates static feature extraction, dynamic time sequence modeling, and feature integration is proposed based on Convolutional Neural Networks (CNNs) and Gated Recurrent Units (GRUs) to describe the state space. CNNs extract features from sonar observation data to improve the accuracy of obstacle recognition. GRUs are combined with CNNs to capture the correlation of long-distance features and extract dynamic features. The spatial and temporal invariance of the neural network architecture enhances the fault tolerance of the UUV collision avoidance system for inputs and adaptability to observation noise and environments. Finally, simulation results show that this method is adaptable to sonar observation noise and unknown environments to solve the problem of forward-looking sonar-based UUV collision avoidance in unknown complex ocean environments. • An end-to-end neural network architecture from observations to motion control is proposed for UUV autonomous collision avoidance. • The neural network architecture is end-to-end trainable, eliminating the manual recognition and labeling of observations. • A multi-input and single-output autonomous collision avoidance neural network for UUV is proposed by combining CNNs and GRUs. • The autonomous collision avoidance method has strong adaptability to uncertain sonar observations and unknown environments. [ABSTRACT FROM AUTHOR]

The paper addresses path planning and path-following problems in an unknown complex environment for an underactuated autonomous underwater vehicle (AUV). The AUV is required to follow a given reference path represented as a sequence of smoothly joined lines and arcs, bypassing obstacles encountered on the path. A two-level control system is proposed with an upper level for event-driven path planning and a lower level for path-following. A discrete event system is designed to identify situations that require planning a new path. An improved waypoint guidance algorithm and a Dubins curves based algorithm are proposed to build paths that allow the AUV to avoid collision with obstacles and to return to the reference path respectively. Both algorithms generate paths that meet the minimum turning radius constraint. A robust parameter-varying controller is designed using sublinear vector Lyapunov functions to solve the path-following problem. The performance of the developed event-based control system is demonstrated in three different simulation scenarios: with a sharp-edged obstacle, with a U-shaped obstacle, and with densely scattered obstacles. The proposed scheme does not require significant computing resources and allows for easy implementation on board.

• Focus on three-dimensional interferometry by self-developed forward-looking sonar. • Propose a subpixel image registration algorithm based on a two-dimensional surface. • Reduce the interference of high coherence noise to achieve high accuracy. • Optimize the step search mode in traditional to achieve low computational complexity. • The experiments show the accurate of seafloor bathymetry and 3D target detection. In recent years, three-dimensional forward-looking sonar (3D FLS) has become a research hotspot. Aiming at the problem of image registration in 3D FLS interferometry, this study designs a 3D FLS receiving array using a multi-subarray structure and proposes a subpixel image registration algorithm based on a two-dimensional surface. The amplitude-phase gradient combination criterion is proposed to select the homonymous points. The main advantages of the proposed algorithm are that it can reduce the interference of high coherence noise and achieve high accuracy and low computational complexity. The effectiveness of the proposed method is verified by the simulation and tank experiments using self-developed FLS. The experimental results show that the proposed algorithm can accurately complete seafloor bathymetry and target in the field of view. The proposed algorithm can greatly improve the mapping and surveying efficiency of underwater vehicles, and provide more accurate position information for underwater obstacle. [ABSTRACT FROM AUTHOR]

Deep-sea mining system is used to collect manganese nodules or other seabed mineral resources. When the deep-sea mining vehicle works, long-term localization is important to realize efficient and safe operation. This study proposes a localization algorithm which can get accurate, real-time localization results for deep-sea mining vehicle. The forward-looking sonar is used here to estimate the motion of deep-sea mining vehicle by image matching. Stable features of objects are identified from the sonar images. The 3-D point clouds of object features are generated by the geometric relationships of the sonar projection model and then used for image matching to obtain relative pose transformation for constructing the odometer. The framework of the data fusion algorithm is based on pose graph, allowing global and relative measurements of position from multiple heterogeneous sensors to be integrated together. The proposed localization algorithm is validated through the dataset acquired during sea trials in July 2021 with "Pioneer I" in the South China Sea. The results of validation show that the proposed localization algorithm can output accurate positioning results for the deep-sea mining vehicle. • Forward-looking sonar is used here to estimate the motion of deep-sea mining vehicle and construct sonar odometer. • A localization algorithm based on pose graph framework with all kinds of underwater sensors is proposed. • The feasibility and applicability of the proposed localization algorithm is validated by sensor data from real sea trials. [ABSTRACT FROM AUTHOR]

A predictive guidance obstacle avoidance algorithm (PGOA) in unknown environments is proposed for autonomous underwater vehicle (AUV) that must adapt to multiple complex obstacle environments. Using the environmental information collected by the Forward-looking Sonar (FLS), the obstacle boundary is simplified by the convex algorithm and Bessel interpolation. Combining the predictive control secondary optimization function and the obstacle avoidance weight function, the predicting obstacle avoidance trajectory parameters are obtained. According to different types of obstacle environments, the corresponding obstacle avoidance rules are formulated. Lastly, combining with the obstacle avoidance parameters and rules, the AUV’s predicting obstacle avoidance trajectory point is obtained. Then AUV can successfully achieve obstacle avoidance using the guidance algorithm. The simulation results show that the PGOA algorithm can better predict the trajectory point of the obstacle avoidance path of AUV, and the secondary optimization function can successfully achieve collision avoidance for different complex obstacle environments. Lastly, comparing the execution efficiency and cost of different algorithms, which deal with various complex obstacle environments, simulation experiment results indicate the high efficiency and great adaptability of the proposed algorithm.

In view of a dynamic obstacle environment with motion uncertainty, we present a dynamic collision avoidance method based on the collision risk assessment and improved velocity obstacle method. First, through the fusion optimization of forward-looking sonar data, the redundancy of the data is reduced and the position, size and velocity information of the obstacles are obtained, which can provide an accurate decision-making basis for next-step collision avoidance. Second, according to minimum meeting time and the minimum distance between the obstacle and unmanned underwater vehicle (UUV), this paper establishes the collision risk assessment model, and screens key obstacles to avoid collision. Finally, the optimization objective function is established based on the improved velocity obstacle method, and a UUV motion characteristic is used to calculate the reachable velocity sets. The optimal collision speed of UUV is searched in velocity space. The corresponding heading and speed commands are calculated, and outputted to the motion control module. The above is the complete dynamic obstacle avoidance process. The simulation results show that the proposed method can obtain a better collision avoidance effect in the dynamic environment, and has good adaptability to the unknown dynamic environment. [ABSTRACT FROM AUTHOR]

Optical cameras give detailed images in clear waters. However, in dark or turbid waters, information coming from electro-optical sensors is insufficient for accurate scene perception. Imaging sonars, also known as acoustic cameras, can provide enhanced target details in these scenarios. In this paper, a computationally efficient 2D high-frequency, forward-looking sonar image simulator is presented. Stages and requirements of the image formation process are explained in detail. For the postprocessing of the returned sonar signals, a novel computation engine is proposed based on the geometric structures of the simulated surfaces. By treating all the continuous surfaces separately, the simulator is able to exactly calculate bright and shadowed zones in the 2D sonar image. [ABSTRACT FROM AUTHOR]