Search

Given the propagation characteristics of sound waves and the complexity of the underwater environment, denoising forward-looking sonar image data presents a formidable challenge. Existing studies often add noise to sonar images and then explore methods for its removal. This approach neglects the inherent complex noise in sonar images, resulting in inaccurate evaluations of traditional denoising methods and poor learning of noise characteristics by deep learning models. To address the lack of high-quality data for FLS denoising model training, we propose a simulation algorithm for forward-looking sonar data based on RGBD data. By utilizing rendering techniques and noise simulation algorithms, high-quality noise-free and noisy sonar data can be rapidly generated from existing RGBD data. Based on these data, we optimize the loss function and training process of the FLS denoising model, achieving significant improvements in noise removal and feature preservation compared to other methods. Finally, this paper performs both qualitative and quantitative analyses of the algorithm's performance using real and simulated sonar data. Compared to the latest FLS denoising models based on traditional methods and deep learning techniques, our method demonstrates significant advantages in denoising capability. All inference results for the Marine Debris Dataset (MDD) have been made open source, facilitating subsequent research and comparison. [ABSTRACT FROM AUTHOR]

【Objective】Cage biometrics and statistics are one of the key reference factors for marine pasture farming management. Aiming at the interference of reverberation noise and complex background, this paper constructs fish detection data sets under different lighting conditions, and uses forward-looking sonar imaging technology to propose a fish recognition lightweight model based on YOLOV5-MobilenetV3 and sonar images (LAPR-Net) to realize fish recognition in water cages in turbid or dark scenes.【Method】Taking tilapia as the research object, based on the frame structure of the YOLOV5 model, the backbone network module ado pts the lightweight Mob ileNetV3 bneck block, using the linear bottleneck inverse residual structure and depth separable convolution extract the features of fish in sonar images, applying the attention mechanism SE-Net to obtain multi-scale semantic features of sonar images and enhance the correlation between features; the neck network adopts the path aggregation network structure to perform multi-scale fusion of target features, to enhance the feature fusion ability; the prediction part adopts the maximum local search based on the non-maximum suppression method, removes the redundant detection frame, screens the detection frame with the highest confidence, and finally outputs and displays the detection result of the fish, including the position, category and detection probability of detecting an object.【Result】Four other mainstream detection models were selected for comparative experiments, including YOLOV3-ting (Darknet53), YOLOV5 (CSPdarknet53), YOLOV5 (Repvgg), and YOLOV5s (Transformer). It proposes the model parameter quantityof 3 545 453, FLOPs of 6.3 G, and the mAP of 0.957, and the average inference speed of each picture of the model is 0.08868 s. Compared with the YOLOV5 model, the mAP of the improved model has increased by 9.7%.【Conclusion】The proposed network improves the speed of training and recognition, reduces the requirements for hardware equipment, and provides a reference for the detection model of cage cultured fish in marine pastures. [ABSTRACT FROM AUTHOR]

In subsea environments, sound navigation and ranging (SONAR) images are widely used for exploring and monitoring infrastructures due to their robustness and insensitivity to low-light conditions. However, their quality can degrade during acquisition and transmission, where standard SONAR image processing techniques can hardly produce high-quality outcomes. An effective image quality assessment (IQA) method can assess their usefulness and aid to develop refinement techniques by identifying the degradation issues, ensuring the reliability of SONAR data. Existing methods often fail to account for degradations from noise, distortion, and resolution changes simultaneously. To address this challenge, we propose a new blind quality assessment method that measures the overall quality of SONAR images by quantifying both the perceptual and utility qualities using the micro- and macro-scale texture and contour features derived from the wavelet domain. By combining the local binary pattern (LBP) micro-scale texture features with the proposed histograms of Schmid Gabor-like edge maps as macro-scale features, a support vector regression model is learned to map from these features to subjective quality scores. Extensive experiments have demonstrated the superiority of our method over existing SONAR IQA techniques on distorted and reconstructed super-resolution side-scan, acoustic lens, and forward-looking SONAR images. Specifically, our method achieves Pearson's and Spearman's correlation metrics of 0.8616 and 0.8541, respectively, for distorted SONAR images, demonstrating improvements of 4.69% and 4.8%. For reconstructed super-resolution SONAR images, our method attains correlation metrics of 0.9415 and 0.9408, reflecting improvements of 0.8% and 1.6% over the second-best method, respectively. To facilitate ease of access, a comprehensive list of key abbreviations and their full names is provided in Table A.9 in the Appendix section. The source code of the proposed method will be shared at https://github.com/hfarhaditolie/BSIQA. • Micro-/macro- wavelet-domain texture/gradient features for representing SONAR images. • Gradient-maps based contour descriptors from Schmid Gabor-like filter bank in wavelet. • Optimised predicting SONAR image quality by training various regression models. [ABSTRACT FROM AUTHOR]

Precise localization of unmanned underwater vehicle (UUV) on multi-beam forward-looking sonar (MFLS) is a key technology in underwater robotic exploration. However, large appearance change and weak features of targets in MFLS image, pose a serious challenge on target detection. This paper proposes an efficient UUV detection model with an enhanced training scheme and convolutional block attention mechanism. The proposed training scheme is composed of dual-path gradient information optimization and adaptive loss adjustment, which improves the network's detection capabilities and the underwater environment adaptability. The convolutional block attention module (CBAM) is developed for extracting channel and spatially weighted feature maps, and providing a mechanism to adaptive feature refinement. We construct a comprehensive sonar detection dataset from field experiments. By comparing existing state-of-the-art detection algorithms, our proposed UUVDNet outperforms Faster R-CNN, Cascade R-CNN, CenterNet, SSD, YOLOv8n, DPFIN, and MBSNN in terms of mAP 50 – 95 by 39.5%, 35.7%, 8.4%, 14.0%, 2.5%, 13.6%, and 21.8%, respectively. Furthermore, our inference model size is the most compact among all detection models, with a size of 5.1M. Final ablation experiments effectively showcase the proposed components' efficiency. [Display omitted] • A novel detection framework called UUVDNet considering enhanced training scheme and attention mechanism is proposed for underwater FLS image detection task, which improves the detection precision and speed. • Enhanced training with dual-path optimization and DCIoU loss boosts underwater detection. • The convolutional block attention module is proposed to refine the spatial and channel feature maps of deep networks. • A new multibeam forward-looking sonar target detection dataset has been constructed, encompassing a rich variety of detection environments and UUV target quantities. [ABSTRACT FROM AUTHOR]

Forward-looking sonar is a technique widely used for underwater detection. However, most sonar images have underwater noise and low resolution due to their acoustic properties. In recent years, the semantic segmentation model U-Net has shown excellent segmentation performance, and it has great potential in forward-looking sonar image segmentation. However, forward-looking sonar images are affected by noise, which prevents the existing U-Net model from segmenting small objects effectively. Therefore, this study presents a forward-looking sonar semantic segmentation model called Feature Pyramid U-Net with Attention (FPUA). This model uses residual blocks to improve the training depth of the network. To improve the segmentation accuracy of the network for small objects, a feature pyramid module combined with an attention structure is introduced. This improves the model's ability to learn deep semantic and shallow detail information. First, the proposed model is compared against other deep learning models and on two datasets, of which one was collected in a tank environment and the other was collected in a real marine environment. To further test the validity of the model, a real forward-looking sonar system was devised and employed in the lake trials. The results show that the proposed model performs better than the other models for small-object and few-sample classes and that it is competitive in semantic segmentation of forward-looking sonar images. [ABSTRACT FROM AUTHOR]

Imaging and mapping sonars such as forward-looking sonars (FLS) and side-scan sonars (SSS) are sensors frequently used onboard autonomous underwater vehicles. To acquire information from around the vehicle, it is desirable for these sonar systems to insonify a large area; thus, the sonar transmit beampattern should have a wide field of view. In this work, we study the problem of the optimization of wide transmission beampatterns. We consider the conventional phased-array beampattern design problem where all array elements transmit an identical waveform. The complex weight vector is adjusted to create the desired beampattern shape. In our experiments, we consider wide transmission beampatterns (≥20 ∘ ) with uniform output power. In this paper, we introduce a new iterative-convex optimization method for narrowband linear phased arrays and compare it to existing approaches for convex and concave–convex optimization. In the iterative-convex method, the phase of the weight parameters is allowed to be complex as in disciplined convex–concave programming (DCCP). Comparing the iterative-convex optimization method and DCCP to the standard convex optimization, we see that the former methods archive optimized beampatterns closer to the desired beampatterns. Furthermore, for the same number of iterations, the proposed iterative-convex method achieves optimized beampatterns, which are closer to the desired beampattern than the beampatterns achieved by optimization with DCCP. [ABSTRACT FROM AUTHOR]

Forward-looking sonar (FLS) image registration is a key step in many underwater applications such as underwater target detection, ocean observation, and mapping. However, low resolution, low signal-to-noise ratio, and the complex nonlinear transformation relationship between FLS images from two different viewpoints have brought great challenges to register them. In order to better cope with this challenge, we propose a global perspective and local flow registration (GPLFR) method for FLS images. GPLFR consists of two networks, i.e., a regression correction network (RCNet) and a deformable network (IRRDNet) with the iterative refinement of the residual. For a given pair of FLS images, RCNet is used to estimate the global transformation parameters to achieve global registration, and then, IRRDNet is used to estimate the deformation field or flow field to realize local alignment. The experimental results on real FLS image and 2D face expression image registration tasks demonstrate the effectiveness and robustness of the proposed method. [ABSTRACT FROM AUTHOR]

Three-dimensional reconstruction is a crucial technique for mapping and object-search tasks, but it is challenging in sonar imaging because of the nature of acoustics. In underwater sensing, many advanced studies have introduced approaches that have included feature-based methods and multiple imaging at different locations. However, most existing methods are prone to environmental conditions, and they are not adequate for continuous data acquisition on moving autonomous underwater vehicles (AUVs). This paper proposes a sensor fusion method for 3D reconstruction using acoustic sonar data with two sonar devices that provide complementary features. The forward-looking multibeam sonar (FLS) is an imaging sonar capable of short-range scanning with a high horizontal resolution, and the profiling sonar (PS) is capable of middle-range scanning with high reliability in vertical information. Using both sonars, which have different data acquisition planes and times, we propose a probabilistic sensor fusion method. First, we extract the region of interest from the background and develop a sonar measurement model. Thereafter, we utilize the likelihood field generated by the PS and estimate the elevation ambiguity using importance sampling. We also present the evaluation of our method in a ray-tracing-based sonar simulation environment and the generation of the pointclouds. The experimental results indicate that the proposed method can provide a better accuracy than that of the conventional method. Because of the improved accuracy of the generated pointclouds, this method can be expanded for pointcloud-based mapping and classification methods. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel method to generate realistic acoustic datasets for forward-looking sonars. A forward-looking sonar, which is also known as an acoustic camera, outperforms other imaging sensor when applied in underwater tasks as it can provide more accurate and detailed information about the environment, even in dark or turbid water. However, the difficulty and high cost of acquiring acoustic images in real experiments encourage researchers to consider the generation of simulated acoustic image datasets. In particular, deep learning-based methods demonstrated high performance in computer vision tasks, such as in object detection. However, a large dataset is necessary in most cases. In the proposed method, we first build a novel user-friendly acoustic image simulator based on 3D modeling software. Then, the CycleGAN is applied to generate realistic acoustic images based on the generated dataset from the simulator. The experimental results demonstrate that our method can generate effective and realistic acoustic datasets with relatively simple operations. [ABSTRACT FROM AUTHOR]

The deep-sea mining vehicle is one of the key equipment for deep-sea mining system, which needs to complete tasks such as traveling on the seabed, ore crushing, ore collection, and ore transportation. During the operation of deep-sea mining vehicles, real-time mapping of the surrounding terrain environment is a necessary guarantee for safety. Forward-looking sonars can perform real-time imaging of underwater environments without being affected by water turbidity, which has been gradually used in underwater environment exploration. In this study, a novel seabed terrain mapping algorithm based on occupancy-elevation grid map for deep-sea mining vehicles, which uses a wide aperture forward-looking imaging sonar is proposed. This paper proposes an inverse sensor model for forward-looking sonar, which can map pixels in the image to the occupied probability and elevation information corresponding to the real space, and use it to generate the occupancy-elevation grid map. Then, a local occupancy-elevation grid map can be constructed as a submap with the inverse sensor model and position information from sonar odometer or other sensors. And the poses of different local submaps are optimized based on factor graph to generate the global occupancy-elevation grid map. The proposed mapping algorithm is validated by both simulation experiments and real data from forward-looking sonar during sea trials. • Proposing an inverse sensor model of the forward-looking sonar to estimate the occupied probability and elevation. • Using 2-D probabilistic space carving to reduce the redundant impact of uncertainty on obstacle areas in sonar images. • Both the simulation and the dataset from sea trials proved that the proposed method can generate a reliable occupancy-elevation grid map successfully. [ABSTRACT FROM AUTHOR]

Forward-looking sonar (FLS) has been gaining attractions in the realm of near-bottom, close-range underwater inspections, owing to its high-resolution and rapid framerate capabilities. Although automatic target recognition (ATR) algorithms have been tentatively employed for object detection tasks, the necessity for human oversight remains vital, particularly in sensitive areas. A comprehensive FLS mosaic encapsulating all relevant information is highly desired to aid experts in managing an extensive array of perception data. Yet, prior research has often presumed that FLS operates under an ideal system configuration, assuming optimal sonar imaging setups and the availability of precise positioning data. Without these assumptions, intra-frame and inter-frame artifacts would emerge, deteriorating the quality of the resultant mosaic by obscuring essential information. In this study, we propose an innovative blending method specifically for FLS mosaicing aiming preserve important information. We first develop a long–short time sliding window (LST-SW) to rectify the local statistics of raw sonar images. These statistics then serve to create a global variance map (GVM). This step helps to emphasize the useful information contained in images in the blending phase by classifying the informative and featureless pixels, thereby enhancing the quality of final mosaic. This approach is substantiated through data collected in real-world scenarios. The results show that our method can preserve more details in FLS mosaics for human inspection purposes in practice. • Analyze the intra- and inter-frame artifacts of forward-looking sonar. • Develop a method to ameliorate the influence from these artifacts on the mosaic. • The method is validated through extensive field experiments. [ABSTRACT FROM AUTHOR]

This paper proposes a 3D autonomous collision avoidance method based on convolutional gated recurrent units to improve the autonomy of Unmanned Underwater Vehicle (UUV). The state equations of the UUV autonomous collision avoidance system are constructed by studying its mechanism and integrating dynamic/static obstacle recognition, dynamic obstacle motion prediction, collision risk assessment, and collision avoidance. Then a multi-input single-output neural network architecture that integrates static feature extraction, dynamic time sequence modeling, and feature integration is proposed based on Convolutional Neural Networks (CNNs) and Gated Recurrent Units (GRUs) to describe the state space. CNNs extract features from sonar observation data to improve the accuracy of obstacle recognition. GRUs are combined with CNNs to capture the correlation of long-distance features and extract dynamic features. The spatial and temporal invariance of the neural network architecture enhances the fault tolerance of the UUV collision avoidance system for inputs and adaptability to observation noise and environments. Finally, simulation results show that this method is adaptable to sonar observation noise and unknown environments to solve the problem of forward-looking sonar-based UUV collision avoidance in unknown complex ocean environments. • An end-to-end neural network architecture from observations to motion control is proposed for UUV autonomous collision avoidance. • The neural network architecture is end-to-end trainable, eliminating the manual recognition and labeling of observations. • A multi-input and single-output autonomous collision avoidance neural network for UUV is proposed by combining CNNs and GRUs. • The autonomous collision avoidance method has strong adaptability to uncertain sonar observations and unknown environments. [ABSTRACT FROM AUTHOR]

• Focus on three-dimensional interferometry by self-developed forward-looking sonar. • Propose a subpixel image registration algorithm based on a two-dimensional surface. • Reduce the interference of high coherence noise to achieve high accuracy. • Optimize the step search mode in traditional to achieve low computational complexity. • The experiments show the accurate of seafloor bathymetry and 3D target detection. In recent years, three-dimensional forward-looking sonar (3D FLS) has become a research hotspot. Aiming at the problem of image registration in 3D FLS interferometry, this study designs a 3D FLS receiving array using a multi-subarray structure and proposes a subpixel image registration algorithm based on a two-dimensional surface. The amplitude-phase gradient combination criterion is proposed to select the homonymous points. The main advantages of the proposed algorithm are that it can reduce the interference of high coherence noise and achieve high accuracy and low computational complexity. The effectiveness of the proposed method is verified by the simulation and tank experiments using self-developed FLS. The experimental results show that the proposed algorithm can accurately complete seafloor bathymetry and target in the field of view. The proposed algorithm can greatly improve the mapping and surveying efficiency of underwater vehicles, and provide more accurate position information for underwater obstacle. [ABSTRACT FROM AUTHOR]

Deep-sea mining system is used to collect manganese nodules or other seabed mineral resources. When the deep-sea mining vehicle works, long-term localization is important to realize efficient and safe operation. This study proposes a localization algorithm which can get accurate, real-time localization results for deep-sea mining vehicle. The forward-looking sonar is used here to estimate the motion of deep-sea mining vehicle by image matching. Stable features of objects are identified from the sonar images. The 3-D point clouds of object features are generated by the geometric relationships of the sonar projection model and then used for image matching to obtain relative pose transformation for constructing the odometer. The framework of the data fusion algorithm is based on pose graph, allowing global and relative measurements of position from multiple heterogeneous sensors to be integrated together. The proposed localization algorithm is validated through the dataset acquired during sea trials in July 2021 with "Pioneer I" in the South China Sea. The results of validation show that the proposed localization algorithm can output accurate positioning results for the deep-sea mining vehicle. • Forward-looking sonar is used here to estimate the motion of deep-sea mining vehicle and construct sonar odometer. • A localization algorithm based on pose graph framework with all kinds of underwater sensors is proposed. • The feasibility and applicability of the proposed localization algorithm is validated by sensor data from real sea trials. [ABSTRACT FROM AUTHOR]

In view of a dynamic obstacle environment with motion uncertainty, we present a dynamic collision avoidance method based on the collision risk assessment and improved velocity obstacle method. First, through the fusion optimization of forward-looking sonar data, the redundancy of the data is reduced and the position, size and velocity information of the obstacles are obtained, which can provide an accurate decision-making basis for next-step collision avoidance. Second, according to minimum meeting time and the minimum distance between the obstacle and unmanned underwater vehicle (UUV), this paper establishes the collision risk assessment model, and screens key obstacles to avoid collision. Finally, the optimization objective function is established based on the improved velocity obstacle method, and a UUV motion characteristic is used to calculate the reachable velocity sets. The optimal collision speed of UUV is searched in velocity space. The corresponding heading and speed commands are calculated, and outputted to the motion control module. The above is the complete dynamic obstacle avoidance process. The simulation results show that the proposed method can obtain a better collision avoidance effect in the dynamic environment, and has good adaptability to the unknown dynamic environment. [ABSTRACT FROM AUTHOR]

Optical cameras give detailed images in clear waters. However, in dark or turbid waters, information coming from electro-optical sensors is insufficient for accurate scene perception. Imaging sonars, also known as acoustic cameras, can provide enhanced target details in these scenarios. In this paper, a computationally efficient 2D high-frequency, forward-looking sonar image simulator is presented. Stages and requirements of the image formation process are explained in detail. For the postprocessing of the returned sonar signals, a novel computation engine is proposed based on the geometric structures of the simulated surfaces. By treating all the continuous surfaces separately, the simulator is able to exactly calculate bright and shadowed zones in the 2D sonar image. [ABSTRACT FROM AUTHOR]

In this paper, the scale ambiguity in underwater monocular visual odometry is solved with 2-D sonar measurements obtained from three viewpoints. Each measurement value is formulated as geometrical constraint and the recovered scale can be expressed in closed-form. Different from the existing method which assumes the features being on the same plane in order to meet the pre-condition of quasi-homography transformation, our method allows the observation points to be distributed spatially (not limited to a plane), thus extending the proposed method to more general application environments. An error contribution criterion of sonar measurement is proposed to weight the constraints, which further reduces the influence of noisy input. Experiments are performed in both simulated and real environments. For nonplanar case, the proposed method gives improved accuracy after employing constraint weighting. As for planar case, our method achieves comparable accuracy to the existing method, while the time consumption is much lower. • Forward looking sonar is used to estimate translation scale for monocular visual odometry. • Sonar measurements at three viewpoints are used to establish geometrical constraints. • The method is applicable to both general scene and planar scene. • Constraint weighting is proposed to improve the estimated results. [ABSTRACT FROM AUTHOR]

Underwater fishing nets represent a danger faced by autonomous underwater vehicles (AUVs). To avoid irreparable damage to the AUV caused by fishing nets, the AUV needs to be able to identify and locate them autonomously and avoid them in advance. Whether the AUV can avoid fishing nets successfully depends on the accuracy and efficiency of detection. In this paper, we propose an object detection multiple receptive field network (MRF-Net), which is used to recognize and locate fishing nets using forward-looking sonar (FLS) images. The proposed architecture is a center-point-based detector, which uses a novel encoder-decoder structure to extract features and predict the center points and bounding box size. In addition, to reduce the interference of reverberation and speckle noises in the FLS image, we used a series of preprocessing operations to reduce the noises. We trained and tested the network with data collected in the sea using a Gemini 720i multi-beam forward-looking sonar and compared it with state-of-the-art networks for object detection. In order to further prove that our detector can be applied to the actual detection task, we also carried out the experiment of detecting and avoiding fishing nets in real-time in the sea with the embedded single board computer (SBC) module and the NVIDIA Jetson AGX Xavier embedded system of the AUV platform in our lab. The experimental results show that in terms of computational complexity, inference time, and prediction accuracy, MRF-Net is better than state-of-the-art networks. In addition, our fishing net avoidance experiment results indicate that the detection results of MRF-Net can support the accurate operation of the later obstacle avoidance algorithm. [ABSTRACT FROM AUTHOR]

• A novel data set containing underwater sonar images. • Two novel deep learning architectures to detect underwater objects. • Three novel representations for bounding box regression. Autonomous underwater vehicles (AUVs) are often used to inspect the condition of submerged structures in oil and gas fields. Because the use of global positioning systems to aid AUV navigation is not feasible, object detection is an alternative method of supporting underwater inspection missions by detecting landmarks. Objects are detected not only to plan the trajectory of the AUVs, but their inspection can be the ultimate goal of the mission. In both cases, detecting an object's distance and orientation with respect to the AUV provides clues for the vehicle's navigation. Accordingly, we introduce a novel multi-object detection system that outputs object position and rotation from sonar images to support AUV navigation. To achieve this aim, two novel convolutional neural network-based architectures are proposed to detect and estimate rotated bounding boxes: an end-to-end network (RBoxNet), and a pipeline comprised of two networks (YOLOv2+RBoxDNet). Both proposed networks are structured from one of three novel representations of rotated bounding boxes regressed deep inside. Experimental analyses were performed by comparing several configurations of our proposed methods (by varying the backbone, regression representation, and architecture) with state-of-the-art methods using real sonar images. Results showed that RBoxNet presents the optimum trade-off between accuracy and speed, reaching an averaged mAP@[.5,.95] of 90.3% at 8.58 frames per second (FPS), while YOLOv2+RBoxDNet is the fastest solution, running at 16.19 FPS but with a lower averaged mAP@[.5,.95] of 77.5%. Both proposed methods are robust to additive Gaussian noise variations, and can detect objects even when the noise level is up to 0.10. [ABSTRACT FROM AUTHOR]

A predictive guidance obstacle avoidance algorithm (PGOA) in unknown environments is proposed for autonomous underwater vehicle (AUV) that must adapt to multiple complex obstacle environments. Using the environmental information collected by the Forward-looking Sonar (FLS), the obstacle boundary is simplified by the convex algorithm and Bessel interpolation. Combining the predictive control secondary optimization function and the obstacle avoidance weight function, the predicting obstacle avoidance trajectory parameters are obtained. According to different types of obstacle environments, the corresponding obstacle avoidance rules are formulated. Lastly, combining with the obstacle avoidance parameters and rules, the AUV's predicting obstacle avoidance trajectory point is obtained. Then AUV can successfully achieve obstacle avoidance using the guidance algorithm. The simulation results show that the PGOA algorithm can better predict the trajectory point of the obstacle avoidance path of AUV, and the secondary optimization function can successfully achieve collision avoidance for different complex obstacle environments. Lastly, comparing the execution efficiency and cost of different algorithms, which deal with various complex obstacle environments, simulation experiment results indicate the high efficiency and great adaptability of the proposed algorithm. [ABSTRACT FROM AUTHOR]