Search

In complex environments with long-term changes such as light, seasonal, and viewpoint changes, robust, accurate, and high-frequency global positioning based on light detection and ranging (LiDAR) map is still a challenge, which is crucial for autonomous vehicles or robots. To this end, a novel observation model that relies on the siamese multitask convolutional neural networks (CNNs) with multimodule cascaded is creatively presented in this article. In particular, a new pseudoimage representing LiDAR submap is designed to enrich scene texture and enhance rotation invariance. Besides, novel siamese CNNs that are coupled by NeXtVLAD and long short-term memory is designed for the first time, which can reliably predict similarity and quaternion at the same time. Finally, the predicted quaternion observation is integrated into the extended Kalman filter framework for multisensor fusion to achieve robust high-frequency global pose estimation. Extensive evaluations on KITTI, NCLT, and real-world datasets suggest that the proposed method not only obtains the remarkable precision-recall performance, but also effectively enhances and improves the robustness and accuracy of long-term positioning. [ABSTRACT FROM AUTHOR]

Good responsiveness of autonomous vehicle computing systems is crucial to safety and performance of the vehicles. For example, an autonomous vehicle (AV) may cause an accident if the end-to-end latency from sensing a pedestrian to emergency stop is too high. However, the AV software stacks are too complex to probabilistically analye the end-to-end latency on a multi-core system. They consist of a graph of tasks with different periods, and have a large variability in the task execution times, which may lead to the maximum core utilization $U^{\max }$ U max greater than 1.0 on some cores. This paper proposes a novel stochastic analysis of the end-to-end latency over the AV stacks that allows $U^{\max }$ U max to exceed 1.0 on each core. The proposed analysis models the entire stack as a graph of task graphs under a multi-core partitioned scheduling and provides a probabilistic guarantee that the analyzed latency distribution upper-bounds the one observed from a real system under the assumption of independent task execution times. Using the Autoware stack with inter-task dependent execution times, it is shown that our analysis, combined with a task grouping to mitigate the inter-task correlations, can give a latency distribution for each task path that almost upper-bounds the observed one. [ABSTRACT FROM AUTHOR]

Emerging autonomous vehicles equip multiple high-throughput sensors to enable automatic driving, such as multiline lidars and high-definition cameras. Existing automotive embedded systems usually employ software stacks to receive and preprocess high-throughput sensor data, which brings high latency and CPU consumption. Most research is devoted to accelerators for data processing but ignores the latency overhead caused by sensor data access. Therefore, this article proposes SENTunnel to build fast path from sensors to the corresponding processing units by offloading redundant software stacks into hardware. Specifically, SENTunnel builds fast path for sensor data access to processors/accelerators through two hardware modules. First, the unified access module is used to receive, parse, and transmit raw sensor data. Second, SENTunnel performs necessary preprocessing of different sensor data with the preprocessors module. Based on the design of SENTunnel, we implement a prototype for accessing the data of multiline lidars to the processor and a dedicated accelerator on FPGA. Experimental results indicate that SENTunnel reduces the latency by 55.5% for the data path to processors and reduces the CPU usage caused by the preprocessing driver by 45.9% on average. Compared to the original and partially offloaded data path to accelerators, SENTunnel reduces the latency by 93.8% and 93%, respectively, and eliminates the CPU costs. [ABSTRACT FROM AUTHOR]

The increasing availability of multimodal data holds many promises for developments in millimeter-wave (mmWave) multiple-antenna systems by harnessing the potential for enhanced situational awareness. Specifically, inclusion of non-RF modalities to complement RF-only data in communications-related decisions like beam selection may speed up decision making in situations where an exhaustive search, spanning all candidate options, is required by the standard. However, to accelerate research in this topic, there is a need to collect real-world datasets in a principled manner. This article presents an experimentally obtained dataset, composed of 23 GB of data, which aids in beam selection in vehicle-to-everything mmWave bands, with the goal of facilitating machine learning (ML) in the wireless communication required for autonomous driving. Beyond this specific example, the article describes methodologies of creating such datasets that use time synchronized and heterogeneous types of LiDAR, GPS, and camera images, paired with the RF ground truth data of selected beams in the mmWave band. While we use beam selection as the primary demonstrator, we also discuss how multimodal datasets may be used in other ML-based PHY-layer optimization areas, such as beamforming and localization. [ABSTRACT FROM AUTHOR]

For drone vision and navigation, low-power 3-D depth sensing with robust operations against strong/weak light and various weather conditions is crucial. CMOS image sensor (CIS) and light detection and ranging (LiDAR) can provide high-fidelity imaging. However, CIS lacks depth sensing and has difficulty in low light conditions. LiDAR is expensive with issues of dealing with strong direct interference sources. Ultrasound imaging system (UIS), on the other hand, is robust in various weather and light conditions and is cost-effective. However, in air channel, it often suffers from long image reconstruction latency and low framerate. To address these issues, we present a UIS application-specific integrated circuit (ASIC) that adopts the one-shot transmitter (TX) and on-chip per-voxel receiver (RX) beamfocusing (PV-RXBF) image reconstruction scheme. The ASIC adopts the designs of fully differential charge-reuse high-voltage TX (FDCR-HVTX), digital back-end (DBE), and an on-chip power management unit (PMU). FDCR-HVTX generates 28 $\text{V}_{\mathrm {pp}}$ pulses and reduces the average power consumption by 25% by charge reuse (CR). The DBE achieves 7.76- $\mu \text{s}$ processing latency and 9.83M-FocalPoint/s throughput to effectively translate real-time 3-D image streaming at 24 frames/s. A prototype UIS, with an $8\times 8$ bulk piezo transducer array, is assembled with the proposed ASIC and a wireless data transmission module [field-programmable gate array (FPGA) + ESP32] on an entry-level consumer drone, and the real-time wireless 3-D image streaming at 24 frames/s with a range of 7 m is verified while the drone is flying. The ASIC implemented in 180-nm 1P6M Standard CMOS occupies 32.5 mm2 and consumes 142.3 mW. [ABSTRACT FROM AUTHOR]

This article presents a flash light detection and ranging (LiDAR) sensor featuring an in-pixel histogramming time-to-digital converter (hTDC) based on a delta-intensity quaternary search (DIQS) technique. The proposed 12-b DIQS hTDC is a two-step converter consisting of a 6-b coarse hTDC and a 7-b fine hTDC with 1-b redundancy. The DIQS hTDC synthesizes depth maps with three subframes from the coarse mode and a single subframe from the fine mode, achieving 100-ps resolution without a clock frequency of a few gigahertz. The DIQS repeats dividing the time range of a current step into four periods and finding the location where a target object is placed by comparing the number of events in each period, which is similar to the binary search method but doubles its operating speed. Two time-of-flight (ToF) bits are consecutively determined in every coarse step, and seven ToF bits are estimated by the indirect ToF technique with photon counts. An up–down counter is employed to reduce the memory size by half and enable the delta-intensity technique that can extend the dynamic range by suppressing the uniform background light. The prototype LiDAR with an $80\times60$ pixel array was fabricated in a 110-nm CMOS image sensor (CIS) process and fully characterized. The maximum detectable range is measured to 45 m with a success rate of 100% at night and 60% under 70-klux background light. The depth accuracy and precision are 2.5 and 1.5 cm from 3 to 4.5 m indoor, respectively, and the precision is maintained to 1.8 cm for the target located at a 1.5-m distance under 60-klux background light. Inherent time-gating and differential signaling of the DIQS hTDC effectively suppress common-mode noise, accomplishing real-time acquisition of depth images with 30 frames/s in a 9-m range at 30-klux background light. [ABSTRACT FROM AUTHOR]

Recent advances in robotics and deep learning demonstrate promising 3-D perception performances via fusing the light detection and ranging (LiDAR) sensor and camera data, where both spatial calibration and temporal synchronization are generally required. While the LiDAR–camera calibration problem has been actively studied during the past few years, LiDAR–camera synchronization has been less studied and mainly addressed by employing a conventional pipeline consisting of clock synchronization and temporal synchronization. The conventional pipeline has certain potential limitations, which have not been sufficiently addressed and could be a bottleneck for the potential wide adoption of low-cost LiDAR–camera platforms. Different from the conventional pipeline, in this article, we propose the LiCaS3, the first deep-learning-based framework, for the LiDAR–camera synchronization task via self-supervised learning. The proposed LiCaS3 does not require hardware synchronization or extra annotations and can be deployed both online and offline. Evaluated on both the KITTI and Newer College datasets, the proposed method shows promising performances. The code will be publicly available at https://github.com/KleinYuan/LiCaS3. [ABSTRACT FROM AUTHOR]

State-of-the-art methods for driving-scene LiDAR-based perception (including point cloud semantic segmentation, panoptic segmentation and 3D detection, etc.) often project the point clouds to 2D space and then process them via 2D convolution. Although this cooperation shows the competitiveness in the point cloud, it inevitably alters and abandons the 3D topology and geometric relations. A natural remedy is to utilize the 3D voxelization and 3D convolution network. However, we found that in the outdoor point cloud, the improvement obtained in this way is quite limited. An important reason is the property of the outdoor point cloud, namely sparsity and varying density. Motivated by this investigation, we propose a new framework for the outdoor LiDAR segmentation, where cylindrical partition and asymmetrical 3D convolution networks are designed to explore the 3D geometric pattern while maintaining these inherent properties. The proposed model acts as a backbone and the learned features from this model can be used for downstream tasks such as point cloud semantic and panoptic segmentation or 3D detection. In this paper, we benchmark our model on these three tasks. For semantic segmentation, we evaluate the proposed model on several large-scale datasets, i.e., SemanticKITTI, nuScenes and A2D2. Our method achieves the state-of-the-art on the leaderboard of SemanticKITTI (both single-scan and multi-scan challenge), and significantly outperforms existing methods on nuScenes and A2D2 dataset. Furthermore, the proposed 3D framework also shows strong performance and good generalization on LiDAR panoptic segmentation and LiDAR 3D detection. [ABSTRACT FROM AUTHOR]

Semantic scene completion is the task of jointly estimating 3D geometry and semantics of objects and surfaces within a given extent. This is a particularly challenging task on real-world data that is sparse and occluded. We propose a scene segmentation network based on local Deep Implicit Functions as a novel learning-based method for scene completion. Unlike previous work on scene completion, our method produces a continuous scene representation that is not based on voxelization. We encode raw point clouds into a latent space locally and at multiple spatial resolutions. A global scene completion function is subsequently assembled from the localized function patches. We show that this continuous representation is suitable to encode geometric and semantic properties of extensive outdoor scenes without the need for spatial discretization (thus avoiding the trade-off between level of scene detail and the scene extent that can be covered). We train and evaluate our method on semantically annotated LiDAR scans from the Semantic KITTI dataset. Our experiments verify that our method generates a powerful representation that can be decoded into a dense 3D description of a given scene. The performance of our method surpasses the state of the art on the Semantic KITTI Scene Completion Benchmark in terms of geometric completion intersection-over-union (IoU). [ABSTRACT FROM AUTHOR]

Collaboration localization and mapping in GPS-denied environments is a challenging problem, due to the computing power and communication bandwidth constraints of onboard equipment. In this article, a voxel-based localization and mapping system is developed, which achieves real-time trajectory estimation for the multirobot system utilizing only onboard sensors, computing power, and communication module. The key idea of the voxel-based matching approach is to formulate the registration problem as a maximum a posteriori (MAP) problem and to simplify it to a nonlinear least-squares problem using Gauss approximation and principal component analysis (PCA). The voxel-based matching method significantly improves the computational efficiency and reduces the communication bandwidth of the localization and mapping system. The effectiveness of the proposed method is demonstrated through experiments in real-world environments for various scenarios. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel system named Disp R-CNN for 3D object detection from stereo images. Many recent works solve this problem by first recovering point clouds with disparity estimation and then apply a 3D detector. The disparity map is computed for the entire image, which is costly and fails to leverage category-specific prior. In contrast, we design an instance disparity estimation network (iDispNet) that predicts disparity only for pixels on objects of interest and learns a category-specific shape prior for more accurate disparity estimation. To address the challenge from scarcity of disparity annotation in training, we propose to use a statistical shape model to generate dense disparity pseudo-ground-truth without the need of LiDAR point clouds, which makes our system more widely applicable. Experiments on the KITTI dataset show that, when LiDAR ground-truth is not used at training time, Disp R-CNN outperforms previous state-of-the-art methods based on stereo input by 20 percent in terms of average precision for all categories. The code and pseudo-ground-truth data are available at the project page: https://github.com/zju3dv/disprcnn. [ABSTRACT FROM AUTHOR]

Stereo matching is one of the longest-standing problems in computer vision with close to 40 years of studies and research. Throughout the years the paradigm has shifted from local, pixel-level decision to various forms of discrete and continuous optimization to data-driven, learning-based methods. Recently, the rise of machine learning and the rapid proliferation of deep learning enhanced stereo matching with new exciting trends and applications unthinkable until a few years ago. Interestingly, the relationship between these two worlds is two-way. While machine, and especially deep, learning advanced the state-of-the-art in stereo matching, stereo itself enabled new ground-breaking methodologies such as self-supervised monocular depth estimation based on deep networks. In this paper, we review recent research in the field of learning-based depth estimation from single and binocular images highlighting the synergies, the successes achieved so far and the open challenges the community is going to face in the immediate future. [ABSTRACT FROM AUTHOR]

Due to the large amount of data involved in the three-dimensional (3D) LiDAR point clouds, point cloud compression (PCC) becomes indispensable to many real-time applications. In autonomous driving of connected vehicles for example, point clouds are constantly acquired along the time and subjected to be compressed. Among the existing PCC methods, very few of them have effectively removed the temporal redundancy inherited in the point clouds. To address this issue, a novel lossy LiDAR PCC system is proposed in this paper, which consists of the inter-frame coding and the intra-frame coding. For the former, a deep-learning approach is proposed to conduct bi-directional frame prediction using an asymmetric residual module and 3D space-time convolutions; the proposed network is called the bi-directional prediction network (BPNet). For the latter, a novel range-adaptive floating-point coding (RAFC) algorithm is proposed for encoding the reference frames and the B-frame prediction residuals in the 32-bit floating-point precision. Since the pixel-value distribution of these two types of data are quite different, various encoding modes are designed for providing adaptive selection. Extensive simulation experiments have been conducted using multiple point cloud datasets, and the results clearly show that our proposed PCC system consistently outperforms the state-of-the-art MPEG G-PCC in terms of data fidelity and localization, while delivering real-time performance. [ABSTRACT FROM AUTHOR]

We present an approach to automatically generate semantic labels for real recordings of automotive range-Doppler (RD) radar spectra. Such labels are required when training a neural network for object recognition from radar data. The automatic labeling approach rests on the simultaneous recording of camera and lidar data in addition to the radar spectrum. By warping radar spectra into the camera image, state-of-the-art object recognition algorithms can be applied to label relevant objects, such as cars, in the camera image. The warping operation is designed to be fully differentiable, which allows backpropagating the gradient computed on the camera image through the warping operation to the neural network operating on the radar data. As the warping operation relies on accurate scene flow estimation, we further propose a novel scene flow estimation algorithm which exploits information from camera, lidar and radar sensors. The proposed scene flow estimation approach is compared against a state-of-the-art scene flow algorithm, and it outperforms it by approximately 30% w.r.t. mean average error. The feasibility of the overall framework for automatic label generation for RD spectra is verified by evaluating the performance of neural networks trained with the proposed framework for Direction-of-Arrival estimation. [ABSTRACT FROM AUTHOR]

This article introduces a high fill factor (FF) and low dark count rate (DCR) back-illuminated single photon avalanche diode (SPAD) pixel array fabricated in customized 55nm CMOS technology. In the SPAD array, all pixels use the same deep N-well (DNW) and adjacent pixels share the same N+ and N-well. The fill factor reaches 41.6%. Furthermore, deep trench isolation (DTI) is used to reduce optical crosstalk. For a single SPAD design, the shallow trench isolation (STI) is isolated away from avalanche multiplication zone to reduce the DCR. The array contains of ${8}\,\,\times \,\,{8\,4}\,\,\mu \text{m}$ pixels whose active area diameter is $2.6~\mu \text{m}$. At room temperature, the DCR of the SPAD is only 0.03Hz $/ \mu \text{m}^{{2}}$ at 1.0 V excess bias and 0.36Hz $/ \mu \text{m}^{{2}}$ at 2.5 V excess bias. The full width at half maximum (FWHM) jitter at 640nm wavelength is 180 ps and the photon detection efficiency (PDE) at 905nm wavelength is 5%. The SPAD array shows excellent performance for time-of-flight (ToF) applications. [ABSTRACT FROM AUTHOR]

Micro-opto-electro-mechanical systems (MOEMS) based mirrors with a suitable actuation range are often used for optical beam scanning applications. While several MOEMS scanners have been reported, they often have limitations with respect to their range of motion, field of view, mass, size and fabrication cost. In this work, we present a rotary micro-polygon scanner based on an electrostatic variable capacitance micromotor for laser beam steering applications. The miniaturized scanner consists of a hollow micro-polyhedron mirror with a base radius of $200 {\mu }\text{m}$ and a height of $400 {\mu }\text{m}$. It is mounted on the rotor of a variable capacitance micromotor using an epoxy-based adhesive. The rotor has a diameter of $600 {\mu }\text{m}$ and is $2.5 {\mu }\text{m}$ thick. The micromotor was fabricated using a commercial three-layer polysilicon surface micro-machining process (PolyMUMPs) and the micro-polyhedrons were fabricated using a commercial bulk micro-machining process (PiezeoMUMPs). After fabrication of the components, the micro-scanner was assembled and operated under atmospheric conditions using a three-phase square wave excitation ranging from 0 V to 200 V and achieved rotational speeds of up to 2100 rpm. An optical laser beam with a wavelength of 632 nm was reflected off the sidewall of the rotating micro-polyhedron, achieving an experimentally measured scanning angular range of 57.88°, an angular scan rate of 220 radians per second and a scan speed of 25,200 lines per minute or 420 lines per second with a 12 sided micro-mirror. The ratio of the output power over the input optical power is measured over a scanning range of 22°, which is limited by the size of the photo-detector, yielding a ratio spanning from 0.18 to 0.22. This rotary micro-scanner can provide high scanning speeds and a wide scanning field of view with a footprint of 1.25 mm2 and a thickness of 0.83 mm, which is suitable to implement chip-level high-performance micro-beam steering systems for various applications. [2022-0020] [ABSTRACT FROM AUTHOR]

Light detection and ranging (Lidar) has become the core device in a simultaneous localization and mapping system, which has attracted much attention in recent years. However, the point clouds acquired by Lidar are sparse and mutually uncorrelated, making it difficult to calculate correspondences between consecutive frames and resulting in data drifting. In this situation, we propose a tightly coupled Lidar + IMU fusion system with loop optimization to address the above-mentioned problems. First, the ground Lidar points are segmented and removed. Second, under our framework, an inertial measurement unit is used to deskew the motion distortion of Lidar. Third, a voxel grid filtering process is implemented to further eliminate the redundant points, and feature matching is performed by identifying lines and planes. Finally, loop closure detection is realized to correct pose estimation and localization error. Experimental results on both public dataset and field test demonstrate the effectiveness of our algorithm. [ABSTRACT FROM AUTHOR]

Multimodal data fusion, e.g., hyperspectral image (HSI) and light detection and ranging (LiDAR) data fusion, plays an important role in object recognition and classification tasks. However, existing methods pay little attention to the specificity of HSI spectral channels and the complementarity of HSI and LiDAR spatial information. In addition, the utilized feature extraction modules tend to consider the feature transmission processes among different modalities independently. Therefore, a new data fusion network named AM3Net is proposed for multimodal data classification; it includes three parts. First, an involution operator slides over the input HSI’s spectral channels, which can independently measure the contribution rate of the spectral channel of each pixel to the spectral feature tensor construction. Furthermore, the spatial information of HSI and LiDAR data is integrated and excavated in an adaptively fused, modality-oriented manner. Second, a spectral-spatial mutual-guided module is designed for the feature collaborative transmission among spectral features and spatial information, which can increase the semantic relatedness connection through adaptive, multiscale, and mutual-learning transmission. Finally, the fused spatial-spectral features are embedded into a classification module to obtain the final results, which determines whether to continue updating the network weights. Experimental evaluations on HSI-LiDAR datasets indicate that AM3Net possesses a better feature representation ability than the state-of-the-art methods. Additionally, AM3Net still maintains considerable performance when its input is replaced with multispectral and synthetic aperture radar data. The result indicates that the proposed data fusion framework is compatible with diversified data types. [ABSTRACT FROM AUTHOR]

Existing LiDAR point cloud compression (PCC) methods tend to treat compression as a fidelity issue, without sufficiently addressing its machine perception aspect. The latter issue is often encountered by the decoder agents that might aim to conduct scene-understanding related tasks only, such as computing the localization information. For tackling this challenge, a novel LiDAR PCC system is proposed to compress the point cloud geometry, which contains a back channel for allowing the decoder to initiate such request to the encoder. The key success of our PCC method lies in our proposed semantic prior representation (SPR) and its lossy encoding algorithm with variable precision to generate the final bitstream; the entire process is fast and achieves real-time performance. Note that our SPR is a compact and effective representation of three-dimensional (3D) input point clouds, and it consists of labels, predictions, and residuals. These information can be generated by first exploiting a scene-aware object segmentation to a set of 2D range images (frames) individually, which were generated from the 3D point clouds via a projection process. Based on the generated labels, the pixels associated with those moving objects are considered as noisy information and should be removed for not only saving bit budget on transmission but also, most importantly, improving the accuracy of localization computed at the decoder. Experimental results conducted on the commonly-used test dataset have shown that our proposed system outperforms the MPEG’s G-PCC (TMC13-v14.0) in a large bitrate range. In fact, the performance gap will become even larger when more and/or large moving objects are involved in the input point clouds. [ABSTRACT FROM AUTHOR]

Road detection is a critically important task for self-driving cars. By employing LiDAR data, recent works have significantly improved the accuracy of road detection. However, relying on LiDAR sensors limits the application of those methods when only cameras are available. In this paper, we propose a novel road detection approach with RGB images being the only input. Specifically, we exploit pseudo-LiDAR using depth estimation and propose a feature fusion network in which RGB images and learned depth information are fused for improved road detection. To optimize the network architecture and improve the efficiency of our network, we propose a method to search for the information propagation paths. Finally, to reduce the computational cost, we design a modality distillation strategy to avoid using depth estimation networks during inference. The resulting model eliminates the reliance on LiDAR sensors and achieves state-of-the-art performance on two challenging benchmarks, KITTI and R2D. [ABSTRACT FROM AUTHOR]

This article presents FAST-LIO2: a fast, robust, and versatile LiDAR-inertial odometry framework. Building on a highly efficient tightly coupled iterated Kalman filter, FAST-LIO2 has two key novelties that allow fast, robust, and accurate LiDAR navigation (and mapping). The first one is directly registering raw points to the map (and subsequently update the map, i.e., mapping) without extracting features. This enables the exploitation of subtle features in the environment and, hence, increases the accuracy. The elimination of a hand-engineered feature extraction module also makes it naturally adaptable to emerging LiDARs of different scanning patterns; the second main novelty is maintaining a map by an incremental k-dimensional (k-d) tree data structure, incremental k-d tree (ikd-Tree), that enables incremental updates (i.e., point insertion and delete) and dynamic rebalancing. Compared with existing dynamic data structures (octree, R $^\ast$ -tree, and nanoflann k-d tree), ikd-Tree achieves superior overall performance while naturally supports downsampling on the tree. We conduct an exhaustive benchmark comparison in 19 sequences from a variety of open LiDAR datasets. FAST-LIO2 achieves consistently higher accuracy at a much lower computation load than other state-of-the-art LiDAR-inertial navigation systems. Various real-world experiments on solid-state LiDARs with small field of view are also conducted. Overall, FAST-LIO2 is computationally efficient (e.g., up to 100 Hz odometry and mapping in large outdoor environments), robust (e.g., reliable pose estimation in cluttered indoor environments with rotation up to 1000 deg/s), versatile (i.e., applicable to both multiline spinning and solid-state LiDARs, unmanned aerial vehicle (UAV) and handheld platforms, and Intel- and ARM-based processors), while still achieving a higher accuracy than existing methods. Our implementation of the system FAST-LIO2 and the data structure ikd-Tree are both open-sourced on Github. [ABSTRACT FROM AUTHOR]

Loop closure detection is an essential component of simultaneous localization and mapping (SLAM) systems, which reduces the drift accumulated over time. Over the years, several deep learning approaches have been proposed to address this task; however, their performance has been subpar compared to handcrafted techniques, especially while dealing with reverse loops. In this article, we introduce the novel loop closure detection network (LCDNet) that effectively detects loop closures in light detection and ranging (LiDAR) point clouds by simultaneously identifying previously visited places and estimating the six degrees of freedom relative transformation between the current scan and the map. LCDNet is composed of a shared encoder, a place recognition head that extracts global descriptors, and a relative pose head that estimates the transformation between two point clouds. We introduce a novel relative pose head based on the unbalanced optimal transport theory that we implement in a differentiable manner to allow for end-to-end training. Extensive evaluations of LCDNet on multiple real-world autonomous driving datasets show that our approach outperforms state-of-the-art loop closure detection and point cloud registration techniques by a large margin, especially while dealing with reverse loops. Moreover, we integrate our proposed loop closure detection approach into a LiDAR SLAM library to provide a complete mapping system and demonstrate the generalization ability using different sensor setup in an unseen city. [ABSTRACT FROM AUTHOR]

We provide an overview of our progress on the development of linear mode avalanche photodiodes (LmAPDs) on InP substrates using antimony (Sb)-based multipliers for short-wavelength infrared (SWIR) spectral region. We identify the key figures of merit of LmAPDs to provide higher sensitivity and speed for applications like light detection and ranging (LiDAR) and remote sensing. We discuss the design of separate absorption, charge, and multiplication (SACM) APDs that are used for narrow gap absorption. We summarize our results on the impact ionization, multiplication gain, dark current, and excess noise of AlGaAsSb and AlInAsSb multipliers lattice-matched to InP substrates. Finally, we identify the key technical challenges associated with the development of SACM APDs on InP substrates. [ABSTRACT FROM AUTHOR]

Single-wavelength bathymetric light detection and ranging (LiDAR) (532 nm) can provide seamless meter- and submeter-scale digital elevation model (DEMs) of both the terrestrial surface and seafloor. However, mixed terrestrial and bathymetric surfaces obtained by this sensor are challenging for full-waveform (FW) signal detection. This study addresses the issues in two FW mixed surfaces: accurate classification of terrestrial and nonterrestrial waveforms from the original waveforms without auxiliary information and flexible detection of peaks based on a new FW theoretical model. A novel FW signal detection model (FWSD) for single-wavelength bathymetric LiDAR is proposed without complex feature extraction and iterative procedure through waveform classification and segmentation. The raw FWs are divided into five categories for subsequent signal detection using a convolutional neural network that merges local descriptors with contextual information. The signal detection task is then split into FW segment recognition and peak extraction using a new FW model, which integrates a leapfrog sliding window FW segmentation, an improved extreme learning machine (ELM) algorithm for FW segment recognition, and a flexible signal detection framework. To search for the optimal initial parameters for ELM, a self-annealing particle swarm optimization (SAPSO) algorithm is introduced, and the output weight is adjusted by online sequence to improve its generalization. When combined with the Richardson–Lucy deconvolution (RLD) algorithm, FWSD can be adapted to deal with shallow water waveforms. Finally, a test demonstration with an airborne dataset shows that FWSD has higher detection efficiency and higher accuracy than Levenberg–Marquardt algorithm optimized generalized Gaussian model (LM-GGM) and RLD algorithm. [ABSTRACT FROM AUTHOR]

Connected and Autonomous Vehicles (CAVs) are set to be the next global revolution in transportation. One of the major challenges in a CAV environment is handling the positional-forgery of Basic Safety Messages while calculating critical traffic gaps at a Stop Sign Gap Assist controlled intersection. Traditionally, positional-forgery has been handled by isolated instances of physical layer detection and correction mechanisms, which proved unreliable during the multi-hop and majority malicious vehicle scenarios. In this work, we propose a novel framework called Positional-forgEry Resistant traFfic gap Estimation for Connected inTersection management (PERFECT) that performs optimal traffic gap estimation even during multi-hop scenarios and majority malicious positional-forgery attacks. The framework reduces traffic gap estimation error during majority malicious scenarios by over 60%, keeps the average error within 8.17%–10.62% and minimizes the maximum error across a range of malicious situations. [ABSTRACT FROM AUTHOR]

Due to the huge volume of point cloud data, storing and transmitting it is currently difficult and expensive in autonomous driving. Learning from the high-efficiency video coding (HEVC) framework, we propose a novel compression scheme for large-scale point cloud sequences, in which several techniques have been developed to remove the spatial and temporal redundancy. The proposed strategy consists mainly of three parts: intracoding, intercoding, and residual data coding. For intracoding, inspired by the depth modeling modes (DMMs), in 3-D HEVC (3-D-HEVC), a cluster-based prediction method is proposed to remove the spatial redundancy. For intercoding, a point cloud registration algorithm is utilized to transform two adjacent point clouds into the same coordinate system. By calculating the residual map of their corresponding depth image, the temporal redundancy can be removed. Finally, the residual data are compressed either by lossless or lossy methods. Our approach can deal with multiple types of point cloud data, from simple to more complex. The lossless method can compress the point cloud data to 3.63% of its original size by intracoding and 2.99% by intercoding without distance distortion. Experiments on the KITTI dataset also demonstrate that our method yields better performance compared with recent well-known methods. Note to Practitioners—This article deals with the problem of efficient compression of point cloud sequences that come from light detection and ranging (LiDARs) mounted on autonomous mobile robots. The vast amount of point cloud data could be an important bottleneck for transmission and storage. Inspired by the HEVC algorithm, we develop a novel coding architecture for the point cloud sequence. The scans are divided into intraframe and interframe, which are encoded separately using different techniques. Our method can be used for the compression of LiDAR point cloud sequences or dense LiDAR point cloud map and will significantly reduce the transmission bandwidth and storage spaces. We have to admit that although our method is less effective for real-time solutions, it can be highly efficient for off-line applications. Future studies will concentrate on further optimizing the coding algorithm to reduce the computational complexity and trying to find a balance between them. [ABSTRACT FROM AUTHOR]

Due to the increased popularity of augmented and virtual reality experiences, the interest in capturing high-resolution real-world point clouds has never been higher. Loss of details and irregularities in point cloud geometry can occur during the capturing, processing, and compression pipeline. It is essential to address these challenges by being able to upsample a low Level-of-Detail (LoD) point cloud into a high LoD point cloud. Current upsampling methods suffer from several weaknesses in handling point cloud upsampling, especially in dense real-world photo-realistic point clouds. In this paper, we present a novel geometry upsampling technique, PU-Dense, which can process a diverse set of point clouds including synthetic mesh-based point clouds, real-world high-resolution point clouds, real-world indoor LiDAR scanned objects, as well as outdoor dynamically acquired LiDAR-based point clouds. PU-Dense employs a 3D multiscale architecture using sparse convolutional networks that hierarchically reconstruct an upsampled point cloud geometry via progressive rescaling and multiscale feature extraction. The framework employs a UNet type architecture that downscales the point cloud to a bottleneck and then upscales it to a higher level-of-detail (LoD) point cloud. PU-Dense introduces a novel Feature Extraction Unit that incorporates multiscale spatial learning by employing filters at multiple sampling rates and receptive fields. The architecture is memory efficient and is driven by a binary voxel occupancy classification loss that allows it to process high-resolution dense point clouds with millions of points during inference time. Qualitative and quantitative experimental results show that our method significantly outperforms the state-of-the-art approaches by a large margin while having much lower inference time complexity. We further test our dataset on high-resolution photo-realistic datasets. In addition, our method can handle noisy data well. We further show that our approach is memory efficient compared to the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

We propose a deep fine-grained multi-level fusion architecture for monocular 3D object detection, with an additionally designed anti-occlusion optimization process. Conventional monocular 3D object detection methods usually leverage geometry constraints such as keypoints, object shape relationships, and 3D to 2D optimizations to offset the lack of accurate depth information. However, these methods still struggle against directly extracting rich information for fusion from the depth estimation. To solve the problem, we integrate the monocular 3D features with the pseudo-LiDAR filter generation network between fine-grained multi-level layers. Our network utilizes the inherent multi-scale and promotes depth and semantic information flow in different stages. The new architecture can obtain features that incorporate more reliable depth information. At the same time, the problem of occlusion among objects is prevalent in natural scenes yet remains unsolved mainly. We propose a novel loss function that aims at alleviating the problem of occlusion. Extensive experiments have proved that the framework demonstrates a competitive performance, especially for the complex scenes with occlusion. [ABSTRACT FROM AUTHOR]

In this study, we develop a novel deep hierarchical vision transformer (DHViT) architecture for hyperspectral and light detection and ranging (LiDAR) data joint classification. Current classification methods have limitations in heterogeneous feature representation and information fusion of multi-modality remote sensing data (e.g., hyperspectral and LiDAR data), these shortcomings restrict the collaborative classification accuracy of remote sensing data. The proposed deep hierarchical vision transformer architecture utilizes both the powerful modeling capability of long-range dependencies and strong generalization ability across different domains of the transformer network, which is based exclusively on the self-attention mechanism. Specifically, the spectral sequence transformer is exploited to handle the long-range dependencies along the spectral dimension from hyperspectral images, because all diagnostic spectral bands contribute to the land cover classification. Thereafter, we utilize the spatial hierarchical transformer structure to extract hierarchical spatial features from hyperspectral and LiDAR data, which are also crucial for classification. Furthermore, the cross attention (CA) feature fusion pattern could adaptively and dynamically fuse heterogeneous features from multi-modality data, and this contextual aware fusion mode further improves the collaborative classification performance. Comparative experiments and ablation studies are conducted on three benchmark hyperspectral and LiDAR datasets, and the DHViT model could yield an average overall classification accuracy of 99.58%, 99.55%, and 96.40% on three datasets, respectively, which sufficiently certify the effectiveness and superior performance of the proposed method. [ABSTRACT FROM AUTHOR]

Deep learning has attracted much attention in the field of hyperspectral image classification recently, due to its powerful representation and generalization abilities. Most of the current deep learning models are trained in a supervised manner, which requires large amounts of labeled samples to achieve the state-of-the-art performance. Unfortunately, pixel-level labeling in hyperspectral imageries is difficult, time-consuming, and human-dependent. To address this issue, we propose an unsupervised feature learning model using multimodal data, hyperspectral, and light detection and ranging (LiDAR) in particular. It takes advantage of the relationship between hyperspectral and LiDAR data to extract features, without using any label information. After that, we design a dual fine-tuning strategy to transfer the extracted features for hyperspectral image classification with small numbers of training samples. Such a strategy is able to explore not only the semantic information but also the intrinsic structure information of training samples. In order to test the performance of our proposed model, we conduct comprehensive experiments on three hyperspectral and LiDAR datasets. Experimental results show that our proposed model can achieve better performance than several state-of-the-art deep learning models. [ABSTRACT FROM AUTHOR]

Accurate localization is an important component of the vehicle’s autonomous navigation. The appearance of the moving objects may lead to a feature-matching error with the map features, thereby causing a serious decline in localization accuracy. A neuromorphic vision (NeuroIV) sensor is a kind of dynamic vision sensor with the properties of high temporal resolution, movement capture, and lightweight computation. In view of this, this research proposes to combine the NeuroIV and LiDAR points to acquire the static landmark features and robust navigation localization. However, as a younger and smaller research field compared to RGB computer vision, NeuroIV vision is rarely associated with the intelligent vehicle. For this purpose, we built a novel dataset recorded by NeuroIV sensor, and a state-of-the-art YOLO-small network is designed to detect the moving objects with the dataset. In order to completely deduct the whole dynamic zones, a sensors’ novel fusion model is built by the zones’ segmentation and matching, so the LiDAR’s static environment is obtained completely by the remained points. By evaluating different types of LiDAR points, the feature-matching error can be alleviated further, making the localization more accurate. Together with qualitative and quantitative results, this work provides a moving objects’ detection improvement of 14.13% mAP with the new NeuroIV dataset and an obvious localization accuracy improvement with LiDAR points’ evaluation. [ABSTRACT FROM AUTHOR]

Beam selection for millimeter-wave links in a vehicular scenario is a challenging problem, as an exhaustive search among all candidate beam pairs cannot be assuredly completed within short contact times. We solve this problem via a novel expediting beam selection by leveraging multimodal data collected from sensors like LiDAR, camera images, and GPS. We propose individual modality and distributed fusion-based deep learning (F-DL) architectures that can execute locally as well as at a mobile edge computing center (MEC), with a study on associated tradeoffs. We also formulate and solve an optimization problem that considers practical beam-searching, MEC processing and sensor-to-MEC data delivery latency overheads for determining the output dimensions of the above F-DL architectures. Results from extensive evaluations conducted on publicly available synthetic and home-grown real-world datasets reveal 95% and 96% improvement in beam selection speed over classical RF-only beam sweeping, respectively. F-DL also outperforms the state-of-the-art techniques by 20-22% in predicting top-10 best beam pairs. [ABSTRACT FROM AUTHOR]

Large bandwidth at millimeter wave (mm-wave) is crucial for fifth generation (5G) and beyond, but the high path loss (PL) requires highly accurate PL prediction for network planning and optimization. Statistical models with slope-intercept fit fall short in capturing large variations seen in urban canyons, whereas ray tracing, capable of characterizing site-specific features, faces challenges in describing foliage and street clutter and associated reflection/diffraction ray calculation. Machine learning (ML) is promising but faces three key challenges in PL prediction: 1) insufficient measurement data; 2) lack of extrapolation to new streets; 3) overwhelmingly complex features/models. We propose an ML-based urban canyon PL prediction model based on extensive 28 GHz measurements from Manhattan where street clutters are modeled via a light detection and ranging (LiDAR) point cloud dataset and buildings by a mesh-grid building dataset. We extract expert knowledge-driven street clutter features from the point cloud and aggressively compress the 3-D building information using a convolutional autoencoder. Using a new street-by-street training and testing procedure to improve generalizability, the proposed model using both clutter and building features achieves a prediction error [root-mean-square error (RMSE)] of 4.8 ± 1.1 dB compared to 10.6 ± 4.4 and 6.5 ± 2.0 dB for 3GPP line of sight (LOS) and slope-intercept prediction, respectively, where the standard deviation indicates street-by-street variation. By only using four most influential clutter features, the RMSE of 5.5 ± 1.1 dB is achieved. [ABSTRACT FROM AUTHOR]

In this article, an active multireceiver low-THz imaging radar is used to generate 3-D images of automotive environments by combining time (difference) of arrival (TDoA) information with backprojection techniques to give height profiles of the objects in the scene. The large bandwidth of this system facilitates noncoherent height reconstruction using time of arrival alone. In doing so, the constraints on receiver coherency are relaxed compared to many height-finding techniques. 3-D reconstructions of common roadside targets and scenes are generated through a modified form of backprojection, adapted to use the time of arrival information of the signals from multiple receivers with different sensitivities, which are arranged in a quasi-multistatic configuration. Here, it is experimentally shown that the use of three vertically-aligned receivers is sufficient to provide a fine resolution in range, cross-range, and height. The system is used to generate 3-D reconstructions of a number of different scenarios, including pedestrians moving through a scene, and complex real-world road scenes. [ABSTRACT FROM AUTHOR]

In stochastic estimation problems, we aim to estimate an unknown state of interest given a set of measurements received from noisy sensory devices, such as radar, light detection and ranging, etc. A common model of the measurements’ random error is white Gaussian noise. This noise model is usually used to derive an estimator of the unknown state; accordingly, the state is described as a random variable with a Gaussian density. In some applications, the unknown state is known to take only positive values, e.g., estimating size or dimension. In such a case, the classical approaches based on Gaussian densities might fail, i.e., produce a negative value. In this article, a recursive Bayesian filter is proposed based on modeling the unknown state as Wishart distributed random matrix. This model ensures that the probability densities of the random variables are restricted to positive real values, even though the measurement’s noise is still modeled as white Gaussian noise. The feasibility of the proposed Bayesian–Wishart filter is demonstrated within the framework of extended target tracking. A target contour measurement model of a symmetric star-convex shape is presented and integrated in the proposed filter to track the target’s extent. [ABSTRACT FROM AUTHOR]

This article provides a tutorial introduction to the direct Time-of-Flight (dToF) signal chain and typical artifacts introduced due to detector and processing electronic limitations. We outline the memory requirements of embedded histograms related to desired precision and detectability, which are often the limiting factor in the array resolution. A survey of integrated CMOS dToF arrays is provided highlighting future prospects to further scaling through process optimization or smart embedded processing. [ABSTRACT FROM AUTHOR]

Panoptic segmentation of point clouds is a crucial task that enables autonomous vehicles to comprehend their vicinity using their highly accurate and reliable LiDAR sensors. Existing top–down approaches tackle this problem by either combining independent task-specific networks or translating methods from the image domain ignoring the intricacies of LiDAR data and thus often resulting in suboptimal performance. In this article, we present the novel top–down efficient LiDAR panoptic segmentation (EfficientLPS) architecture that addresses multiple challenges in segmenting LiDAR point clouds, including distance-dependent sparsity, severe occlusions, large scale-variations, and reprojection errors. EfficientLPS comprises of a novel shared backbone that encodes with strengthened geometric transformation modeling capacity and aggregates semantically rich range-aware multiscale features. It incorporates new scale-invariant semantic and instance segmentation heads along with the panoptic fusion module which is supervised by our proposed panoptic periphery loss function. Additionally, we formulate a regularized pseudolabeling framework to further improve the performance of EfficientLPS by training on unlabeled data. We benchmark our proposed model on two large-scale LiDAR datasets: nuScenes, for which we also provide ground truth annotations, and SemanticKITTI. Notably, EfficientLPS sets the new state-of-the-art on both these datasets. [ABSTRACT FROM AUTHOR]

Place recognition is a key module in robotic navigation. The existing line of studies mostly focuses on visual place recognition to recognize previously visited places solely based on their appearance. In this article, we address structural place recognition by recognizing a place based on structural appearance, namely from range sensors. Extending our previous work on a rotation invariant spatial descriptor, the proposed descriptor completes a generic descriptor robust to both rotation (heading) and translation when roll–pitch motions are not severe. We introduce two subdescriptors and enable topological place retrieval followed by the 1-degree of freedom semimetric localization, thereby bridging the gap between topological place retrieval and metric localization. The proposed method has been evaluated thoroughly in terms of environmental complexity and scale. The source code is available and can easily be integrated into existing light detection and ranging simultaneous localization and mapping. [ABSTRACT FROM AUTHOR]

The effective fusion of multisource data helps to improve the performance of land cover classification. Most existing convolutional neural network (CNN)-based methods adopt an early/late fusion strategy to fuse the low-/high-level features for classification, which still has two inherent challenges: 1) the conventional convolution operation performs a weighted average operation on each pixel in the receptive field, which will reduce the discriminability of the center pixel due to the influence of the interference pixels and 2) the spatial–spectral features of the hyperspectral image (HSI), the elevation features of light detection and ranging (LiDAR), and the complementary features between the multimodal data are not fully exploited, which results in the reduction of classification accuracy. In this article, an effective multibranch feature fusion network with self- and cross-guided attention (MB2FscgaNet) is proposed for the joint classification of LiDAR and HSI. The main concern of this article is how to accurately estimate more effective spectral–spatial-elevation features and yield more effective transfer in the network. Specifically, MB2FscgaNet adopts a multibranch feature fusion architecture to fully exploit the hierarchical features from LiDAR and HSI level by level. At each level of the network, a self- and cross-guided attention (SCGA) is developed to assign a higher weight to interesting areas and channels of LiDAR and HSI feature maps to obtain refined spectral–spatial-elevation features and provide complementary information cross-guidance between LiDAR and HS. We further designed a spectral supplement module (SeSuM) to improve the discriminative ability of the center pixel. Comparative classification results and ablation studies demonstrate that the proposed MB2FscgaNet achieves competitive performance against state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Precipitation nowcasting is an important task, which can be used in numerous applications. The key challenge of the task lies in radar echo map prediction. Previous studies leverage the convolutional recurrent neural network (ConvRNN) to address the problem. However, the approaches are built upon mean square losses, and the results tend to have inaccurate appearances, shapes, and positions for predictions. To alleviate this problem, we explore the idea of adversarial regularization and systematically compare four types of generative adversarial networks (GANs), which are the combinations of GAN/Wasserstein GAN (WGAN) and its multiscale version. Extensive experiments on a real-world radar dataset and four typical meteorological examples are conducted. The results validate the effectiveness of adversarial regularization. The developed models show superior performances over the existing prediction approaches in the majority of circumstances. Moreover, we find that the WGAN regularization often delivers better results than the GAN regularization due to its robustness, and the multiscale WGAN, in general, performs the best among all the methods. To reproduce the results, we release the source code at https://github.com/luochuyao/MultiScaleGAN and the test system at http://39.97.217.145:80/. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNNs) have been extensively utilized for hyperspectral image (HSI) and light detection and ranging (LiDAR) data classification. However, CNNs have not been much explored for joint HSI and LiDAR image classification. Therefore, this article proposes a joint feature learning (HSI and LiDAR) and fusion mechanism using CNN and spatial morphological blocks, which generates highly accurate land-cover maps. The CNN model comprises three Conv3D layers and is directly applied to the HSIs for extracting discriminative spectral–spatial feature representation. On the contrary, the spatial morphological block is able to capture the information relevant to the height or shape of the different land-cover regions from LiDAR data. The LiDAR features are extracted using morphological dilation and erosion layers that increase the robustness of the proposed model by considering elevation information as an additional feature. Finally, both the obtained features from CNNs and spatial morphological blocks are combined using an additive operation prior to the classification. Extensive experiments are shown with widely used HSIs and LiDAR datasets, i.e., University of Houston (UH), Trento, and MUUFL Gulfport scene. The reported results show that the proposed model significantly outperforms traditional methods and other state-of-the-art deep learning models. The source code for the proposed model will be made available publicly at https://github.com/AnkurDeria/HSI+LiDAR. [ABSTRACT FROM AUTHOR]

As one of the fundamental tasks in scene understanding, 3D object detection from LiDAR point clouds has drawn extensive attention in the past few years. Although the existing voxel-based methods have achieved remarkable performance, how to effectively exploit geometric structure information of the point clouds to boost the detection performance remains to be explored. In this article, we propose a novel structure-information enhanced voxel network (SIEV-Net) for 3D object detection from LiDAR point clouds. The proposed SIEV-Net learns feature representations of 3D objects by jointly considering uneven spatial distribution and height information of the point clouds. Specifically, considering the uneven spatial distribution characteristics of point clouds, a hierarchical-voxel feature encoding module is proposed to effectively extract features of voxels in both sparse and dense regions. Besides, by utilizing the bird’s eye view (BEV) map of point clouds, a height information complement module is designed to minimize the height information lost in the process of point feature aggregation in a voxel network. Experimental results on the widely used KITTI benchmark dataset have demonstrated the efficacy of the proposed SIEV-Net. [ABSTRACT FROM AUTHOR]

Self-driving vehicles are posing new challenges as the automation level defined in the SAE International standards for autonomous driving is increased. A pivotal task in autonomous driving is building a perception of the surrounding environment using optical sensors, which is a long-standing challenge and prompts us to explore the utilization of various sensors. Radar is an older and cheaper type of sensor than alternatives such as lidar for long-range distance coverage, and it is also competitively reliable and robust in adverse weather conditions. However, sparse data and noise are inherent challenges of radar. This study explores the dynamic Gaussian process for occupancy mapping and predicting a drivable path for a self-driving vehicle within the field of view (FOV) of a radar sensor. Gaussian occupancy mapping does not need abundant data for training and is a promising alternative to data-reliant deep learning techniques. The proposed technique optimizes parameters (variational and kernel-based) of the Gaussian process to determine the allowed region within the FOV limits by means of stochastic selection of functional points (pseudoinput) and tuning of threshold values. We have tested the proposed technique in experiments performed under different environmental conditions, such as various road and traffic conditions and diverse weather and illumination conditions. The results verify the efficacy of the proposed technique in diverse weather conditions for finding a drivable path for a self-driving vehicle, with the additional benefits of requiring only a low-cost apparatus and providing coverage of a long distance range. [ABSTRACT FROM AUTHOR]

High-efficiency, high-speed phototransistors applicable in visible wavelength and the near-infrared regime are highly desirable for optical communication links utilized for data centers, high-performance computing, and laser radar application. To overcome the intrinsic weak absorption of silicon material and alleviate the compromise between efficiency and speed of the device, a SiGe/Si heterojunction phototransistor (HPT) with photon-trapping nanoholes is demonstrated in this article. With photon-trapping nanohole structures, the absorption efficiency of the HPT is greatly improved near the whole band ranging from 600- to 1000-nm wavelength compared to one without nanoholes. The responsivity of nanohole-assisted HPT is 25.69, 27.02, and 15.65 A/W at 650-, 850-, and 940-nm wavelength, exhibiting 85.22%, 192.42%, and 848.48% improvement compared with its counterpart without nanoholes. The thin base region also allows a high speed of the HPT with a transient response of 0.45 ns at 850-nm wavelength. [ABSTRACT FROM AUTHOR]

The traditional way to detect the vertical structure of seawater optical properties and chlorophyll-a is mainly through shipboard discrete observations or biogeochemical-argo profiling floats, which requires considerable time to cover a limited area. In this study, the vertical distribution of seawater optical properties and chlorophyll-a concentration across two different optically contrasted sea areas from the East China Sea (ESC) to the South China Sea (SCS) were obtained for the first time using a shipboard integrated Mie–Raman–fluorescence Light Detection and Ranging (LiDAR) for large-scale observations, with a total observation distance of over 3700 km. More than 74000 LiDAR profiles were obtained from September 5 to September 15, 2020. In general, the LiDAR-estimated inherent optical properties (IOPs) and chlorophyll-a values decreased from turbid water in the ECS to clear water in the SCS. Subsurface scattering layers were often observed at depths ranging from 10 to 20 m along the SCS coast. Subsequently, the LiDAR-derived results were compared against in situ measurements. In addition, the diurnal hourly variation in IOPs and chlorophyll-a by LiDAR at a fixed coastal station was monitored for the first time, which was relatively lower in the early morning and midday yet was higher in the evening, while the relative tide height showed the reverse change trend, which revealed that the tide possibly impacted the diurnal variation in IOPs and chlorophyll-a on the SCS coast. Overall, our results indicate that the LiDAR remote sensing technique is effective and feasible to monitor large-scale and long-term subsurface phytoplankton structure over different optically contrasted sea regions, and integration of multiple detection mechanisms will enhance the monitoring capacity. [ABSTRACT FROM AUTHOR]

Significant range differences were identified between shallow and deep channels of the Mapper5000 bathymetry light detection and ranging (LiDAR) system with segmented field-of-view (FOV) receivers. Range difference varied with depth and water optical properties. The main feature was the maximum value in range difference curves, which ranged from 0.3 to 0.6 m and usually exceeded the International Hydrographic Organization (IHO) accuracy standards. Sensitivity analyses based on a semianalytical Monte Carlo simulation model revealed that the scattering phase function and laser beam divergence angle played more important roles in causing pulse dispersion and determining the amplitude and position of maximum range difference than absorption and scattering coefficients. A range difference correction method by fitting existing shallow and deep channel data in the overlapping range with a cubic polynomial was proposed to correct the deep channel data in the entire depth range that LiDAR can detect. Depth discontinuity at the junction of the shallow channel and deep channel measurements was successfully removed, and the mean and standard deviation of corrected range differences were within 0.01 and 0.1 m, respectively. A combination of range difference correction and mean bias corrector can be an alternative method for depth bias correction of segmented-FOV LiDAR when referenced sonar data is not available. [ABSTRACT FROM AUTHOR]

Light detection and ranging (LiDAR) data provide rich elevation information, so it plays an irreplaceable role in ground object classification. Recently, convolutional neural networks (CNNs) have shown excellent performance in LiDAR digital surface models (DSMs) classification. However, the architecture of CNN model relies heavily on manual design, so it has great limitations. In addition, different sensors capture LiDAR datasets with different properties, so the model should be designed to suit for different datasets, which further increases the workload of architecture design. Therefore, this article proposes a method of automatic design of LiDAR DSM classification model. First, attention mechanism is introduced into search space to improve the feature extraction capability of the model. Then, a gradient-based search strategy is used to obtain the optimal architecture from this search space. Second, a learning rate adjustment strategy is proposed to reduce the time spent in the search stage and evaluation stage to improve the classification accuracy of the model. Finally, a regularization scheme is introduced to enhance the robustness of the model and avoid overfitting. Experimental results on three public LiDAR datasets (Bayview Park, Recology, and Houston) obtained from different sensors show that the proposed neural architecture search method achieves the impressive classification performance compared to several state-of-the-art classification methods and improves the classification accuracy under the condition of limited training samples. [ABSTRACT FROM AUTHOR]

The uniform and linear wind models have been commonly used for wind field retrieval in meteorological community. However, the accuracy and robustness of the retrieval results can be quite unsatisfactory due to the mismatch between these models and the real wind distribution, especially under complex wind conditions. In this article, a nonlinear model based on high-order Taylor expansion is proposed to deal with this limitation, and the combination of ridge regression and decomposition-iteration process (denoted as Ridge-DI method) is further introduced to solve the model with high accuracy and robustness. A case study on simulation and field experiment shows that the proposed method with the third-order Taylor expansion can reduce the mean root-mean-square errors (RMSEs) of the retrieved velocities by more than 16.84% in comparison with traditional methods. [ABSTRACT FROM AUTHOR]

In this article, we propose an efficient and generalizable framework based on a deep convolutional neural network (CNN) for multisource remote sensing (RS) data joint classification. While recent methods are mostly based on multistream architectures, we use group convolution (GConv) to construct equivalent network architectures efficiently within a single-stream network. Based on a recent technique called dynamic grouping convolution (DGConv), we further propose a network module named separable DGConv (SepDGConv), to make GConv hyperparameters, and, thus, the overall network architecture, learnable during network training. In the experiments, the proposed method is applied to residual network (ResNet) and UNet, and the adjusted networks are verified on three very diverse benchmark datasets (i.e., Houston2018 data, Berlin data, and MUUFL Gulfport Hyperspectral and LiDAR Airborne Data Set (MUUFL) data). Experimental results demonstrate the effectiveness of the proposed single-stream CNNs, and in particular, SepG-ResNet18 improves the state-of-the-art classification overall accuracy (OA) on hyperspectral–synthetic aperture radar (HS–SAR) Berlin dataset from 62.23% to 68.21%. In the experiments we have two interesting findings. First, using DGConv generally reduces test OA variance. Second, multistream is harmful to model performance if imposed to the first few layers, but becomes beneficial if applied to deeper layers. Altogether, the findings imply that the multistream architecture, instead of being a strictly necessary component in deep learning models for multisource RS data, essentially plays the role of model regularizer. Our code is publicly available at https://github.com/yyyyangyi/CNNs-for-Multi-Source-Remote-Sensing-Data-Fusion. We hope our work can inspire novel research in the future. [ABSTRACT FROM AUTHOR]

The strong demand of autonomous driving in the industry has led to vigorous interest in 3D object detection and resulted in many excellent 3D object detection algorithms. However, the vast majority of algorithms only model single-frame data, ignoring the temporal clue in video sequence. In this work, we propose a new transformer, called Temporal-Channel Transformer (TCTR), to model the temporal-channel domain and spatial-wise relationships for video object detecting from Lidar data. As the special design of this transformer, the information encoded in the encoder is different from that in the decoder. The encoder encodes temporal-channel information of multiple frames while the decoder decodes the spatial-wise information for the current frame in a voxel-wise manner. Specifically, the temporal-channel encoder of the transformer is designed to encode the information of different channels and frames by utilizing the correlation among features from different channels and frames. On the other hand, the spatial decoder of the transformer decodes the information for each location of the current frame. Before conducting the object detection with detection head, a gate mechanism is further deployed for re-calibrating the features of current frame, which filters out the object-irrelevant information by repetitively refining the representation of target frame along with the up-sampling process. Experimental results reveal that TCTR achieves the state-of-the-art performance in grid voxel-based 3D object detection on the nuScenes benchmark. [ABSTRACT FROM AUTHOR]