Search

In recent years, the accuracy of visual SLAM (Simultaneous Localization and Mapping) technology has seen significant improvements, making it a prominent area of research. However, within the current RGB-D SLAM systems, the estimation of 3D positions of feature points primarily relies on direct measurements from RGB-D depth cameras, which inherently contain measurement errors. Moreover, the potential of triangulation-based estimation for ORB (Oriented FAST and Rotated BRIEF) feature points remains underutilized. To address the singularity of measurement data, this paper proposes the integration of the ORB features, triangulation uncertainty estimation and depth measurements uncertainty estimation, for 3D positions of feature points. This integration is achieved using a CI (Covariance Intersection) filter, referred to as the CI-TEDM (Triangulation Estimates and Depth Measurements) method. Vision-based SLAM systems face significant challenges, particularly in environments, such as long straight corridors, weakly textured scenes, or during rapid motion, where tracking failures are common. To enhance the stability of visual SLAM, this paper introduces an improved CI-TEDM method by incorporating wheel encoder data. The mathematical model of the encoder is proposed, and detailed derivations of the encoder pre-integration model and error model are provided. Building on these improvements, we propose a novel tightly coupled visual-inertial RGB-D SLAM with encoder integration of ORB triangulation and depth measurement uncertainties. Validation on open-source datasets and real-world environments demonstrates that the proposed improvements significantly enhance the robustness of real-time state estimation and localization accuracy for intelligent vehicles in challenging environments. [ABSTRACT FROM AUTHOR]

As a key method of ocean exploration, the positioning accuracy of autonomous underwater vehicles (AUVs) directly influences the success of subsequent missions. This study aims to develop a novel method to address the low accuracy in visual simultaneous localization and mapping (SLAM) within underwater environments, enhancing its application in the navigation and localization of AUVs. We propose an underwater multisensor fusion SLAM system based on image enhancement. First, we integrate hybrid attention mechanisms with generative adversarial networks to address the blurring and low contrast in underwater images, thereby increasing the number of feature points. Next, we develop an underwater feature-matching algorithm based on a local matcher to solve the feature tracking problem caused by grayscale changes in the enhanced image. Finally, we tightly couple the Doppler velocity log (DVL) with the SLAM algorithm to better adapt to underwater environments. The experiments demonstrate that, compared to other algorithms, our proposed method achieves reductions in both mean absolute error (MAE) and standard deviation (STD) by up to 68.18% and 44.44%, respectively, when all algorithms are operating normally. Additionally, the MAE and STD of our algorithm are 0.84 m and 0.48 m, respectively, when other algorithms fail to operate properly. [ABSTRACT FROM AUTHOR]

Rolling bearing feature extraction and fault identification techniques using deep learning algorithms have been widely adopted in recent years. We proposed a method for diagnosing composite faults in rolling bearings by employing multisensor decision fusion and convolutional neural networks. Different types of bearing faults and eccentricity faults have different fault eigenfrequencies in vibration signals. In the proposed method, vibration and acoustic signals are collected, their characteristics are analyzed, and multisensor data fusion processing is conducted. A neural network is then used to identify the signals containing bearing fault characteristics to diagnose bearing faults at different rotational speeds. We demonstrated the effectiveness of the proposed method by conducting comparative experiments on existing methods. [ABSTRACT FROM AUTHOR]

We design a scheme for laser-inertial odometry and mapping with bundle adjustment (BA-LIOM), which can greatly mitigate the problem of undesired ground warping due to sparsity of laser scans and significantly reduce odometry drift. Specifically, an Inertial measurement unit (IMU)-assisted adaptive voxel map initialization algorithm is proposed and elaborately integrated with the existing framework LIO-SAM, allowing for accurate registration in the beginning of the localization and mapping process. In addition, to accommodate to fast-moving and structure-less scenarios, we design a tightly coupled odometry, which jointly optimizes both the IMU preintegration constraints and scan matching with adaptive voxel maps. The voxels (edge and plane, respectively) are updated with BA optimization. And then the accurate mapping result is obtained by performing local BA. The proposed BA-LIOM is thoroughly assessed using datasets collected from multiple platforms over a variety of environments. Experimental results show the superiority of BA-LIOM over the state-of-the-art methods in robustness and precision, especially for large-scale scenarios. BA-LIOM improves the accuracy of localization by $61\%$ and $73\%$ on the buildings and lawn datasets, respectively, and has a $29\%$ accuracy improvement over LIO-SAM on the KITTI datasets. A supplementary video can be accessed at https://youtu.be/5l4ZFhTc2sw. [ABSTRACT FROM AUTHOR]

As vital equipment in modern industry, the health state of rotating machinery influences the production process and equipment safety. However, rotating machinery generally operates in a normal state most of the time, which results in limited fault data, thus greatly constraining the performance of intelligent fault diagnosis methods. To solve this problem, this paper proposes a novel fault diagnosis method for rotating machinery with limited multisensor fusion samples based on the fused attention-guided Wasserstein generative adversarial network (WGAN). Firstly, the dimensionality of collected multisensor data is reduced to three channels by principal component analysis, and then the one-dimensional data of each channel are converted into a two-dimensional pixel matrix, of which the RGB images are obtained by fusing the three-channel two-dimensional images. Subsequently, the limited RGB samples are augmented to obtain sufficient samples utilizing the fused attention-guided WGAN combined with the gradient penalty (FAWGAN-GP) method. Lastly, the augmented samples are applied to train a residual convolutional neural network for fault diagnosis. The effectiveness of the proposed method is demonstrated by two case studies. When training samples per class are 50, 35, 25, and 15 on the KAT-bearing dataset, the average classification accuracy is 99.9%, 99.65%, 99.6%, and 98.7%, respectively. Meanwhile, the methods of multisensor fusion and the fused attention mechanism have an average improvement of 1.51% and 1.09%, respectively, by ablation experiments on the WT gearbox dataset. [ABSTRACT FROM AUTHOR]

In recent years, the accuracy of visual SLAM (Simultaneous Localization and Mapping) technology has seen significant improvements, making it a prominent area of research. However, within the current RGB-D SLAM systems, the estimation of 3D positions of feature points primarily relies on direct measurements from RGB-D depth cameras, which inherently contain measurement errors. Moreover, the potential of triangulation-based estimation for ORB (Oriented FAST and Rotated BRIEF) feature points remains underutilized. To address the singularity of measurement data, this paper proposes the integration of the ORB features, triangulation uncertainty estimation and depth measurements uncertainty estimation, for 3D positions of feature points. This integration is achieved using a CI (Covariance Intersection) filter, referred to as the CI-TEDM (Triangulation Estimates and Depth Measurements) method. Vision-based SLAM systems face significant challenges, particularly in environments, such as long straight corridors, weakly textured scenes, or during rapid motion, where tracking failures are common. To enhance the stability of visual SLAM, this paper introduces an improved CI-TEDM method by incorporating wheel encoder data. The mathematical model of the encoder is proposed, and detailed derivations of the encoder pre-integration model and error model are provided. Building on these improvements, we propose a novel tightly coupled visual-inertial RGB-D SLAM with encoder integration of ORB triangulation and depth measurement uncertainties. Validation on open-source datasets and real-world environments demonstrates that the proposed improvements significantly enhance the robustness of real-time state estimation and localization accuracy for intelligent vehicles in challenging environments.

Recently, magnetic levitation systems have been applied and studied in various industrial fields. In particular, in-tracktype magnetic levitation conveyor systems are actively studied since they can effectively minimize electromagnetic effects in processes that require a highly clean environment. In this type of system, diverse and multiple sensors are structurally required so that the control performance of an integrated system is primarily governed by the slowest measuring sensor. This paper proposes a multisensor fusion compensator to integrate the outputs obtained from various sensors into one output with the single fastest time rate. Since the state of the system is estimated at a fast time rate, the optimal controller also guarantees fast performance and stability. The computation of electromagnetic fields and the control performance of the considered superconducting hybrid system were analyzed using a computer simulation based on finite element methods. [ABSTRACT FROM AUTHOR]

As a key method of ocean exploration, the positioning accuracy of autonomous underwater vehicles (AUVs) directly influences the success of subsequent missions. This study aims to develop a novel method to address the low accuracy in visual simultaneous localization and mapping (SLAM) within underwater environments, enhancing its application in the navigation and localization of AUVs. We propose an underwater multisensor fusion SLAM system based on image enhancement. First, we integrate hybrid attention mechanisms with generative adversarial networks to address the blurring and low contrast in underwater images, thereby increasing the number of feature points. Next, we develop an underwater feature-matching algorithm based on a local matcher to solve the feature tracking problem caused by grayscale changes in the enhanced image. Finally, we tightly couple the Doppler velocity log (DVL) with the SLAM algorithm to better adapt to underwater environments. The experiments demonstrate that, compared to other algorithms, our proposed method achieves reductions in both mean absolute error (MAE) and standard deviation (STD) by up to 68.18% and 44.44%, respectively, when all algorithms are operating normally. Additionally, the MAE and STD of our algorithm are 0.84 m and 0.48 m, respectively, when other algorithms fail to operate properly.

As vital equipment in modern industry, the health state of rotating machinery influences the production process and equipment safety. However, rotating machinery generally operates in a normal state most of the time, which results in limited fault data, thus greatly constraining the performance of intelligent fault diagnosis methods. To solve this problem, this paper proposes a novel fault diagnosis method for rotating machinery with limited multisensor fusion samples based on the fused attention-guided Wasserstein generative adversarial network (WGAN). Firstly, the dimensionality of collected multisensor data is reduced to three channels by principal component analysis, and then the one-dimensional data of each channel are converted into a two-dimensional pixel matrix, of which the RGB images are obtained by fusing the three-channel two-dimensional images. Subsequently, the limited RGB samples are augmented to obtain sufficient samples utilizing the fused attention-guided WGAN combined with the gradient penalty (FAWGAN-GP) method. Lastly, the augmented samples are applied to train a residual convolutional neural network for fault diagnosis. The effectiveness of the proposed method is demonstrated by two case studies. When training samples per class are 50, 35, 25, and 15 on the KAT-bearing dataset, the average classification accuracy is 99.9%, 99.65%, 99.6%, and 98.7%, respectively. Meanwhile, the methods of multisensor fusion and the fused attention mechanism have an average improvement of 1.51% and 1.09%, respectively, by ablation experiments on the WT gearbox dataset.

• The Timeline Spans a Wide Range. • This paper extensively reviewed about 142 papers in the history of water hazards detection. These papers spanned 20 years. The previous papers only focused on short-term or specific tasks, and did not consider the entire long-term technological development. • The Research on the Target is Relatively Detailed and Concentrated. • This paper only summarizes and prospects the research on water hazards, and the work is more meticulous, while the research objectives of the previous review literature are relatively broad, which leads to the broad and rough literature arrangement. • Multi-Angle Analysis of a Single Research Object. • This paper classifies and sorts out the detection literature of water hazards from multiple angles. The previous review literature has a relatively single classification method for water hazards detection research. Obstacle detection is a complex task involving detection of obstacle features, identification of appropriate sensor and environmental conditions. The development of water hazard detection in the past two decades can be regarded as a microcosm of the history of typical obstacle detection. This review provides an extensive study of water hazard detection papers spanning a quarter-century (the 1990 s to 2021). The review mainly focuses on the width of water hazards, features of water hazards, sensor types for the detection of water hazards, and environmental light. This paper analyses and summarizes the research overview and status of some research institutions in the field over the past 20 years, hoping to provide some reference for UGV water hazard detection. In addition, the existing water hazard detection problems are summarized, and future development trends are proposed. [ABSTRACT FROM AUTHOR]

Recently, magnetic levitation systems have been applied and studied in various industrial fields. In particular, in-tracktype magnetic levitation conveyor systems are actively studied since they can effectively minimize electromagnetic effects in processes that require a highly clean environment. In this type of system, diverse and multiple sensors are structurally required so that the control performance of an integrated system is primarily governed by the slowest measuring sensor. This paper proposes a multisensor fusion compensator to integrate the outputs obtained from various sensors into one output with the single fastest time rate. Since the state of the system is estimated at a fast time rate, the optimal controller also guarantees fast performance and stability. The computation of electromagnetic fields and the control performance of the considered superconducting hybrid system were analyzed using a computer simulation based on finite element methods.

The Fields of Views (FoVs) of radars in a distributed network partially overlap due to detecting capability, waveform design, and antenna orientation constraints, resulting in observed discrepancies between radars and a significant obstacle to future information fusion. In this paper, we propose a distributed multitarget tracking method under the scene of partially overlapping radar FoVs, based on the Gaussian Mixture Cardinalized Probability Hypothesis Density (GM-CPHD) filter. First, we employ the product of the multitarget densities to split the PHD functions and find the part that characterizes the information of the targets commonly observed by multiple radars. Then, a standard distributed fusion (arithmetic average or geometric average fusion) acts on the splitting information to improve tracking performance, and a compensation fusion acts on the remaining information to expand the observation FoV. The proposed method does not require prior knowledge of the radar’s FoV and may adapt to the scene of distributed multitarget tracking while the FoVs are unknown. Simulations are provided to verify the effectiveness of the proposed approach under the scene of unknown and time-varying radar FoVs, and show that the proposed method has better performance than that of the cluster method based on Gaussian matching.

Indoor localization problems are difficult due to that the information, such as WLAN and GPS, cannot achieve enough precision for indoor issues. This paper presents a novel indoor localization algorithm, GeoLoc, with uncertainty eliminate based on fusion of acceleration, angular rate, and magnetic field sensor data. The algorithm can be deployed in edge devices to overcome the problems of insufficient computing resources and long delay caused by high complexity of location calculation. Firstly, the magnetic map is built and magnetic values are matched. Secondly, orientation updating and position selection are iteratively executed using the fusion data, which gradually reduce uncertainty of orientation. Then, we filter the trajectory from a path set. By gradually reducing uncertainty, GeoLoc can bring a high positioning precision and a smooth trajectory. In addition, this method has an advantage in that it does not rely on any infrastructure such as base stations and beacons. It solves the common problems regarding the non-uniqueness of the geomagnetic fingerprint and the deviation of the sensor measurement. The experimental results show that our algorithm achieves an accuracy of less than 2.5 m in indoor environment, and the positioning results are relatively stable. It meets the basic requirements of indoor location-based services (LBSs). [ABSTRACT FROM AUTHOR]

In this article, we propose a novel federated structure of the invariant extended Kalman filter (IEKF) using the left- and right-invariant observations. A proper invariant error should be chosen to exploit the trajectory-independent property of the IEKF. For the left- and right-invariant observations, the left-IEKF (LIEKF) and the right-IEKF (RIEKF) are designed, respectively. However, in applications, both left- and right-invariant observations are often used together while using multiple aiding sensors to improve estimation performance. If either the LIEKF or RIEKF is selected and implemented as a centralized filter, the measurement matrix contains current state estimates in mismatched observation-related terms. As a result, as the state estimation error increases, the calculation of the Kalman gain is affected, and the filter cannot properly correct the error. For such situations, we propose a federated structure that updates the left- and right-invariant observations separately in the allocated subfilters. In the subfilter assigned with left-invariant observations, the error is corrected using the LIEKF, and in the subfilter assigned with right-invariant observations, the error is corrected using the RIEKF. Since each subfilter uses trajectory-independent Kalman gain, it is expected to improve the estimation performance compared to the centralized structure. The performance of the proposed method is evaluated and compared to the centralized LIEKF and RIEKF through Monte-Carlo simulations for an inertial navigation example. Also, the proposed method is validated by a publicly available dataset of a ground vehicle equipped with a global positioning system (GPS) and wheel encoders. [ABSTRACT FROM AUTHOR]

To improve the validity of magnetic flux leakage (MFL) multisensor signals, anomaly detection has become a significant part of MFL signal processing. The anomalies in MFL are uncertain and have no prior information or labels. Therefore, the detection and location of the anomalies become a difficult issue. Regarding the abovementioned problem, we propose an unsupervised method called multisensor cycle-supervised convolutional neural network (CsCNN). The CsCNN is built including multiple CNNs with the same structure and a cycle-supervised part. The proposed model realizes unsupervised anomaly detection through multiple cycle-supervised CNNs for the first time. Moreover, the latent relationship between multisensor signals is established by CsCNN to take full use of multisensor information. Besides, a dynamic threshold is applied to detect anomalies. In the end, experiments on simulated signals and measured signals are conducted, and CsCNN is compared to the state-of-the-art methods. The results show that the proposed method is effective. [ABSTRACT FROM AUTHOR]

Reinforcement learning has achieved promising results on robotic control tasks but struggles to leverage information effectively from multiple sensory modalities that differ in many characteristics. Recent works construct auxiliary losses based on reconstruction or mutual information to extract joint representations from multiple sensory inputs to improve the sample efficiency and performance of reinforcement learning algorithms. However, the representations learned by these methods could capture information irrelevant to learning a policy and may degrade the performance. We argue that compressing information in the learned joint representations about raw multimodal observations is helpful, and propose a multimodal information bottleneck model to learn task-relevant joint representations from egocentric images and proprioception. Our model compresses and retains the predictive information in multimodal observations for learning a compressed joint representation, which fuses complementary information from visual and proprioceptive feedback and meanwhile filters out task-irrelevant information in raw multimodal observations. We propose to minimize the upper bound of our multimodal information bottleneck objective for computationally tractable optimization. Experimental evaluations on several challenging locomotion tasks with egocentric images and proprioception show that our method achieves better sample efficiency and zero-shot robustness to unseen white noise than leading baselines. We also empirically demonstrate that leveraging information from egocentric images and proprioception is more helpful for learning policies on locomotion tasks than solely using one single modality. [ABSTRACT FROM AUTHOR]

Early detection of age-related disease symptoms in older people by the use of daily activity data is one of the central challenges of home sensor systems. This paper focuses on dementia scale classification from daily activity data collected using sensors that can be deployed in actual residential environments. Activity data collected by four sensors (a door sensor, human motion sensor, location sensor, and sleep sensor) were obtained by recording 56 older adults living in common residences. We analyzed the effects of different types of sensor data, such as time spent in an individual room according to human motion sensors, location in a facility, and sleep patterns, on dementia detection. We then developed a feature extraction method related to daily activity patterns based on a clustering algorithm and analyzed its effectiveness. In the experimental evaluation, we trained binary classification models to classify dementia scale scores based on the Mini-Mental State Examination (MMSE) from these datasets. The experimental results show that a maximum accuracy of 0.871 was obtained with a linear support vector machine (SVM) model by fusing the door, location, and sleep features and by clustering activity patterns using the X-means algorithm.

With the development of modern information technology, intelligent substation technology has been widely used, which greatly promotes the development of power grid. The information integration platform of intelligent substations realizes panoramic data collection and data sharing of substations. With the continuous development of the economy and society, it is crucial to improve the power regulation ability of intelligent substations, make scientific use of intelligent technology, and optimize the patrol level of intelligent substation patrol robots to better meet the operation needs of substations. In the actual substation construction, the flexibility of equipment utilization in intelligent substations should be fully considered. In addition, the number of substations should be controlled to improve the utilization efficiency of patrol robots. This paper mainly summarizes the monitoring technology of intelligent substations, the positioning technology of inspection robots, and the multi-sensor control technology to lay a foundation for the follow-up inspection work of intelligent substations.

The fusion of light detection and ranging (LiDAR) and inertial measurement unit (IMU) sensing information can effectively improve the environment modeling and localization accuracy of navigation systems. To realize the spatiotemporal unification of data collected by the IMU and the LiDAR, a two-step spatiotemporal calibration method combining coarse and fine is proposed. The method mainly includes two aspects: (1) Modeling continuous-time trajectories of IMU attitude motion using B-spline basis functions; the motion of the LiDAR is estimated by using the normal distributions transform (NDT) point cloud registration algorithm, taking the Hausdorff distance between the local trajectories as the cost function and combining it with the hand–eye calibration method to solve the initial value of the spatiotemporal relationship between the two sensors' coordinate systems, and then using the measurement data of the IMU to correct the LiDAR distortion. (2) According to the IMU preintegration, and the point, line, and plane features of the lidar point cloud, the corresponding nonlinear optimization objective function is constructed. Combined with the corrected LiDAR data and the initial value of the spatiotemporal calibration of the coordinate systems, the target is optimized under the nonlinear graph optimization framework. The rationality, accuracy, and robustness of the proposed algorithm are verified by simulation analysis and actual test experiments. The results show that the accuracy of the proposed algorithm in the spatial coordinate system relationship calibration was better than 0.08° (3δ) and 5 mm (3δ), respectively, and the time deviation calibration accuracy was better than 0.1 ms and had strong environmental adaptability. This can meet the high-precision calibration requirements of multisensor spatiotemporal parameters of field robot navigation systems. [ABSTRACT FROM AUTHOR]

Multisensor fusion-based road segmentation plays an important role in the intelligent driving system since it provides a drivable area. The existing mainstream fusion method is mainly to feature fusion in the image space domain which causes the perspective compression of the road and damages the performance of the distant road. Considering the bird’s eye views (BEVs) of the LiDAR remains the space structure in the horizontal plane, this article proposes a bidirectional fusion network (BiFNet) to fuse the image and BEV of the point cloud. The network consists of two modules: 1) the dense space transformation (DST) module, which solves the mutual conversion between the camera image space and BEV space and 2) the context-based feature fusion module, which fuses the different sensors information based on the scenes from corresponding features. This method has achieved competitive results on the KITTI dataset. [ABSTRACT FROM AUTHOR]

Environment mapping is an essential prerequisite for mobile robots to perform different tasks such as navigation and mission planning. With the availability of low-cost 2-D LiDARs, there are increasing applications of such 2-D LiDARs in industrial environments. However, environment mapping in an unknown and featureless environment with such low-cost 2-D LiDARs remains a challenge. The challenge mainly originates from the short range of LiDARs and complexities in performing scan matching in these environments. In order to resolve these shortcomings, we propose to fuse the ultrawideband (UWB) with 2-D LiDARs to improve the mapping quality of a mobile robot. The optimization-based approach is utilized for the fusion of UWB ranging information and odometry to first optimize the trajectory. Then, the LiDAR-based loop closures are incorporated to improve the accuracy of the trajectory estimation. Finally, the optimized trajectory is combined with the LiDAR scans to produce the occupancy map of the environment. The performance of the proposed approach is evaluated in an indoor featureless environment with a size of 20 × 20 m. Obtained results show that the mapping error of the proposed scheme is 85.5% less than that of the conventional GMapping algorithm with short-range LiDAR (for example, Hokuyo URG-04LX in our experiment with a maximum range of 5.6 m). [ABSTRACT FROM AUTHOR]

This article focuses on the problem of resilient $H_{\infty }$ filtering for Takagi–Sugeno fuzzy-model-based nonlinear networked systems with multisensors. A weighted fusion approach is adopted before information from multisensors is transmitted over the network. A novel event-triggered mechanism is proposed, which allows us not only to reduce the data-releasing rate but also to prevent abnormal data being potentially transmitted over the network due to sensor measurement or other practical factors. The problem of denial-of-service (DoS) attacks, which often occurs in a communication network, is also considered, where the DoS attack model is based on an assumption that the periodic attack includes active periods and sleeping periods. By employing the idea of the switching model for filtering error systems to deal with DoS attacks, sufficient conditions are derived to guarantee that the filtering error system is exponentially stable. Simulation results are given to demonstrate the effectiveness of the theoretical analysis and design method. [ABSTRACT FROM AUTHOR]

To overcome the limitations of two-dimensional simultaneous localization and mapping (2D-SLAM) technology, which struggles with closed-loop detection in large spatial environments with unstructured and poor lighting conditions, leading to the accumulation of positioning errors, information loss, and low map construction accuracy, solutions need to be found. We focused on the multimodal fusion algorithm of a depth camera and LiDAR based on particle filter at the front end of SLAM. The precision of map information in complex environments is maintained through the utilization of the multilayer iterative closest point matching method for creating accurate laser odometry and screening the loopback candidate set for closed-loop detection of 2D LiDAR data. The loopback information is calibrated through proximity detection, reducing the global map drift phenomenon and enhancing the operational efficiency of the SLAM algorithm through incremental optimization. The public dataset confirms that the proposed algorithm reduces the average relative pose error by 52% compared with the 2D-SLAM algorithm Cartographer. Moreover, it reduces detection time by an average of 15% while maintaining loopback detection accuracy. The SLAM experiment was validated in a complex real-life environment using the Turtlebot2 robot. The results show that the multimodal fusion SLAM method proposed can accurately reconstruct environmental information and achieve high precision and real-time mapping in complex and expansive scenes. [ABSTRACT FROM AUTHOR]

Summary: In case of sharp road illumination changes, bad weather such as rain, snow or fog, wear or missing of the lane marking, the reflective water stain on the road surface, the shadow obstruction of the tree, and mixed lane markings and other signs, missing detection or wrong detection will occur for the traditional lane marking detection algorithm. In this manuscript, a lane marking detection algorithm based on high‐precision map and multisensor fusion is proposed. The basic principle of the algorithm is to use the centimeter‐level high‐precision positioning combined with high‐precision map data to complete the detection of lane markings. In the process of generating high‐precision maps or in the uncovered areas of high‐precision maps, LIDAR (LIght Detection And Ranging) is used to estimate the curvature of the road to assist in lane marking detection. The experimental results show that the algorithm has lower false detection rate in case of bad road conditions, and the algorithm is robust. [ABSTRACT FROM AUTHOR]

This article proposes a new multisensor fusion fault diagnosis method for gearbox, namely residual gated dynamic sparse network, to improve the multisensor feature learning and fusion ability. Considering that the fault sensitivity of the sensor varies with mounted location and complex transfer path modulation causes information from multisensor redundant, the lightweight channel attention unit is designed to strengthen the feature extraction ability of the network. The developed gated dynamic sparse unit is inserted into the deep architecture to eliminate ineffective components caused by high noise interference. Besides, the loss function is improved with multiple activation criteria to enhance convergence ability. The results of experiments and the engineering application show that the proposed method is more effective than other methods under varying degrees of noise interference. [ABSTRACT FROM AUTHOR]

We consider a challenging scenario in this research, where the sensors may receive spurious sensor data, potentially causing inconsistent state estimates. Covariance union (CU) is a fault-tolerant algorithm that can deal with inconsistent state estimation fusion. However, existing CU algorithms suffer from high computational costs due to optimizing nonlinear cost functions when generating fusion weights. To overcome this deficiency, an efficient CU algorithm named fast covariance union (FCU) is developed. We have proved that the fusion weight of FCU can be optimally generated by a closed-form algorithm without optimizing any nonlinear cost function, leading to better fusion efficiency. In addition, the FCU algorithm ensures the fused estimate be consistent as long as one of the estimates is consistent. Finally, the Monte Carlo simulation results show that the FCU algorithm has higher computational efficiency than the existing CU algorithms and handles the spurious sensor data fusion effectively.

Multisensor fusion of visual-inertial-LiDAR systems (VILSs) has become a research hotspot in recent years. Most VILSs require offline calibration of external parameters and hardware time synchronization to ensure successful initialization of the system. However, such requirements are not easy to achieve in practice. We propose an online initialization method for VILSs that can achieve software time synchronization (temporal) and automatically calculate external parameters (spatial) without prior knowledge of environmental information or special movements. Our approach consists of two phases: (1) novel monocular visual inertial (VI) initialization and (2) LiDAR-added (LI) online initialization. In phase 1, our method considers more complete initial states, including the time offsets, biases of the accelerometer, and translation between the inertial measurement unit (IMU) and camera. We implement faster and more robust VI initialization than VINS-MONO. In phase 2, we linearly interpolate the visual-inertial odometry (VIO) onto the same timestamps of the LiDAR and solve the LI external parameters by aligning the VIO with LiDAR odometry (LO). Our LI initialization phase is more practical due to the current lack of an online LI initialization method. Additionally, to eliminate the bundling effect of the initial state estimation in the two phases, we provide a linear solution in nominal-state space to obtain a rough starting value of the rotation extrinsic parameter first. Then, nonlinear optimization under motion constraints is introduced to obtain the full initial states in true-state space. The performance of the proposed method is verified on both a public dataset and a self-assembled handheld VILS device in real-world experiments. [ABSTRACT FROM AUTHOR]

Assessing building performance related to energy consumption in post-design-occupancy stage requires knowledge of building occupancy pattern. These occupancy data can potentially be collected from trials and used to improve the prediction capability of building performance models. Due to the limitations of passive sensors in detecting an individual's occupancy throughout the building, indoor positioning can provide a viable alternative. Previous work on using indoor positioning techniques for detecting building occupancy mainly focused on passive monitoring through Wi-Fi or BLE proximity sensing by estimating the number of occupants at any given time. This paper extends our previous research and demonstrates the merit of occupancy monitoring through active tracking at an individual level using a smartphone-based multi-floor indoor positioning system. The paper discusses the design of a novel occupancy detection trial setup, mimicking real-world office occupancy and discusses the outcome of the ecologically valid trials using the developed positioning system. In total 50 occupancy trials were carried out by around 22 participants comprising of a variety of routes within the building. The trial results are presented to demonstrate the level of accuracy achievable against a specific set of the performance metric necessary for building occupancy detection and modelling. [ABSTRACT FROM AUTHOR]

Indoor localization problems are difficult due to that the information, such as WLAN and GPS, cannot achieve enough precision for indoor issues. This paper presents a novel indoor localization algorithm, GeoLoc, with uncertainty eliminate based on fusion of acceleration, angular rate, and magnetic field sensor data. The algorithm can be deployed in edge devices to overcome the problems of insufficient computing resources and long delay caused by high complexity of location calculation. Firstly, the magnetic map is built and magnetic values are matched. Secondly, orientation updating and position selection are iteratively executed using the fusion data, which gradually reduce uncertainty of orientation. Then, we filter the trajectory from a path set. By gradually reducing uncertainty, GeoLoc can bring a high positioning precision and a smooth trajectory. In addition, this method has an advantage in that it does not rely on any infrastructure such as base stations and beacons. It solves the common problems regarding the non-uniqueness of the geomagnetic fingerprint and the deviation of the sensor measurement. The experimental results show that our algorithm achieves an accuracy of less than 2.5 m in indoor environment, and the positioning results are relatively stable. It meets the basic requirements of indoor location-based services (LBSs).

As cameras and inertial sensors are becoming ubiquitous in mobile devices and robots, it holds great potential to design visual-inertial navigation systems (VINS) for efficient versatile 3-D motion tracking, which utilize any (multiple) available cameras and inertial measurement units (IMUs) and are resilient to sensor failures or measurement depletion. To this end, rather than the standard VINS paradigm using a minimal sensing suite of a single camera and IMU, in this article, we design a real-time consistent multi-IMU multi-camera (MIMC) VINS estimator that is able to seamlessly fuse multimodal information from an arbitrary number of uncalibrated cameras and IMUs. Within an efficient multi-state constraint Kalman filter framework, the proposed MIMC-VINS algorithm optimally fuses asynchronous measurements from all sensors while providing smooth, uninterrupted, and accurate 3-D motion tracking even if some sensors fail. The key idea of the proposed MIMC-VINS is to perform high-order on-manifold state interpolation to efficiently process all available visual measurements without increasing the computational burden due to estimating additional sensors’ poses at asynchronous imaging times. In order to fuse the information from multiple IMUs, we propagate a joint system consisting of all IMU states while enforcing rigid-body constraints between the IMUs during the filter update stage. Finally, we estimate online both spatiotemporal extrinsic and visual intrinsic parameters to make our system robust to errors in prior sensor calibration. The proposed system is extensively validated in both Monte Carlo simulations and real-world experiments. [ABSTRACT FROM AUTHOR]

Background: With the projected upsurge in the percentage of people with some form of disability, there has been a significant increase in the need for assistive mobility devices. However, for mobility aids to be effective, such devices should be adapted to the user's needs. This can be achieved by improving the confidence of the acquired information (interaction between the user, the environment, and the device) following design specifications. Therefore, there is a need for literature review on the adaptability of assistive mobility devices. Objective: In this study, we aim to review the adaptability of assistive mobility devices and the role of the internet of medical things in terms of the acquired information for assistive mobility devices. We review internet-enabled assistive mobility technologies and non--internet of things (IoT) assistive mobility devices. These technologies will provide awareness of the status of adaptive mobility technology and serve as a source and reference regarding information to health care professionals and researchers. Methods: We performed a literature review search on the following databases of academic references and journals: Google Scholar, ScienceDirect, Institute of Electrical and Electronics Engineers, Springer, and websites of assistive mobility and foundations presenting studies on assistive mobility found through a generic Google search (including the World Health Organization website). The following keywords were used: assistive mobility OR assistive robots, assistive mobility devices, internet-enabled assistive mobility technologies, IoT Framework OR IoT Architecture AND for Healthcare, assisted navigation OR autonomous navigation, mobility AND aids OR devices, adaptability of assistive technology, adaptive mobility devices, pattern recognition, autonomous navigational systems, human-robot interfaces, motor rehabilitation devices, perception, and ambient assisted living. Results: We identified 13,286 results (excluding titles that were not relevant to this study). Then, through a narrative review, we selected 189 potential studies (189/13,286, 1.42%) from the existing literature on the adaptability of assistive mobility devices and IoT frameworks for assistive mobility and conducted a critical analysis. Of the 189 potential studies, 82 (43.4%) were selected for analysis after meeting the inclusion criteria. On the basis of the type of technologies presented in the reviewed articles, we proposed a categorization of the adaptability of smart assistive mobility devices in terms of their interaction with the user (user system interface), perception techniques, and communication and sensing frameworks. Conclusions: We discussed notable limitations of the reviewed literature studies. The findings revealed that an improvement in the adaptation of assistive mobility systems would require a reduction in training time and avoidance of cognitive overload. Furthermore, sensor fusion and classification accuracy are critical for achieving real-world testing requirements. Finally, the trade-off between cost and performance should be considered in the commercialization of these devices. [ABSTRACT FROM AUTHOR]

Fall events among older adults are a serious concern, having an impact on their health and well-being. The development of the Internet of Things (IoT) over the last years has led to the emergence of systems able to track abnormal body movements and falls, thus facilitating fall detection and in some cases prevention. Fusing information from multiple unrelated sources is one of the recent trends in healthcare systems. This work aims to provide a survey of recent methods and trends of multisensor data fusion in fall detection systems and discuss their performance, challenges, and limitations. The paper highlights the benefits of developing multimodal systems for fall detection compared to single-sensor approaches, categorizes the different methods applied to this field, and discusses issues and trends for future work. [ABSTRACT FROM AUTHOR]

The fusion of light detection and ranging (LiDAR) and inertial measurement unit (IMU) sensing information can effectively improve the environment modeling and localization accuracy of navigation systems. To realize the spatiotemporal unification of data collected by the IMU and the LiDAR, a two-step spatiotemporal calibration method combining coarse and fine is proposed. The method mainly includes two aspects: (1) Modeling continuous-time trajectories of IMU attitude motion using B-spline basis functions; the motion of the LiDAR is estimated by using the normal distributions transform (NDT) point cloud registration algorithm, taking the Hausdorff distance between the local trajectories as the cost function and combining it with the hand–eye calibration method to solve the initial value of the spatiotemporal relationship between the two sensors’ coordinate systems, and then using the measurement data of the IMU to correct the LiDAR distortion. (2) According to the IMU preintegration, and the point, line, and plane features of the lidar point cloud, the corresponding nonlinear optimization objective function is constructed. Combined with the corrected LiDAR data and the initial value of the spatiotemporal calibration of the coordinate systems, the target is optimized under the nonlinear graph optimization framework. The rationality, accuracy, and robustness of the proposed algorithm are verified by simulation analysis and actual test experiments. The results show that the accuracy of the proposed algorithm in the spatial coordinate system relationship calibration was better than 0.08° (3δ) and 5 mm (3δ), respectively, and the time deviation calibration accuracy was better than 0.1 ms and had strong environmental adaptability. This can meet the high-precision calibration requirements of multisensor spatiotemporal parameters of field robot navigation systems.

In recent years, the use of wearable devices to track athlete performance has increased sharply. Using onboard sensors, wearable devices can provide critical information about athlete’s performance and well-being. Athlete tracking is an important functionality of wearable devices that rely on positioning data, which also influences the accuracy of numerous other attributes. However, accurate athlete tracking is a challenging task due to the nonlinear nature of the problem and the presence of non-Gaussian noise. In the literature, researchers have used the particle filter (PF) to improve athlete tracking accuracy. While the PF algorithm, in general, works well, they perform poorly when athletes take the sharp change of direction (COD), a common and important movement in the sport that is not currently captured. In this article, we introduce a sensor fusion technique to address this challenge. Our proposed solution combines the positioning data and inertial sensor data to accurately track an athlete’s movements. We then analyze the accuracy using data collected from a commercially used athlete tracking wearable device. We have found that the obtained results are very promising, and the proposed solution performs up to five times better than a conventional PF sensor fusion algorithm for positioning. [ABSTRACT FROM AUTHOR]

Precision measurement and reconstruction of detailed surfaces topography is a challenging task for non-diffuse complex parts. Although coordinate measurement machines (CMM) with the touch-trigger probe are widely used in current industry, the measurement efficiency limits their application in the measurement of complex surfaces. This article proposes a multisensor data fusion strategy by integrating the technical merits of CMM and photometric stereo (PS) to achieve multiscale reconstruction of a complex surface with high efficiency. Considering the complementary measurement characteristics of the two approaches, the sparse points from CMM are used to provide global shape information, and the high-resolution surface normal map from PS is used to provide local detailed structure. A multistage neural network is then proposed to fuse these two kinds of modality information such that the global features from the sparse points and the local features from the surface normal map are fused in a coarse-to-fine multistage process so as to make the training process more stable and the reconstruction more accurate. To enhance the generality of the fusion neural network, a synthetic training data set is also designed to include a large variety of multiscale features enriched surfaces. Experiments are conducted to verify the effectiveness of the proposed multisensor fusion strategy in accurate reconstruction of complex surfaces with high efficiency. [ABSTRACT FROM AUTHOR]

This article presents an adaptive fusion method for a multisensor odometry using a fuzzy inference system and applies it to a visual-inertial measurement unit (IMU)-wheel odometry. This article first presents the mechanism of the multisensor odometry with an error state Kalman filter and points out that the assumption of the constant covariance in the prediction and measurement models are invalid due to dynamical interior and exterior abnormities. The previous work of fuzzy logic Kalman filter can only deal with uncertain uncertainty in the measurement model and thus not suitable for this double stage variant uncertainty problem. This article provides a solution using adaptive network fuzzy inference systems. The abnormal working conditions of the two models can be detected online separately, and the fusion gains are tuned accordingly. The training method of the networks is also provided to further improve the fuzzy inference system when the ground truth of the trajectories is available. Experimental results show that the proposed adaptive fusion method can significantly improve the robustness and accuracy of the fused trajectory estimation, even under severe working conditions. [ABSTRACT FROM AUTHOR]

The track line suffers from track irregularity, subgrade settlement, rail surface defects, and gauge distortion. Measuring the track geometry quality plays an important role in rail administrations to safe operation and track maintenance. In order to meet the requirements of track daily inspection, this article describes a comprehensive detection system for the track geometry, which is designed by four cameras, an inclinometer, and an inertial measurement unit. Moreover, the detection system has the characteristics of simple structure, automatic detection, and multiple measurement parameters. First, to measure track alignment and irregularity, a multisensing calibration method is established to fuse visual sensors and inertial sensors. An accelerometer and a Kalman filter are used to eliminate accumulated unrealistic drifts of accelerometer and gyroscope. Second, to measure the rail surface defects and gauge, the model of laser and machine vision is deduced to obtain the coordinate values in a unified coordinate system, and an improved iterative closest point method is developed to improve the matching accuracy. Finally, a series of experiments is carried out to validate the measuring performance. The results indicate that the proposed method can accurately measure the track geometry, with the maximum deviations of track alignment, track irregularity, rail defects, and gauge that are less than 0.5, 0.4, 0.3, and 0.1 mm, respectively. The detection system is not only simple in structure but also has multiple detection parameters, which is especially suitable for track daily maintenance. [ABSTRACT FROM AUTHOR]

Tool wear damages the surface quality of the workpiece and increases equipment downtime. Tool wear prediction is of great importance for reducing processing costs and improving processing efficiency. This paper applies multisensor fusion technology to predict tool wear. The cutting force, vibration, and acoustic emission signals are collected simultaneously during the milling process. The time domain, frequency domain, and time–frequency domain characteristics of each signal are extracted, reduced, and filtered through correlation analysis. A GSA-BP prediction model is established by a BP neural network in which the weights and thresholds are optimized through the gravitational search algorithm (GSA). The test results show that the prediction results of the GSA-BP model are in good agreement with the actual wear, and the prediction accuracy is higher than that of the traditional BP neural network model. [ABSTRACT FROM AUTHOR]

Stroke is one of the leading causes of death and disability in the world. The rehabilitation of Patients' limb functions has great medical value, for example, the therapy of functional electrical stimulation (FES) systems, but suffers from effective rehabilitation evaluation. In this paper, six gestures of upper limb rehabilitation were monitored and collected using microelectromechanical systems sensors, where data stability was guaranteed using data preprocessing methods, that is, deweighting, interpolation, and feature extraction. A fully connected neural network has been proposed investigating the effects of different hidden layers, and determining its activation functions and optimizers. Experiments have depicted that a three‐hidden‐layer model with a softmax function and an adaptive gradient descent optimizer can reach an average gesture recognition rate of 97.19%. A stop mechanism has been used via recognition of dangerous gesture to ensure the safety of the system, and the lightweight cryptography has been used via hash to ensure the security of the system. Comparison to the classification models, for example, k‐nearest neighbor, logistic regression, and other random gradient descent algorithms, was conducted to verify the outperformance in recognition of upper limb gesture data. This study also provides an approach to creating health profiles based on large‐scale rehabilitation data and therefore consequent diagnosis of the effects of FES rehabilitation. [ABSTRACT FROM AUTHOR]

The spatial location of fuze detonation point is crucial for evaluating the working condition and improving the performance of fuze. Considering the observation safety, the non-contact long-distance accurate measurement technology is essential. In this paper, we propose a method that takes the sensor data of optics, spatial attitude and GPS as input and outputs the spatial position of the fuze detonation point. The proposed method consists of two steps. First, an object detection algorithm with post-processing algorithm is proposed to obtain rich information of the target. The algorithm achieves high-accuracy detection by introducing powerful backbone, attention mechanism, group convolution, and improved multi-scale feature fusion. Second, a Variational Auto-Encoder (VAE) algorithm model improved by dense connection structure and multiple source heterogeneous sensor information fusion structure is proposed as the position regression algorithm. It receives the status information of the observer camera and the output of the object detection algorithm, and then outputs the three-dimensional coordinates of the explosion point. Finally, method validation and performance analysis are realized through virtual scene simulation. Experiment results show the superiority of the proposed object detection algorithm over other typical algorithms on explosion flare detection, with its Average Precision (AP) of 0.889. The positioning error of the spatial location method is 0.896 m, while that of the binocular stereo vision method is 2.863 m. Therefore, the proposed target spatial location method is proved to be effective and accurate. [ABSTRACT FROM AUTHOR]

The map recovery and fusion is a key issue in the application of large scale and long-term augmented reality (AR) scenarios. However, they are still not addressed well in an efficient and precise way, especially for complex industrial environments. In this article, we propose a map recovery and fusion strategy based on vision-inertial simultaneous localization and mapping. We first develop a heuristic strategy that can fast search and match map points among multiple maps, and can be used for efficient map fusion. For map recovery, we leverage the inertial sensors for short time motion estimation, and transform the previous lost map to the current map. Based on this strategy, a novel framework for collaborative AR is implemented and can parallelly run in multiple mobile devices in real time. Extensive experiments have been carried out on a public data set, and the results show that the proposed method can recovery and fuse multiple maps with high completeness and precision. [ABSTRACT FROM AUTHOR]

This paper presents a framework for processing, modeling, and fusing underwater sensor signals to provide a reliable perception for underwater localization in structured environments. Submerged sensory information is often affected by diverse sources of uncertainty that can deteriorate the positioning and tracking. By adopting uncertain modeling and multi-sensor fusion techniques, the framework can maintain a coherent representation of the environment, filtering outliers, inconsistencies in sequential observations, and useless information for positioning purposes. We evaluate the framework using cameras and range sensors for modeling uncertain features that represent the environment around the vehicle. We locate the underwater vehicle using a Sequential Monte Carlo (SMC) method initialized from the GPS location obtained on the surface. The experimental results show that the framework provides a reliable environment representation during the underwater navigation to the localization system in real-world scenarios. Besides, they evaluate the improvement of localization compared to the position estimation using reliable dead-reckoning systems. [ABSTRACT FROM AUTHOR]

The simultaneous use of remote sensing sensors has been of great interest due to reasons such as the weaknesses of each sensor individually and the improvement in results by integrating multisensor information. In this context, decision fusion of multiremote sensing sensors is one of the hot research topics for improvement in land cover classification. This paper investigates a multisensor fusion system based on Choquet fuzzy integral and a modified particle swarm optimization for fusion of high-resolution visible RGB image and thermal infrared hyperspectral. Instead of using common optimization algorithms, a new version of PSO is used to reduce the running time and to increase the accuracy. After some processes and fuzzy classifications, a fuzzy integral–fuzzy measure fusion methodology which uses modified PSO to find the best subset of fuzzy measures is utilized to fuse the resulted decision profiles of soft decision-making system. Also, comparison between proposed method and other successful decision fusion techniques such as Dempsters shafer, AdaBoost and Bagging is investigated. Experiments are executed on high-resolution RGB image and thermal infrared hyperspectral data from Quebec of Canada. The results show that the suggested method obtains 92% of overall accuracy and outperforms the other decision fusion and optimization methods in classification performance. [ABSTRACT FROM AUTHOR]