Search

Ensuring safe human-robot interactions is a crucial concern in robotics research and development. However, controlling the social distance between quadruped robots and humans poses a significant challenge due to the large motion range of robots and the lack of specific social distance rules for human safety. This study proposes an innovative framework that integrates proxemic theory and ISO safety strategy to provide a new social distance control mechanism for human-quadruped robot interactions. The proposed framework uses a gated spike filter neural network (GSN) that fuses LiDAR and RGB-D data to estimate human-robot distance and ensures quantified human-robot social distance with continuous velocity limitations, which current approaches do not offer. The experimental results demonstrate that the GSN algorithm offers high accuracy and is well-suited for large-range human-robot distance control of quadruped robots. It has an average prediction error that is 31.4% lower than traditional Kalman filter methods and offers a more efficient and precise solution for real-time social distance control. Additionally, this study includes a detailed use case that demonstrates how to apply the proposed framework to human-quadruped robot interactions. Overall, this study's findings have implications for improving the safety and efficiency of human-robot interactions in various applications. [ABSTRACT FROM AUTHOR]

This study aims to improve the performance and generalization capability of end-to-end autonomous driving with scene understanding leveraging deep learning and multimodal sensor fusion techniques. The designed end-to-end deep neural network takes as input the visual image and associated depth information in an early fusion level and outputs the pixel-wise semantic segmentation as scene understanding and vehicle control commands concurrently. The end-to-end deep learning-based autonomous driving model is tested in high-fidelity simulated urban driving conditions and compared with the benchmark of CoRL2017 and NoCrash. The testing results show that the proposed approach is of better performance and generalization ability, achieving a 100% success rate in static navigation tasks in both training and unobserved situations, as well as better success rates in other tasks than the prior models. A further ablation study shows that the model with the removal of multimodal sensor fusion or scene understanding pales in the new environment because of the false perception. The results verify that the performance of our model is improved by the synergy of multimodal sensor fusion with scene understanding subtask, demonstrating the feasibility and effectiveness of the developed deep neural network with multimodal sensor fusion. [ABSTRACT FROM AUTHOR]

Transportation and locomotion mode recognition from multimodal smartphone sensors is useful for providing just-in-time context-aware assistance. However, the field is currently held back by the lack of standardized datasets, recognition tasks, and evaluation criteria. Currently, the recognition methods are often tested on the ad hoc datasets acquired for one-off recognition problems and with different choices of sensors. This prevents a systematic comparative evaluation of methods within and across research groups. Our goal is to address these issues by: 1) introducing a publicly available, large-scale dataset for transportation and locomotion mode recognition from multimodal smartphone sensors; 2) suggesting 12 reference recognition scenarios, which are a superset of the tasks we identified in the related work; 3) suggesting relevant combinations of sensors to use based on energy considerations among accelerometer, gyroscope, magnetometer, and global positioning system sensors; and 4) defining precise evaluation criteria, including training and testing sets, evaluation measures, and user-independent and sensor-placement independent evaluations. Based on this, we report a systematic study of the relevance of statistical and frequency features based on the information theoretical criteria to inform recognition systems. We then systematically report the reference performance obtained on all the identified recognition scenarios using a machine-learning recognition pipeline. The extent of this analysis and the clear definition of the recognition tasks enable future researchers to evaluate their own methods in a comparable manner, thus contributing to further advances in the field. The dataset and the code are available online. http://www.shl-dataset.org/.

This article presents a survey of simultaneous localization and mapping (SLAM) and data fusion techniques for object detection and environmental scene perception in unmanned aerial vehicles (UAVs). We critically evaluate some current SLAM implementations in robotics and autonomous vehicles and their applicability and scalability to UAVs. SLAM is envisioned as a potential technique for object detection and scene perception to enable UAV navigation through continuous state estimation. In this article, we bridge the gap between SLAM and data fusion in UAVs while also comprehensively surveying related object detection techniques such as visual odometry and aerial photogrammetry. We begin with an introduction to applications where UAV localization is necessary, followed by an analysis of multimodal sensor data fusion to fuse the information gathered from different sensors mounted on UAVs. We then discuss SLAM techniques such as Kalman filters and extended Kalman filters to address scene perception, mapping, and localization in UAVs. The findings are summarized to correlate prevalent and futuristic SLAM and data fusion for UAV navigation, and some avenues for further research are discussed.

With a large number of video surveillance systems installed for the requirement from industrial security, the task of object tracking, which aims to locate objects of interest in videos, is very important. Although numerous tracking algorithms for RGB videos have been developed in the decade, the tracking performance and robustness of these systems may be degraded dramatically when the information from RGB video is unreliable (e.g., poor illumination conditions or very low resolution). To address this issue, this paper presents a new tracking system, which aims to combine the information from RGB and infrared modalities for object tracking. The proposed tracking systems is based on our proposed machine learning model. Particularly, the learning model can alleviate the modality discrepancy issue under the proposed modality consistency constraint from both representation patterns and discriminability, and generate discriminative feature templates for collaborative representations and discrimination in heterogeneous modalities. Experiments on a variety of challenging RGB-infrared videos demonstrate the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Localisation of geometric features like holes, edges, slots, etc. is vital to robotic planning in industrial automation settings. Low-cost 3D scanners are crucial in terms of improving accessibility, but pose a practical challenge to feature localisation because of poorer resolution and consequently affect robotic planning. In this work, we address the possibility of enhancing the quality of a 3D scan by a manual ’touch-up’ of task-relevant features, to ensure their automatic detection prior to automation. We propose a framework whereby the operator (i) has access to both the actual work-piece and its 3D scan; (ii) evaluates the missing salient features from the scan; (iii) uses a haptic stylus to physically interact with the actual work-piece, around such specific features; (iv) interactively updates the scan using the position and force information from the haptic stylus. The contribution of this work is the use of haptic mismatch for geometric update. Specifically, the geometry from the 3D scan is used to predict haptic feedback at a point on the work-piece surface. The haptic mismatch is derived as a measure of error between this prediction and the real interaction forces from physical contact at that point on the work-piece. The geometric update is driven until the haptic mismatch is minimised. Convergence of the proposed algorithm is first numerically verified on an analytical surface with simulated physical interaction. Error analysis of the surface position and orientations were also plotted. Experiments were conducted using a motion capture system providing sub-mm accuracy in position and a 6 axis F/T sensor. Missing features are successfully detected after the update of the scan using the proposed method in an experiment.

Medical imaging sensors, such as positron emission tomography and single-photon emission computed tomography, can provide rich information, but each has its inherent drawbacks. In this scenario, multimodal sensor medical image fusion becomes an effective solution. The chief objective of medical imaging is to extract as much preponderant and complementary information as possible from the source into a single output that can play a critical role in medical diagnosis and clinical operations. In this paper, a novel fusion method is presented for multimodal sensor medical images, based on local difference (LD) in non-subsampled domain. In this method, the source medical images are first decomposed into low-frequency and high-frequency subimages, via non-subsampled schemes. Then, the coefficients of sub-bands are fused by an operator, called LD. The final fused image is reconstructed, via the inverse non-subsampled schemes, with all composite coefficients. The proposed fusion method was applied in several clinical studies, and the results show that it is a much more straightforward and effective method than some of the state-of-the-art methods, in terms of both subjective visual performance and objective evaluation results. Also, the performance of the proposed method was compared with that of two non-subsampled schemes, namely, non-subsampled contourlet transform and non-subsampled shearlet transform. [ABSTRACT FROM AUTHOR]

Detail information on objects of interest plays a vital role in current medical diagnosis. However, the existing multimodal sensor fusion methods cause problems of low contrast and color distortion during the process of integration. Therefore, the preservation of detail information in high contrast is worthy of investment in the field of medical image fusion. This paper presents a new multiscale fusion-based framework using the local Laplacian pyramid transform (LLP) and adaptive cloud model (ACM). The proposed framework, LLP+ACM, includes three key modules. First, the input images are decomposed into detail-enhanced approximate and residual images using LLP. Second, ACM is adopted to fuse the approximate images. A salience match tool is used to fuse the residual images. Third, the fused image is reconstructed using the inversed LLP. Experiments show that the proposed LLP+ACM significantly improves detail information with high contrast and reduces the color distortion of the fused images in both subjective and objective evaluations. [ABSTRACT FROM AUTHOR]

The problem of information fusion from multiple data-sets acquired by multimodal sensors has drawn significant research attention over the years. In this paper, we focus on a particular problem setting consisting of a physical phenomenon or a system of interest observed by multiple sensors. We assume that all sensors measure some aspects of the system of interest with additional sensor-specific and irrelevant components. Our goal is to recover the variables relevant to the observed system and to filter out the nuisance effects of the sensor-specific variables. We propose an approach based on manifold learning, which is particularly suitable for problems with multiple modalities, since it aims to capture the intrinsic structure of the data and relies on minimal prior model knowledge. Specifically, we propose a nonlinear filtering scheme, which extracts the hidden sources of variability captured by two or more sensors, that are independent of the sensor-specific components. In addition to presenting a theoretical analysis, we demonstrate our technique on real measured data for the purpose of sleep stage assessment based on multiple, multimodal sensor measurements. We show that without prior knowledge on the different modalities and on the measured system, our method gives rise to a data-driven representation that is well correlated with the underlying sleep process and is robust to noise and sensor-specific effects. [ABSTRACT FROM AUTHOR]

In object detection systems for autonomous driving, LIDAR sensors provide very useful information. However, problems occur because the object representation is greatly distorted by changes in distance. To solve this problem, we propose a LIDAR shape set that reconstructs the shape surrounding the object more clearly by using the LIDAR point information projected on the object. The LIDAR shape set restores object shape edges from a bird’s eye view by filtering LIDAR points projected on a 2D pixel-based front view. In this study, we use this shape set for two purposes. The first is to supplement the shape set with a LIDAR Feature map, and the second is to divide the entire shape set according to the gradient of the depth and density to create a 2D and 3D bounding box proposal for each object. We present a multimodal fusion framework that classifies objects and restores the 3D pose of each object using enhanced feature maps and shape-based proposals. The network structure consists of a VGG -based object classifier that receives multiple inputs and a LIDAR-based Region Proposal Networks (RPN) that identifies object poses. It works in a very intuitive and efficient manner and can be extended to other classes other than vehicles. Our research has outperformed object classification accuracy (Average Precision, AP) and 3D pose restoration accuracy (3D bounding box recall rate) based on the latest studies conducted with KITTI data sets. [ABSTRACT FROM AUTHOR]

In this paper, we perform a systematic study about the on-body sensor positioning and data acquisition details for Human Activity Recognition (HAR) systems. We build a testbed that consists of eight body-worn Inertial Measurement Units (IMU) sensors and an Android mobile device for activity data collection. We develop a Long Short-Term Memory (LSTM) network framework to support training of a deep learning model on human activity data, which is acquired in both real-world and controlled environments. From the experiment results, we identify that activity data with sampling rate as low as 10 Hz from four sensors at both sides of wrists, right ankle, and waist is sufficient in recognizing Activities of Daily Living (ADLs) including eating and driving activity. We adopt a two-level ensemble model to combine class-probabilities of multiple sensor modalities, and demonstrate that a classifier-level sensor fusion technique can improve the classification performance. By analyzing the accuracy of each sensor on different types of activity, we elaborate custom weights for multimodal sensor fusion that reflect the characteristic of individual activities.

In object detection systems for autonomous driving, LIDAR sensors provide very useful information. However, problems occur because the object representation is greatly distorted by changes in distance. To solve this problem, we propose a LIDAR shape set that reconstructs the shape surrounding the object more clearly by using the LIDAR point information projected on the object. The LIDAR shape set restores object shape edges from a bird’s eye view by filtering LIDAR points projected on a 2D pixel-based front view. In this study, we use this shape set for two purposes. The first is to supplement the shape set with a LIDAR Feature map, and the second is to divide the entire shape set according to the gradient of the depth and density to create a 2D and 3D bounding box proposal for each object. We present a multimodal fusion framework that classifies objects and restores the 3D pose of each object using enhanced feature maps and shape-based proposals. The network structure consists of a VGG -based object classifier that receives multiple inputs and a LIDAR-based Region Proposal Networks (RPN) that identifies object poses. It works in a very intuitive and efficient manner and can be extended to other classes other than vehicles. Our research has outperformed object classification accuracy (Average Precision, AP) and 3D pose restoration accuracy (3D bounding box recall rate) based on the latest studies conducted with KITTI data sets.

Over the past years, inverse perspective mapping has been successfully applied to several problems in the field of Intelligent Transportation Systems. In brief, the method consists of mapping images to a new coordinate system where perspective effects are removed. The removal of perspective associated effects facilitates road and obstacle detection and also assists in free space estimation. There is, however, a significant limitation in the inverse perspective mapping: the presence of obstacles on the road disrupts the effectiveness of the mapping. The current paper proposes a robust solution based on the use of multimodal sensor fusion. Data from a laser range finder is fused with images from the cameras, so that the mapping is not computed in the regions where obstacles are present. As shown in the results, this considerably improves the effectiveness of the algorithm and reduces computation time when compared with the classical inverse perspective mapping. Furthermore, the proposed approach is also able to cope with several cameras with different lenses or image resolutions, as well as dynamic viewpoints. [ABSTRACT FROM AUTHOR]

Multimodal medical image fusion plays a vital role in different clinical imaging sensor applications. This paper presents a novel multimodal medical image fusion method that adopts a multiscale geometric analysis of the nonsubsampled contourlet transform (NSCT) with type-2 fuzzy logic techniques. First, the NSCT was performed on preregistered source images to obtain their high- and low-frequency subbands. Next, an effective type-2 fuzzy logic-based fused rule is proposed for fusion of the high-frequency subbands. In the presented fusion approach, the local type-2 fuzzy entropy is introduced to automatically select high-frequency coefficients. However, for the low-frequency subbands, they were fused by a local energy algorithm based on the corresponding image’s local features. Finally, the fused image was constructed by the inverse NSCT with all composite subbands. Both subjective and objective evaluations showed better contrast, accuracy, and versatility in the proposed approach compared with state-of-the-art methods. Besides, an effective color medical image fusion scheme is also given in this paper that can inhibit color distortion to a large extent and produce an improved visual effect. [ABSTRACT FROM PUBLISHER]

[Display omitted] • Multimodal occupancy detection framework for residential buildings. • Two variants of Occ-STPN models with feature level fusion and decision level fusion. • Feature extraction and occupancy detection models for various data modalities. • Transferability of a trained occupancy detection model from one building to another. • Occupancy prediction performance evaluation on three open-sourced datasets. For several years now, smart building energy systems have been a research area of intensive activity. In light of the increasing need for sustainable buildings and energy systems, this trend motivates an increasing need for a solution to reduce carbon dioxide emissions and improve energy efficiency. This paper proposes a high-performing and transferable occupancy detection framework that combines sensor data from different data modalities, including time series environmental data (temperature, humidity, and illuminance), image data, and acoustic energy data using ensemble method. To draw out the best prediction performance in each modality, the proposed framework was developed, including various models that were designed to learn the occupancy patterns reflected in the physical data streams. To tackle the time series environmental data, we designed two variants of an occupancy detection spatiotemporal pattern network (Occ-STPN) that performs both feature level and decision level fusion, respectively. We also propose a new metric; the fading memory mean square error (FMMSE), that provides a fair evaluation and penalization of delayed occupancy predictions. Multiple open-sourced datasets, including the Electricity Consumption and Occupancy and the University of California, Irvine's (UCI) building occupancy detection dataset, along with our own real data collected from six different houses, were used to validate the algorithms' performance. The experimental results presented herein break down the performance for each sensing modality, and a detailed analysis of the performance is also discussed. [ABSTRACT FROM AUTHOR]

Our research goal is to prove the hypothesis that using multiple modalities, dialog management, context, and semantics improves HRI over using a single modality by testing the hypothesis using a single implementation and application. We tested the hypothesis by simulating a domestic robot that can be taught to clean a house using a multi-modal interface. The multi-modal interface included speech, pointing, field of vision, and head nodding. The contexts included real world context and dialog history. We tested three different dialog-grounding strategies: optimistic, cautious, and pessimistic. We measured the learning process in two ways: dialog succinctness and learning accuracy. Experimental evidence shows that: multiple modalities and contexts improve HRI over speech by itself for various ratios of accuracy to succinctness; the dialog grounding strategy that improves HRI the best depends on the desired ratio of accuracy to succinctness, with optimistic grounding being the best for ratios ≤0.8, and pessimistic grounding being the best otherwise; and the Pareto principle applies to which grounding strategy is the best. [ABSTRACT FROM AUTHOR]

• A therapy performance assessment method for upper limb rehabilitation. • Patient participation evaluation based on a multimodal sensor fusion. • Performance, tiredness, and slacking assessment using fuzzy inference system. • System validation by using sEMG signals. The patient's active participation in exercises is a crucial factor to increase the functional outputs received from therapy. For improving the patient's active and voluntary involvement, the difficulty levels of therapy tasks and the device assistance are adjusted based on the patient's therapy performance. However, the existing performance evaluation methods are based on either some specific device designs or certain therapy tasks. In this work, a patient performance evaluation method is proposed based on a multimodal sensor fusion formed by the trajectory tracking error signal and the patient's physiological responses (heart rate and skin conductance) during the upper extremity rehabilitation. The novelty of the system is assessing the patient's performance independently from any device designs or therapy tasks. The developed system also evaluates the patient's tiredness and slacking, which are substantial factors affecting the therapy performance. The patient's upper limb joints' angles are measured via inertial measurement unit sensors. Arm movements of the patient are estimated by an upper limb kinematic module for evaluating the trajectory tracking. The patient's performance, tiredness, and slacking are assessed by a fuzzy inference system using physiological responses and exercise profile. The developed system is tested experimentally with healthy subjects on five therapy tasks. Also, for demonstrating the proposed method efficacy, additional experiments have been performed for different cases while measuring the sEMG signals of the subjects. The experimental results showed that the proposed system estimates subjects' participation successfully and adjusts the therapy tasks according to subjects' performance and tiredness. [ABSTRACT FROM AUTHOR]

Localisation of geometric features like holes, edges, slots, etc. is vital to robotic planning in industrial automation settings. Low-cost 3D scanners are crucial in terms of improving accessibility, but pose a practical challenge to feature localisation because of poorer resolution and consequently affect robotic planning. In this work, we address the possibility of enhancing the quality of a 3D scan by a manual 'touch-up' of task-relevant features, to ensure their automatic detection prior to automation. We propose a framework whereby the operator (i) has access to both the actual work-piece and its 3D scan; (ii) evaluates the missing salient features from the scan; (iii) uses a haptic stylus to physically interact with the actual work-piece, around such specific features; (iv) interactively updates the scan using the position and force information from the haptic stylus. The contribution of this work is the use of haptic mismatch for geometric update. Specifically, the geometry from the 3D scan is used to predict haptic feedback at a point on the work-piece surface. The haptic mismatch is derived as a measure of error between this prediction and the real interaction forces from physical contact at that point on the work-piece. The geometric update is driven until the haptic mismatch is minimised. Convergence of the proposed algorithm is first numerically verified on an analytical surface with simulated physical interaction. Error analysis of the surface position and orientations were also plotted. Experiments were conducted using a motion capture system providing sub-mm accuracy in position and a 6 axis F/T sensor. Missing features are successfully detected after the update of the scan using the proposed method in an experiment. [ABSTRACT FROM AUTHOR]

In this paper, we perform a systematic study about the on-body sensor positioning and data acquisition details for Human Activity Recognition (HAR) systems. We build a testbed that consists of eight body-worn Inertial Measurement Units (IMU) sensors and an Android mobile device for activity data collection. We develop a Long Short-Term Memory (LSTM) network framework to support training of a deep learning model on human activity data, which is acquired in both real-world and controlled environments. From the experiment results, we identify that activity data with sampling rate as low as 10 Hz from four sensors at both sides of wrists, right ankle, and waist is sufficient in recognizing Activities of Daily Living (ADLs) including eating and driving activity. We adopt a two-level ensemble model to combine class-probabilities of multiple sensor modalities, and demonstrate that a classifier-level sensor fusion technique can improve the classification performance. By analyzing the accuracy of each sensor on different types of activity, we elaborate custom weights for multimodal sensor fusion that reflect the characteristic of individual activities. [ABSTRACT FROM AUTHOR]