Search

This letter presents a novel and retractable ring-shaped quadrotor called Ring-Rotor that can adjust the vehicle's length and width simultaneously. Unlike other morphing quadrotors with high platform complexity and poor controllability, Ring-Rotor uses only one servo motor for morphing but reduces the largest dimension of the vehicle by approximately 31.4\%. It can guarantee passibility while flying through small spaces in its compact form and energy saving in its standard form. Meanwhile, the vehicle breaks the cross configuration of general quadrotors with four arms connected to the central body and innovates a ring-shaped mechanical structure with spare central space. Based on this, an ingenious whole-body aerial grasping and transportation scheme is designed to carry various shapes of objects without the external manipulator mechanism. Moreover, we exploit a nonlinear model predictive control (NMPC) strategy that uses a time-variant physical parameter model to adapt to the quadrotor morphology. Above mentioned applications are performed in real-world experiments to demonstrate the system's high versatility., 8 pages, accepted by IEEE Robotics and Automation Letters on January, 2023

Modern efficient Convolutional Neural Networks (CNNs) are able to perform semantic segmentation both swiftly and accurately, which covers typically separate detection tasks desired by Intelligent Vehicles (IV) in a unified way. Most of the current semantic perception frameworks are designed to work with pinhole cameras and benchmarked against public datasets with narrow Field-of-View (FoV) images. However, there is a large accuracy downgrade when a pinhole-yielded CNN is taken to omnidirectional imagery, causing it unreliable for surrounding perception. In this paper, we propose an omnisupervised learning framework for efficient CNNs, which bridges multiple heterogeneous data sources that are already available in the community, bypassing the labor-intensive process to have manually annotated panoramas, while improving their reliability in unseen omnidirectional domains. Being omnisupervised, the efficient CNN exploits both labeled pinhole images and unlabeled panoramas. The framework is based on our specialized ensemble method that considers the wide-angle and wrap-around features of omnidirectional images, to automatically generate panoramic labels for data distillation. A comprehensive variety of experiments demonstrates that the proposed solution helps to attain significant generalizability gains in panoramic imagery domains. Our approach outperforms state-of-the-art efficient segmenters on highly unconstrained IDD20K and PASS datasets.

With the development of computer vision and mobile computing, assistive navigation for people with visual impairment arouses the attention of research communities. As two key challenges of assistive navigation, “Where am I?” and “What are the surroundings?” are still to be resolved by taking advantage of visual information. In this paper, we leverage the prevailing compact network as the backbone to build a unified network featuring two branches that implement scene description and scene recognition separately. Based on the unified network, the proposed pipeline performs scene recognition and visual localization simultaneously in the scenario of assistive navigation. The visual localization pipeline involves image retrieval and sequence matching. In the experiments, different configurations of the proposed pipeline are tested on public datasets to search for the optimal parameters. Moreover, on the real-world datasets captured by the wearable assistive device, the proposed assistive navigation pipeline is proved to achieve satisfactory performance. On the challenging dataset, the top-5 precision of scene recognition is more than 80%, and the visual localization precision is over 60% under a recall of 60%. The related codes and datasets are open-source online at https://github.com/chengricky/ScenePlaceRecognition.

Aerial pixel-wise scene perception of the surrounding environment is an important task for UAVs (Unmanned Aerial Vehicles). Previous research works mainly adopt conventional pinhole cameras or fisheye cameras as the imaging device. However, these imaging systems cannot achieve large Field of View (FoV), small size, and lightweight at the same time. To this end, we design a UAV system with a Panoramic Annular Lens (PAL), which has the characteristics of small size, low weight, and a 360-degree annular FoV. A lightweight panoramic annular semantic segmentation neural network model is designed to achieve high-accuracy and real-time scene parsing. In addition, we present the first drone-perspective panoramic scene segmentation dataset Aerial-PASS, with annotated labels of track, field, and others. A comprehensive variety of experiments shows that the designed system performs satisfactorily in aerial panoramic scene parsing. In particular, our proposed model strikes an excellent trade-off between segmentation performance and inference speed suitable, validated on both public street-scene and our established aerial-scene datasets., Comment: Our dataset will be made publicly available at: http://wangkaiwei.org/downloadeg.html

In this paper, we propose panoramic annular simultaneous localization and mapping (PA-SLAM), a visual SLAM system based on panoramic annular lens. A hybrid point selection strategy is put forward in the tracking front-end, which ensures repeatability of keypoints and enables loop closure detection based on the bag-of-words approach. Every detected loop candidate is verified geometrically and the $Sim(3)$ relative pose constraint is estimated to perform pose graph optimization and global bundle adjustment in the back-end. A comprehensive set of experiments on real-world datasets demonstrates that the hybrid point selection strategy allows reliable loop closure detection, and the accumulated error and scale drift have been significantly reduced via global optimization, enabling PA-SLAM to reach state-of-the-art accuracy while maintaining high robustness and efficiency., Accepted to Applied Optics. 12 pages, 11 figures, 3 tables

Depth estimation, as a necessary clue to convert 2D images into the 3D space, has been applied in many machine vision areas. However, to achieve an entire surrounding 360-degree geometric sensing, traditional stereo matching algorithms for depth estimation are limited due to large noise, low accuracy, and strict requirements for multi-camera calibration. In this work, for a unified surrounding perception, we introduce panoramic images to obtain larger field of view. We extend PADENet first appeared in our previous conference work for outdoor scene understanding, to perform panoramic monocular depth estimation with a focus for indoor scenes. At the same time, we improve the training process of the neural network adapted to the characteristics of panoramic images. In addition, we fuse traditional stereo matching algorithm with deep learning methods and further improve the accuracy of depth predictions. With a comprehensive variety of experiments, this research demonstrates the effectiveness of our schemes aiming for indoor scene perception., Comment: Accepted to Applied Optics. Code will be made publicly available at https://github.com/zzzkkkyyy/PADENet

Combining the advantages of quantum computing and neural networks, quantum neural networks (QNNs) have gained considerable attention recently. However, because of the lack of quantum resource, it is costly to train QNNs. In this work, we presented a simple unsupervised method for quantum-classical convolutional networks to learn a hierarchy of quantum feature extractors. Each level of the resulting feature extractors consist of multiple quanvolution filters, followed by a pooling layer. The main contribution of the proposed approach is to use the $K$-means clustering to maximize the difference of quantum properties in quantum circuit ansatz. One experiment on the bearing fault detection task shows the effectiveness of the proposed method.

Scene sonification is a powerful technique to help Visually Impaired People (VIP) understand their surroundings. Existing methods usually perform sonification on the entire images of the surrounding scene acquired by a standard camera or on the priori static obstacles acquired by image processing algorithms on the RGB image of the surrounding scene. However, if all the information in the scene are delivered to VIP simultaneously, it will cause information redundancy. In fact, biological vision is more sensitive to moving objects in the scene than static objects, which is also the original intention of the event-based camera. In this paper, we propose a real-time sonification framework to help VIP understand the moving objects in the scene. First, we capture the events in the scene using an event-based camera and cluster them into multiple moving objects without relying on any prior knowledge. Then, sonification based on MIDI is enabled on these objects synchronously. Finally, we conduct comprehensive experiments on the scene video with sonification audio attended by 20 VIP and 20 Sighted People (SP). The results show that our method allows both participants to clearly distinguish the number, size, motion speed, and motion trajectories of multiple objects. The results show that our method is more comfortable to hear than existing methods in terms of aesthetics.

Nowadays, information technology is indispensable to various education levels, as the cutting-edge technology of modern society. Moreover, Artificial intelligence has significantly affected the educational process of colleges and universities, especially for the practical teaching model. Therefore, starting with the online experimental teaching of electronic information majors, a teaching model that integrates artificial intelligence and information technology is proposed to provide references for future adaptive online experimental learning. The model uses a variety of assisted platforms to provide students and teachers with multimedia experimental guidance. Specifically, it fully collects all kinds of big data in the experiment process, and uses a variety of artificial intelligence algorithms to help students and teachers continue to improve experimental teaching. The quantitative results gathered from real online experimental class show that the teaching mode improves the efficiency of the teaching and learning.

Visual localization in vehicle navigation remains a crucial image retrieval task to determine the best matched image. Developing an efficient algorithm to address the localization issues of vehicle is highly difficult, for severe appearance variations with vehicles moving around can bring about significant challenges and big obstacles. In this paper, we propose the CFVL framework which takes panoramas into use in the localizer and the system processes from coarse to fine, in order to attain more robust and stable descriptors. NetVALD descriptors based on explicit panorama construction, which are regarded robust to appearance changes, are extracted in the coarse stage, while Geodesc keypoint descriptors, which are believed to detect more detailed information, are utilized in the fine stage, so as to perceive the accurate localization. A comprehensive set of experiments is carried on several datasets with different appearances across seasonal cycling, illumination variations, diverse traversals, and so on, to verify the effectiveness of the coarse stage and fine stage in our system. Brute Force (BF) matching and Fundamental Matrix mapping are utilized to match and locate correct locations after coarse stage and after fine stage. The accuracy of the coarse matching and fine matching are verified separately. Our system is demonstrated to be with high location recall, generalization capacity across different environments.

Semantic segmentation renders a unified way of surrounding perception, where most of driving scene detection tasks can be covered by running a single efficient ConvNet through a forward pass. However, current frameworks posit the closed-world paradigm expressed as a single source of distribution over a predetermined set of visual classes, forgetting that a deep model must be deployed in the wild facing unseen domains and unforeseen hazards. In spite of being accurate in its comfort zone, the segmentation model may not generalize well to a new domain. In addition, a model trained with single dataset is heavily limited in terms of recognizable classes. In this paper, we propose an omni-supervised learning framework for semantic segmentation which is able to leverage heterogeneous data sources. Our omni-supervised training framework incorporates all available labeled and unlabeled data, meanwhile bridges multiple training sets to be capable of recognizing more classes that are needed for autonomous navigation application at hand in the new domain. A comprehensive variety of experiments shows that with the proposed multi-source omni-supervised learning solution, an efficient ConvNet like our ERF-PSPNet attains significant robustness gains in open domains that are of critical relevance to real deployment of vision algorithms. Our approach surpasses the state of the art on the highly unconstrained PASS and IDD20K datasets.

Semantically interpreting the traffic scene is crucial for autonomous transportation and robotics systems. However, state-of-the-art semantic segmentation pipelines are dominantly designed to work with pinhole cameras and train with narrow Field-of-View (FoV) images. In this sense, the perception capacity is severely limited to offer higher-level confidence for upstream navigation tasks. In this paper, we propose a network adaptation framework to achieve Panoramic Annular Semantic Segmentation (PASS), which allows to re-use conventional pinhole-view image datasets, enabling modern segmentation networks to comfortably adapt to panoramic images. Specifically, we adapt our proposed SwaftNet to enhance the sensitivity to details by implementing attention-based lateral connections between the detail-critical encoder layers and the context-critical decoder layers. We benchmark the performance of efficient segmenters on panoramic segmentation with our extended PASS dataset, demonstrating that the proposed real-time SwaftNet outperforms state-of-the-art efficient networks. Furthermore, we assess real-world performance when deploying the Detail-Sensitive PASS (DS-PASS) system on a mobile robot and an instrumented vehicle, as well as the benefit of panoramic semantics for visual odometry, showing the robustness and potential to support diverse navigational applications., 8 pages, 10 figures

Semantic segmentation is a critical method in the field of autonomous driving. When performing semantic image segmentation, a wider field of view (FoV) helps to obtain more information about the surrounding environment, making automatic driving safer and more reliable, which could be offered by fisheye cameras. However, large public fisheye datasets are not available, and the fisheye images captured by the fisheye camera with large FoV comes with large distortion, so commonly-used semantic segmentation model cannot be directly utilized. In this paper, a seven degrees of freedom (DoF) augmentation method is proposed to transform rectilinear image to fisheye image in a more comprehensive way. In the training process, rectilinear images are transformed into fisheye images in seven DoF, which simulates the fisheye images taken by cameras of different positions, orientations and focal lengths. The result shows that training with the seven-DoF augmentation can improve the model's accuracy and robustness against different distorted fisheye data. This seven-DoF augmentation provides a universal semantic segmentation solution for fisheye cameras in different autonomous driving applications. Also, we provide specific parameter settings of the augmentation for autonomous driving. At last, we tested our universal semantic segmentation model on real fisheye images and obtained satisfactory results. The code and configurations are released at https://github.com/Yaozhuwa/FisheyeSeg., SMC2020 recieved

Semantic Segmentation (SS) is promising for outdoor scene perception in safety-critical applications like autonomous vehicles, assisted navigation and so on. However, traditional SS is primarily based on RGB images, which limits the reliability of SS in complex outdoor scenes, where RGB images lack necessary information dimensions to fully perceive unconstrained environments. As preliminary investigation, we examine SS in an unexpected obstacle detection scenario, which demonstrates the necessity of multimodal fusion. Thereby, in this work, we present EAFNet, an Efficient Attention-bridged Fusion Network to exploit complementary information coming from different optical sensors. Specifically, we incorporate polarization sensing to obtain supplementary information, considering its optical characteristics for robust representation of diverse materials. By using a single-shot polarization sensor, we build the first RGB-P dataset which consists of 394 annotated pixel-aligned RGB-Polarization images. A comprehensive variety of experiments shows the effectiveness of EAFNet to fuse polarization and RGB information, as well as the flexibility to be adapted to other sensor combination scenarios., Comment: Accepted by Optics Express. 18 pages, 16 figures, 3 tables, 9 equations. Code will be made publicly available at https://github.com/Katexiang/EAFNet

An optical coherence tomography (OCT) image is becoming the standard imaging modality in diagnosing retinal diseases and the assessment of their progression. However, the manual evaluation of the volumetric scan is time consuming, expensive and the signs of the early disease are easy to miss. In this paper, we mainly present an attention-based deep learning method for the retinal disease classification in OCT images, which can assist the large-scale screening or the diagnosis recommendation for an ophthalmologist. First, according to the unique characteristic of a retinal OCT image, we design a customized pre-processing method to improve image quality. Second, in order to guide the network optimization more effectively, a specially designed attention model, which pays more attention to critical regions containing pathological anomalies, is integrated into a typical deep learning network. We evaluate our proposed method on two data sets, and the results consistently show that it outperforms the state-of-the-art methods. We report an overall four-class accuracy of 97.4%, a two-class sensitivity of 100.0%, and a two-class specificity of 100.0% on a public data set shared by Zhang et al. with 1,000 testing B-scans in four disease classes. Compared to their work, our method improves the numbers by 0.8%, 2.2%, and 2.6% respectively.

A nuclear cataract is an age-related chronic and priority ophthalmic disease in which a clouding of the lens in the human eye affects vision. Automatic segmentation of nuclear region based on slit-lamp photographs is a basic step for computer-aided diagnosis such as nuclear cataract grading. However, slit-lamp photographs collected from a clinic scenario often have complex background containing the eyelids, sclera and cornea with spectral highlights. The existing efforts using traditional image processing that have unsatisfactory results, and the deep learning method using standard Faster R-CNN tends to obtain a bigger nuclear contour. In this paper, we propose a coarse-to-fine deep learning solution to localize nuclear regions by cascading the Faster R-CNN in a two-stage framework. First, a nuclear ROI (region of interest) predictor is pre-trained to localize a rough position and remove complex backgrounds. Then, a fine nuclear locator is applied to predict a more compact nuclear bounding box. Finally, an ellipse-like nuclear contour is fitted based on its bounding box. Evaluated on a clinical dataset of 884 slit-lamp photographs, the proposed method outperforms the state-of-the-art, improving the overlapping rate (IoU) by 0.33% from 67.98% to 68.31%, and increasing the success rate by 2.55% from 85.71% to 88.26%.

In recent years, with the vigorous development of artificial intelligence and autonomous driving technology, the importance of scene perception technology is increasing. Unsupervised deep learning based methods have demonstrated a certain level of robustness and accuracy in some challenging scenes. By inferring depth from a single input image without any ground truth label, a lot of time and resources can be saved. However, unsupervised depth estimation has defects in robustness and accuracy under complex environment which could be improved by modifying network structure and incorporating other modal information. In this paper, we propose an unsupervised, monocular depth estimation network achieving high speed and accuracy, and a learning framework with our depth estimation network to improve depth performance by incorporating transformed images across different modalities. The depth estimator is an encoder-decoder network to generate the multi-scale dense depth map. The sub-pixel convolutional layer is adopted to obtain depth super-resolution by replacing the up-sample branches. The cross-modal depth estimation using near-infrared image and RGB image enhances the performance of depth estimation than pure RGB image. The training mode is to transfer both images to the same modality and then carry out super-resolved depth estimation for each stereo camera pair. Compared with the initial results of depth estimation using only RGB images, the experiment verifies that our depth estimation network with the cross-modal fusion system designed in this paper achieves better performance on public datasets and a multi-modal dataset collected by our stereo vision sensor.

The Visually Impaired People (VIP) have the difficulty in perceiving the accurate localization in their daily life. Developing an efficient algorithm to address the localization issues of the VIP is crucial. Visual Place Recognition (VPR) refers to using the image retrieval algorithms to determine the location of a query image in the database, which is promising to help the VIP solve their localization problems. However, the accuracy of VPR is directly affected by the changes of scene appearances such as illumination, seasons and viewpoints. Therefore, finding a method to extract robust image descriptors under the changes of scene appearance is one of the most critical tasks in current VPR research. In this paper, we propose a VPR approach to assist the localization and navigation of visually impaired pedestrians. The core of our proposal is a combination of multi-level descriptors by using appropriate descriptors: the whole image, local regions and key-points, aimed to enhance the robustness of VPR. The matching procedure between query images and database images includes three steps. Firstly, we obtain the Convolutional Neural Networks (CNN) features of the whole images from a pre-trained GoogLeNet, and the Euclidean distances between the query images and the database images are computed to determine the top 10 good matches. Secondly, local salient regions are detected from the top-10 best-matched images with Non-Maximum Suppression (NMS) to control the number of bounding boxes. Thirdly, we detect the SIFT key-points and extract the geodesc descriptors of the key-points, from the local salient region, and determine the top 1 among the top 10 good matches. In order to verify our approach, a comprehensive set of experiments has been conducted on dataset with challenging environmental changes, such as the GardensPointWalking dataset.

For Visually Impaired People (VIP), it’s very difficult to perceive their surroundings. To address this problem, we propose a scene understanding system to aid VIP in indoor and outdoor environments. Semantic segmentation performance is generally sensitive to the environment and illumination changes, including the change between indoor and outdoor environments and the change across different weather conditions. Meanwhile, most existing methods have paid more attention on either the accuracy or the efficiency, instead of the balance between both of them. In the proposed system, the training dataset is preprocessed by using an illumination-invariant transformation to weaken the impact of illumination changes and improve the robustness of the semantic segmentation network. Regarding the structure of semantic segmentation network, the lightweight networks such as MobileNetV2 and ShuffleNet V2 are employed as the backbone of DeepLabv3+ to improve the accuracy with little increasing of computation, which is suitable for mobile assistance device. We evaluate the robustness of the segmentation model across different environments on the Gardens Point Walking dataset, and demonstrate the extremely positive effect of the illumination-invariant pre-transformation in challenging real-world domain. The network trained on computer achieves a relatively high accuracy on ADE20K relabeled into 20 classes. The frame rate of the proposed system is up to 83 FPS on a 1080Ti GPU.

Place recognition plays a crucial role in navigational assistance, and is also a challenging issue of assistive technology. The place recognition is prone to erroneous localization owing to various changes between database and query images. Aiming at the wearable assistive device for visually impaired people, we propose an open-sourced place recognition algorithm OpenMPR, which utilizes the multimodal data to address the challenging issues of place recognition. Compared with conventional place recognition, the proposed OpenMPR not only leverages multiple effective descriptors, but also assigns different weights to those descriptors in image matching. Incorporating GNSS data into the algorithm, the cone-based sequence searching is used for robust place recognition. The experiments illustrate that the proposed algorithm manages to solve the place recognition issue in the real-world scenarios and surpass the state-of-the-art algorithms in terms of assistive navigation performance. On the real-world testing dataset, the online OpenMPR achieves 88.7% precision at 100% recall without illumination changes, and achieves 57.8% precision at 99.3% recall with illumination changes. The OpenMPR is available at https://github.com/chengricky/OpenMultiPR., The paper is accepted by the special issue of Measurement Science and Engineering

Stereo cameras are widely used in wearable visually impaired assistance devices (VIADs). However, the inevitable vibration, shock, and mechanical stress may make the camera pair become misaligned and cause a sharp decline in the quality of the acquired depth map, which significantly influences the performance of VIADs. In this paper, we propose an epipolar-constraint-based unconstrained self-calibration method that requires neither user involvement nor specific environment, while achieving a rotation accuracy of 0.83 mrad and a translation accuracy of 0.42 mm. Several approaches are proposed to address the image matching issues, including blurred images removal, mismatched key points removal, etc. Based on correctly matched key point pairs, a planar quadric-distribution approach is proposed to ensure the quality and consistency of the final key point group. These collection approaches ensure the reliability of key point pairs, which is the most important factor to realize high accuracy with minimum constraint. A comprehensive set of experiments demonstrates the high robustness of the proposed methods, which are suitable for VIADs. We also present a field test with blindfolded users to validate the flexibility and applicability of the approach.

Panoramic images have advantages in information capacity and scene stability due to their large field of view (FoV). In this paper, we propose a method to synthesize a new dataset of panoramic image. We managed to stitch the images taken from different directions into panoramic images, together with their labeled images, to yield the panoramic semantic segmentation dataset denominated as SYNTHIA-PANO. For the purpose of finding out the effect of using panoramic images as training dataset, we designed and performed a comprehensive set of experiments. Experimental results show that using panoramic images as training data is beneficial to the segmentation result. In addition, it has been shown that by using panoramic images with a 180 degree FoV as training data the model has better performance. Furthermore, the model trained with panoramic images also has a better capacity to resist the image distortion., Comment: 15 pages, 12 figures, SPIE Security + Defence International Symposium

Compared to RGB semantic segmentation, RGBD semantic segmentation can achieve better performance by taking depth information into consideration. However, it is still problematic for contemporary segmenters to effectively exploit RGBD information since the feature distributions of RGB and depth (D) images vary significantly in different scenes. In this paper, we propose an Attention Complementary Network (ACNet) that selectively gathers features from RGB and depth branches. The main contributions lie in the Attention Complementary Module (ACM) and the architecture with three parallel branches. More precisely, ACM is a channel attention-based module that extracts weighted features from RGB and depth branches. The architecture preserves the inference of the original RGB and depth branches, and enables the fusion branch at the same time. Based on the above structures, ACNet is capable of exploiting more high-quality features from different channels. We evaluate our model on SUN-RGBD and NYUDv2 datasets, and prove that our model outperforms state-of-the-art methods. In particular, a mIoU score of 48.3\% on NYUDv2 test set is achieved with ResNet50. We will release our source code based on PyTorch and the trained segmentation model at https://github.com/anheidelonghu/ACNet., Accepted to be published in 2019 IEEE International Conference on Image Processing (ICIP 2019), Sep 22-25, 2019, Taipei. IEEE Copyright notice added, 5 pages, 5 figures

Currently, semantic segmentation shows remarkable efficiency and reliability in standard scenarios such as daytime scenes with favorable illumination conditions. However, in face of adverse conditions such as the nighttime, semantic segmentation loses its accuracy significantly. One of the main causes of the problem is the lack of sufficient annotated segmentation datasets of nighttime scenes. In this paper, we propose a framework to alleviate the accuracy decline when semantic segmentation is taken to adverse conditions by using Generative Adversarial Networks (GANs). To bridge the daytime and nighttime image domains, we made key observation that compared to datasets in adverse conditions, there are considerable amount of segmentation datasets in standard conditions such as BDD and our collected ZJU datasets. Our GAN-based nighttime semantic segmentation framework includes two methods. In the first method, GANs were used to translate nighttime images to the daytime, thus semantic segmentation can be performed using robust models already trained on daytime datasets. In another method, we use GANs to translate different ratio of daytime images in the dataset to the nighttime but still with their labels. In this sense, synthetic nighttime segmentation datasets can be generated to yield models prepared to operate at nighttime conditions robustly. In our experiment, the later method significantly boosts the performance at the nighttime evidenced by quantitative results using Intersection over Union (IoU) and Pixel Accuracy (Acc). We show that the performance varies with respect to the proportion of synthetic nighttime images in the dataset, where the sweet spot corresponds to most robust performance across the day and night., 13 pages, 7 figures, 2 tables, 2 equations. Artificial Intelligence and Machine Learning in Defense Applications, SPIE Security + Defence 2019, Strasbourg, France, September 2019

2019 IEEE Intelligent Vehicles Symposium (IV), Paris, France, 9-12 June 2019, Pixel-wise semantic segmentation unifies distinct scene perception tasks in a coherent way, and has catalyzed notable progress in autonomous and assisted navigation, where a whole surrounding perception is vital. However, current mainstream semantic segmenters are normally benchmarked against datasets with narrow Field of View (FoV), and most visionbased navigation systems use only a forward-view camera. In this paper, we propose a Panoramic Annular Semantic Segmentation (PASS) framework to perceive the entire surrounding based on a compact panoramic annular lens system and an online panorama unfolding process. To facilitate the training of PASS models, we leverage conventional FoV imaging datasets, bypassing the effort entailed to create dense panoramic annotations. To consistently exploit the rich contextual cues in the unfolded panorama, we adapt our real-time ERF-PSPNet to predict semantically meaningful feature maps in different segments and fuse them to fulfill smooth and seamless panoramic scene parsing. Beyond the enlarged FoV, we extend focal length-related and style transfer-based data augmentations, to robustify the semantic segmenter against distortions and blurs in panoramic imagery. A comprehensive variety of experiments demonstrates the qualified robustness of our proposal for realworld surrounding understanding., Ministerio de Economía y Competitividad, Comunidad de Madrid

Semantic Segmentation (SS) is the task to assign a semantic label to each pixel of the observed images, which is of crucial significance for autonomous vehicles, navigation assistance systems for the visually impaired, and augmented reality devices. However, there is still a long way for SS to be put into practice as there are two essential challenges that need to be addressed: efficiency and evaluation criterions for practical application. For specific application scenarios, different criterions need to be adopted. Recall rate is an important criterion for many tasks like autonomous vehicles. For autonomous vehicles, we need to focus on the detection of the traffic objects like cars, buses, and pedestrians, which should be detected with high recall rates. In other words, it is preferable to detect it wrongly than miss it, because the other traffic objects will be dangerous if the algorithm miss them and segment them as safe roadways. In this paper, our main goal is to explore possible methods to attain high recall rate. Firstly, we propose a real-time SS network named Swift Factorized Network (SFN). The proposed network is adapted from SwiftNet, whose structure is a typical U-shape structure with lateral connections. Inspired by ERFNet and Global convolution Networks (GCNet), we propose two different blocks to enlarge valid receptive field. They do not take up too much calculation resources, but significantly enhance the performance compared with the baseline network. Secondly, we explore three ways to achieve higher recall rate, i.e. loss function, classifier and decision rules. We perform a comprehensive set of experiments on state-of-the-art datasets including CamVid and Cityscapes. We demonstrate that our SS convolutional neural networks reach excellent performance. Furthermore, we make a detailed analysis and comparison of the three proposed methods on the promotion of recall rate., 14 pages, 11figures, SPIE Security + Defence 2019

Automatic head posture recognition is essential in the human-centered AI applications such as counting watching people in a bus or lift advertising system. A partially-occluded head posture estimation algorithm based on a histogram of gradient orientation (HoG) in a pyramid setting is proposed. We first apply face detection to a 2D image, and then divide the detected face region into two sub-regions, mouth and eye areas, to predict whether there is an occlusion in these two sub-regions individually. According to the predicted occlusion status, pyramid HoG features are extracted from non-occluded face sub-region. Finally, a support vector machine is applied for model training. Experimental results show that our proposed method has high accuracy, speed, and robustness in terms of illumination, shadow, occlusion, and complex background. The average recognition accuracy on the CMU-PIE dataset is 95.24% (94.22% with full mouth area occlusion), and that on the CAS-PEAL-R1 dataset is 95.61%. In addition, the contribution of the eye area is significantly bigger than that of the mouth area. Our method has potential commercial values and broad application prospects for intelligent human-machine interaction.

It is very difficult for visually impaired people to perceive and avoid obstacles at a distance. To address this problem, the unified framework of multiple target detection, recognition, and fusion is proposed based on the sensor fusion system comprising a low-power millimeter wave (MMW) radar and an RGB-Depth (RGB-D) sensor. In this paper, the Mask R-CNN and the single shot multibox detector network are utilized to detect and recognize the objects from color images. The obstacles' depth information is obtained from the depth images using the MeanShift algorithm. The position and velocity information on the multiple target is detected by the MMW radar based on the principle of a frequency modulated continuous wave. The data fusion based on the particle filter obtains more accurate state estimation and richer information by fusing the detection results from the color images, depth images, and radar data compared with using only one sensor. The experimental results show that the data fusion enriches the detection results. Meanwhile, the effective detection range is expanded compared to using only the RGB-D sensor. Moreover, the data fusion results keep high accuracy and stability under diverse range and illumination conditions. As a wearable system, the sensor fusion system has the characteristics of versatility, portability, and cost-effectiveness.

In the automatic detection for surface defects of optical components, the digs and dust particles exhibit similar features: point-like shape and variable intensity reflectivity. On this condition, these two types with entirely different damages are easily confused so that misjudgments will be induced. To solve this problem, a polarization-characteristics-based classification method of digs and dust particles (PCCDD) is proposed based on the polarimetric imaging technique and dark-field imaging technique. First, a dark-field imaging system equipped with a polarization state generator (PSG) and a polarization state analyzer (PSA) is employed to measure and establish normalized Mueller matrices' datasets of digs and dust particles. And by a nonlinear global search combined with a separability evaluation method, the optimal number of acquisitions and corresponding polarization measurement states of the PSG and the PSA are obtained, as well as the parameters of classification function. Then, multiple polarization images are acquired under the optimal states to extract a multidimensional feature description that relates only to the polarization characteristics of the defect; this subsequently acts as the input vector of the classifier to finally achieve the classification. This method takes full advantage of both the difference in polarization properties between digs and dust particles and the characteristic that the polarization properties of digs are relatively invariant while those of dust particles have a large variability. The classification process involves only simple matrix operations. Compared to the traditional discrimination method based on intensity images, the features obtained by this method have a higher separability. Experiments show that the classification accuracy reaches over 90%. This method can be further applied to the recognition and discrimination of other defects in the field of surface defects' detection.

Pixel-wise semantic segmentation is capable of unifying most of driving scene perception tasks, and has enabled striking progress in the context of navigation assistance, where an entire surrounding sensing is vital. However, current mainstream semantic segmenters are predominantly benchmarked against datasets featuring narrow Field of View (FoV), and a large part of vision-based intelligent vehicles use only a forward-facing camera. In this paper, we propose a Panoramic Annular Semantic Segmentation (PASS) framework to perceive the whole surrounding based on a compact panoramic annular lens system and an online panorama unfolding process. To facilitate the training of PASS models, we leverage conventional FoV imaging datasets, bypassing the efforts entailed to create fully dense panoramic annotations. To consistently exploit the rich contextual cues in the unfolded panorama, we adapt our real-time ERF-PSPNet to predict semantically meaningful feature maps in different segments, and fuse them to fulfill panoramic scene parsing. The innovation lies in the network adaptation to enable smooth and seamless segmentation, combined with an extended set of heterogeneous data augmentations to attain robustness in panoramic imagery. A comprehensive variety of experiments demonstrates the effectiveness for real-world surrounding perception in a single PASS, while the adaptation proposal is exceptionally positive for state-of-the-art efficient networks., Ministerio de Economía y Competitividad, Comunidad de Madrid

A macular fovea is a physiological structure of the human retina, which is an essential optical center. The distance between the lesion area and the foveal center determines the severity degree of visual impacts. Therefore, accurate fovea localization is the basis of the computer-aided ophthalmic diagnosis and vision screening. A simple but effective fovea localization algorithm based on the Faster R-CNN and physiological structure prior is presented. First, a fovea localization model and an optic disc localization model are trained separately. Then, for each fundus photograph, both candidate areas of the fovea and the location of the optic disc are predicted using two pre-trained models. Next, prior knowledge of the physiological adjacent relationship between a fovea and an optic disc is applied to eliminate unreasonable candidate bounding boxes. Finally, the ultimate bounding box of the fovea is determined by the best candidate. Experiments were conducted on a private dataset with 5,203 fundus photographs and the public Messidor dataset including 1200 fundus photographs. The accuracy of the foveal location in the offset scale of 1/2 optic disc diameter on the Messidor is 99.58%, which is 0.71% higher than the state-of-the-art (98.87%).

Visual localization is an attractive problem that estimates the camera localization from database images based on the query image. It is a crucial task for various applications, such as autonomous vehicles, assistive navigation and augmented reality. The challenging issues of the task lie in various appearance variations between query and database images, including illumination variations, dynamic object variations and viewpoint variations. In order to tackle those challenges, Panoramic Annular Localizer into which panoramic annular lens and robust deep image descriptors are incorporated is proposed in this paper. The panoramic annular images captured by the single camera are processed and fed into the NetVLAD network to form the active deep descriptor, and sequential matching is utilized to generate the localization result. The experiments carried on the public datasets and in the field illustrate the validation of the proposed system., Comment: Accepted by ITSC 2019

Intersection navigation comprises one of the major ingredient of Intelligent Transportation Systems (ITS) for Visually Impaired Pedestrians (VIP), who are the most vulnerable road users that should be protected with a high priority in metropolitan areas. Robotic vision-based assistive technologies sprung up over the past few years, which focused on specific scene objects using monocular detectors or depth sensors. These dividual solutions have reached impressive detectable range and accuracy with relatively short running time, and enhanced the intersection perception to a large degree. However, simultaneously enabling all detectors incurs a long delay and becomes computationally prohibitive on wearable embedded systems. In this work, we propose to seize CNN-based per-pixel semantic segmenter to cover navigational perception needs in a unified way. This is not only critical to perceive crosswalk position (where to cross roads), traffic light signal (when to cross roads), but also to analyze the states of other pedestrians and vehicles (whether safe to cross roads). At the centroid of our unification proposal is a deep learning architecture, aspired to attain efficient and robust semantic understanding. A comprehensive variety of experiments demonstrates the advanced accuracy over state-of-art algorithms/segmenters while maintaining high inference speed on a real-world navigation assistance system.

Traversability perception constitutes an important task for robotics and visually impaired people, which aims to detect obstacle-free paths that allow individuals to ambulate with suitable navigational aids. However, approaches that help prevent stepping into water areas are rare in state-of-art autonomous or assisted navigation systems. To address water hazard detection, this paper proposes a pRGB-D-SS perception framework, which incorporates: Polarization imaging (p), RGB-D sensory awareness and real-time Semantic Segmentation (SS). More specifically, as large water areas and small water puddles exhibit different characteristics, the detection of these two kinds of hazards pursue different pipelines. In our contribution, large water areas are detected together with traversable areas through pixel-wise semantic segmentation. Comparatively, the detection of water puddles extends the convolutional neural network based segmentation by using polarized RGB-D information as the primary cue. Beyond enhanced traversability awareness, it enables a unified framework of water hazard detection, which has been evaluated in a comprehensive variety of real-world surroundings on two wearable systems including a pair of commercial intelligent spectacles and a customized prototype, demonstrating the speeded-up efficiency and augmented sensorial/environmental robustness.

Semantic perception is a key enabler in robotics, which supposes a very resourceful and efficient manner of applying vision information for upper-level navigation and manipulation tasks. Given the challenges on specular semantics such as water hazards, transparent glasses and metallic surfaces, polarization imaging has been explored to complement the RGB-based pixel-wise semantic segmentation because it reflects surface characteristics and provides additional attributes. However, polarimetric measurements generally entail prohibitively expensive cameras and highly accurate calibrations. Inspired by the representation power of Convolutional Neural Networks (CNNs), we propose to predict polarization information from monocular RGB images, precisely per-pixel polarization difference. The core of our approach is a cluster of efficient deep architectures building on factorized convolutions, hierarchical dilations and pyramid representations, aimed to produce both semantic and polarimetric estimations in real time. Comprehensive experiments demonstrate the qualified accuracy on a wearable exoskeleton humanoid robot.

Localization is one of the crucial issues in assistive technology for visually impaired people. In this paper, we propose a novel hierarchical visual localization pipeline based on the wearable assistive navigation device for visually impaired people. The proposed pipeline involves the deep descriptor network, 2D-3D geometric verification and online sequence matching. Images in different modalities (RGB, Infrared and Depth) are fed into Dual Desc network to generate robust attentive global descriptors and local features. The global descriptors are leveraged to retrieve the coarse candidates of query images. The 2D local features, as well as 3D sparse point cloud, are used in geometric verification to select the optimal results from the retrieved candidates. Finally, sequence matching robustifies the localization results by synthesizing the verified results of successive frames. The proposed unified descriptor network Dual Desc surpasses the state-of-the-art NetVLAD and its variant on the task of image description. Validated on the real-world dataset captured by the wearable assistive device, the proposed visual localization utilizes multimodal images to overcome the disadvantages of RGB images and robustifies the localization performance by deep descriptor network and hierarchical pipeline. In the challenging scenarios of the Yuquan dataset, the proposed method achieves the F 1 score of 0.77 and the mean localization error of 2.75, which is satisfactory in practical use.

With the development of deep learning, various fields of computer vision have made huge progress. Among them, depth estimation is an important part of scene perception, therefore receives much interest and is widely used in daily life with the assistance of GPUs. Besides, the ways to obtain depth maps have also been improved, from using multiple images to a single image to obtain depth, which is called monocular depth estimation task. In this paper, we design a convolutional neural network called ERF-PSPNet to perform the task. We prove that by using unsupervised training, monocular depth estimation’s result learned from large-scale dataset is close to the result of stereo matching. We also show that the monocular depth estimation model proposed in this paper can achieve a satisfying precision while maintaining a certain real-time frame rate for day-night driving scenes, which confirms the practical applicability of our design and result.

With the increasing demands of visually impaired people, developing assistive technology to help them travel effectively and safely has been a research hotspot. Red, Green, Blue and Depth (RGB-D) sensor has been widely used to help visually impaired people, but the detection and recognition of glass objects is still a challenge, considering the depth information of glass cannot be obtained correctly. In order to overcome the limitation, we put forward a method to detect glass objects in natural indoor scenes in this paper, which is based on the fusion of ultrasonic sensor and RGB-D sensor on a wearable prototype. Meanwhile, the erroneous depth map of glass object computed by the RGB-D sensor could also be densely recovered. In addition, under some special circumstances, such as facing a mirror or an obstacle within the minimum detectable range of the RGB-D sensor, we use a similar processing method to regain depth information in the invalid area of the original depth map. The experimental results show that the detection range and precision of the RGB-D sensor have been significantly improved with the aid of ultrasonic sensor. The proposed method is proved to be able to detect and recognize common glass obstacles for visually impaired people in real time, which is suitable for real-world indoor navigation assistance.

Visually Impaired (VI) people around the world have difficulties in socializing and traveling due to the limitation of traditional assistive tools. In recent years, practical assistance systems for scene text detection and recognition allow VI people to obtain text information from surrounding scenes. However, real-world scene text features complex background, low resolution, variable fonts as well as irregular arrangement which make it difficult to achieve robust scene text detection and recognition. In this paper, a scene text recognition system to help VI people is proposed. Firstly, we propose a high-performance neural network to detect and track objects, which is applied to specific scenes to obtain Regions of Interest (ROI). In order to achieve real-time detection, a light-weight deep neural network has been built using depth-wise separable convolutions that enables the system to be integrated into mobile devices with limited computational resources. Secondly, we train the neural network using the textural features to improve the precision of text detection. Our algorithm suppresses the effects of spatial transformation (including translation, scaling, rotation as well as other geometric transformations) based on the spatial transformer networks. Open-source optical character recognition (OCR) is used to train scene texts individually to improve the accuracy of text recognition. The interactive system eventually transfers the number and distance information of inbound buses to visually impaired people. Finally, a comprehensive set of experiments on several benchmark datasets demonstrates that our algorithm has achieved an extraordinary trade-off between precision and resource usage.

According to the data from the World Health Organization, 285 million people are estimated to be visually impaired worldwide, and 39 million are blind. It is very difficult for visually impaired people to perceive and avoid obstacles at a distance during their travelling. To address this problem, we propose a sensor fusion system, which combines the RGBDepth sensor and millimeter wave radar sensor, to detect the surrounding obstacles. The range and velocity of multiple obstacles are acquired by the millimeter wave radar based on the principle of frequency modulated continuous wave. The positions of the obstacles are verified by the RGB-Depth sensor based on the contour extraction and MeanShift algorithm. The data fusion algorithm based on particle filters obtains accurate state estimation by fusing RGB-Depth data with millimeter wave radar data. The experiment results show that multiple obstacles with different ranges and angles are successfully detected by the proposed system. The measurement uncertainties are reduced by the data fusion system, meanwhile the effective detectable range is expanded compared to the detection with only RGB-Depth sensor. Moreover, the measurement results are stable when the illumination varies. As a wearable prototype, the sensor fusion system has the characteristics of versatility, portability and cost-effectiveness, which is very suitable for blind navigation application.

Feature matching is at the base of many computer vision algorithms such as SLAM, which is a technology widely used in the area from intelligent vehicles (IV) to assistance for the visually impaired (VI). This article presents an improved detector and a novel semantic-visual descriptor, coined SORB (Semantic ORB), combining binary semantic labels and traditional ORB descriptor. Compared to the original ORB feature, the new SORB performs better in uniformity of distribution and accuracy of matching. We demonstrate it through experiments on some open source datasets and several real-world images obtained by RealSense.

In recent years, with development of computer vision and robotics, a wide variety of localization approaches have been proposed. However, it is still challenging to design a localization algorithm that performs well in both indoor and outdoor environment. In this paper, an algorithm that fuses camera, IMU, GPS, as well as digital compass is proposed to solve this problem. Our algorithm includes two phases: (1) the monocular RGB camera and IMU are fused together as a VIO that estimates the approximate orientation and position; (2) the absolute position and orientation measured by GPS and digital compass are merged with the position and orientation estimated in first phase to get a refined result in the world coordinate. A bag-of-word based algorithm is utilized to realize loop detection and relocalization. We also built a prototype and did two experiments to evaluate the effectiveness and robustness of the localization algorithm in both indoors and outdoors environment.

Assistive technologies aim at enhancing personal mobility of individuals with disabilities to improve their independence and access to social life. For the visually impaired, perception during navigation comprises a major ingredient of independent living. With the development of computer vision, it is possible to meet the richer needs of visually impaired people. However, research on navigation assistance for the visually impaired is still relatively unexplored when compared with the active progress in autonomous driving which is already in full swing. In respond to this issue, we aim to leverage the study of the Stixel-World for automotive systems and transfer it to develop assistive technology for visually impaired people. The impressive research results of deep learning also suppose benefits for vision-based technology. Precisely, semantic segmentation is a task that enables identification of different objects uniformly. Inspired by these observations, we design a set of wearable visual aids, while the core algorithm is based on the stixel representations for three-dimensional world combined with pixel-wise semantic segmentation. Predetermined conditions for stixels in automotive research, such as camera angles, position fixes, and unsuitable assumptions made about the real world are optimized to fit the needs of navigation assistance in our algorithm, along with the incorporation of traversability-related semantic information. We also propose a sound mapping scheme, so that the environmental awareness about geographic and semantic information are conveyed to the visually impaired through acoustic feedback.

On the off-the-shelf navigational assistance devices, the localization precision is limited to the signal error of global navigation satellite system (GNSS). During travelling outdoors, the inaccurately localization perplexes visually impaired people, especially at key positions, such as gates, bus stations or intersections. The visual localization is a feasible approach to improving the positioning precision of assistive devices. Using multiple image descriptors, the paper proposes a robust and efficient visual localization algorithm, which takes advantage of priori GNSS signals and multi-modal images to achieve the accurate localization of key positions. In the experiments, we implement the approach on the wearable system and test the performance of visual localization under practical scenarios., Comment: This paper has been accepted by International Conference on Pattern Recognition (ICPR) 2018

The introduction of RGB-depth (RGB-D) sensors harbors a revolutionary power in the field of navigational assistance for the visually impaired. However, RGB-D sensors are limited by a minimum detectable distance of about 800 mm. This paper proposes an effective approach to decrease the minimum range for navigational assistance based on a RGB-D sensor of RealSense R200. A large-scale stereo matching between two infrared (IR) images and a cross-modal stereo matching between one IR image and RGB image are incorporated for short-range depth acquisition. The minimum range reduction is critical not only for avoiding obstacles up close, but also in the enhancement of traversability awareness. Overall, the minimum detectable distance of RealSense is reduced from 650 mm to 60 mm with qualified accuracy. A traversable line is created to give feedback to visually impaired individuals through stereo sound. The approach is proved to have usefulness and reliability by a comprehensive set of experiments and field tests in real-world scenarios involving real visually impaired participants.

The application of RGB-D sensors provides a revolutionary force in the research field of computer vision, where traversable area detection and obstacle avoidance are the fundamental topics to aid visually impaired people. However, the detection using RGB-D sensors is limited by the sparse depth map and the narrow field of view, which hampers longer traversability awareness and lower obstacle negotiation. This paper proposes an effective approach to combine a pair of wearable smart glasses and a waist-worn pathfinder to assist the visually impaired. Traversable lines are generated to feedback the visually impaired through real-time stereo sound after real-world coordinates adjustment with a RealSense R200 (RGB-D sensor) and an IMU integrated in the pair of smart glasses. Low-lying obstacles are detected by using another RGB-D sensor of RealSense, the RS410, as the waist-worn pathfinder to provide early warning. Indoor and outdoor detection results as well as a field test demonstrate the usefulness and effectiveness of our approach.

The introduction of RGB-D sensor has attracted attention from researchers majored in computer vision. With real-time depth measurement provided by RGB-D sensor, a better navigational assistance than traditional aiding tools can be offered for visually impaired people. However, nowadays RGB-D sensor usually has a limited detecting range, and fails in performing depth measurement on objects with special surfaces, such as absorbing, specular, and transparent surfaces. In this paper, a novel algorithm using two RealSense R200 simultaneously to build a short-baseline color stereo vision system is developed. This algorithm enhances depth estimation by fusing color stereo depth map and original RealSense depth map, which is obtained by infrared stereo vision. Moreover, the minimum range is decreased by up to 84.6%, from 650mm to 100mm. We anticipate out algorithm to provide better assistance for visually impaired individuals.

Utilizing RGB-Depth images acquired by a wearable system, we propose an integrated assistive navigation for visually impaired people at urban intersection, which provides with crosswalk position (where to cross roads), crossing light signal (when to cross roads) and pedestrian state (whether safe to cross roads). Verified by the experiment results on datasets and in field, the proposed approach detects multiple targets at urban intersections robustly and provides visually impaired people with effective assistance.

The use of RGB-Depth (RGB-D) sensors for assisting visually impaired people (VIP) has been widely reported as they offer portability, function-diversity and cost-effectiveness. However, polarization cues to assist traversability awareness without precautions against stepping into water areas are weak. In this paper, a polarized RGB-Depth (pRGB-D) framework is proposed to detect traversable area and water hazards simultaneously with polarization-color-depth-attitude information to enhance safety during navigation. The approach has been tested on a pRGB-D dataset, which is built for tuning parameters and evaluating the performance. Moreover, the approach has been integrated into a wearable prototype which generates a stereo sound feedback to guide visually impaired people (VIP) follow the prioritized direction to avoid obstacles and water hazards. Furthermore, a preliminary study with ten blindfolded participants suggests its effectivity and reliability.