Search

The smart unmanned vending machine using machine vision technology suffers from the sharp decrease of detection accuracy due to the incomplete image collection of items by monocular camera in complex environment, and the lack of obvious features in dense stacking of items. In this paper, a binocular camera system is designed to effectively solve the problems of distortion and coverage caused by monocular camera. Besides, an image-stitching algorithm is developed to splice the images captured by the camera, which reliefs the burden of computation for back-end recognition processing brought by the binocular camera. A new model YOLOv3-TinyE is proposed based on YOLOv3-tiny model. Based on the data set of 21, 000 images captured in real scenarios that containing 20 different type of beverages, the comparison experimental results show that YOLOv3-TinyE model achieves the mean average precision of 99.15%, and the inference speed is 2.91 times faster than that of YOLOv3 model, and the detection accuracy of YOLOv3-TinyE model based on binocular vision is higher than that based on monocular vision. The results suggest that the designed method achieves the goal in terms of inference speed and average precision, that is, it is able to satisfy the requirements for real-world applications.

This article presents a robust brightness gradient-based estimation strategy for small aerial robots. The proposed nonlinear observer is capable of estimating the flight altitude and ego-motion by the fusion of monocular vision and inertial measurement unit feedback. The novelty primarily lies in the implementation of the gradient-based featureless approach and the direct use of photometric feedback in the state and output vectors. Under the single-plane assumption, the proposed framework permits the entire estimation process to be accomplished efficiently in a single iterative step without the need for feature detection and tracking or pre-computation of optic flow as commonly seen in conventional methods. The nonlinear and featureless implementation reduces the computational demand, enlarges the region of attraction, and markedly improves the robustness of the ego-motion estimation against scenes with scarce features when compared to Kalman-based estimators and feature-based methods. We conducted extensive flight experiments with different flying patterns and textures to evaluate the performance of the proposed observer. The results reveal that the root-mean-square error in the altitude estimates is approximately 10% for flights at ≈40-130 cm above the ground, comparable to feature-based estimators. Nevertheless, the devised observer does better than feature-based methods when deployed on low-textured scenes or with low-resolution blurry images.

Cameras have been widely used in traffic operations. While many technologically smart camera solutions in the market can be integrated into Intelligent Transport Systems (ITS) for automated detection, monitoring and data generation, many Network Operations (a.k.a Traffic Control) Centres still use legacy camera systems as manual surveillance devices. In this paper, we demonstrate effective use of these older assets by applying computer vision techniques to extract traffic data from videos captured by legacy cameras. In our proposed vision-based pipeline, we adopt recent state-of-the-art object detectors and transfer-learning to detect vehicles, pedestrians, and cyclists from monocular videos. By weakly calibrating the camera, we demonstrate a novel application of the image-to-world homography which gives our monocular vision system the efficacy of counting vehicles by lane and estimating vehicle length and speed in real-world units. Our pipeline also includes a module which combines a convolutional neural network (CNN) classifier with projective geometry information to classify vehicles. We have tested it on videos captured at several sites with different traffic flow conditions and compared the results with the data collected by piezoelectric sensors. Our experimental results show that the proposed pipeline can process 60 frames per second for pre-recorded videos and yield high-quality metadata for further traffic analysis.

The attitude of moving objects has a predictive function for navigation and positioning in a dim indoor environment. Currently, the accuracy of vision and inertial sensors’ attitude estimation is low, as it mainly affected by factors such as motion image blur, attitude angle processing algorithms, and data synchronization. Firstly, in this paper we propose a novel continuous multi-frame evaluation and registration algorithm to obtain high signal-to-noise ratio (SNR) images in regular indoor environments without global positioning system (GPS). The coordinates of vanishing points and plumb lines were extracted in the strong texture image. Then, a visual attitude model was constructed based on the characteristic points and lines. A pre-integrated gyro sensor was used to establish the inertial attitude model. The visual and gyro attitude models were used to obtain high-precision attitude information using multi-rate filtering. Finally, we build a hardware attachment composed of a consumer camera, an inexpensive inertial sensor and hardware circuit based on digital signal processing, which effectively demonstrates that the experiment result compares favourably with the more traditional methods.

Initialization is a fundamental task for the state estimation of multi-sensor integration. In this paper, we propose a novel approach to achieve a fast and accurate initialization for the integrated system. With the outputs from GNSS, the rough roll, pitch, yaw, and direction of gravity are calculated. Based on these initial states, the GNSS/INS fusion and Visual-Inertial Odometry (VIO) are simultaneously launched whenever the observations are available. Subsequently, a coarse scale is retrieved by assigning the initial states of GNSS/INS fusion to the VIO procedure. The scale can be utilized to get poses and points very close to the real world. Once the VIO estimation converges, the transformation parameters between the poses of VIO and GNSS/INS fusion are estimated via non-linear optimization. Thus, we can align these two trajectories and evaluate the scale error. Through six vehicular tests, we analyze the performance of the proposed method with different illumination, obstructions, and motion patterns, and compare the results of the proposed approach with the state-of-the-art algorithms. The results indicated that the Visual-Inertial parameters converge within 9 seconds in our method. The scale errors of both our method are mostly within 10%. The alignment errors are mostly at centimeter-level in our method, which is comparable to the existing approaches.

Introduction This study aims to compare preclinical visual-quality metrics and halo size of intraocular lenses (IOL) with enhanced intermediate vision to a standard monofocal lens. Methods Three monofocal- IOL models with an extended-depth-of-focus (EDoF) intended for monocular implantation (Tecnis ICB00, AE2UV/ZOE, and IsoPure) and one for monovision (RayOne EMV) were compared against a standard monofocal lens (Tecnis ZCB00). An optical-metrology station was used in the assessment of IOLs' optical quality in polychromatic light. The imaging quality was compared with metrics derived from the optical transfer function. Halo size was estimated from the projection of the point spread function under scotopic pupil. Results The monofocal IOL showed the highest image quality at the far focus. The ICB00’s, the AE2UV/ZOE’s, and the IsoPure’s performance at − 1D was superior to that of the monofocal lens. The monocular defocus tolerance of the RayOne EMV was comparable with that of the ZCB00. The RayOne EMV’s intermediate range was improved in a monovision configuration (− 1D offset). This approach, however, yielded the largest halo area, i.e., 53% of the ZCB00’s halo, compared to 34% for the IsoPure, 14% for the AE2UV/ZOE, and 8% for the ICB00. Conclusion The mono-EDoF models have a clear advantage over the standard monofocal lens by expanded imaging capability beyond − 0.5D. Although the RayOne EMV provided the largest (binocular) visual-range extension, it was at the expense of monocular vision and higher susceptibility to halo. The ICB00’s and the AE2UV/ZOE’s halo-profile was similar to that of the ZCB00, indicating their low potential to induce photic phenomena. Supplementary Information The online version contains supplementary material available at 10.1007/s40123-021-00411-9.

In order to solve the problem of high measurement cost and complex operation of position-independent geometric errors (PIGEs) calibration on a five-axis motion platform, this paper first proposes a low-cost pose measurement method, based on monocular vision, which can accurately determine the pose in the environment, even with image shadow and noise. Next, an improved method, combining pose measurement and kinematic parameters identification, is proposed to calibrate a five-axis motion platform. The kinematic error model of the platform and the pose planning of automatic image acquisition are established, providing the pose data and motor position data, required for calibration. Combined with the kinematic loop method, the kinematic parameters of the five-axis motion platform are identified, while the geometric structure parameters are accurately calibrated. Before and after calibration, a circular trajectory of the target coordinate system (TCS) origin, relative to the camera coordinate system (CCS), is used to test the comprehensive accuracy evolution of the five-axis motion platform, by comparing the position and orientation errors of the theoretical circle trajectory to the actual one. The experimental data show that, before and after calibration, the average position error of the five-axis motion platform is reduced by 79.46%, while the average direction error is reduced by 86.53%. The experimental results clearly demonstrate that the proposed calibration method significantly improves the comprehensive motion accuracy of the five-axis motion platform, and they verify the practical value and effectiveness of the calibration scheme.

Purpose This paper aims to develop a robust person tracking method for human following robots. The tracking system adopts the multimodal fusion results of millimeter wave (MMW) radars and monocular cameras for perception. A prototype of human following robot is developed and evaluated by using the proposed tracking system. Design/methodology/approach Limited by angular resolution, point clouds from MMW radars are too sparse to form features for human detection. Monocular cameras can provide semantic information for objects in view, but cannot provide spatial locations. Considering the complementarity of the two sensors, a sensor fusion algorithm based on multimodal data combination is proposed to identify and localize the target person under challenging conditions. In addition, a closed-loop controller is designed for the robot to follow the target person with expected distance. Findings A series of experiments under different circumstances are carried out to validate the fusion-based tracking method. Experimental results show that the average tracking errors are around 0.1 m. It is also found that the robot can handle different situations and overcome short-term interference, continually track and follow the target person. Originality/value This paper proposed a robust tracking system with the fusion of MMW radars and cameras. Interference such as occlusion and overlapping are well handled with the help of the velocity information from the radars. Compared to other state-of-the-art plans, the sensor fusion method is cost-effective and requires no additional tags with people. Its stable performance shows good application prospects in human following robots.

Distance perception is the basis and necessary prerequisite of environment perception, attitude perception and obstacle avoidance for both intelligent vehicle and unmanned vehicle. Passive ranging is a critical part of machine vision measurement. Most passive ranging methods based on machine vision apply binocular technology that needs strict hardware conditions and lacks universality. Therefore, the monocular vision ranging method is one of the mainstream distance sensing methods at present. In order to improve the accuracy of monocular vision ranging, a monocular vision ranging method based on pixel area and aspect ratio is proposed. Subsequently, this method improves the stability of real-time target detection by introducing Kalman filter processing. Experimental results display that that ranging used by this method has higher accuracy. The mean relative error of the depth measurement is 5% when it is 3-10 m. After introducing Kalman filter, the stability of real-time ranging processing is improved by 25.21%. In this paper, the ROS smart car with real-time target tracking is also realized by the method based on the combination of SIFT-KCF target detection and tracking and monocular ranging.

This paper designed an intelligent service robot system in highway based on multi-sensor fusion. The mobile robot attempts to fuse the lidar information and monocular vision information to estimate the pose of itself and obtain an environmental map. It adapts a new SLAM method which combines lidar and vision information. Lidar is used to obtain the 2D occupancy grid map and the monocular vision SLAM algorithm uses the Extended Kalman Filter (EKF) to magnify the pose estimation. The 3-DOF pose provided by lidar is obtained through Cartographer algorithm and the monocular vision SLAM who offers the 6-DOF pose is realized with ORB-SLAM. The experimental results show that the system is effective in application as an intelligent service robot of highway.

In intelligent video surveillance under complex scenes, it is vital to identify the current actions of multi-target human bodies accurately and in real time. In this paper, a real-time multi-person action recognition method with monocular vision is proposed based on sequence models. Firstly, the key points of multi-target human body skeleton in the video are extracted by using the OpenPose algorithm. Then, the human action features are constructed, including limb direction vector and the skeleton height-width ratio. The multi-target human bodies tracking is then achieved by using the tracking algorithm. Next, the tracking results are matched with the action features, and the action recognition model is constructed, which includes the spatial branch based on Deep neural networks and the temporal branch based on Bi-directional RNN and Bi-directional long short-term memory networks. After pre-training, the model can be used to recognize the human body action from action features, and a recognition stabilizer is designed to minimize false alarms. Finally, extensive evaluations on the JHMDB dataset validate the effectiveness and the superiority of the proposed approach.

In order to improve the accuracy of conical target pose parameters measured by monocular vision, we propose a high precision visual measurement method for conical target based on the minimum reconstruction error of 3D. Firstly, we select the main and auxiliary features used for pose parameters’ measurement by analyzing target’s imaging process. Secondly, the algebraic and geometric methods for measuring pose parameters are determined based on the selected features. Finally, we utilize the geometric constraints of target to establish the reconstruction error formula and optimize the pose parameters based on the minimum reconstruction error principle to improve the measurement accuracy. To verify the effectiveness of our method, we use 3ds Max software to generate target’s simulation images under different pose parameters and compare the pose parameters before and after optimization. Experimental results show that the proposed method can significantly improve the accuracy of pose parameters when the measurement distance is far (more than 30 m), which is up to about 40%.

The camera calibration in monocular vision represents the relationship between the pixels’ units which is obtained from a camera and the object in the real world. As an essential procedure, camera calibration calculates the three-dimensional geometric information from the captured two-dimensional images. Therefore, a modified camera calibration method based on polynomial regression is proposed to simplify. In this method, a parameter vector is obtained by pixel coordinates of obstacles and corresponding distance values using polynomial regression. The set of parameter’s vectors can measure the distance between the camera and the ground object in the field of vision under the camera’s posture and position. The experimental results show that the lowest accuracy of this focal length calibration method for measurement is 97.09%, and the average accuracy was 99.02%.

As the indoor dynamic target localization does not detect and repair the circumferential jump value in time, which leads to large position and attitude errors and low-velocity stability, a combined Global Navigation Satellite System/Simultaneous Localization and Mapping- (GNSS/SLAM-) based high-precision indoor dynamic target localization method is proposed. The method uses Empirical Mode Decomposition (EMD) to decompose the noisy signal, obtains the noise energy as the dominant mode from the decomposed components, extracts the useful signal energy as the main dividing point, removes the high-frequency signal, constructs the low-frequency component to realize low-pass filtering and denoising, selects a suitable threshold processing function to make the high-frequency signal component retain the detailed signal effectively to realize high-frequency component denoising, detects and fixes the circumferential jump of the observed data, and detects and fixes the circumferential jump of each frequency. The indoor dynamic target positioning method is established by combining GNSS/SLAM to achieve high accuracy target positioning. The experimental results show that the position and attitude errors are small, and the velocity is stable, which indicates that the position information is closer to the dynamic target, i.e., the target positioning is more accurate. To address the problems of scale drift and frequent initialization due to environmental factors in monocular vision SLAM, an Ultra Wideband (UWB)/vision fusion localization model that takes into account the scale factor is proposed, which can give full play to the complementary characteristics between UWB and vision; solve the problems of visual initialization, scale ambiguity, and absolute spatial reference; and improve UWB localization accuracy and localization frequency as well as reducing the number of base stations. The model can reliably achieve 0.2 m accuracy in indoor environments with sparse texture or frequent light changes.

Dynamic acquisition of crop morphology is beneficial to real-time variable decision of precise spraying operations in fields. However, the existing spraying quantity regulation has high tolerance on the statistical characteristics of regional morphology, so expensive LiDAR and ultrasonic radar can't make full use of their high accuracy, and can reduce decision speed because of too much detail of branches and leaves. Therefore, designing a novel recognition system embedded machine learning with low-cost monocular vision is more feasible, especially in China, where the agricultural implements are medium sizes and cost-sensitive. In addition, we found that the growth period of crops is an important reference index for guiding spraying. So, taking cotton as a case study, a cotton morphology acquisition by a single camera is established, and a cotton growth period recognition algorithm based on Convolution Neural Network (CNN) is proposed in this paper. Through the optimization process based on confusion matrix and recognition efficiency, an optimized CNN model structure is determined from 9 different model structures, and its reliability was verified by changing training sets and test sets many times based on the idea of k-fold test. The accuracy, precision, recall, F1-score and recognition speed of this CNN model are 93.27%, 95.39%, 94.31%, 94.76% and 71.46 ms per image, respectively. In addition, compared with the performance of VGG16 and AlexNet, the convolution neural network model proposed in this paper has better performance. Finally, in order to verify the reliability of the designed recognition system and the feasibility of the spray decision-making algorithm based on CNN, spraying deposition experiments were carried out with 3 different growth-periods of cotton. The experiments’ results validate that after the optimal spray parameters were applied at different growth periods respectively, the average optimum index in 3 growth periods was 42.29%, which was increased up to 62.24% than the operations without distinguishing growth periods.

Autonomous robots for medical and emergency supplies are a potential way to avoid contact with people in quarantine and control the spread of contagious diseases in an indoor scene. However, scene understanding and reconstruction through a single low-cost camera remains a challenge. It is known that absolute precise depth cannot be calculated accurately from a single image, but the relative pose of different planes, which can be inferred from geometric features in a 2-D image, are more likely to be used in understanding scenes and its reconstruction. In this article, we present an interpretable model to bridge the gap between 2-D scene understanding and three-dimensional (3-D) reconstruction without prior training or any precise depth data. Based on 2-D semantic information in our previous works, the 3-D relative pose of estimated planes can be estimated. At that point, indoor scenes are approximated in the reconstruction. The approach behaves as an interpretable characteristic and requires no prior training or knowledge of the camera's internal parameters. We compare the quantitative performance on the percentage of incorrectly reconstructed planes by relative pose estimation. The results demonstrated that the method can successfully understand and reconstruct indoor scenes including both Manhattan and curved non-Manhattan structures.

The rapid development of unmanned aerial vehicles (UAV) puts forward a higher requirement for autonomous obstacle avoidance. Due to the limited payload and power supply, small UAVs such as quadrotors usually carry simple sensors and computation units, which makes traditional methods more challenging to implement. In this paper, a novel framework is demonstrated to control a quadrotor flying through crowded environments autonomously with monocular vision. The framework adopts a two-stage architecture, consisting of a sensing module and a decision module. The sensing module is based on an unsupervised deep learning method. And the decision module uses dueling double deep recurrent Q-learning to eliminate the adverse effects of limited observation capacity of an on-board monocular camera. The framework enables the quadrotor to realize autonomous obstacle avoidance without any prior environment information or labeled datasets for training. The trained model shows a high success rate in the simulation and a good generalization ability for transformed scenarios., 23 pages, 10 figures

Recently, real-time obstacle detection by monocular vision exhibits a promising prospect in enhancing the safety of unmanned surface vehicles (USVs). Since the obstacles that may threaten USVs generally appear below the water edge, most existing methods first detect the horizon line and then search for obstacles below the estimated horizon line. However, these methods detect horizon line only using edge or line features, which are susceptible to interference edges from clouds, waves, and land, eventually resulting in poor obstacle detection. To avoid being affected by interference edges, in this paper, we propose a novel horizon line detection method based on semantic segmentation. The method assumes a Gaussian mixture model (GMM) with spatial smoothness constraints to fit the semantic structure of marine images and simultaneously generate a water segmentation mask. The horizon line is estimated from the water boundary points via straight line fitting. Further, inspired by human visual attention mechanisms, an efficient saliency detection method based on background prior and contrast prior is presented to detect obstacles below the estimated horizon line. To reduce false positives caused by sun glitter, waves and foam, the continuity of the adjacent frames is employed to filter the detected obstacles. An extensive evaluation was conducted on a large marine image dataset collected by our ‘Jinghai VIII’ USV. The experimental results show that the proposed method significantly outperformed the recent state-of-the-art marine obstacle method by 22.07% in terms of F-score while running over 24 fps on an NVIDIA GTX1080Ti GPU.

In this work, we propose a method for AUV to reconstruct the docking ring autonomously under the guidance of visual information. The proposed method is based on the knowledge of self-reconfiguration of two AUVs in underwater environment. In order to enable long distance performance, we make use of blue-green light beacons. The light beacons are detected by using YOLO V3 algorithm. In addition, P4P algorithm is applied to resolve the problem of relative poses of AUVs. In case of short distances, we use Aruco marker to precisely guide the AUV towards the docking ring. In addition, we use KCF algorithm to shorten the recognition time, and Kalman filter method to eliminate the interference of occlusions and jitters. The feasibility of the proposed method is verified by performing multiple experiments. Under the reasonable control flow and visual algorithm, the proposed solution avoids the situation where the accuracy of long-distance monocular recognition is low and is easy to be misjudged. The underwater multi-target integrated guidance method based on monocular vision is convenient for the recovery of underwater AUV, and can effectively take into account the advantages of wide effective guidance range and high precision of close range guidance.

An adaptive machine-learning object-localization system was proposed in this study and applied to an automated carrier robot equipped with monocular vision, and the problem of object-distance localization in vision was improved using machine learning. When executing a project involving small robots, the calculation of the target object by the object-distance algorithm depends on the system-defined parameters, such as camera height, field of view, and inclination angle. When the carrier encounters an inclination that causes changes in the camera’s external parameters, the parameters defined in the system differ from those under real conditions, causing severe errors in calculation results. This system combined a triaxial sensor and machine-learning regression to identify the functional relationship between the output signal of the accelerometer and the actual carrier attitude. The regression network model was trained by recording several sets of signal features and carrier angles as the data set. During object-distance calculation, the adaptive machine-learning object-localization system predicted the current carrier angle by using the output of the accelerometer and the trained regression network model to update the algorithm’s angle parameters. The experimental results indicated that the system possessed stability and accuracy in calculating the distance of the target object when the carrier angle changed.

Forward collision warning (FCW) system is an advanced driver assistance safety system that continually monitors the ego vehicle’s position, predicts a collision and update it to the driver. It is challenging as it requires to detect and track moving vehicles on-road. In this paper, a novel monocular vision-based FCW system with range prediction is proposed. Firstly, the on-road moving vehicles are detected using a pre-trained YOLO (You only look once) model by setting it as a regression problem to predict the bounding boxes and its class probabilities. The detected vehicles are then assigned a unique ID using Hungarian algorithm and is tracked using Kalman filter. Secondly, the distance between the detected vehicle and the ego-vehicle is calculated by mapping the image into a two-dimensional orthogonal top-down view using inverse perspective mapping (IPM). Road boundary detection is carried out to obtain the calibration points for IPM. Quantitative evaluation carried out on recorded video clips in traffic environment shows that the proposed FCW system effectively estimate the inter-vehicle distance and its relative velocity to predict a collision with great accuracy.

In 3-D attitude angle estimation, monocular vision-based methods are often utilized for the advantages of short-time and high efficiency. However, the limitations of these methods lie in the complexity of the algorithm and the specificity of the scene, which needs to match the characteristics of the cooperation object and the scene. In this article, we propose a fully connected attitude detection network (FADN), which combines neural network and traditional algorithms for 3-D attitude angle estimation. FADN provides a whole process from the input of a single frame image in the industrial video stream to the output of the corresponding 3-D attitude angle estimation. Benefiting from the end-to-end estimation framework, FADN avoids tedious matching algorithms and thus has certain portability. A series of comparative experiments based on the rendering software 3-D Studio Max (3d Max) have been carried out to evaluate the performance of FADN. The experimental results show that FADN has high estimation accuracy and fast running speed. At the same time, the simulation results reliably prove the feasibility of FADN, and also promote the research in real scenarios.

Retinal migraine is a chronic, disabling headache disorder that strikes sufferers with recurrent, transient episodes of monocular vision loss before the onset of a throbbing headache on the same side as the visual disturbance. The author discusses the experience of a patient with type 2 diabetes perceiving these frightening episodes of transient visual disturbances during the coronavirus disease 2019 pandemic in which stressors and missed meals precipitated and exacerbated these attacks. With an increased understanding of the characteristics of retinal migraines, nurse practitioners can work closely with other health care professionals to provide comprehensive and patient-centered care to these individuals.

Traditional monocular vision localization methods are usually suitable for short-range area and indoor relative positioning tasks. This paper presents MGG, a novel monocular global geolocation method for outdoor long-range targets. This method takes a single RGB image combined with necessary navigation parameters as input and outputs targets' GPS information under the Global Navigation Satellite System (GNSS). In MGG, we first design a camera pose correction method via pixel mapping to correct the pose of the camera. Then, we use anchor-based methods to improve the detection ability for long-range targets with small image regions. Next, the local monocular vision model (LMVM) with a local structure coefficient is proposed to establish an accurate 2D-to-3D mapping relationship. Subsequently, a soft correspondence constraint (SCC) is presented to solve the local structure coefficient, which can weaken the coupling degree between detection and localization. Finally, targets can be geolocated through optimization theory-based methods and a series of coordinate transformations. Furthermore, we demonstrate the importance of focal length on solving the error explosion problem in locating long-range targets with monocular vision. Extensive experiments on the challenging KITTI dataset as well as applications in outdoor environments with targets located at a long range of up to 150 meters show the superiority of our method.

Depth estimation is an essential component in computer vision systems for achieving 3D scene understanding. Efficient and accurate depth map estimation has numerous applications including self-driving vehicles and virtual reality tools. This paper presents a new deep network, called D-Net, for depth estimation from a single RGB image. The proposed network can be trained end-to-end, and its structure can be customised to meet different requirements in model size, speed, and prediction accuracy. Our approach gathers strong global and local contextual features at multiple resolutions, and then transfers these to high resolutions for clearer depth maps. For the encoder backbone, D-Net can utilise many state-of-the-art models including EfficientNet, HRNet and Swin Transformer to obtain dense depth maps. The proposed D-net is designed to have minimal parameters and reduced computational complexity. Extensive evaluations on the NYUv2 and KITTI benchmark datasets show that our model is highly accurate across multiple backbones, and it achieves state-of-the-art performance on both benchmarks when combined with the Swin Transformer and HRNets.

Obstacle distance measurement is one of the key technologies for autonomous navigation of high-voltage transmission line inspection robots. To address the robustness of obstacle distance measurement under varying illumination conditions, this article develops a research method that fuses image enhancement with robot monocular vision so that the robot can adapt to various levels of illumination running along the transmission line. During the inspection of high-voltage transmission lines in such an overexposed (excessively bright) environment, a specular highlight suppression method is proposed to suppress the specular reflections in an image; when scene illumination is insufficient, a robust low-light image enhancement method based on a tone mapping algorithm with weighted guided filtering is presented. Based on the monocular vision measurement principle, the error generation mechanism is analyzed through experiments, and we introduce the parameter modification mechanism. The two proposed image enhancement methods outperform other state-of-the-art enhancement algorithms in qualitative and quantitative analyses. The experimental results show that the measurement error is less than 3% for static distance measurements and less than 5% for dynamic distance measurements within 6 m. The proposed method can meet the requirements of high-accuracy positioning, real-time performance and strong robustness. This method greatly contributes to the sustainable development of inspection robots in the power industry.

Vision-based manipulation has been largely used in various robot applications. Normally, in order to obtain the spatial information of the operated target, a carefully calibrated stereo vision system is required. However, it limits the application of robots in the unstructured environment which limits both the number and the pose of the camera. In this study, a calibration-free monocular vision-based robot manipulation approach is proposed based on domain randomization and deep reinforcement learning (DRL). Firstly, a learning strategy combined domain randomization is developed to estimate the spatial information of the target from a single monocular camera arbitrarily mounted in a large area of the manipulation environment. Secondly, to address the monocular occlusion problem which regularly happens during robot manipulations, an occlusion awareness DRL policy has been designed to control the robot to avoid occlusions actively in the manipulation tasks. The performance of our method has been evaluated on two common manipulation tasks, reaching and lifting of a target building block, which show the efficiency and effectiveness of our proposed approach.

This study proposes a single-axis registration system based on monocular vision and CAD modelling without employing three-dimensional scanning or reconstruction techniques. The developed system use...

This article presents the Spacecraft Pose Network (SPN), the first neural network-based method for on-board estimation of the pose, i.e., the relative position and attitude, of a known noncooperative spacecraft using monocular vision. In contrast to other state-of-the-art pose estimation approaches for spaceborne applications, the SPN method does not require the formulation of hand-engineered features and only requires a single grayscale image to determine the pose of the spacecraft relative to the camera. The SPN method uses a convolutional neural network (CNN) with three branches to solve the problem of relative attitude estimation. The first branch of the CNN bootstraps a state-of-the-art object detection algorithm to detect a 2-D bounding box around the target spacecraft in the input image. The region inside the 2-D bounding box is then used by the other two branches of the CNN to determine the relative attitude by initially classifying the input region into discrete coarse attitude labels before regressing to a finer estimate. The SPN method then estimates the relative position by using the constraints imposed by the detected 2-D bounding box and the estimated relative attitude. Further, with the detection of 2-D bounding boxes of subcomponents of the target spacecraft, the SPN method is easily generalizable to estimate the pose of multiple target geometries. Finally, to facilitate integration with navigation filters and perform continuous pose tracking, the SPN method estimates the uncertainty associated with the estimated pose. The secondary contribution of this article is the generation of the Spacecraft PosE Estimation Dataset (SPEED), which is used to train and evaluate the performance of the SPN method. SPEED consists of synthetic as well as actual camera images of a mock-up of the Tango spacecraft from the PRISMA mission. The synthetic images are created by fusing OpenGL-based renderings of the spacecraft's 3-D model with actual images of the Earth captured by the Himawari-8 meteorological satellite. The actual camera images are created using a seven degrees-of-freedom robotic arm, which positions and orients a vision-based sensor with respect to a full-scale mock-up of the Tango spacecraft with submillimeter and submillidegree accuracy. The SPN method, trained only on synthetic images, produces degree-level relative attitude error and cm-level relative position errors when evaluated on the actual camera images with a different distribution not used during training.

The combination of a small unmanned ground vehicle (UGV) and a large unmanned carrier vehicle allows more flexibility in real applications such as rescue in dangerous scenarios. The autonomous recovery system, which is used to guide the small UGV back to the carrier vehicle, is an essential component to achieve a seamless combination of the two vehicles. This paper proposes a novel autonomous recovery framework with a low-cost monocular vision system to provide accurate positioning and attitude estimation of the UGV during navigation. First, we introduce a light-weight convolutional neural network called UGV-KPNet to detect the keypoints of the small UGV from the images captured by a monocular camera. UGV-KPNet is computationally efficient with a small number of parameters and provides pixel-level accurate keypoints detection results in real-time. Then, six degrees of freedom pose is estimated using the detected keypoints to obtain positioning and attitude information of the UGV. Besides, we are the first to create a large-scale real-world keypoints dataset of the UGV. The experimental results demonstrate that the proposed system achieves state-of-the-art performance in terms of both accuracy and speed on UGV keypoint detection, and can further boost the 6-DoF pose estimation for the UGV., IET Image Processing, code: https://github.com/waterljwant/UGV-KPNet

Intelligent vehicle technology has become a research hot issue in recent ten years, the reason is that intelligent vehicles can not only be used as a flexible weapon platform in the military. And in life, it is also a system that provides convenience and security for people. For example, driverless cars and advanced driver assistance systems (ADAS). Information processing is the key to the degree of intelligence, and the detection and recognition of traffic safety information based on monocular vision is the core of information processing, it’s also the bottleneck problem. Because of the complexity and diversity of the environment have brought great challenges to this problem. In this paper, the existing lane detection methods in structured and semi-structured roads do not specifically consider the problem of weak line detection, two models are proposed. Fuzzy LDA enhancement model is used to enhance the contrast of lane area, another brightness contrast saliency model can be used for robust Lane extraction. Then, two models are applied to lane detection, a two-stage lane detection method is proposed and a blind area vehicle detection method is designed. Firstly, the vehicle area is roughly extracted based on road gray statistics, and then the typical vehicle features are screened finely. Finally, the extracted features and SVM classifiers are used to confirm the candidate regions. Experiments show that: The proposed method can detect the vehicle in the blind area very well and is insensitive to the shape distortion and size change of the vehicle.

Localization is one of the most challenging requirements needed for a mobile robots. Successful localization represents success in meeting the other principal requirements, such as perception and navigation. This article proposes a new approach for the localization of autonomous mobile robots using a Kinect sensor and the concept of Transfer Learning linked to Convolutional Neural Networks (CNNs). The images acquired from the sensor are applied in a mosaic form, which consists of the three color channels provided along with the depth information of the environment. Topological maps were used for indoor environment localization. The proposed computer vision system employed the Bayesian Classifier, k-Nearest Neighbor, Random Forest, Multi-layer Perceptron, and Support Vector Machine as the classifiers. The main focus of this work is the use of a unique configuration of RGB-D images transformed into a mosaic image, combined with the descriptive power of CNNs, in order to estimate the location of a mobile robot in an indoor environments. The results show that the proposed approach proved to be a convincing method for the tasks of localization and supporting the navigation of mobile robots; the results achieved 100% in Accuracy and in the F1-Score. The values obtained for the processing times were also suitable for a computer vision system, with 31.885ms, 0.040s, and 0.044s for the extraction, training, and classification stages, respectively. All these results were from the RGB-D mosaic images. The data demonstrated the relevance of the proposed work, providing a successful localization and, consequently, a successful navigation for mobile robots.

Most existing machine vision-based location methods mainly focus on the spatial positioning schemes using one or two cameras along with non-vision sensors. To achieve an accurate location, both schemes require processing a large amount of data. In this study, the authors propose a novel method, which requires much less amount of data to be processed for measuring target distance using monocular vision. Based on the geometric model of camera imaging, the parameters of the camera (such as camera's focal length and equivalent focal length.), as well as the principle of analogue signal being transformed into a digital signal, the authors derive the relationship among the target distance, field of view, equivalent focal length and camera resolution. Experimental results show that the proposed method can effectively and accurately achieve the target distance measurement.

Multirotor unmanned aerial systems (UASs) are often used to transport a variety of payloads. However, the maximum time that the cargo can remain airborne is limited by the flight endurance of the UAS. In this letter, we present a novel approach for two multirotors to transfer a payload between them in-air, while keeping the payload aloft and stationary. Our framework is built on a visual-feedback and grasping pipeline that enables one UAS to grasp the payload held by another, thereby allowing the UASs to act as swappable carriers. By connecting the payload outwards along a single rigid link, and allowing the UASs to maneuver about it, we let the payload remain online while it is transferred to a different carrier. Furthermore, building entirely on monocular vision, the approach does not rely on precise extrinsic localization systems. We demonstrate our proposed strategy in a variety of indoor and GPS-free outdoor experiments, and explore the range of operating limits for our system.

Two common methods exist for solving indoor autonomous navigation and obstacle-avoidance problems using monocular vision: the traditional simultaneous localization and mapping (SLAM) method, which requires complex hardware, heavy calculations, and is prone to errors in low texture or dynamic environments; and deep-learning algorithms, which use the fully connected layer for classification or regression, resulting in more model parameters and easy over-fitting. Among the latter ones, the most advanced indoor navigation algorithm divides a single image frame into multiple parts for prediction, resulting in doubled reasoning time. To solve these problems, we propose a multi-task deep network based on feature map region division for monocular indoor autonomous navigation. We divide the feature map instead of the original image to avoid repeated information processing. To reduce model parameters, we use convolution instead of the fully connected layer to predict the navigable probability of the left, middle, and right parts. We propose that the linear velocity is determined by combining three prediction probabilities to reduce collision risk. Experimental evaluation shows that the proposed method is nine times smaller than the previous state-of-the-art methods; further, its processing speed and navigation capability increase more than five and 1.6 times, respectively.

In the lunar surface sampling mission, the drop operation that transferring the packaging container into the sample return container is a vital step. In this paper, for supporting the drop operation, we proposed a method to measure the coaxiality between the two containers based on the monocular vision pose measurement using ellipse and line features. The method first extracts ellipse and line features in the 2D image to calculate the initial parameters. Then, the initial parameters and the point set are used to fit the curve by the cylinder perspective contours fitting method, where the constraints between ellipse and line features are used to improve the robustness to varying noises. Next, given the fitting ellipse and camera parameters, the pose of the packaging container is solved by using circle-based method. Finally, the coaxiality of the two containers is calculated based on the relative pose between them. Simulation experiments were carried out and the results show the effectiveness and reliability of the proposed method.

Perceiving the distance between vehicles is a crucial issue for advanced driving assistance systems. However, most vision-based distance estimation methods do not consider the influence of the change in camera attitude angles during driving or only use the vanishing point detected by lane lines to correct the pitch angle. This paper proposed an improved pinhole distance estimation model based on the road vanishing point without the lane line information. First, the road vanishing point is detected based on the dominant texture orientation, and the yaw and pitch angles of the camera are estimated. Then, a distance estimation model considering attitude angle compensation is established. Finally, the experimental results show that the proposed method can effectively correct the influence of the camera attitude angle on the distance estimation results.

The design and implementation of a real-time object motion sensing and trajectory replicating system, dependent on the image-based visual servoing capabilities of an electrohydraulic eye-in-hand parallel manipulator has been described in this paper. The tracking system consists of a monocular vision apparatus affixed to the top platform of a six-cylinder parallel manipulator constructed with industry grade double-acting single rod hydraulic actuators and budget proportional valves. The control algorithm is an amalgamation of three principles: machine vision based object tracking, image based visual servoing and actuator-level feedback control utilizing the inverse kinematics of the parallel manipulator. The real-time experiments are performed on a laboratory scale parallel manipulator setup with the aid of a virtual tracking object rendered on a computer monitor. The results for a few reference trajectories reveal that accurate tracking of target motions can be achieved with visual servoing algorithms designed for parallel manipulators in real-time.

Advances in deep learning make monocular vision approaches attractive for the autonomous driving domain. This work investigates a method for estimating the speed of the ego-vehicle using state-of-the-art deep neural network based optical flow and single-view depth prediction models. Adopting a straightforward intuitive approach and approximating a single scale factor, several application schemes of the deep networks are evaluated and meaningful conclusions are formulated, such as: combining depth information with optical flow improves speed estimation accuracy as opposed to using optical flow alone; the quality of the deep neural network results influences speed estimation performance; using the depth and optical flow data from smaller crops of wide images degrades performance. With these observations in mind, a RMSE of less than 1 m/s for ego-speed estimation was achieved on the KITTI benchmark using monocular images as input. Limitations and possible future directions are discussed as well.

In order to solve the problem that the blurred image of a moving object decreases accuracy in the process of detecting the payload swing angle of an overhead crane based on vision, and the tracking failure caused by the drastic change of grey targets, a robust real-time detection method of the load swing angle of a bridge crane is proposed. This method uses a spherical marker attached to the load, which is insensitive to rotation and tilt when it is detected. First, it uses the mean shift algorithm combined with Kalman filter to track the moving objects in the image plane continuously, and then integrates the method of minimum area circle to detect the spherical marker image in the region of interest accurately and quickly. Finally, combined with the geometric method, the real-time swing angle is calculated. In addition, the angle diagram method is used to increase the speed of calculating the swing angle. The experimental results show that the method is effective for detecting the load target swing angle of different trolley motion speed.

This paper addresses the problem of stabilizing the machine at a pose specified by a reference image and four laser sensor readings. The system configuration combining monocular vision and laser pointers is applied for pose estimation of the end-effector as well as providing the control objective. Specific features are selected to reflect the pose of the end-effector in Cartesian space, and an adaptive controller is designed to implement the alignment between the end-effector and a stationary object for further assembly. The complete control scheme is assessed through simulations and experiments conducted on an automatic drilling and riveting system, the results demonstrate that the assembly task can be accomplished accurately and efficiently even with coarsely calibrated camera and laser pointers. Furthermore, decoupled results can be gained thanks to the selected features.

Background The Heimann-Bielschowsky phenomenon (HBP) is an underrecognized condition characterized by slow, pendular, vertical oscillations of the eye accompanying monocular vision loss. Hypotropic dissociated vertical deviation (DVD) is another rare condition induced by asymmetric visual input. This report documents a rare case of HBP with hypotropic DVD. Case presentation This report describes a case of a 58-year-old woman with HBP and hypotropic DVD, having suffered monocular vision loss in the left eye due to blunt trauma at the age of 10. Preoperatively, she was orthophoric at near fixation and demonstrated an intermittent, slow hypotropia of the left eye upon distance fixation that never rose above the midline. She underwent a 7 mm recession of the inferior rectus muscle in the left eye. After surgery, intermittent, downward drifts became constant vertical oscillations at both distance and near fixation. Conclusions This case describes the clinical manifestation of an eye movement disorder related to prolonged monocular vision loss.

PurposeThe purpose of this paper is to present a visual detection approach to predict the poses of target objects placed in arbitrary positions before completing the corresponding tasks in mobile robotic manufacturing systems.Design/methodology/approachA hybrid visual detection approach that combines monocular vision and laser ranging is proposed based on an eye-in-hand vision system. The laser displacement sensor is adopted to achieve normal alignment for an arbitrary plane and obtain depth information. The monocular camera measures the two-dimensional image information. In addition, a robot hand-eye relationship calibration method is presented in this paper.FindingsFirst, a hybrid visual detection approach for mobile robotic manufacturing systems is proposed. This detection approach is based on an eye-in-hand vision system consisting of one monocular camera and three laser displacement sensors and it can achieve normal alignment for an arbitrary plane and spatial positioning of the workpiece. Second, based on this vision system, a robot hand-eye relationship calibration method is presented and it was successfully applied to a mobile robotic manufacturing system designed by the authors’ team. As a result, the relationship between the workpiece coordinate system and the end-effector coordinate system could be established accurately.Practical implicationsThis approach can quickly and accurately establish the relationship between the coordinate system of the workpiece and that of the end-effector. The normal alignment accuracy of the hand-eye vision system was less than 0.5° and the spatial positioning accuracy could reach 0.5 mm.Originality/valueThis approach can achieve normal alignment for arbitrary planes and spatial positioning of the workpiece and it can quickly establish the pose relationship between the workpiece and end-effector coordinate systems. Moreover, the proposed approach can significantly improve the work efficiency, flexibility and intelligence of mobile robotic manufacturing systems.

Most autonomous vehicles build their perception systems on expensive sensors, such as LIDAR, RADAR, and high-precision Global Positioning System (GPS). However, cameras can provide richer sensing at a considerably lower cost, this makes them a more appealing alternative. A driving assistance system (DAS) based on monocular vision has gradually become a research hotspot, and inter-vehicle distance estimation based on monocular vision is an important technology in DAS. There are still constrains in the existing methods for estimating the inter-vehicle distance based on monocular vision, such as low accuracy when distance is larger, unstable accuracy for different types vehicles, and significantly poor performance on distance estimation for severely occluded vehicles. To improve the accuracy and robustness of ranging results, this study proposes a monocular vision end-to-end inter-vehicle distance estimation method based on 3D detection. The actual area of the rare view of the vehicle and the corresponding projection area in the image are obtained by 3D detection method. An area-distance geometric model is then established on the basis of the camera projection principle to recover distance. Our method shows its potential in complex traffic scenarios by testing the test set data provided on the real-world computer vision benchmark, KITTI. The experimental results have superior performance than the existing published methods. Moreover, the accuracy of occluded vehicle ranging results can reach approximately $98\%$ , while the accuracy deviation between vehicles with different visual angles is less than $2\%$ .

An obstacle real-time sensing method, which is based on deep learning and target tracking method and integrated with monocular vision and binocular vision, is proposed for unmanned aerial vehicles (UAVs) in this paper. Firstly, it uses the deep learning method to detect and recognize the first-frame figure collected by cameras. Then, it uses the target tracking algorithm to track the detection results for the first-frame figure in real time to improve the real-time performance of the detection system. Meanwhile, it uses the binocular vision technology to execute the three-dimensional reconstruction for the current frame of the entire figure to obtain the environmental spatial information. Subsequently, combined with the points clustering strategy and the information fusion method, it can resolve the types, spatial locations, and outlines of obstacles. Finally, to verify the proposed method, we developed a physical prototype, and the results showed that the real-time sensing for obstacles can be realized under the condition that UAVs are equipped with one binocular camera.

How to work effectively under occlusion-rich environments remains a challenge for airborne vision-based ground target tracking, due to the natural limitation of monocular vision. Given this, a novel cross-drone binocular coordination approach, inspired by the efficient coordination of human eyes, is proposed and developed. The idea, derived from neural models of the human visual system, is to utilize distributed target measurements to overcome occlusion effects. Eventually, a binocular coordination controller is developed. It enables two distributed pan-tilt cameras to execute synergistic movements similar to human eyes. The proposed approach is able to work based on binocular or monocular vision, and hence it is practically appropriate for various environments. Both testbed experiments and field experiments are conducted for performance evaluation. Testbed experiments highlight its advantages over independent tracking in terms of accuracy while being robust to a partial perception ratio of up to 43%. Field experiments with a pair of drones further demonstrate its effectiveness in the real-world scenarios.