Search

The smart unmanned vending machine using machine vision technology suffers from the sharp decrease of detection accuracy due to the incomplete image collection of items by monocular camera in complex environment, and the lack of obvious features in dense stacking of items. In this paper, a binocular camera system is designed to effectively solve the problems of distortion and coverage caused by monocular camera. Besides, an image-stitching algorithm is developed to splice the images captured by the camera, which reliefs the burden of computation for back-end recognition processing brought by the binocular camera. A new model YOLOv3-TinyE is proposed based on YOLOv3-tiny model. Based on the data set of 21, 000 images captured in real scenarios that containing 20 different type of beverages, the comparison experimental results show that YOLOv3-TinyE model achieves the mean average precision of 99.15%, and the inference speed is 2.91 times faster than that of YOLOv3 model, and the detection accuracy of YOLOv3-TinyE model based on binocular vision is higher than that based on monocular vision. The results suggest that the designed method achieves the goal in terms of inference speed and average precision, that is, it is able to satisfy the requirements for real-world applications.

This article presents a robust brightness gradient-based estimation strategy for small aerial robots. The proposed nonlinear observer is capable of estimating the flight altitude and ego-motion by the fusion of monocular vision and inertial measurement unit feedback. The novelty primarily lies in the implementation of the gradient-based featureless approach and the direct use of photometric feedback in the state and output vectors. Under the single-plane assumption, the proposed framework permits the entire estimation process to be accomplished efficiently in a single iterative step without the need for feature detection and tracking or pre-computation of optic flow as commonly seen in conventional methods. The nonlinear and featureless implementation reduces the computational demand, enlarges the region of attraction, and markedly improves the robustness of the ego-motion estimation against scenes with scarce features when compared to Kalman-based estimators and feature-based methods. We conducted extensive flight experiments with different flying patterns and textures to evaluate the performance of the proposed observer. The results reveal that the root-mean-square error in the altitude estimates is approximately 10% for flights at ≈40-130 cm above the ground, comparable to feature-based estimators. Nevertheless, the devised observer does better than feature-based methods when deployed on low-textured scenes or with low-resolution blurry images.

Cameras have been widely used in traffic operations. While many technologically smart camera solutions in the market can be integrated into Intelligent Transport Systems (ITS) for automated detection, monitoring and data generation, many Network Operations (a.k.a Traffic Control) Centres still use legacy camera systems as manual surveillance devices. In this paper, we demonstrate effective use of these older assets by applying computer vision techniques to extract traffic data from videos captured by legacy cameras. In our proposed vision-based pipeline, we adopt recent state-of-the-art object detectors and transfer-learning to detect vehicles, pedestrians, and cyclists from monocular videos. By weakly calibrating the camera, we demonstrate a novel application of the image-to-world homography which gives our monocular vision system the efficacy of counting vehicles by lane and estimating vehicle length and speed in real-world units. Our pipeline also includes a module which combines a convolutional neural network (CNN) classifier with projective geometry information to classify vehicles. We have tested it on videos captured at several sites with different traffic flow conditions and compared the results with the data collected by piezoelectric sensors. Our experimental results show that the proposed pipeline can process 60 frames per second for pre-recorded videos and yield high-quality metadata for further traffic analysis.

The attitude of moving objects has a predictive function for navigation and positioning in a dim indoor environment. Currently, the accuracy of vision and inertial sensors’ attitude estimation is low, as it mainly affected by factors such as motion image blur, attitude angle processing algorithms, and data synchronization. Firstly, in this paper we propose a novel continuous multi-frame evaluation and registration algorithm to obtain high signal-to-noise ratio (SNR) images in regular indoor environments without global positioning system (GPS). The coordinates of vanishing points and plumb lines were extracted in the strong texture image. Then, a visual attitude model was constructed based on the characteristic points and lines. A pre-integrated gyro sensor was used to establish the inertial attitude model. The visual and gyro attitude models were used to obtain high-precision attitude information using multi-rate filtering. Finally, we build a hardware attachment composed of a consumer camera, an inexpensive inertial sensor and hardware circuit based on digital signal processing, which effectively demonstrates that the experiment result compares favourably with the more traditional methods.

Initialization is a fundamental task for the state estimation of multi-sensor integration. In this paper, we propose a novel approach to achieve a fast and accurate initialization for the integrated system. With the outputs from GNSS, the rough roll, pitch, yaw, and direction of gravity are calculated. Based on these initial states, the GNSS/INS fusion and Visual-Inertial Odometry (VIO) are simultaneously launched whenever the observations are available. Subsequently, a coarse scale is retrieved by assigning the initial states of GNSS/INS fusion to the VIO procedure. The scale can be utilized to get poses and points very close to the real world. Once the VIO estimation converges, the transformation parameters between the poses of VIO and GNSS/INS fusion are estimated via non-linear optimization. Thus, we can align these two trajectories and evaluate the scale error. Through six vehicular tests, we analyze the performance of the proposed method with different illumination, obstructions, and motion patterns, and compare the results of the proposed approach with the state-of-the-art algorithms. The results indicated that the Visual-Inertial parameters converge within 9 seconds in our method. The scale errors of both our method are mostly within 10%. The alignment errors are mostly at centimeter-level in our method, which is comparable to the existing approaches.

In order to solve the problem of high measurement cost and complex operation of position-independent geometric errors (PIGEs) calibration on a five-axis motion platform, this paper first proposes a low-cost pose measurement method, based on monocular vision, which can accurately determine the pose in the environment, even with image shadow and noise. Next, an improved method, combining pose measurement and kinematic parameters identification, is proposed to calibrate a five-axis motion platform. The kinematic error model of the platform and the pose planning of automatic image acquisition are established, providing the pose data and motor position data, required for calibration. Combined with the kinematic loop method, the kinematic parameters of the five-axis motion platform are identified, while the geometric structure parameters are accurately calibrated. Before and after calibration, a circular trajectory of the target coordinate system (TCS) origin, relative to the camera coordinate system (CCS), is used to test the comprehensive accuracy evolution of the five-axis motion platform, by comparing the position and orientation errors of the theoretical circle trajectory to the actual one. The experimental data show that, before and after calibration, the average position error of the five-axis motion platform is reduced by 79.46%, while the average direction error is reduced by 86.53%. The experimental results clearly demonstrate that the proposed calibration method significantly improves the comprehensive motion accuracy of the five-axis motion platform, and they verify the practical value and effectiveness of the calibration scheme.

Purpose This paper aims to develop a robust person tracking method for human following robots. The tracking system adopts the multimodal fusion results of millimeter wave (MMW) radars and monocular cameras for perception. A prototype of human following robot is developed and evaluated by using the proposed tracking system. Design/methodology/approach Limited by angular resolution, point clouds from MMW radars are too sparse to form features for human detection. Monocular cameras can provide semantic information for objects in view, but cannot provide spatial locations. Considering the complementarity of the two sensors, a sensor fusion algorithm based on multimodal data combination is proposed to identify and localize the target person under challenging conditions. In addition, a closed-loop controller is designed for the robot to follow the target person with expected distance. Findings A series of experiments under different circumstances are carried out to validate the fusion-based tracking method. Experimental results show that the average tracking errors are around 0.1 m. It is also found that the robot can handle different situations and overcome short-term interference, continually track and follow the target person. Originality/value This paper proposed a robust tracking system with the fusion of MMW radars and cameras. Interference such as occlusion and overlapping are well handled with the help of the velocity information from the radars. Compared to other state-of-the-art plans, the sensor fusion method is cost-effective and requires no additional tags with people. Its stable performance shows good application prospects in human following robots.

Traditional navigation systems rely mainly on satellite navigation, but internal positioning and navigation cannot be achieved. This is mainly due to the inability to penetrate the wall due to the complex internal environment and the signal reaching the ground. Therefore, this paper proposes the research of indoor positioning algorithm based on improved convolutional neural network. The main method of this paper is based on the monocular vision indoor positioning based on the improved convolutional neural network. This method is an intelligent solution to the problems of traditional methods such as high cost, poor anti-interference ability, weak robustness, and poor compatibility. Positioning aids. The traditional computer vision camera pose estimation method is affected by the complex background in the image. When extracting feature corners, it is very affected by non-interest corners, so this paper adds an improved convolutional neural network algorithm to add in complex indoor scenes. With the region limitation, camera pose estimation in the region of interest is better to achieve low-cost, high accuracy and more stability. The improved model in this paper was tested on a test set of indoor datasets with markers, and the recognition accuracy reached 98.1%. In addition, this paper improves the PnP solution method, compares and analyzes the traditional RPnP, EPnP, and CEPPnP algorithms, and seeks the optimal camera pose estimation algorithm to locate the world coordinates of the current shooting position. The experimental results show that this paper proposes the positioning algorithm has high stability.

Distance perception is the basis and necessary prerequisite of environment perception, attitude perception and obstacle avoidance for both intelligent vehicle and unmanned vehicle. Passive ranging is a critical part of machine vision measurement. Most passive ranging methods based on machine vision apply binocular technology that needs strict hardware conditions and lacks universality. Therefore, the monocular vision ranging method is one of the mainstream distance sensing methods at present. In order to improve the accuracy of monocular vision ranging, a monocular vision ranging method based on pixel area and aspect ratio is proposed. Subsequently, this method improves the stability of real-time target detection by introducing Kalman filter processing. Experimental results display that that ranging used by this method has higher accuracy. The mean relative error of the depth measurement is 5% when it is 3-10 m. After introducing Kalman filter, the stability of real-time ranging processing is improved by 25.21%. In this paper, the ROS smart car with real-time target tracking is also realized by the method based on the combination of SIFT-KCF target detection and tracking and monocular ranging.

This paper designed an intelligent service robot system in highway based on multi-sensor fusion. The mobile robot attempts to fuse the lidar information and monocular vision information to estimate the pose of itself and obtain an environmental map. It adapts a new SLAM method which combines lidar and vision information. Lidar is used to obtain the 2D occupancy grid map and the monocular vision SLAM algorithm uses the Extended Kalman Filter (EKF) to magnify the pose estimation. The 3-DOF pose provided by lidar is obtained through Cartographer algorithm and the monocular vision SLAM who offers the 6-DOF pose is realized with ORB-SLAM. The experimental results show that the system is effective in application as an intelligent service robot of highway.

In intelligent video surveillance under complex scenes, it is vital to identify the current actions of multi-target human bodies accurately and in real time. In this paper, a real-time multi-person action recognition method with monocular vision is proposed based on sequence models. Firstly, the key points of multi-target human body skeleton in the video are extracted by using the OpenPose algorithm. Then, the human action features are constructed, including limb direction vector and the skeleton height-width ratio. The multi-target human bodies tracking is then achieved by using the tracking algorithm. Next, the tracking results are matched with the action features, and the action recognition model is constructed, which includes the spatial branch based on Deep neural networks and the temporal branch based on Bi-directional RNN and Bi-directional long short-term memory networks. After pre-training, the model can be used to recognize the human body action from action features, and a recognition stabilizer is designed to minimize false alarms. Finally, extensive evaluations on the JHMDB dataset validate the effectiveness and the superiority of the proposed approach.

In the present work, we develop an adaptive dynamic controller based on monocular vision for the tracking of objects with a three-degrees of freedom (DOF) Scara robot manipulator. The main characteristic of the proposed control scheme is that it considers the robot dynamics, the depth of the moving object, and the mounting of the fixed camera to be unknown. The design of the control algorithm is based on an adaptive kinematic visual servo controller whose objective is the tracking of moving objects even with uncertainties in the parameters of the camera and its mounting. The design also includes a dynamic controller in cascade with the former one whose objective is to compensate the dynamics of the manipulator by generating the final control actions to the robot even with uncertainties in the parameters of its dynamic model. Using Lyapunov’s theory, we analyze the two proposed adaptive controllers for stability properties, and, through simulations, the performance of the complete control scheme is shown.

In order to improve the accuracy of conical target pose parameters measured by monocular vision, we propose a high precision visual measurement method for conical target based on the minimum reconstruction error of 3D. Firstly, we select the main and auxiliary features used for pose parameters’ measurement by analyzing target’s imaging process. Secondly, the algebraic and geometric methods for measuring pose parameters are determined based on the selected features. Finally, we utilize the geometric constraints of target to establish the reconstruction error formula and optimize the pose parameters based on the minimum reconstruction error principle to improve the measurement accuracy. To verify the effectiveness of our method, we use 3ds Max software to generate target’s simulation images under different pose parameters and compare the pose parameters before and after optimization. Experimental results show that the proposed method can significantly improve the accuracy of pose parameters when the measurement distance is far (more than 30 m), which is up to about 40%.

Micro Aerial Vehicles (MAVs) have the great potential to be deployed in commercial or health care services such as e-commerce package delivery, transportation of medicines, same-day food delivery, and other time-sensitive transportation tasks. A team of MAVs can cooperatively transport objects to overcome the physical limitations of a single vehicle, while concurrently increasing the system's resilience to vehicles’ failures. In this letter, we address the state estimation, control and trajectory tracking problems of cooperative transportation of cable suspended rigid body payloads with MAVs using monocular vision and inertial sensing. The key contributions are (a) a distributed vision-based coordinated control of the cable-suspended rigid body payload on SE(3), (b) a distributed estimation approach that allows each agent to estimate its cable direction and velocity independently, and (c) a new cooperative estimation scheme that can infer the payload's full 6-DoF states. This is obtained by sharing the robots’ local position estimates and their relative position with respect to their corresponding attachment points on the payload. It allows to infer the payload's 6-DoF state when it is not directly measurable by each vehicle. The proposed solution runs in real-time on board and is validated through experimental results with multiple quadrotors transporting a rigid body payload via cables.

The camera calibration in monocular vision represents the relationship between the pixels’ units which is obtained from a camera and the object in the real world. As an essential procedure, camera calibration calculates the three-dimensional geometric information from the captured two-dimensional images. Therefore, a modified camera calibration method based on polynomial regression is proposed to simplify. In this method, a parameter vector is obtained by pixel coordinates of obstacles and corresponding distance values using polynomial regression. The set of parameter’s vectors can measure the distance between the camera and the ground object in the field of vision under the camera’s posture and position. The experimental results show that the lowest accuracy of this focal length calibration method for measurement is 97.09%, and the average accuracy was 99.02%.

As the indoor dynamic target localization does not detect and repair the circumferential jump value in time, which leads to large position and attitude errors and low-velocity stability, a combined Global Navigation Satellite System/Simultaneous Localization and Mapping- (GNSS/SLAM-) based high-precision indoor dynamic target localization method is proposed. The method uses Empirical Mode Decomposition (EMD) to decompose the noisy signal, obtains the noise energy as the dominant mode from the decomposed components, extracts the useful signal energy as the main dividing point, removes the high-frequency signal, constructs the low-frequency component to realize low-pass filtering and denoising, selects a suitable threshold processing function to make the high-frequency signal component retain the detailed signal effectively to realize high-frequency component denoising, detects and fixes the circumferential jump of the observed data, and detects and fixes the circumferential jump of each frequency. The indoor dynamic target positioning method is established by combining GNSS/SLAM to achieve high accuracy target positioning. The experimental results show that the position and attitude errors are small, and the velocity is stable, which indicates that the position information is closer to the dynamic target, i.e., the target positioning is more accurate. To address the problems of scale drift and frequent initialization due to environmental factors in monocular vision SLAM, an Ultra Wideband (UWB)/vision fusion localization model that takes into account the scale factor is proposed, which can give full play to the complementary characteristics between UWB and vision; solve the problems of visual initialization, scale ambiguity, and absolute spatial reference; and improve UWB localization accuracy and localization frequency as well as reducing the number of base stations. The model can reliably achieve 0.2 m accuracy in indoor environments with sparse texture or frequent light changes.

Dynamic acquisition of crop morphology is beneficial to real-time variable decision of precise spraying operations in fields. However, the existing spraying quantity regulation has high tolerance on the statistical characteristics of regional morphology, so expensive LiDAR and ultrasonic radar can't make full use of their high accuracy, and can reduce decision speed because of too much detail of branches and leaves. Therefore, designing a novel recognition system embedded machine learning with low-cost monocular vision is more feasible, especially in China, where the agricultural implements are medium sizes and cost-sensitive. In addition, we found that the growth period of crops is an important reference index for guiding spraying. So, taking cotton as a case study, a cotton morphology acquisition by a single camera is established, and a cotton growth period recognition algorithm based on Convolution Neural Network (CNN) is proposed in this paper. Through the optimization process based on confusion matrix and recognition efficiency, an optimized CNN model structure is determined from 9 different model structures, and its reliability was verified by changing training sets and test sets many times based on the idea of k-fold test. The accuracy, precision, recall, F1-score and recognition speed of this CNN model are 93.27%, 95.39%, 94.31%, 94.76% and 71.46 ms per image, respectively. In addition, compared with the performance of VGG16 and AlexNet, the convolution neural network model proposed in this paper has better performance. Finally, in order to verify the reliability of the designed recognition system and the feasibility of the spray decision-making algorithm based on CNN, spraying deposition experiments were carried out with 3 different growth-periods of cotton. The experiments’ results validate that after the optimal spray parameters were applied at different growth periods respectively, the average optimum index in 3 growth periods was 42.29%, which was increased up to 62.24% than the operations without distinguishing growth periods.

Autonomous robots for medical and emergency supplies are a potential way to avoid contact with people in quarantine and control the spread of contagious diseases in an indoor scene. However, scene understanding and reconstruction through a single low-cost camera remains a challenge. It is known that absolute precise depth cannot be calculated accurately from a single image, but the relative pose of different planes, which can be inferred from geometric features in a 2-D image, are more likely to be used in understanding scenes and its reconstruction. In this article, we present an interpretable model to bridge the gap between 2-D scene understanding and three-dimensional (3-D) reconstruction without prior training or any precise depth data. Based on 2-D semantic information in our previous works, the 3-D relative pose of estimated planes can be estimated. At that point, indoor scenes are approximated in the reconstruction. The approach behaves as an interpretable characteristic and requires no prior training or knowledge of the camera's internal parameters. We compare the quantitative performance on the percentage of incorrectly reconstructed planes by relative pose estimation. The results demonstrated that the method can successfully understand and reconstruct indoor scenes including both Manhattan and curved non-Manhattan structures.

The rapid development of unmanned aerial vehicles (UAV) puts forward a higher requirement for autonomous obstacle avoidance. Due to the limited payload and power supply, small UAVs such as quadrotors usually carry simple sensors and computation units, which makes traditional methods more challenging to implement. In this paper, a novel framework is demonstrated to control a quadrotor flying through crowded environments autonomously with monocular vision. The framework adopts a two-stage architecture, consisting of a sensing module and a decision module. The sensing module is based on an unsupervised deep learning method. And the decision module uses dueling double deep recurrent Q-learning to eliminate the adverse effects of limited observation capacity of an on-board monocular camera. The framework enables the quadrotor to realize autonomous obstacle avoidance without any prior environment information or labeled datasets for training. The trained model shows a high success rate in the simulation and a good generalization ability for transformed scenarios., 23 pages, 10 figures

Recently, real-time obstacle detection by monocular vision exhibits a promising prospect in enhancing the safety of unmanned surface vehicles (USVs). Since the obstacles that may threaten USVs generally appear below the water edge, most existing methods first detect the horizon line and then search for obstacles below the estimated horizon line. However, these methods detect horizon line only using edge or line features, which are susceptible to interference edges from clouds, waves, and land, eventually resulting in poor obstacle detection. To avoid being affected by interference edges, in this paper, we propose a novel horizon line detection method based on semantic segmentation. The method assumes a Gaussian mixture model (GMM) with spatial smoothness constraints to fit the semantic structure of marine images and simultaneously generate a water segmentation mask. The horizon line is estimated from the water boundary points via straight line fitting. Further, inspired by human visual attention mechanisms, an efficient saliency detection method based on background prior and contrast prior is presented to detect obstacles below the estimated horizon line. To reduce false positives caused by sun glitter, waves and foam, the continuity of the adjacent frames is employed to filter the detected obstacles. An extensive evaluation was conducted on a large marine image dataset collected by our ‘Jinghai VIII’ USV. The experimental results show that the proposed method significantly outperformed the recent state-of-the-art marine obstacle method by 22.07% in terms of F-score while running over 24 fps on an NVIDIA GTX1080Ti GPU.

In this work, we propose a method for AUV to reconstruct the docking ring autonomously under the guidance of visual information. The proposed method is based on the knowledge of self-reconfiguration of two AUVs in underwater environment. In order to enable long distance performance, we make use of blue-green light beacons. The light beacons are detected by using YOLO V3 algorithm. In addition, P4P algorithm is applied to resolve the problem of relative poses of AUVs. In case of short distances, we use Aruco marker to precisely guide the AUV towards the docking ring. In addition, we use KCF algorithm to shorten the recognition time, and Kalman filter method to eliminate the interference of occlusions and jitters. The feasibility of the proposed method is verified by performing multiple experiments. Under the reasonable control flow and visual algorithm, the proposed solution avoids the situation where the accuracy of long-distance monocular recognition is low and is easy to be misjudged. The underwater multi-target integrated guidance method based on monocular vision is convenient for the recovery of underwater AUV, and can effectively take into account the advantages of wide effective guidance range and high precision of close range guidance.

An adaptive machine-learning object-localization system was proposed in this study and applied to an automated carrier robot equipped with monocular vision, and the problem of object-distance localization in vision was improved using machine learning. When executing a project involving small robots, the calculation of the target object by the object-distance algorithm depends on the system-defined parameters, such as camera height, field of view, and inclination angle. When the carrier encounters an inclination that causes changes in the camera’s external parameters, the parameters defined in the system differ from those under real conditions, causing severe errors in calculation results. This system combined a triaxial sensor and machine-learning regression to identify the functional relationship between the output signal of the accelerometer and the actual carrier attitude. The regression network model was trained by recording several sets of signal features and carrier angles as the data set. During object-distance calculation, the adaptive machine-learning object-localization system predicted the current carrier angle by using the output of the accelerometer and the trained regression network model to update the algorithm’s angle parameters. The experimental results indicated that the system possessed stability and accuracy in calculating the distance of the target object when the carrier angle changed.

In diverse applications of the industrial Internet of Things (IIoT), the six degrees of freedom (6-DoF) information is essential, which determines the attitude and position of a 3-D object. Nevertheless, due to the complexity and variability of the scenarios, higher requirements are imposed on the 6-DoF estimation. Among them, occlusion is one of the knottiest problems, which causes significant performance degradation and needs to be solved urgently. Therefore, in this article, we propose a completely new and universal multibranch network (MBN) for industrial applications. Our method is based on monocular vision system and convolutional neural network (CNN) framework. First and foremost, it reduces occlusion interference by focusing on the physical area characteristics of the image. Compared with the traditional CNN-based method, it owns higher accuracy and lower estimation error under occlusion. Second, we propose five algorithms to process the predictions of the independent branches, further effectively improving performance. Third, we optimize the marker to solve the inequality problem in attitude angle estimation. Furthermore, we design and conduct a series of experiments, and the experimental results sufficiently prove the superiority of MBN

Forward collision warning (FCW) system is an advanced driver assistance safety system that continually monitors the ego vehicle’s position, predicts a collision and update it to the driver. It is challenging as it requires to detect and track moving vehicles on-road. In this paper, a novel monocular vision-based FCW system with range prediction is proposed. Firstly, the on-road moving vehicles are detected using a pre-trained YOLO (You only look once) model by setting it as a regression problem to predict the bounding boxes and its class probabilities. The detected vehicles are then assigned a unique ID using Hungarian algorithm and is tracked using Kalman filter. Secondly, the distance between the detected vehicle and the ego-vehicle is calculated by mapping the image into a two-dimensional orthogonal top-down view using inverse perspective mapping (IPM). Road boundary detection is carried out to obtain the calibration points for IPM. Quantitative evaluation carried out on recorded video clips in traffic environment shows that the proposed FCW system effectively estimate the inter-vehicle distance and its relative velocity to predict a collision with great accuracy.

In 3-D attitude angle estimation, monocular vision-based methods are often utilized for the advantages of short-time and high efficiency. However, the limitations of these methods lie in the complexity of the algorithm and the specificity of the scene, which needs to match the characteristics of the cooperation object and the scene. In this article, we propose a fully connected attitude detection network (FADN), which combines neural network and traditional algorithms for 3-D attitude angle estimation. FADN provides a whole process from the input of a single frame image in the industrial video stream to the output of the corresponding 3-D attitude angle estimation. Benefiting from the end-to-end estimation framework, FADN avoids tedious matching algorithms and thus has certain portability. A series of comparative experiments based on the rendering software 3-D Studio Max (3d Max) have been carried out to evaluate the performance of FADN. The experimental results show that FADN has high estimation accuracy and fast running speed. At the same time, the simulation results reliably prove the feasibility of FADN, and also promote the research in real scenarios.

Athletes’ sports detection has a heavy pressure on athletes’ training and post-injury rehabilitation. In the traditional mobilization test, there is no effective combination of exercise and rehabilitation, which directly leads to the athlete’s physical health cannot be guaranteed. Based on this, this study combines the current situation of the athletes’ field and the training ground, and uses monocular vision as the video input interface, and combines the monocular vision technology in the research. Moreover, in the research, this paper combines the human body model to construct an athlete’s human body model that adapts to monocular vision. At the same time, this paper combines the image processing technology to transform the image of the monocular visual athlete into a skeleton model, so as to realize the modeling of the athlete’s movement. In addition, this paper combines the model to explore the indoor and outdoor athlete recovery techniques and validates the model by experiment. The research shows that the research model has certain effects, which can meet the actual needs, and can provide theoretical reference for subsequent related research.

Depth estimation is an essential component in computer vision systems for achieving 3D scene understanding. Efficient and accurate depth map estimation has numerous applications including self-driving vehicles and virtual reality tools. This paper presents a new deep network, called D-Net, for depth estimation from a single RGB image. The proposed network can be trained end-to-end, and its structure can be customised to meet different requirements in model size, speed, and prediction accuracy. Our approach gathers strong global and local contextual features at multiple resolutions, and then transfers these to high resolutions for clearer depth maps. For the encoder backbone, D-Net can utilise many state-of-the-art models including EfficientNet, HRNet and Swin Transformer to obtain dense depth maps. The proposed D-net is designed to have minimal parameters and reduced computational complexity. Extensive evaluations on the NYUv2 and KITTI benchmark datasets show that our model is highly accurate across multiple backbones, and it achieves state-of-the-art performance on both benchmarks when combined with the Swin Transformer and HRNets.

Obstacle distance measurement is one of the key technologies for autonomous navigation of high-voltage transmission line inspection robots. To address the robustness of obstacle distance measurement under varying illumination conditions, this article develops a research method that fuses image enhancement with robot monocular vision so that the robot can adapt to various levels of illumination running along the transmission line. During the inspection of high-voltage transmission lines in such an overexposed (excessively bright) environment, a specular highlight suppression method is proposed to suppress the specular reflections in an image; when scene illumination is insufficient, a robust low-light image enhancement method based on a tone mapping algorithm with weighted guided filtering is presented. Based on the monocular vision measurement principle, the error generation mechanism is analyzed through experiments, and we introduce the parameter modification mechanism. The two proposed image enhancement methods outperform other state-of-the-art enhancement algorithms in qualitative and quantitative analyses. The experimental results show that the measurement error is less than 3% for static distance measurements and less than 5% for dynamic distance measurements within 6 m. The proposed method can meet the requirements of high-accuracy positioning, real-time performance and strong robustness. This method greatly contributes to the sustainable development of inspection robots in the power industry.

Vision-based manipulation has been largely used in various robot applications. Normally, in order to obtain the spatial information of the operated target, a carefully calibrated stereo vision system is required. However, it limits the application of robots in the unstructured environment which limits both the number and the pose of the camera. In this study, a calibration-free monocular vision-based robot manipulation approach is proposed based on domain randomization and deep reinforcement learning (DRL). Firstly, a learning strategy combined domain randomization is developed to estimate the spatial information of the target from a single monocular camera arbitrarily mounted in a large area of the manipulation environment. Secondly, to address the monocular occlusion problem which regularly happens during robot manipulations, an occlusion awareness DRL policy has been designed to control the robot to avoid occlusions actively in the manipulation tasks. The performance of our method has been evaluated on two common manipulation tasks, reaching and lifting of a target building block, which show the efficiency and effectiveness of our proposed approach.

Due to its low cost, low power, and non-cooperative advantage, monocular vision has become a crucial solution for enhancing the Sense and Avoid capability of unmanned aerial vehicles (UAVs). However, due to the restricted optical measurement properties of monocular vision, there are drawbacks to monocular-vision-based obstacle perception; namely, the capability for obstacle range information perception and global obstacle perception is insufficient. In this paper, a local-bearing-angle-based trajectory planning algorithm is proposed for UAV collision avoidance. First, the observability of monocular-vision-based obstacle range perception is theoretically analyzed. The local coordinate system is selected for collision avoidance, and the bearing angle is selected as the observation. Second, the problem of UAV monocular vision collision avoidance trajectory planning is mathematically formulated. The quadratic Bezier curve is selected to model the trajectory, and the trajectory planning problem is transformed into optimization based on a single objective function. The collision avoidance constraint based on the relative bearing angle rate is designed, and the collision avoidance trajectory segment is optimized on the basis of the bearing angle observed within a limited time period. Finally, the receding horizon is adopted to optimize the trajectory sequence, and the sequences are connected to formulate the near-globally optimal trajectory. Simulation experiments reveal that the collision avoidance trajectory planning method is capable of avoiding both static and dynamic obstacles. Compared with global-information-based collision avoidance, the convergence time of the method proposed in this paper is reduced, and the near-globally optimal trajectory can be acquired in real time, which is in accordance with the perception capability of monocular vision.

This article presents the Spacecraft Pose Network (SPN), the first neural network-based method for on-board estimation of the pose, i.e., the relative position and attitude, of a known noncooperative spacecraft using monocular vision. In contrast to other state-of-the-art pose estimation approaches for spaceborne applications, the SPN method does not require the formulation of hand-engineered features and only requires a single grayscale image to determine the pose of the spacecraft relative to the camera. The SPN method uses a convolutional neural network (CNN) with three branches to solve the problem of relative attitude estimation. The first branch of the CNN bootstraps a state-of-the-art object detection algorithm to detect a 2-D bounding box around the target spacecraft in the input image. The region inside the 2-D bounding box is then used by the other two branches of the CNN to determine the relative attitude by initially classifying the input region into discrete coarse attitude labels before regressing to a finer estimate. The SPN method then estimates the relative position by using the constraints imposed by the detected 2-D bounding box and the estimated relative attitude. Further, with the detection of 2-D bounding boxes of subcomponents of the target spacecraft, the SPN method is easily generalizable to estimate the pose of multiple target geometries. Finally, to facilitate integration with navigation filters and perform continuous pose tracking, the SPN method estimates the uncertainty associated with the estimated pose. The secondary contribution of this article is the generation of the Spacecraft PosE Estimation Dataset (SPEED), which is used to train and evaluate the performance of the SPN method. SPEED consists of synthetic as well as actual camera images of a mock-up of the Tango spacecraft from the PRISMA mission. The synthetic images are created by fusing OpenGL-based renderings of the spacecraft's 3-D model with actual images of the Earth captured by the Himawari-8 meteorological satellite. The actual camera images are created using a seven degrees-of-freedom robotic arm, which positions and orients a vision-based sensor with respect to a full-scale mock-up of the Tango spacecraft with submillimeter and submillidegree accuracy. The SPN method, trained only on synthetic images, produces degree-level relative attitude error and cm-level relative position errors when evaluated on the actual camera images with a different distribution not used during training.

The combination of a small unmanned ground vehicle (UGV) and a large unmanned carrier vehicle allows more flexibility in real applications such as rescue in dangerous scenarios. The autonomous recovery system, which is used to guide the small UGV back to the carrier vehicle, is an essential component to achieve a seamless combination of the two vehicles. This paper proposes a novel autonomous recovery framework with a low-cost monocular vision system to provide accurate positioning and attitude estimation of the UGV during navigation. First, we introduce a light-weight convolutional neural network called UGV-KPNet to detect the keypoints of the small UGV from the images captured by a monocular camera. UGV-KPNet is computationally efficient with a small number of parameters and provides pixel-level accurate keypoints detection results in real-time. Then, six degrees of freedom pose is estimated using the detected keypoints to obtain positioning and attitude information of the UGV. Besides, we are the first to create a large-scale real-world keypoints dataset of the UGV. The experimental results demonstrate that the proposed system achieves state-of-the-art performance in terms of both accuracy and speed on UGV keypoint detection, and can further boost the 6-DoF pose estimation for the UGV., IET Image Processing, code: https://github.com/waterljwant/UGV-KPNet

Anticipating the intentions of vulnerable road users (VRUs) such as pedestrians and cyclists is critical for performing safe and comfortable driving maneuvers. This is the case for human driving and, thus, should be taken into account by systems providing any level of driving assistance, from advanced driver assistant systems (ADAS) to fully autonomous vehicles (AVs). In this paper, we show how the latest advances on monocular vision-based human pose estimation, i.e. those relying on deep Convolutional Neural Networks (CNNs), enable to recognize the intentions of such VRUs. In the case of cyclists, we assume that they follow traffic rules to indicate future maneuvers with arm signals. In the case of pedestrians, no indications can be assumed. Instead, we hypothesize that the walking pattern of a pedestrian allows to determine if he/she has the intention of crossing the road in the path of the ego-vehicle, so that the ego-vehicle must maneuver accordingly (e.g. slowing down or stopping). In this paper, we show how the same methodology can be used for recognizing pedestrians and cyclists' intentions. For pedestrians, we perform experiments on the JAAD dataset. For cyclists, we did not found an analogous dataset, thus, we created our own one by acquiring and annotating videos which we share with the research community. Overall, the proposed pipeline provides new state-of-the-art results on the intention recognition of VRUs., Comment: Paper accepted by IEEE Trans. on Intelligent Transportation Systems. arXiv admin note: substantial text overlap with arXiv:1807.10580

Intelligent vehicle technology has become a research hot issue in recent ten years, the reason is that intelligent vehicles can not only be used as a flexible weapon platform in the military. And in life, it is also a system that provides convenience and security for people. For example, driverless cars and advanced driver assistance systems (ADAS). Information processing is the key to the degree of intelligence, and the detection and recognition of traffic safety information based on monocular vision is the core of information processing, it’s also the bottleneck problem. Because of the complexity and diversity of the environment have brought great challenges to this problem. In this paper, the existing lane detection methods in structured and semi-structured roads do not specifically consider the problem of weak line detection, two models are proposed. Fuzzy LDA enhancement model is used to enhance the contrast of lane area, another brightness contrast saliency model can be used for robust Lane extraction. Then, two models are applied to lane detection, a two-stage lane detection method is proposed and a blind area vehicle detection method is designed. Firstly, the vehicle area is roughly extracted based on road gray statistics, and then the typical vehicle features are screened finely. Finally, the extracted features and SVM classifiers are used to confirm the candidate regions. Experiments show that: The proposed method can detect the vehicle in the blind area very well and is insensitive to the shape distortion and size change of the vehicle.

Localization is one of the most challenging requirements needed for a mobile robots. Successful localization represents success in meeting the other principal requirements, such as perception and navigation. This article proposes a new approach for the localization of autonomous mobile robots using a Kinect sensor and the concept of Transfer Learning linked to Convolutional Neural Networks (CNNs). The images acquired from the sensor are applied in a mosaic form, which consists of the three color channels provided along with the depth information of the environment. Topological maps were used for indoor environment localization. The proposed computer vision system employed the Bayesian Classifier, k-Nearest Neighbor, Random Forest, Multi-layer Perceptron, and Support Vector Machine as the classifiers. The main focus of this work is the use of a unique configuration of RGB-D images transformed into a mosaic image, combined with the descriptive power of CNNs, in order to estimate the location of a mobile robot in an indoor environments. The results show that the proposed approach proved to be a convincing method for the tasks of localization and supporting the navigation of mobile robots; the results achieved 100% in Accuracy and in the F1-Score. The values obtained for the processing times were also suitable for a computer vision system, with 31.885ms, 0.040s, and 0.044s for the extraction, training, and classification stages, respectively. All these results were from the RGB-D mosaic images. The data demonstrated the relevance of the proposed work, providing a successful localization and, consequently, a successful navigation for mobile robots.

Most existing machine vision-based location methods mainly focus on the spatial positioning schemes using one or two cameras along with non-vision sensors. To achieve an accurate location, both schemes require processing a large amount of data. In this study, the authors propose a novel method, which requires much less amount of data to be processed for measuring target distance using monocular vision. Based on the geometric model of camera imaging, the parameters of the camera (such as camera's focal length and equivalent focal length.), as well as the principle of analogue signal being transformed into a digital signal, the authors derive the relationship among the target distance, field of view, equivalent focal length and camera resolution. Experimental results show that the proposed method can effectively and accurately achieve the target distance measurement.

Multirotor unmanned aerial systems (UASs) are often used to transport a variety of payloads. However, the maximum time that the cargo can remain airborne is limited by the flight endurance of the UAS. In this letter, we present a novel approach for two multirotors to transfer a payload between them in-air, while keeping the payload aloft and stationary. Our framework is built on a visual-feedback and grasping pipeline that enables one UAS to grasp the payload held by another, thereby allowing the UASs to act as swappable carriers. By connecting the payload outwards along a single rigid link, and allowing the UASs to maneuver about it, we let the payload remain online while it is transferred to a different carrier. Furthermore, building entirely on monocular vision, the approach does not rely on precise extrinsic localization systems. We demonstrate our proposed strategy in a variety of indoor and GPS-free outdoor experiments, and explore the range of operating limits for our system.

Picking up and utilizing visual information is thought to be of high importance in the control of trampoline skills. Yet, the question arises how information from the different vision systems contributes to trampoline gymnastics performance. The aim of this study was to examine the role of binocular and monocular visual information in trampoline gymnastics. N = 12 gymnasts performed straight leaps on the trampoline under full vision, monocular vision, and under occluded vision. Gymnasts’ preferred flight duration as well as gymnasts’ variable error in feet placement on the trampoline bed were analysed by means of an optic movement analysis system. Results revealed that gymnasts exhibited longer flight duration in the binocular vision condition and the monocular vision condition, as compared to the occluded vision condition. Gymnasts furthermore exhibited a larger variable error in feet placement with less visual information available. It is argued that gymnasts benefit from the availability of binocular information in order to perform precise leaps on the trampoline. Nevertheless, it is argued that utilizing visual cues in trampoline gymnastics is likely to depend on the current configuration of task-constraints (i.e., performing straight leaps vs. somersaults with and without twists).

Obtaining a high-precision 3-D burden surface (BS) shape of a blast furnace (BF) is of significance for improving energy efficiency in the ironmaking process, and for stabilizing the operation of the BF. However, existing methods used for detecting the BS cannot provide precise data to guide the burden distribution operation of the BF effectively, because they cannot overcome the harsh environment inside the BF. Therefore, to obtain a reliable BS shape under a high-temperature, high-pressure, high-dust, and low-light environment, a 3-D high-temperature industrial endoscope with a large depth of field and wide field of view is proposed in this article. First, image-forming optical system indices are obtained by analyzing the environmental factors and geometry of the BF. Based on these indices, an optical system with a large depth of field, wide field of view, and low-light imaging capacity is designed. Next, a 3-D depth calculation method based on a virtual and real twin binocular optical system is used to obtain 3-D depth of feature points on the burden distribution ring. Furthermore, a functional relationship model between the image defocusing degree and depth of the BS is established, based on the 3-D depth. Finally, a monocular vision 3-D reconstruction method based on the depth from defocus parameter is used to realize the 3-D reconstruction of the BS. Experimental industrial applications of the proposed device demonstrate that it can provide data support for the BF burden distribution operation and has a significant industrial application value.

Two common methods exist for solving indoor autonomous navigation and obstacle-avoidance problems using monocular vision: the traditional simultaneous localization and mapping (SLAM) method, which requires complex hardware, heavy calculations, and is prone to errors in low texture or dynamic environments; and deep-learning algorithms, which use the fully connected layer for classification or regression, resulting in more model parameters and easy over-fitting. Among the latter ones, the most advanced indoor navigation algorithm divides a single image frame into multiple parts for prediction, resulting in doubled reasoning time. To solve these problems, we propose a multi-task deep network based on feature map region division for monocular indoor autonomous navigation. We divide the feature map instead of the original image to avoid repeated information processing. To reduce model parameters, we use convolution instead of the fully connected layer to predict the navigable probability of the left, middle, and right parts. We propose that the linear velocity is determined by combining three prediction probabilities to reduce collision risk. Experimental evaluation shows that the proposed method is nine times smaller than the previous state-of-the-art methods; further, its processing speed and navigation capability increase more than five and 1.6 times, respectively.

In the lunar surface sampling mission, the drop operation that transferring the packaging container into the sample return container is a vital step. In this paper, for supporting the drop operation, we proposed a method to measure the coaxiality between the two containers based on the monocular vision pose measurement using ellipse and line features. The method first extracts ellipse and line features in the 2D image to calculate the initial parameters. Then, the initial parameters and the point set are used to fit the curve by the cylinder perspective contours fitting method, where the constraints between ellipse and line features are used to improve the robustness to varying noises. Next, given the fitting ellipse and camera parameters, the pose of the packaging container is solved by using circle-based method. Finally, the coaxiality of the two containers is calculated based on the relative pose between them. Simulation experiments were carried out and the results show the effectiveness and reliability of the proposed method.

Perceiving the distance between vehicles is a crucial issue for advanced driving assistance systems. However, most vision-based distance estimation methods do not consider the influence of the change in camera attitude angles during driving or only use the vanishing point detected by lane lines to correct the pitch angle. This paper proposed an improved pinhole distance estimation model based on the road vanishing point without the lane line information. First, the road vanishing point is detected based on the dominant texture orientation, and the yaw and pitch angles of the camera are estimated. Then, a distance estimation model considering attitude angle compensation is established. Finally, the experimental results show that the proposed method can effectively correct the influence of the camera attitude angle on the distance estimation results.

The design and implementation of a real-time object motion sensing and trajectory replicating system, dependent on the image-based visual servoing capabilities of an electrohydraulic eye-in-hand parallel manipulator has been described in this paper. The tracking system consists of a monocular vision apparatus affixed to the top platform of a six-cylinder parallel manipulator constructed with industry grade double-acting single rod hydraulic actuators and budget proportional valves. The control algorithm is an amalgamation of three principles: machine vision based object tracking, image based visual servoing and actuator-level feedback control utilizing the inverse kinematics of the parallel manipulator. The real-time experiments are performed on a laboratory scale parallel manipulator setup with the aid of a virtual tracking object rendered on a computer monitor. The results for a few reference trajectories reveal that accurate tracking of target motions can be achieved with visual servoing algorithms designed for parallel manipulators in real-time.

Advances in deep learning make monocular vision approaches attractive for the autonomous driving domain. This work investigates a method for estimating the speed of the ego-vehicle using state-of-the-art deep neural network based optical flow and single-view depth prediction models. Adopting a straightforward intuitive approach and approximating a single scale factor, several application schemes of the deep networks are evaluated and meaningful conclusions are formulated, such as: combining depth information with optical flow improves speed estimation accuracy as opposed to using optical flow alone; the quality of the deep neural network results influences speed estimation performance; using the depth and optical flow data from smaller crops of wide images degrades performance. With these observations in mind, a RMSE of less than 1 m/s for ego-speed estimation was achieved on the KITTI benchmark using monocular images as input. Limitations and possible future directions are discussed as well.

In order to solve the problem that the blurred image of a moving object decreases accuracy in the process of detecting the payload swing angle of an overhead crane based on vision, and the tracking failure caused by the drastic change of grey targets, a robust real-time detection method of the load swing angle of a bridge crane is proposed. This method uses a spherical marker attached to the load, which is insensitive to rotation and tilt when it is detected. First, it uses the mean shift algorithm combined with Kalman filter to track the moving objects in the image plane continuously, and then integrates the method of minimum area circle to detect the spherical marker image in the region of interest accurately and quickly. Finally, combined with the geometric method, the real-time swing angle is calculated. In addition, the angle diagram method is used to increase the speed of calculating the swing angle. The experimental results show that the method is effective for detecting the load target swing angle of different trolley motion speed.

This paper addresses the problem of stabilizing the machine at a pose specified by a reference image and four laser sensor readings. The system configuration combining monocular vision and laser pointers is applied for pose estimation of the end-effector as well as providing the control objective. Specific features are selected to reflect the pose of the end-effector in Cartesian space, and an adaptive controller is designed to implement the alignment between the end-effector and a stationary object for further assembly. The complete control scheme is assessed through simulations and experiments conducted on an automatic drilling and riveting system, the results demonstrate that the assembly task can be accomplished accurately and efficiently even with coarsely calibrated camera and laser pointers. Furthermore, decoupled results can be gained thanks to the selected features.

PurposeThe purpose of this paper is to present a visual detection approach to predict the poses of target objects placed in arbitrary positions before completing the corresponding tasks in mobile robotic manufacturing systems.Design/methodology/approachA hybrid visual detection approach that combines monocular vision and laser ranging is proposed based on an eye-in-hand vision system. The laser displacement sensor is adopted to achieve normal alignment for an arbitrary plane and obtain depth information. The monocular camera measures the two-dimensional image information. In addition, a robot hand-eye relationship calibration method is presented in this paper.FindingsFirst, a hybrid visual detection approach for mobile robotic manufacturing systems is proposed. This detection approach is based on an eye-in-hand vision system consisting of one monocular camera and three laser displacement sensors and it can achieve normal alignment for an arbitrary plane and spatial positioning of the workpiece. Second, based on this vision system, a robot hand-eye relationship calibration method is presented and it was successfully applied to a mobile robotic manufacturing system designed by the authors’ team. As a result, the relationship between the workpiece coordinate system and the end-effector coordinate system could be established accurately.Practical implicationsThis approach can quickly and accurately establish the relationship between the coordinate system of the workpiece and that of the end-effector. The normal alignment accuracy of the hand-eye vision system was less than 0.5° and the spatial positioning accuracy could reach 0.5 mm.Originality/valueThis approach can achieve normal alignment for arbitrary planes and spatial positioning of the workpiece and it can quickly establish the pose relationship between the workpiece and end-effector coordinate systems. Moreover, the proposed approach can significantly improve the work efficiency, flexibility and intelligence of mobile robotic manufacturing systems.