Search

Simultaneous Localization and Mapping (SLAM) is one of the most crucial aspects of autonomous navigation in mobile robots. The core components of the SLAM system can be depth perception and pose tracking. However, the existing unsupervised learning visual odometry framework cannot fully meet the actual requirements of the visual odometry information, particularly on the scale uncertainty in visual odometry estimation. It is still lacking in the geometric constraints during the autonomous operation of greenhouse mobile robots. In this study, an unsupervised optical flow-based visual odometry was presented. An optical flow estimation network was trained in an unsupervised manner using image warping. The optical flow between stereo images (disparity) was used to calculate the absolute depth of scenes. The optical flow between adjacent frames of left images was combined with the scene depth, in order to solve the frame-to-frame pose transformation matrix using Perspective-n-Point (PnP) algorithm. The reliable correspondences were selected in the solving process using forward and backward flow consistency checking to recover the absolute pose. A compact deep neural network was built with the convolutional modules to serve as the backbone of the flow model. This improved network was designed, according to the well-established principles: pyramidal processing, warping, and the use of a cost volume. At the same time, the cost volume normalization in the network was estimated with high values to alleviate the feature activations at higher levels than before. Furthermore, the local geometric consistency constraints were designed for the objective function of flow models. Meanwhile, a pyramid distilling loss was introduced to provide the supervision for the intermediate optical flows via distilling the finest final flow field as pseudo labels. A series of experiments were conducted using a wheeled mobile robot in a tomato greenhouse. The results showed that the better performance was achieved in the improved model. The local geometric consistency constraints improved the optical flow estimation accuracy. The endpoint error (EPE) of inter-frame and stereo optical flow was reduced by 8.89% and 8.96%, respectively. The pyramid distillation loss significantly reduced the optical flow estimation error of the flow model, in which the EPEs of the inter-frame and stereo optical flow decreased by 11.76% and 11.45%, respectively. The EPEs of the inter-frame and stereo optical flow were reduced by 12.50% and 7.25%, respectively, after cost volume normalization. Particularly, the price decreased by 1.28% for the calculation speed of the optical flow network. This improved model showed a 9.52% and 9.80% decrease in the root mean square error (RMSE) and mean absolute error (MAE) of relative translation error (RTE), respectively, compared with an existing unsupervised flow model. The decrease was 43.0% and 43.21%, respectively, compared with the Monodepth2. The pose tracking accuracy of this improved model was lower than that of ORB-SLAM3. The pure multi-view geometry shared the predicting dense depth maps of a scene. The relative error of depth estimation was 5.28% higher accuracy than the existing state-of-the-art self-supervised joint depth-pose learning. The accuracy of pose tracking depended mainly on the motion speed of robots. The performance of pose tracking at 0.2 m/s low speed and 0.8 m/s fast speed was significantly lower than that at 0.4-0.6 m/s. The resolution of the input image greatly impacted the pose tracking accuracy, with the errors decreasing gradually as the resolution increased. The MAE of RTE was not higher than 3.6 cm with the input image resolution of 832×512 pixels and the motion scope of 1 m, whereas, the MAE of relative rotation error (RRE) was not higher than 1.3º. These findings can provide technical support to design the vision system of greenhouse mobile robots. [ABSTRACT FROM AUTHOR]

Simultaneous localization and mapping (SLAM) play a vital role in implementing autonomous navigation of mobile robots in an unknown environment. Especially, visual odometry (VO) is a core component for a localization module in the SLAM system. The pose and velocity of a robot can, therefore, be estimated using computational geometry. Furthermore, the learning-based VO has gained great success in joint estimation camera ego-motion and depth from videos. In this study, a novel self-supervised VO model was proposed to realize the autonomous operation of a mobile robot in a greenhouse. The consistency constraint of temporal depth was also introduced for the learning framework using the binocular baseline supervision. Stereo video sequences were selected to train the model. The pose network after training was then used for pose estimation. A pre-test found that the stillness between video frames caused the prediction value of the model to shrink. Therefore, a soft mask was used in photometric re-projection error to remove the static region from the apparent difference measurement, and the non-rigidity scene and occlusion were further solved with normalized mask planes. Meanwhile, a new type of star dilated convolution (SDC) was also designed, where the filter was used to extract image features from the center 3×3 solid kernel and eight directions of 1-D kernel. The computational cost of SDC was thus less than that of the regular convolution of the same receptive field. Moreover, SDC was superimposed on spatial dimensions using depth-wise convolution with different dilation rates, particularly without the necessary to modify the existing deep learning framework. A convolutional auto-encoder (CAE) with residual network architecture was constructed using the SDC and inverse residual module (IRM), further serving as the backbone network for the VO model. With the aid of a binocular camera, the video sequences were collected in the solar greenhouses with tomato as the crop. The stereo video dataset was constructed to carry out the training and testing experiments. The static samples of the video sequence were removed from the image apparent difference measurement with a soft mask. The results showed that the mean relative errors (MREs) of translation and rotation estimation in the model were cut down by 5.06% and 11.05%, respectively, while the mean square root errors (RMSE) were reduced by 24.78% and 30.65%, respectively. Once a normalized mask plane was utilized in the model to deal with non-rigidity scenes and occlusion, the MREs of translation and rotation estimation were reduced by 4.15% and 3.86%, respectively. It inferred that both masks significantly improved the accuracy of the model. Meanwhile, the SDC-based IRM (SDC-IRM) reduced the MRE of rotation by 7.54% under the unchanged network parameters. Since the SDC-IRM structure presented significant effectiveness in reducing model error, the increase of perceptive field was an effective way to improve the accuracy of the model. The MRE and RMSE of rotation estimation were reduced by 2.74% and 36.48%, respectively, whereas, the mean cumulative rotation error per hundred frames (MCRE) decreased by 54.75%, when the consistency constraint of temporal depth was used in the model, indicating high accuracy and stability of pose estimation. The MRE of rotation estimation was reduced by 7.30% when extending the expansion factor of IRMs. The data demonstrated that the increase of receptive fields in the SDC kernel contributed to the higher accuracy of rotation estimation. Nevertheless, there was no longer obvious improvement of the model, when the maximum dilation rate was more than 6. More importantly, the calculation speed was up to 56.5 frames per second in the final pose estimation network. The MREs of translation and rotation estimation were 8.29% and 5.71%, respectively. The pose estimation performed better, compared with the previous VO model under similar input settings. This finding can provide sound support for the design of the navigation system for mobile robots in a greenhouse. [ABSTRACT FROM AUTHOR]

Copyright of Transactions of the Chinese Society of Agricultural Engineering is the property of Chinese Society of Agricultural Engineering and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Depth information acquisition is the key to mobile robots which realize autonomous operation in the greenhouse. This study proposed an unsupervised model that used binocular images for training and testing based on dense convolutional auto-encoder. This model enabled the neural network to perform plant image depth estimation and defined a loss function for the depth estimation with convolution feature comparison and regularization constraints. Aiming at the problem of pixel vanished due to the different perspective and occlusion, a disparity confidence prediction was introduced to suppress the problem gradient caused by the image reconstruction loss. In the meantime, a dense block was designed based on separable convolution and built a convolutional auto-encoder as the backbone network for the model. In the greenhouse of tomato planting, a large number of binocular images were collected when tomato planting was growing on an overcast, cloudy and sunny days. An unsupervised plant image depth estimation network was also designed with a Python application interface, which was implemented by adopting Microsoft Cognitive Tools (CNTK) v2.7, a deep learning computing framework. The experiments of training and testing which were used image feature similarity, depth estimation error, and threshold precision as the criteria were carried out, and the binocular images of tomato planting were also taken as examples, on Tesla K40c graphic device. The results showed that the auto-encoder based on the separable convolution dense block which was compared with the regular convolution could effectively reduce the number of network weight parameters. Compared with the other activations which included ReLU (Rectified Linear Unit), Param-ReLU, ELU (Exponential Linear Unit), and SELU (Scaled-ELU), the network model with Leaky-ReLU as the nonlinear transformation had the minimum depth error and the maximum threshold precision. Also, the results showed that the network structures had significant impacts on the accuracy of prediction disparity. The introduction of separable convolution dense block in the skip connection between the encoder and decoder of auto-encoder had a certain effect on improving the accuracy of depth estimation. Meanwhile, by making the depth estimation model predict the disparity confidence which was used to restrain the problem gradient backpropagation, the error of depth estimation was remarkably decreased, Mean Absolute Error (MAE), and Root Mean Square Error (RMSE) were reduced by 55.2% and 33.0% respectively. The accuracy of depth estimation was significantly improved by using these processing methods, such as image reconstruction, loss function calculation after up-sampling the disparity map to the input image scale and splicing the multi-scale disparity map predicted by the network to the feature map of its encoder, as well as sending the combination feature map to the next prediction module. The performance of depth estimation was improved by increasing the depth and width of the convolutional auto-encoder. The error of depth estimation decreased significantly with the reduction of the spatial point depth. When the spatial point depth was within 9 m, the MAE of the estimated depth was less than 14.1 cm. And when the depth was within 3 m, the MAE was less than 7 cm. The influence of illumination conditions on the accuracy of this study depth estimation model was not significant. The method in this study was robust to the change of the luminous environment. The highest test speed of this study model was 14.2 FPS (Frames Per Second), which was near real-time. Compared with the existing researches, the mean relative error, MAE, and Mean Range Error (MRE) of depth estimation in this study were reduced by 46.0%, 26.0%, and 25.5% respectively. This research could provide a reference for the design of the vision system of greenhouse mobile robots. [ABSTRACT FROM AUTHOR]

Depth estimation is critical to 3D reconstruction and object location in intelligent agricultural machinery vision system, and a common method in it is stereo matching. Traditional stereo matching method used low-quality image extracted manually. Because the color and texture in the image of field plant is nonuniform, the artificial features in the image are poorly distinguishable and mismatching could occur as a result. This would compromise the accuracy of the depth of the map. While the supervised learning-based convolution neural network (CNN) is able to estimate the depth of each pixel in plant image directly, it is expensive to annotate the depth data. In this paper, we present a depth estimation model based on the self-supervised learning to phenotype tomato canopy. The tasks of the depth estimation method were to reconstruct the image. The dense disparity maps were estimated indirectly using the rectified stereo pair of images as the network input, from which a bilinear interpolation was used to sample the input images to reconstruct the warping images. We developed three channel wise group convolutional (CWGC) modules, including the dimension invariable convolution module, the down-sampling convolution module and the up-sampling convolution module, and used them to construct the convolutional auto-encoder - a key infrastructure in the depth estimation method. Considering the shortage of manual features for comparing image similarity, we used the loss in image convolutional feature similarity as one objective of the network training. A CWGC-based CNN classification network (CWGCNet) was developed to extract the low-level features automatically. In addition to the loss in image convolutional feature similarity, we also considered the whole training loss, which include the image appearance matching loss, disparity smoothness loss and left-right disparity consistency loss. A stereo pair of images of tomato was sampled using a binocular camera in a greenhouse. After epipolar rectification, the pair of images was constructed for training and testing of the depth estimation model. Using the Microsoft Cognitive Toolkit (CNTK), the CWGCNet and the depth estimation network of the tomato images were calculated using Python. Both training and testing experiments were conducted in a computer with a Tesla K40c GPU (graphics processing unit). The results showed that the shallow convolutional layer of the CWGCNet successfully extracted the low-level multiformity image features to calculate the loss in image convolutional feature similarity. The convolutional auto-encoder developed in this paper was able to significantly improve the disparity map estimated by the depth estimation model. The loss function in image convolutional feature similarity had a remarkable effect on accuracy of the image depth. The accuracy of the disparity map estimated by the model increased with the number of convolution modules for calculating the loss in convolutional feature similarity. When sampled within 9.0 m, the root means square error (RMSE) and the mean absolute error (MAE) of the corner distance estimated by the model were less than 2.5 cm and 1.8 cm, respectively, while when sampled within 3.0m, the associated errors were less than 0.7cm and 0.5cm, respectively. The coefficient of determination (R²) of the proposed model was 0.8081, and the test speed was 28 fps (frames per second). Compared with the existing models, the proposed model reduced the RMSE and MAE by 33.1% and 35.6% respectively, while increased calculation speed by 52.2%. [ABSTRACT FROM AUTHOR]