Search

In the realm of computer vision, the integration of advanced techniques into the pre-processing of RGB-D camera inputs poses a significant challenge, given the inherent complexities arising from diverse environmental conditions and varying object appearances. Therefore, this paper introduces FusionVision, an exhaustive pipeline adapted for the robust 3D segmentation of objects in RGB-D imagery. Traditional computer vision systems face limitations in simultaneously capturing precise object boundaries and achieving high-precision object detection on depth maps, as they are mainly proposed for RGB cameras. To address this challenge, FusionVision adopts an integrated approach by merging state-of-the-art object detection techniques, with advanced instance segmentation methods. The integration of these components enables a holistic (unified analysis of information obtained from both color RGB and depth D channels) interpretation of RGB-D data, facilitating the extraction of comprehensive and accurate object information in order to improve post-processes such as object 6D pose estimation, Simultanious Localization and Mapping (SLAM) operations, accurate 3D dataset extraction, etc. The proposed FusionVision pipeline employs YOLO for identifying objects within the RGB image domain. Subsequently, FastSAM, an innovative semantic segmentation model, is applied to delineate object boundaries, yielding refined segmentation masks. The synergy between these components and their integration into 3D scene understanding ensures a cohesive fusion of object detection and segmentation, enhancing overall precision in 3D object segmentation. [ABSTRACT FROM AUTHOR]

Detecting parcels accurately and efficiently has always been a challenging task when unloading from trucks onto conveyor belts because of the diverse and complex ways in which parcels are stacked. Conventional methods struggle to quickly and accurately classify the various shapes and surface patterns of unordered parcels. In this paper, we propose a parcel-picking surface detection method based on deep learning and image processing for the efficient unloading of diverse and unordered parcels. Our goal is to develop a systematic image processing algorithm that emphasises the boundaries of parcels regardless of their shape, pattern, or layout. The core of the algorithm is the utilisation of RGB-D technology for detecting the primary boundary lines regardless of obstacles such as adhesive labels, tapes, or parcel surface patterns. For cases where detecting the boundary lines is difficult owing to narrow gaps between parcels, we propose using deep learning-based boundary line detection through the You Only Look at Coefficients (YOLACT) model. Using image segmentation techniques, the algorithm efficiently predicts boundary lines, enabling the accurate detection of irregularly sized parcels with complex surface patterns. Furthermore, even for rotated parcels, we can extract their edges through complex mathematical operations using the depth values of the specified position, enabling the detection of the wider surfaces of the rotated parcels. Finally, we validate the accuracy and real-time performance of our proposed method through various case studies, achieving mAP (50) values of 93.8% and 90.8% for randomly sized and rotationally covered boxes with diverse colours and patterns, respectively. [ABSTRACT FROM AUTHOR]

Salient Object Detection (SOD) in RGB-D images plays a crucial role in the field of computer vision, with its central aim being to identify and segment the most visually striking objects within a scene. However, optimizing the fusion of multi-modal and multi-scale features to enhance detection performance remains a challenge. To address this issue, we propose a network model based on semantic localization and multi-scale fusion (SLMSF-Net), specifically designed for RGB-D SOD. Firstly, we designed a Deep Attention Module (DAM), which extracts valuable depth feature information from both channel and spatial perspectives and efficiently merges it with RGB features. Subsequently, a Semantic Localization Module (SLM) is introduced to enhance the top-level modality fusion features, enabling the precise localization of salient objects. Finally, a Multi-Scale Fusion Module (MSF) is employed to perform inverse decoding on the modality fusion features, thus restoring the detailed information of the objects and generating high-precision saliency maps. Our approach has been validated across six RGB-D salient object detection datasets. The experimental results indicate an improvement of 0.20~1.80%, 0.09~1.46%, 0.19~1.05%, and 0.0002~0.0062, respectively in maxF, maxE, S, and MAE metrics, compared to the best competing methods (AFNet, DCMF, and C2DFNet). [ABSTRACT FROM AUTHOR]

According to the problem of the current RGB-D feature extraction network used for pose estimation was too large, this paper proposed a lightweight improved XYZNet RGB-D feature extraction network. First, this paper designed a lightweight sub-network BaseNet to replace ResNet18 in XYZNet, which made the network scale significantly reduced and obtained more powerful performance. Then, this paper proposed a re-parameterized multiscale convolutional attention (Rep-MSCA) sub-module based on depth separable convolution, which enhanced the ability of BaseNet to extract contextual information of different scales, and constrained the amount of parameters in the model. Finally, in order to improve the geometric feature extraction ability of PointNet in XYZNet with a small parameter cost, this paper designed a re-parameterized residual multi-layer perceptron (Rep-ResP) module. The floating point operations (FLOPs) and parameters of the improved network are 60.8% and 64.8% lower, the inference speed is 21.2% higher, and the accuracy of the mainstream datasets LineMOD and YCB-Video is 0.5% and 0.6% higher. The proposed model is more suitable for deployment in scenarios where hardware resources are tight. [ABSTRACT FROM AUTHOR]

Intelligent pruning is an effective way to improve the efficiency of fruit tree pruning and reduce production costs, where the positioning of fruit tree pruning points is the key. In this study, a pruning point localisation method based on deep learning and red-green-blue-depth (RGB-D) is proposed for dormant tall spindle apple trees, which can quickly and accurately identify the pruning points on the primary branches. Firstly, red-green-blue (RGB) images and depth images of apple trees were acquired by using the Realsense D435i depth camera, and the SOLOv2 instance segmentation model was used to segment the trunks, branches, and supports in RGB images. Secondly, the manual pruning rules were adapted to improve the pruning methods; then using OpenCV image processing method, the junction points of branches and trunk and potential pruning points were gained, and the world coordinates of the points were obtained according to the coordinate transformation to calculate the length of branch diameter and spacing. Finally, the pruning points were determined according to the pruning rules. The results show that SOLOv2 has better segmentation effect compared with Mask R–CNN and Cascade Mask R–CNN. The mean absolute error between the estimated and manually measured values of branch diameter and spacing are 1.10 mm and 16.06 mm, and the recognition accuracy of pruning points is 87.2%, with a recognition time of about 3.8 s for each image. It is shown that the method can measure branch diameter and spacing and quickly locate pruning points with high reliability and accuracy, and the study provides a basis for the development of apple tree pruning robots. • Determining apple tree pruning points by RGB-D and deep learning. • Deep learning was used to segment branches and trunks. • RGB-D sensor and image processing were used to determine branch diameter and spacing. • Improved pruning methods were used to determine pruning points quickly and accurately. [ABSTRACT FROM AUTHOR]

Efficient and accurate identification and position of compound buds can greatly contribute to the orchard robots for automatic bud removal operations, thereby improving the efficiency of flower and fruit thinning. This study aims to accurately identify and position the small objects in complex backgrounds for grape bud removal in the field. (1) A visual recognition system was proposed for the concurrent bud and secondary bud using a combination of far and near visual. Euclidean distance was applied on the visual mapping plane. The concurrent bud was then identified in the far view, whereas, the secondary bud was identified in the near view within the concurrent bud mapping area, in order to improve the recognition accuracy of small target objects in the field. The original images (2 132 images) were randomly divided into the training set (1 735 images) and validation set (397 images) in the ratio of 8:2. The training set and validation set were uniformly cropped to 2944×1656 pixel size, and then scaled to 1280×720 pixel size with 2.3 times of equal proportion. Data augmentation was carried out by three random combinations of augmentation through scaling, flipping, and color gamut transformation for the concurrent and secondary bud datasets. Once the augmentation scale was too large for the target features, the images were manually excluded from the dataset. As such, the training set was enlarged to 15 840 images. YOLOv5m and YOLOv5s were selected as the network models for juxtaposed bud and parabudding detection. The sizes of the trained models were 42.1 and 14.4 MB, respectively, with AP of 0.702 and 0.773, F1 scores of 0.685 and 0.765 on the test set of concurrent and secondary bud images, respectively, where the average inference time per image was 10.62 and 7.01 ms. 60 compound bud images showed that the overall average accuracy of this improved model was 0.905, and the average detection time was 18.1 ms for each process; (2) An online detection and position were then proposed for the compound bud in the field. The online detection and position were introduced in the prediction stage to improve the light-resistant ability of the depth camera when positioning in a complex environment, in order to reduce the fluctuation and error of the depth information. The same compound bud was online detected and positioned for 5 consecutive frames. The initial frame and the latest frame image data were deleted and updated in real time, until the relative error of positioning coordinates in 5 consecutive frames was less than the error threshold, in order to obtain the average positioning coordinates of the compound bud. Better performance was achieved in the positioning accuracy of compound bud in natural background and the resistance to light coupling in complex light environment. The test results showed that the positioning accuracy of the improved model was ±0.916 and ±0.654 mm for the concurrent and secondary bud, respectively. The repeatability and accuracy can fully meet the demand for the secondary bud wiping, indicating a reliable and effective system. The finding can provide high-quality technical support for the online positioning of the concurrent and secondary buds for sprout wiping operation in sprouting robots, in order to realize the automatic sprout wiping in orchards. [ABSTRACT FROM AUTHOR]

Scene classification in autonomous navigation is a highly complex task due to variations, such as light conditions and dynamic objects, in the inspected scenes; it is also a challenge for small-factor computers to run modern and highly demanding algorithms. In this contribution, we introduce a novel method for classifying scenes in simultaneous localization and mapping (SLAM) using the boundary object function (BOF) descriptor on RGB-D points. Our method aims to reduce complexity with almost no performance cost. All the BOF-based descriptors from each object in a scene are combined to define the scene class. Instead of traditional image classification methods such as ORB or SIFT, we use the BOF descriptor to classify scenes. Through an RGB-D camera, we capture points and adjust them onto layers than are perpendicular to the camera plane. From each plane, we extract the boundaries of objects such as furniture, ceilings, walls, or doors. The extracted features compose a bag of visual words classified by a support vector machine. The proposed method achieves almost the same accuracy in scene classification as a SIFT-based algorithm and is 2.38× faster. The experimental results demonstrate the effectiveness of the proposed method in terms of accuracy and robustness for the 7-Scenes and SUNRGBD datasets. [ABSTRACT FROM AUTHOR]

RGB-D sensors can be a low-cost solution for an accurate silo's content monitoring which is fundamental for its efficient management. Some reference information such as the position and orientation of the sensor with respect to the silo's geometry is fundamental for obtaining correct content measurements from acquired data. Since in real cases this information is not always known, a new method to obtain these measurements is proposed. This, taking as input sensor acquired data (represented as a point cloud), automatically computes the silo's axis to provide a new reference system from which the point cloud can be easily processed. The z-axis of this reference system coincides with the gravity axis and the xy-plane is parallel to the ground plane. It is obtained in a six-step process that exploits the silo geometry properties and an estimation of the shape tensors of the acquired points. The method has been implemented and tested on both synthetic and real silos, considering a complete silo's discharge process and different camera positions. Data acquired at each discharge has been transformed using the new reference system and compared with the silo's ground truth (manually obtained for the real silos). To evaluate the accuracy the input point cloud to adjusted point cloud average distance has been considered. In all the tests, the well-performance of the proposal has been demonstrated, achieving a maximum average distance error of less than 6 cm. [ABSTRACT FROM AUTHOR]

Object pose estimation has multiple important applications, such as robotic grasping and augmented reality. We present a new method to estimate the 6D pose of objects that improves upon the accuracy of current proposals and can still be used in real-time. Our method uses RGB-D data as input to segment objects and estimate their pose. It uses a neural network with multiple heads to identify the objects in the scene, generate the appropriate masks and estimate the values of the translation vectors and the quaternion that represents the objects' rotation. These heads leverage a pyramid architecture used during feature extraction and feature fusion. We conduct an empirical evaluation using the two most common datasets in the area, and compare against state-of-the-art approaches, illustrating the capabilities of MPF6D. Our method can be used in real-time with its low inference time and high accuracy. [ABSTRACT FROM AUTHOR]

• Bilateral cross-modal interaction is achieved and hybrid contexts are exploited to improve segmentation accuracy. • A proposed FEM is embedded to highlight unimodally useful features. • An HPDC is designed to effectively collect diverse contextual information for improving the capability of BCINet. • A CGM is designed to transmit contextual information. Depth cue has proven to be useful information in the indoor scene understanding of RGB-D images for providing a geometric counterpart to RGB representation. However, because of the differences between RGB-D image pairs, utilizing cross-modal data effectively is a key issue. Most methods exclusively leverage depth data to unilaterally complement RGB data for better feature representation; they invariably ignore the fact that RGB and depth data can bilaterally complement each other. Herein, a novel RGB-D scene-understanding network called BCINet is presented, in which RGB and depth data bilaterally complement each other via a proposed bilateral cross-modal interaction module (BCIM). The BCIM helps to capture cross-modal complementary cues by crossly fusing enhanced features from one modality to the counterpart modality through a feature enhanced module. Meanwhile, exploiting the long-range dependencies of RGB-D features is also significant for accurate scene understanding. Specifically, we design a hybrid pyramid dilated convolution module to enlarge the receptive fields along both the vertical and horizontal spatial directions to adaptively capture diverse contexts with different shapes. Additionally, we propose a context-guided module to aggregate these diverse higher-level contexts with lower-level features in the encoder to guide the information flow for progressively refining the segmentation map. Experimental results on two indoor scene datasets demonstrate the superiority and effectiveness of the proposed BCINet over several state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Three colour and depth (RGB-D) devices were compared, to assess the effect of depth image misalignment, resulting from simultaneous localisation and mapping (SLAM) error, due to forest structure complexity. Urban parkland (S1) was used to assess stem density, and understory vegetation (≤1.3 m) was assessed in native woodland (S2). Individual stem and continuous capture approaches were used, with stem diameter at breast height (DBH) estimated. Misalignment was present within point clouds; however, no significant differences in DBH were observed for stems captured at S1 with either approach (Kinect p = 0.16; iPad p = 0.27; Zed p = 0.79). Using continuous capture, the iPad was the only RGB-D device to maintain SLAM in all S2 plots. There was significant correlation between DBH error and surrounding understory vegetation with the Kinect device (p = 0.04). Conversely, there was no significant relationship between DBH error and understory vegetation for the iPad (p = 0.55) and Zed (p = 0.86). The iPad had the lowest DBH root-mean-square error (RMSE) across both individual stem (RMSE = 2.16 c m) and continuous (RMSE = 3.23 c m) capture approaches. The results suggest that the assessed RGB-D devices are more capable of operation within complex forest environments than previous generations. [ABSTRACT FROM AUTHOR]

Smart farming (SF) applications rely on robust and accurate computer vision systems. An important computer vision task in agriculture is semantic segmentation, which aims to classify each pixel of an image and can be used for selective weed removal. State-of-the-art implementations use convolutional neural networks (CNN) that are trained on large image datasets. In agriculture, publicly available RGB image datasets are scarce and often lack detailed ground-truth information. In contrast to agriculture, other research areas feature RGB-D datasets that combine color (RGB) with additional distance (D) information. Such results show that including distance as an additional modality can improve model performance further. Therefore, we introduce WE3DS as the first RGB-D image dataset for multi-class plant species semantic segmentation in crop farming. It contains 2568 RGB-D images (color image and distance map) and corresponding hand-annotated ground-truth masks. Images were taken under natural light conditions using an RGB-D sensor consisting of two RGB cameras in a stereo setup. Further, we provide a benchmark for RGB-D semantic segmentation on the WE3DS dataset and compare it with a solely RGB-based model. Our trained models achieve up to 70.7% mean Intersection over Union (mIoU) for discriminating between soil, seven crop species, and ten weed species. Finally, our work confirms the finding that additional distance information improves segmentation quality. [ABSTRACT FROM AUTHOR]

Stairs are common vertical traffic structures in buildings, and stair detection tasks are important in environmental perception for autonomous mobile robots. Most existing algorithms have difficulty combining the visual information from binocular sensors effectively and ensuring reliable detection at night and in the case of extremely fuzzy visual clues. To solve these problems, we propose a stair detection network with red-green-blue (RGB) and depth inputs. Specifically, we design a selective module, which can make the network learn the complementary relationship between the RGB feature maps and the depth feature maps and fuse the features effectively in different scenes. In addition, we propose several postprocessing algorithms, including a stair line clustering algorithm and a coordinate transformation algorithm, to obtain the stair geometric parameters. Experiments show that our method has better performance than existing the state-of-the-art deep learning method, and the accuracy, recall, and runtime are improved by 5.64%, 7.97%, and 3.81 ms, respectively. The improved indexes show the effectiveness of the multimodal inputs and the selective module. The estimation values of stair geometric parameters have root mean square errors within 15 mm when ascending stairs and 25 mm when descending stairs. Our method also has extremely fast detection speed, which can meet the requirements of most real-time applications. [ABSTRACT FROM AUTHOR]

Motion analysis is an area with several applications for health, sports, and entertainment. The high cost of state-of-the-art equipment in the health field makes it unfeasible to apply this technique in the clinics' routines. In this vein, RGB-D and RGB equipment, which have joint tracking tools, are tested with portable and low-cost solutions to enable computational motion analysis. The recent release of Google MediaPipe, a joint inference tracking technique that uses conventional RGB cameras, can be considered a milestone due to its ability to estimate depth coordinates in planar images. In light of this, this work aims to evaluate the measurement of angular variation from RGB-D and RGB sensor data against the Qualisys Tracking Manager gold standard. A total of 60 recordings were performed for each upper and lower limb movement in two different position configurations concerning the sensors. Google's MediaPipe usage obtained close results compared to Kinect V2 sensor in the inherent aspects of absolute error, RMS, and correlation to the gold standard, presenting lower dispersion values and error metrics, which is more positive. In the comparison with equipment commonly used in physical evaluations, MediaPipe had an error within the error range of short- and long-arm goniometers. [ABSTRACT FROM AUTHOR]

Simple Summary: Feather coverage reflects the production efficiency and animal welfare of poultry. Monitoring the feather-cover condition of chickens is of great significance. Infrared thermography can be used to evaluate the probable existence of inflammatory or tissue damage processes due to the variation in skin temperature, which can be used to objectively determine the depth of the feather damage. In this study, a 3D reconstruction pipeline of chicken monitoring was developed, with color, depth and thermal information for the comprehensive feather damage monitoring of chickens. The results demonstrated that the proposed method can better assess the feather damage compared to a 2D color image or thermal infrared image. The depth of chicken feather damage can be assessed by the 3D model. The method provided ideas for automation and intelligent feather-damage monitoring in poultry farming. Feather damage is a continuous health and welfare challenge among laying hens. Infrared thermography is a tool that can evaluate the changes in the surface temperature, derived from an inflammatory process that would make it possible to objectively determine the depth of the damage to the dermis. Therefore, the objective of this article was to develop an approach to feather damage assessment based on visible light and infrared thermography. Fusing information obtained from these two bands can highlight their strengths, which is more evident in the assessment of feather damage. A novel pipeline was proposed to reconstruct the RGB-Depth-Thermal maps of the chicken using binocular color cameras and a thermal infrared camera. The process of stereo matching based on binocular color images allowed for a depth image to be obtained. Then, a heterogeneous image registration method was presented to achieve image alignment between thermal infrared and color images so that the thermal infrared image was also aligned with the depth image. The chicken image was segmented from the background using a deep learning-based network based on the color and depth images. Four kinds of images, namely, color, depth, thermal and mask, were utilized as inputs to reconstruct the 3D model of a chicken with RGB-Depth-Thermal maps. The depth of feather damage can be better assessed with the proposed model compared to the 2D thermal infrared image or color image during both day and night, which provided a reference for further research in poultry farming. [ABSTRACT FROM AUTHOR]

Pod phenotypic traits are closely related to grain yield and quality. Pod phenotype detection in soybean populations in natural environments is important to soybean breeding, cultivation, and field management. For an accurate pod phenotype description, a dynamic detection method is proposed based on an improved YOLO-v5 network. First, two varieties were taken as research objects. A self-developed field soybean three-dimensional color image acquisition vehicle was used to obtain RGB and depth images of soybean pods in the field. Second, the red–green–blue (RGB) and depth images were registered using an edge feature point alignment metric to accurately distinguish complex environmental backgrounds and establish a red–green–blue-depth (RGB-D) dataset for model training. Third, an improved feature pyramid network and path aggregation network (FPN+PAN) structure and a channel attention atrous spatial pyramid pooling (CA-ASPP) module were introduced to improve the dim and small pod target detection. Finally, a soybean pod quantity compensation model was established by analyzing the influence of the number of individual plants in the soybean population on the detection precision to statistically correct the predicted pod quantity. In the experimental phase, we analyzed the impact of different datasets on the model and the performance of different models on the same dataset under the same test conditions. The test results showed that compared with network models trained on the RGB dataset, the recall and precision of models trained on the RGB-D dataset increased by approximately 32% and 25%, respectively. Compared with YOLO-v5s, the precision of the improved YOLO-v5 increased by approximately 6%, reaching 88.14% precision for pod quantity detection with 200 plants in the soybean population. After model compensation, the mean relative errors between the predicted and actual pod quantities were 2% to 3% for the two soybean varieties. Thus, the proposed method can provide rapid and massive detection for pod phenotyping in soybean populations and a theoretical basis and technical knowledge for soybean breeding, scientific cultivation, and field management. [ABSTRACT FROM AUTHOR]

With the development of 3D vision techniques, RGB-D cameras are increasingly used to allow easier and cheaper access to the third dimension. In this paper, we focus on testing the potential of the Kinect Azure RGB-D camera in the 3D reconstruction of indoor scenes. First, a series of investigations of the hardware was performed to evaluate its accuracy and precision. The results show that the measurements made with the Azure could be exploited for close-range survey applications. Second, we performed a methodological workflow for indoor reconstruction based on the Open3D framework, which was applied to two different indoor scenes. Based on the results, we can state that the quality of 3D reconstruction significantly depends on the architecture of the captured scene. This was supported by a comparison of the point cloud from the Kinect Azure with that from a terrestrial laser scanner and another from a mobile laser scanner. The results show that the average differences do not exceed 8 mm, which confirms that the Kinect Azure can be considered a 3D measurement system at least as reliable as a mobile laser scanner. [ABSTRACT FROM AUTHOR]

To simultaneously estimate the number of heads and locate heads with bounding boxes, we resort to detection-based crowd counting by leveraging RGB-D data and design a dual-path guided detection network (DPDNet). Specifically, to improve the performance of detection-based approaches for dense/tiny heads, we propose a density map guided detection module, which leverages density map to improve the head/non-head classification in detection network where the density implies the probability of a pixel being a head, and a depth-adaptive kernel that considers the variances in head sizes is also introduced to generate high-fidelity density map for more robust density map regression. In order to prevent dense heads from being filtered out during post-processing, we utilize such a density map for post-processing of head detection and propose a density map guided NMS strategy. Meanwhile, to improve the ability of detecting small heads, we also propose a depth-guided detection module to generate a dynamic dilated convolution to extract features of heads of different scales, and a depth-aware anchor is further designed for better initialization of anchor sizes in the detection framework. Then we use the bounding boxes whose sizes are generated with depth to train our DPDNet. Considering that existing RGB-D datasets are too small and not suitable for performance evaluation of data-driven based approaches, we collect two large-scale RGB-D crowd counting datasets, which comprise a synthetic dataset and a real-world dataset, respectively. Since the depth value at long-distance positions cannot be obtained in the real-world dataset, we further propose a depth completion method with meta learning, which fully utilizes the synthetic depth data to complete the depth value at long-distance positions. Extensive experiments on our proposed two RGB-D datasets and the MICC RGB-D counting dataset show that our method achieves the best performance for RGB-D crowd counting and localization. Further, our method can be easily extended to RGB image based crowd counting and achieves comparable or even better performance on the RGB datasets for both head counting and localization. [ABSTRACT FROM AUTHOR]

In order to improve the estimation accuracy of the road slope during the movement of the robot, this paper proposed a fusion slope estimation algorithm for RGB-D（red green blue-depth） moving robot. Firstly, the method used random sampling consistency algorithm to complete the point cloud processing. Secondly, the normal vector estimation followed an improve plane fitting method. Finally, the cosine clustering and cumulative average method were used to accurately complete the road slope estimation. Experimental results showed that compared with the least squares method and the sparse subspace method under the data set, the estimation error of the algorithm is reduced by 1.21% and 2.13% respectively, in the actual environment, the average error is reduced by 1.43° compared with the least squares method, which verifies the feasibility and effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Cerebral palsy, the most common childhood neuromotor disorder, is often diagnosed through visual assessment of general movements (GM) in infancy. This skill requires extensive training and is thus difficult to implement on a large scale. Automated analysis of GM performed using low-cost instrumentation in the home may be used to estimate quantitative metrics predictive of movement disorders. This study explored if infants' GM may be successfully evaluated in a familiar environment by processing the 3D trajectories of points of interest (PoI) obtained from recordings of a single commercial RGB-D sensor. The RGB videos were processed using an open-source markerless motion tracking method which allowed the estimation of the 2D trajectories of the selected PoI and a purposely developed method which allowed the reconstruction of their 3D trajectories making use of the data recorded with the depth sensor. Eight infants' GM were recorded in the home at 3, 4, and 5 months of age. Eight GM metrics proposed in the literature in addition to a novel metric were estimated from the PoI trajectories at each timepoint. A pediatric neurologist and physiatrist provided an overall clinical evaluation from infants' video. Subsequently, a comparison between metrics and clinical evaluation was performed. The results demonstrated that GM metrics may be meaningfully estimated and potentially used for early identification of movement disorders. [ABSTRACT FROM AUTHOR]

This paper describes the methods submitted for evaluation to the SHREC 2022 track on pothole and crack detection in the road pavement. A total of 7 different runs for the semantic segmentation of the road surface are compared, 6 from the participants plus a baseline method. All methods exploit Deep Learning techniques and their performance is tested using the same environment (i.e., a single Jupyter notebook). A training set, composed of 3836 semantic segmentation image/mask pairs and 797 RGB-D video clips collected with the latest depth cameras was made available to the participants. The methods are then evaluated on the 496 image/masks pairs in the validation set, on the 504 pairs in the test set and finally on 8 video clips. The analysis of the results is based on quantitative metrics for image segmentation and qualitative analysis of the video clips. The participation and the results show that the scenario is of great interest and that the use of RGB-D data is still challenging in this context. [Display omitted] • A dataset of image/mask pairs and RGB-D video clips of road damage. • A fair evaluation of the proposed methods implementing them on a single Jupyter notebook. • A qualitative evaluation of videos of road damages from different points of view. [ABSTRACT FROM AUTHOR]

This paper proposed a new network framework for RGB-D salient object detection (SR-Net). In order to effectively integrate the complementarity of multi-model features, this paper took the depth feature extraction as an independent branch, use the Convolutional Block Attention Module (CBAM) to enhance the depth feature, and integrate the complementary information of the enhanced depth feature and RGB feature. Then, in order to remove feature redundancy and reduce the interference of background noise on the prediction results, it proposed a sequential refining network in the up-sampling network, that is, first, the primary global features are obtained by integrating the complementarity of multi-level and multiscale features, and used the Primary Global Feature Weight Matrix Acquisition Module (PFW) which based on the channel weight to obtains the weight matrix of the primary global feature, and then uses the obtained weight matrix to refine the features of each level to suppress the interference which caused by background noise. Finally, in order to better optimize the whole network, it proposed a new loss function. The experimental results on four public datasets show that the model is superior to nine advanced methods in different model evaluation indexes, and achieves more advanced performance. [ABSTRACT FROM AUTHOR]

Obstacle detection is an essential task for the autonomous navigation by robots. The task becomes more complex in a dynamic and cluttered environment. In this context, the RGB-D camera sensor is one of the most common devices that provides a quick and reasonable estimation of the environment in the form of RGB and depth images. This work proposes an efficient obstacle detection and tracking method using depth images to facilitate quick dynamic obstacle detection. To achieve early detection of dynamic obstacles and stable estimation of their states, as in previous methods, we applied a u-depth map for obstacle detection. Unlike existing methods, the present method provides dynamic thresholding facilities on the u-depth map to detect obstacles more accurately. Here, we propose a restricted v-depth map technique, using post-processing after the u-depth map processing to obtain a better prediction of the obstacle dimension. We also propose a new algorithm to track obstacles until they are within the field of view (FOV). We evaluate the performance of the proposed system on different kinds of data sets. The proposed method outperformed the vision-based state-of-the-art (SoA) methods in terms of state estimation of dynamic obstacles and execution time. [ABSTRACT FROM AUTHOR]

RGB-D salient object detection (SOD) demonstrates its superiority in detecting in complex environments due to the additional depth information introduced in the data. Inevitably, an independent stream is introduced to extract features from depth images, leading to extra computation and parameters. This methodology sacrifices the model size to improve the detection accuracy which may impede the practical application of SOD problems. To tackle this dilemma, we propose a dynamic knowledge distillation (DKD) method, along with a lightweight structure, which significantly reduces the computational burden while maintaining validity. This method considers the factors of both teacher and student performance within the training stage and dynamically assigns the distillation weight instead of applying a fixed weight on the student model. We also investigate the issue of RGB-D early fusion strategy in distillation and propose a simple noise elimination method to mitigate the impact of distorted training data caused by low quality depth maps. Extensive experiments are conducted on five public datasets to demonstrate that our method can achieve competitive performance with a fast inference speed (136FPS) compared to 12 prior methods. [ABSTRACT FROM AUTHOR]

Tomato sucker or axillary shoots should be removed to increase the yield and reduce the disease on tomato plants. It is an essential step in the tomato plant care process. It is usually performed manually by farmers. An automated approach can save a lot of time and labor. In the literature review, we see that semantic segmentation is a process of recognizing or classifying each pixel in an image, and it can help machines recognize and localize tomato suckers. This paper proposes a semantic segmentation neural network that can detect tomato suckers quickly by the tomato plant images. We choose RGB-D images which capture not only the visual of objects but also the distance information from objects to the camera. We make a tomato RGB-D image dataset for training and evaluating the proposed neural network. The proposed semantic segmentation neural network can run in real-time at 138.2 frames per second. Its number of parameters is 680, 760, much smaller than other semantic segmentation neural networks. It can correctly detect suckers at 80.2%. It requires low system resources and is suitable for the tomato dataset. We compare it to other popular non-real-time and real-time networks on the accuracy, time of execution, and sucker detection to prove its better performance. [ABSTRACT FROM AUTHOR]

The navel orange planting area in Gannan region of China has reached the first in the world, with an annual output of one million tons. However, most navel oranges were picked manually with high labor intensity and low efficiency. It was very urgent to introduce intelligent picking robots. In the process of navel orange picking, the picking robot needed to obtain the spatial position and fruit size information of navel orange fruit in real time to achieve efficient and rapid intelligent grading picking. This study proposed an algorithm framework--OrangePointSge based on the improved YOLACT model and the least-square sphere fitting method. The real-time instance segmentation algorithm YOLACT generated an instance mask and cropped the registered depth image to obtain the fruit depth point cloud. Then, the least-square method was used to fit the navel orange fruit shape, and the centroid coordinates and radius under the camera coordinate system were obtained. Firstly, the RGB-D data of navel orange fruit was collected by Microsoft's latest consumer-level depth camera (Azure Kinect DK), and the navel orange fruit instance segmentation dataset and enhancement dataset were established and free to open source. The dataset contained 2 178 images, including 8 712 samples. Among those samples, the number of non-occluded fruit was 4 682, and the number of slightly occluded fruit was 4 030. Then the YOLACT algorithm was improved to adapt to the detection of navel orange fruit, and the instance mask of the fruit was output. We modified the original ResNet (deep residual network) backbone network to HRNet (High-Resolution Net) to simplify the model, and used the hrnet_w48 structure for feature extraction to improve the model detection accuracy. We also optimized the non-maximum suppression process to improve the detection speed. At the same time, several groups of comparative tests were also set up, including the performance of YOLACT with different backbone networks and the performance of different algorithms in different scenarios. To improve the generalization ability of the training model, we carried out a series of enhancement algorithms for color images, mainly including scale change, color change, brightness change, and adding noise. For long-range small targets, we used oversampling method for enhancement training to improve their detection accuracy. Therefore, the recognition speed and mask accuracy of navel orange were higher than other algorithms. The detection speed was 44.63 frames/s and the average accuracy was 31.15%. The mask generated by improved YOLACT will be cropped with the depth image registered with the color image to obtain the depth information of each fruit to generate the fruit depth point cloud from the depth image. The least-square method was used to fit the point cloud to obtain the centroid coordinates and radius of the fruit, which were used to guide the robot to carry out grading picking. In addition, we added the comparison with RANSAC (Random Sample Consensus) algorithm. The results show that the least-square method is faster than RANSAC while ensuring the appropriate accuracy. When the number of point clouds was 1 400-2 000, the fitting time was 1.99 ms, the positioning error was 0.49 cm, the fitting radius root mean square error was 0.43 cm, and the volume root mean square error was 52.6 mL. Through the experiment of setting different distances on the fitting accuracy, it was found that when the number of point clouds was more than 800 and the distance was less than 1.0 m, the positioning error was controlled within 0.46 cm. Finally, by introducing parallel computing and combining the above two processing flows, the overall processing speed of OrangePointSge was 29.4 frames/s, which can better balance the accuracy and speed. Thus, the proposed algorithm is conducive to practical application and engineering deployment. [ABSTRACT FROM AUTHOR]

The detection of moving objects is one of the key problems in the field of computer vision. It is very important to detect moving objects accurately and rapidly for automatic driving. In this paper, we propose an improved moving object detection method to overcome the disadvantages of the RGB information-only-based method in detecting moving objects that are susceptible to shadow interference and illumination changes by adding depth information. Firstly, a convolutional neural network (CNN) based on the color edge-guided super-resolution reconstruction of depth maps is proposed to perform super-resolution reconstruction of low-resolution depth images obtained by depth cameras. Secondly, the RGB-D moving object detection algorithm is based on fusing the depth information of the same scene with RGB features for detection. Finally, in order to evaluate the effectiveness of the algorithm proposed in this paper, the Middlebury 2005 dataset and the SBM-RGBD dataset are successively used for testing. The experimental results show that our super-resolution reconstruction algorithm achieves the best results among the six commonly used algorithms, and our moving object detection algorithm improves the detection accuracy by up to 18.2%, 9.87% and 40.2% in three scenes, respectively, compared with the original algorithm, and it achieves the best results compared with the other three recent RGB-D-based methods. The algorithm proposed in this paper can better overcome the interference caused by shadow or illumination changes and detect moving objects more accurately. [ABSTRACT FROM AUTHOR]

Augmented reality is currently a research hotspot in the field of computer vision and computer graphics, and its applications are becoming more and more extensive. One of its key technologies is the three-dimensional registration of virtual objects. Three-dimensional registration requires accurate camera pose estimation and scene three-dimensional reconstruction. Therefore, this paper studies the 3D registration based on visual SLAM, and mainly contributes the following three aspects. (1) A dense matching method based on a depth camera is proposed, which can be well applied to scenes where the camera moves fast or rotates strongly, such as augmented reality. (2) A dense reconstruction method based on Voxel Hashing is designed, which alleviates the low computational efficiency and low precision of the existing RGB-D SLAM method. (3) A simple augmented reality system was designed to verify the effects of the registration and fusion of virtual objects. Experiments show that compared with the state-of-the-art methods, the algorithm proposed in this paper generates a more refined and smooth reconstructed model, and the virtual-real fusion effect based on this model is also more realistic. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNNs) are good at extracting contexture features within certain receptive fields, while transformers can model the global long-range dependency features. By absorbing the advantage of transformer and the merit of CNN, Swin Transformer shows strong feature representation ability. Based on it, we propose a cross-modality fusion model, SwinNet, for RGB-D and RGB-T salient object detection. It is driven by Swin Transformer to extract the hierarchical features, boosted by attention mechanism to bridge the gap between two modalities, and guided by edge information to sharp the contour of salient object. To be specific, two-stream Swin Transformer encoder first extracts multi-modality features, and then spatial alignment and channel re-calibration module is presented to optimize intra-level cross-modality features. To clarify the fuzzy boundary, edge-guided decoder achieves inter-level cross-modality fusion under the guidance of edge features. The proposed model outperforms the state-of-the-art models on RGB-D and RGB-T datasets, showing that it provides more insight into the cross-modality complementarity task. [ABSTRACT FROM AUTHOR]

This paper focuses on indoor semantic segmentation based on RGB-D images. Semantic segmentation is a pixel-level classification task that has made steady progress based on fully convolutional networks (FCNs). However, we find there is still room for improvements in the following three aspects. The first is related to multi-scale feature extraction. Recent state-of-the-art works forcibly concatenate multi-scale feature representations extracted by spatial pyramid pooling, dilated convolution or other architectures, regardless of the spatial extent for each pixel. The second is regarding RGB-D modal fusion. Most successful methods treat RGB and depth as two separate modalities and force them to be joined together regardless of their different contributions to the final prediction. The final aspect is about the modeling ability of extracted features. Due to the "local grid" defined by the receptive field, the learned feature representation lacks the ability to model spatial dependencies. In addition to these modules, we design a depth estimation module to encourage the RGB network to extract more effective features. To solve the above challenges, we propose four modules to address them: scale-aware module, modality-aware module, attention module and depth estimation module. Extensive experiments on the NYU-Depth v2 and SUN RGB-D datasets demonstrate that our method is effective against RGB-D indoor semantic segmentation. [ABSTRACT FROM AUTHOR]

The success of vision transformer demonstrates that the transformer structure is also suitable for various vision tasks, including high-level classification tasks and low-level dense prediction tasks. Salient object detection (SOD) is a pixel-level dense prediction task that simulates the most salient objects in human visual recognition scenarios. In recent years, depth images have been widely used for salient object detection. Compared with RGB SOD, the key point of RGB-D SOD is the effective fusion of depth information. As RGB-D SOD requires extracting depth features and fusing cross-modal information, additional computation is involved. However, except for lightweight models, most RGB-D SOD methods tend to obtain better prediction maps by consuming more computational resources. We propose a cross-modal dense cooperative fusion net, which provides state-of-the-art performance with less computation and parameters. We take advantage of the ability of the transformer structure to model long sequence dependencies to extract saliency features from RGB images. Since there is less information in the depth image than in the RGB image, it is not necessary to use the same structure in the depth stream. For the sake of reducing parameters and computation, we consider the asymmetric architecture. It is enough to meet our needs that deep features extracted by lightweight MobileV2Net. Our decoder can perform dense cooperative fusion of cross-modal information while decoding features. It can both effectively fuse cross-modal information and save computation. Comprehensive experiments on multiple benchmark datasets for RGB-D SOD show that compared with SOTA methods, our method performs much better with less computation and parameters. [ABSTRACT FROM AUTHOR]

Fruit size is crucial for growers as it influences consumer willingness to buy and the price of the fruit. Fruit size and growth along the seasons are two parameters that can lead to more precise orchard management favoring production sustainability. In this study, a Python-based computer vision system (CVS) for sizing apples directly on the tree was developed to ease fruit sizing tasks. The system is made of a consumer-grade depth camera and was tested at two distances among 17 timings throughout the season, in a Fuji apple orchard. The CVS exploited a specifically trained YOLOv5 detection algorithm, a circle detection algorithm, and a trigonometric approach based on depth information to size the fruits. Comparisons with standard-trained YOLOv5 models and with spherical objects were carried out. The algorithm showed good fruit detection and circle detection performance, with a sizing rate of 92%. Good correlations (r > 0.8) between estimated and actual fruit size were found. The sizing performance showed an overall mean error (mE) and RMSE of + 5.7 mm (9%) and 10 mm (15%). The best results of mE were always found at 1.0 m, compared to 1.5 m. Key factors for the presented methodology were: the fruit detectors customization; the HoughCircle parameters adaptability to object size, camera distance, and color; and the issue of field natural illumination. The study also highlighted the uncertainty of human operators in the reference data collection (5–6%) and the effect of random subsampling on the statistical analysis of fruit size estimation. Despite the high error values, the CVS shows potential for fruit sizing at the orchard scale. Future research will focus on improving and testing the CVS on a large scale, as well as investigating other image analysis methods and the ability to estimate fruit growth. [ABSTRACT FROM AUTHOR]

This work investigates the performance of five depth cameras in relation to their potential for grape yield estimation. The technologies used by these cameras include structured light (Kinect V1), active infrared stereoscopy (RealSense D415), time of flight (Kinect V2 and Kinect Azure), and LiDAR (Intel L515). To evaluate their suitability for grape yield estimation, a range of factors were investigated including their performance in and out of direct sunlight, their ability to accurately measure the shape of the grapes, and their potential to facilitate counting and sizing of individual berries. The depth cameras' performance was benchmarked using high-resolution photogrammetry scans. All the cameras except the Kinect V1 were able to operate in direct sunlight. Indoors, the RealSense D415 camera provided the most accurate depth scans of grape bunches, with a 2 mm average depth error relative to photogrammetric scans. However, its performance was reduced in direct sunlight. The time of flight and LiDAR cameras provided depth scans of grapes that had about an 8 mm depth bias. Furthermore, the individual berries manifested in the scans as pointed shape distortions. This led to an underestimation of berry sizes when applying the RANSAC sphere fitting but may help with the detection of individual berries with more advanced algorithms. Applying an opaque coating to the surface of the grapes reduced the observed distance bias and shape distortion. This indicated that these are likely caused by the cameras' transmitted light experiencing diffused scattering within the grapes. More work is needed to investigate if this distortion can be used for enhanced measurement of grape properties such as ripeness and berry size. [ABSTRACT FROM AUTHOR]

To accomplish the task of aerial manipulator’s autonomous grasping, it is very important to recognize and locate the object. At present, most of the recognition algorithms of aerial manipulators use traditional feature extraction methods. In order to improve the accuracy and efficiency of object recognition and positioning, this paper designs a visual recognition and positioning algorithm based on YOLOv5 deep learning object detection algorithm and RGB‑D sensor. The algorithm can detect the object in real‑time and estimate its pose, which serves for the grasping task of aerial manipulator. At the same time, aiming at the problems that the deep learning algorithm has a huge amount of calculation and cannot achieve high‑performance real‑time detection in the embedded system, model quantization is introduced to optimize the algorithm, which can greatly improve the processing speed of the algorithm. This paper introduces the overall framework and implementation process of the algorithm and verifies the effectiveness of some algorithms of object detection and pose estimation by using the COCO dataset and motion capture system. [ABSTRACT FROM AUTHOR]

In clinical practice, functional limitations in patients with low back pain are subjectively assessed, potentially leading to misdiagnosis and prolonged pain. This paper proposes an objective deep learning (DL) markerless motion capture system that uses a red–green–blue-depth (RGB-D) camera to measure the kinematics of the spine during flexion–extension (FE) through: 1) the development and validation of a DL semantic segmentation algorithm that segments the back into four anatomical classes and 2) the development and validation of a framework that uses these segmentations to measure spine kinematics during FE. Twenty participants performed ten cycles of FE with drawn-on point markers while being recorded with an RGB-D camera. Five of these participants also performed an additional trial where they were recorded with an optical motion capture (OPT) system. The DL algorithm was trained to segment the back and pelvis into four anatomical classes: upper back, lower back, spine, and pelvis. A kinematic framework was then developed to refine these segmentations into upper spine, lower spine, and pelvis masks, which were used to measure spine kinematics after obtaining 3D global coordinates of the mask corners. The segmentation algorithm achieved high accuracy, and the root mean square error (RMSE) between ground truth and predicted lumbar kinematics was < 4°. When comparing markerless and OPT kinematics, RMSE values were < 6°. This work demonstrates the feasibility of using markerless motion capture to assess FE spine movement in clinical settings. Future work will expand the studied movement directions and test on different demographics. [ABSTRACT FROM AUTHOR]

Camouflaged object detection (COD) presents a persistent challenge in accurately identifying objects that seamlessly blend into their surroundings. However, most existing COD models overlook the fact that visual systems operate within a genuine 3D environment. The scene depth inherent in a single 2D image provides rich spatial clues that can assist in the detection of camouflaged objects. Therefore, we propose a novel depth-perception attention fusion network that leverages the depth map as an auxiliary input to enhance the network's ability to perceive 3D information, which is typically challenging for the human eye to discern from 2D images. The network uses a trident-branch encoder to extract chromatic and depth information and their communications. Recognizing that certain regions of a depth map may not effectively highlight the camouflaged object, we introduce a depth-weighted cross-attention fusion module to dynamically adjust the fusion weights on depth and RGB feature maps. To keep the model simple without compromising effectiveness, we design a straightforward feature aggregation decoder that adaptively fuses the enhanced aggregated features. Experiments demonstrate the significant superiority of our proposed method over other states of the arts, which further validates the contribution of depth information in camouflaged object detection. The code will be available at https://github.com/xinran-liu00/DAF-Net. • An advanced RGB-D camouflaged object detection model is proposed. • Knowledge about depth can greatly assist in recognizing camouflaged objects. • Presentation of a fusion method that adaptively incorporates the generated depth. [ABSTRACT FROM AUTHOR]

This paper addresses the critical environmental issue of effectively managing construction and demolition waste (CDW), which has seen a global surge due to rapid urbanization. With the advent of deep learning-based computer vision, this study focuses on improving intelligent identification of valuable recyclables from cluttered and heterogeneous CDW streams in material recovery facilities (MRFs) by optimally leveraging both visual and spatial features (depth). A high-quality CDW RGB-D dataset was curated to capture MRF stream complexities often overlooked in prior studies, and comprises over 3500 images for each modality and more than 160,000 dense object instances of diverse CDW materials with high resource value. In contrast to former studies which directly concatenate RGB and depth features, this study introduces a new depth fusion strategy that utilizes computationally efficient convolutional operations at the end of the conventional waste segmentation architecture to effectively fuse colour and depth information. This avoids cross-modal interference and maximizes the use of distinct information present in the two different modalities. Despite the high clutter and diversity of waste objects, the proposed RGB-DL architecture achieves a 13% increase in segmentation accuracy and a 36% reduction in inference time when compared to the direct concatenation of features. The findings of this study emphasize the benefit of effectively incorporating geometrical features to complement visual cues. This approach helps to deal with the cluttered and varied nature of CDW streams, enhancing automated waste recognition accuracy to improve resource recovery in MRFs. This, in turn, promotes intelligent solid waste management for efficiently managing environmental concerns. [Display omitted] • Features a high-quality CDW RGB-D dataset curated to capture MRF stream complexities. • Direct feature fusion results in large feature maps and loss of distinct characteristics. • Advanced downstream fusion mitigates cross-modal interference. • Proposed RGB-D L fusion strategy improves accuracy without incurring high computational cost. • Most significant improvements are observed in small object segmentation. [ABSTRACT FROM AUTHOR]

Point clouds consist of 3D data points and are among the most considerable data formats for 3D representations. Their popularity is due to their broad application areas, such as robotics and autonomous driving, and their employment in basic 3D vision tasks such as segmentation, classification, and detection. However, processing point clouds is challenging compared to other visual forms such as images, mainly due to their unstructured nature. Deep learning (DL) has been established as a powerful tool for data processing, reporting remarkable performance enhancements compared to traditional methods for all basic 2D vision tasks. However new challenges are emerging when it comes to processing unstructured 3D point clouds. This work aims to guide future research by providing a systematic review of DL on 3D point clouds, holistically covering all 3D vision tasks. 3D technologies of point cloud formation are reviewed and compared to each other. The application of DL methods for point cloud processing is discussed, and state-of-the-art models’ performances are compared focusing on challenges and solutions. Moreover, in this work the most popular 3D point cloud benchmark datasets are summarized based on their task-oriented applications, aiming to highlight existing constraints and to comparatively evaluate them. Future research directions and upcoming trends are also highlighted. [ABSTRACT FROM AUTHOR]

Established surgical navigation systems for pedicle screw placement have been proven to be accurate, but still reveal limitations in registration or surgical guidance. Registration of preoperative data to the intraoperative anatomy remains a time-consuming, error-prone task that includes exposure to harmful radiation. Surgical guidance through conventional displays has well-known drawbacks, as information cannot be presented in-situ and from the surgeon's perspective. Consequently, radiation-free and more automatic registration methods with subsequent surgeon-centric navigation feedback are desirable. In this work, we present a marker-less approach that automatically solves the registration problem for lumbar spinal fusion surgery in a radiation-free manner. A deep neural network was trained to segment the lumbar spine and simultaneously predict its orientation, yielding an initial pose for preoperative models, which then is refined for each vertebra individually and updated in real-time with GPU acceleration while handling surgeon occlusions. An intuitive surgical guidance is provided thanks to the integration into an augmented reality based navigation system. The registration method was verified on a public dataset with a median of 100% successful registrations, a median target registration error of 2.7 mm, a median screw trajectory error of 1.6°and a median screw entry point error of 2.3 mm. Additionally, the whole pipeline was validated in an ex-vivo surgery, yielding a 100% screw accuracy and a median target registration error of 1.0 mm. Our results meet clinical demands and emphasize the potential of RGB-D data for fully automatic registration approaches in combination with augmented reality guidance. [Display omitted] • RGB-D data allows for automatic, fast and accurate registration in open spine surgery. • Radiation-free registration in open spine surgery can reach submillimeter accuracy. • A 2D U-Net can segment the lumbar spine and predict its orientation in RGB images. • GPU acceleration enables simultaneous real-time pose updates of multiple vertebrae. • Virtual twin augmented reality guidance is accurate in cadaveric specimens. [ABSTRACT FROM AUTHOR]

Rails are critical to the safe transportation of railway system, and their surface quality is a vital aspect to consider. Existing defect detection methods struggle to identify irregular defect boundaries and distinguish the similarity of foreground and background. To address these issues, depth images are introduced to detect rail defects. However, the existing RGB-D SOD methods usually fuse two modalities without considering modality-specific characteristics. In this paper, we propose a specificity autocorrelation integration network (SAINet) for surface defect detection of rails. SAINet enhances defect detection performance by exploring autocorrelation features of a single modality and the specificity of each modality. Two decoders are carefully designed to capture the specific characteristics of each modality. Moreover, we propose a cross-modal autocorrelation attention fusion (CAAF) to effectively utilize the two modalities of information. It obtains autocorrelation features of RGB images through dilated convolution and attention modules, introducing depth features to locate defects more accurately. We design a multi-modal feature integration block (MFIB) to supplement the cross-modal features with modality-specific features output by each individual decoder, in order to boost SOD performance. SAINet's performance is verified on the NEU RSDDS-AUG dataset. Our network achieves the best results compared to the 25 state-of-the-art methods. We also validate SAINet's generalization performance on six other benchmark datasets, where the experiments show it competes well on these datasets. The code and results are available at https://github.com/VDT-2048/SAINet. • Propose SAINet for rail surface defect detection to explore autocorrelation features. • Design CAAF to obtain high-quality cross-modal features. • Develop MFIB to utilizes modality-specific characteristics effectively. [ABSTRACT FROM AUTHOR]

Multi-modal fusion has been proved to help enhance the performance of scene classification tasks. This paper presents a 2D–3D Fusion stage that combines 3D Geometric Features with 2D Texture Features obtained by 2D Convolutional Neural Networks. To get a robust 3D Geometric embedding, a network that uses two novel layers is proposed. The first layer, Multi-Neighbourhood Graph Convolution, aims to learn a more robust geometric descriptor of the scene combining two different neighbourhoods: one in the Euclidean space and the other in the Feature space. The second proposed layer, Nearest Voxel Pooling, improves the performance of the well-known Voxel Pooling. Experimental results, using NYU-Depth-V2 and SUN RGB-D datasets, show that the proposed method outperforms the current state-of-the-art in RGB-D indoor scene classification task. • Multi-Neighbourhood Graph Neural Network to extract enriched 3D Geometric features. • Nearest Voxel Pooling that improves the performance of the well-known Voxel Pooling. • 2D–3D multi-modal fusion stage that allows to exploit the benefits of 2D and 3D Networks simultaneously. • A novel 3D random crop that generates random crops of a point cloud. [ABSTRACT FROM AUTHOR]

[Display omitted] • We propose a SCAM that integrates the information from both modes to reduce the fusion ambiguity caused by unreliable depth maps. • The CDCM can extract additional cues from another branch to adaptively alter the sampling locations. • The MFRM can fuse cross-modality features in multiple scales and even aggregate unique cues in tiny sized features. Effective exploration of useful contextual information in multi-modal images is an essential task in salient object detection. Nevertheless, the existing methods based on the early-fusion or the late-fusion schemes cannot address this problem as they are unable to effectively resolve the distribution gap and information loss. In this paper, we propose an adaptive multi-level deformable fusion network (AMDFNet) to exploit the cross-modality information. We use a cross-modality deformable convolution module to dynamically adjust the boundaries of salient objects by exploring the extra input from another modality. This enables incorporating the existing features and propagating more contexts so as to strengthen the model's ability to perceiving scenes. To accurately refine the predicted maps, a multi-scaled feature refinement module is proposed to enhance the intermediate features with multi-level prediction in the decoder part. Furthermore, we introduce a selective cross-modality attention module in the fusion process to exploit the attention mechanism. This module captures dense long-range cross-modality dependencies from a multi-modal hierarchical feature's perspective. This strategy enables the network to select more informative details and suppress the contamination caused by the negative depth maps. Experimental results on eight benchmark datasets demonstrate the effectiveness of the components in our proposed model, as well as the overall saliency model. [ABSTRACT FROM AUTHOR]

In order to solve the problem that the original depth image data acquired by RGB-D camera contains noise and holes, a depth image filtering algorithm based on color image guidance is proposed in this study. The proposed algorithm is based on the principle of bilateral filtering. First, the color image is used as a guide map, and the gray pixel change weight of the color image is multiplied by the Gaussian filter weight to obtain the comprehensive filter weight. Then, the algorithm uses the separation characteristic of the Gaussian kernel function to reduce the computational complexity and improve the running speed of the algorithm through fast Gaussian transformation. Experimental results show that after filtering the original depth image, the algorithm can significantly improve the denoising effect of depth image. Additionally, the algorithm can repair the small-area void area of the depth image while maintaining the advantages of bilateral filter denoising and edge preservation, and shorten the running time. [ABSTRACT FROM AUTHOR]

To solve problems in 3D modelling, including unclear 3D edges in point clouds, lack of geometric semantics, confusing topological relations between 3D models, and low degree of automation in traditional model-driven 3D modelling, a method for high-accuracy and automatic monomer 3D modelling must be developed. This paper is the first to propose a 3D modelling strategy, called combining data-and-model-driven 3D modelling (CDMD3DM), for small regular objects in indoor scenes using RGB-D data. The proposed method's workflow is as follows: generation of initial 3D point cloud data using data-driven Kinect v2; segmentation of point cloud data, based on deep learning, that improves the accuracy and automation of geometric model recognition; definition of the initial model-driven parameters based on the instance segmentation results; optimization of geometric model parameters, based on generalized point photogrammetry theory, to generate monomer models in indoor scenes to overcome the shortcomings of confusing topological relationships and inaccurate 3D model edges; and finally, fusion of the results of data-driven and model-driven 3D modelling. The experimental results demonstrate that CDMD3DM is feasible, automatic, more accurate, more reliable, semantically richer and capable of producing clearer topological relationships than current 3D modelling results using indoor RGB-D data. These outcomes promote interdisciplinary integration between computer vision and photogrammetry. [ABSTRACT FROM AUTHOR]

• We propose a novel MCI-Net for RGB-D salient object detection, which can effectively exploit the correlation between RGB images and depth cues, while also exploring multi-level information to boost the SOD performance. • To exploit the correlations across RGB images and depth cues, a CMRM is proposed to integrate cross-modal features. The CMRM is carried out in various of levels of feature spaces for further multi-level feature fusion. • An MLFM is proposed to fuse the features of each level in a bottom-up manner, and an FIM is proposed to propagate the features of the last convolutional layer back to the previous layers to reduce the information lost during downsampling. • Extensive experimental results on six RGB-D SOD benchmark datasets demonstrate the effectiveness of our MCI-Net over 14 state-of-the-art methods. Depth cues with affluent spatial information have been proven beneficial in boosting salient object detection (SOD), while the depth quality directly affects the subsequent SOD performance. However, it is inevitable to obtain some low-quality depth cues due to thelimitations of its acquisition devices, which can inhibit the SOD performance. Besides, existing methods tend to combine RGB images and depth cues in a direct fusion or a simple fusion module, making them not effectively exploit the complex correlations between the two sources. Moreover, few methods design an appropriate module to fully fuse multi-level features, resulting in cross-level feature interaction insufficient. To address these issues, we propose a novel Multi-level Cross-modal Interaction Network (MCI-Net) for RGB-D based SOD. Our MCI-Net includes two key components: 1) a cross-modal feature learning network, which is used to learn the high-level features for the RGB images and depth cues, effectively enabling the correlations between the two sources to be exploited; and 2) a multi-level interactive integration network, which integrates multi-level cross-modal features to boost the SOD performance. Extensive experiments on six benchmark datasets demonstrate the superiority of our MCI-Net over 14 state-of-the-art methods, and validate the effectiveness of different components in our MCI-Net. More important, our MCI-Net significantly improves the SOD performance as well as has a higher FPS. [ABSTRACT FROM AUTHOR]

The goal of this work is to present a systematic solution for RGB-D salient object detection, which addresses the following three aspects with a unified framework: modal-specific representation learning, complementary cue selection, and cross-modal complement fusion. To learn discriminative modal-specific features, we propose a hierarchical cross-modal distillation scheme, in which we use the progressive predictions from the well-learned source modality to supervise learning feature hierarchies and inference in the new modality. To better select complementary cues, we formulate a residual function to incorporate complements from the paired modality adaptively. Furthermore, a top-down fusion structure is constructed for sufficient cross-modal cross-level interactions. The experimental results demonstrate the effectiveness of the proposed cross-modal distillation scheme in learning from a new modality, the advantages of the proposed multi-modal fusion pattern in selecting and fusing cross-modal complements, and the generalization of the proposed designs in different tasks. [ABSTRACT FROM AUTHOR]

• A new RGB-D shape-based hand gesture recognition method is proposed. • A new hand shape descriptor is proposed with emphasized finger feature. • Different recognition engines are explored and compared for different applications. • Sota accuracies are achieved on several benchmarks, as well as excellent efficiency. • A demo video is given for a real life application of human-computer interaction. The development of depth cameras, e.g., the Kinect sensor, provides new opportunities for human computer interaction (HCI). Although the Kinect sensor has been extensively applied for human tracking, human action recognition and hand gesture recognition, real time hand gesture recognition is still a challenging problem. In this paper, a new real time hand gesture recognition method is proposed. Since fingers are the most important clue for hand gesture classification, a finger-emphasized multi-scale descriptor is proposed. The proposed descriptor incorporates three types of parameters of multiple scales to make a discriminative representation of the hand shape. Furthermore, the features of fingers are emphasized for hand gesture analysis. Three solutions to hand gesture recognition are then investigated with DTW, SVM, and neural network. Extensive experiments are conducted and the results show that the proposed method is robust to noise, articulations and rigid transformations. The comparison with state-of-the-art methods verifies the accuracy and efficiency of our method. [ABSTRACT FROM AUTHOR]

• We propose Cross-Modal Transformer (CMFormer), a Transformer-based cross-modal semantic segmentation model, which achieves better cross-modal information interaction by capturing long-range contextual dependencies. The CMFormer includes the MS-CAC module and the GFA module, and achieves the state-of-the-art results in RGB-D semantic segmentation task. • We design the MS-CAC module and the GFA module by exploiting the cross-modal properties of Transformer. The MS-CAC module preserves multi-scale information while realizing channel feature correction, while the GFA module realizes spatial feature fusion, and fully considers global and local features, which make the CMFormer achieve better cross-modal information interaction by capturing long-range contextual dependencies. • The effectiveness of the CMFormer is evaluated with extensive experiments on the SOP dataset and NYU Depth v2 dataset. The results show that the CMFormer achieves the state-of-the-art results on both datasets. In the experiment of SOP dataset, CMFormer can achieve 96.74% MPA and 92.98% mIoU. At the same time, the real-time performance of CMFormer can reach 43 FPS(Frames Per Second), which has good real-time performance while meeting the requirements of high precision. Scene understanding in a production workshop is an important technology to improve its intelligence level, semantic segmentation of production workshop objects is an effective method for realizing scene understanding. Since the varieties of information of production workshop, making full use of the complementary information of RGB image and depth image can effectively improve the semantic segmentation accuracy of production workshop objects. Aiming at solving the multi-scale and real-time problems of segmenting the production workshop objects, this paper proposes Cross-Modal Transformer (CMFormer), a Transformer-based cross-modal semantic segmentation model. Its key feature correction and feature fusion parts are composed of the Multi-Scale Channel Attention Correction(MS-CAC) module and the Global Feature Aggregation(GFA) module. By improving Multi-Head Self-Attention(MHSA) in Transformer, we design Cross-Modal Multi-Head Self-Attention(CM-MHSA) to build long-range interaction between RGB image and depth image, and further design the MS-CAC module and the GFA module on the basis of the CM-MHSA module, to achieve cross-modal information interaction in the channel and spatial dimensions. Among them, the MS-CAC module enriches the multi-scale features of each channel and achieve more accurate channel attention correction between the two modals; the GFA module interacts with RGB feature and depth feature in the spatial dimension and fuses global and local features at the same time. In the experiments on the NYU Depth v2 dataset, the CMFormer reached 68.00% MPA(Mean Pixel Accuracy) and 55.75% mIoU(Mean Intersection over Union), achieves the state-of-the-art results. While in the experiments on the Scene Objects for Production workshop dataset(SOP), t he CMFormer achieves 96.74% MPA, 92.98% mIoU and 43 FPS(Frames Per Second), which has high precision and good real-time performance. Code is available at: https://github.com/FutureIAI/CMFormer [ABSTRACT FROM AUTHOR]

A smartphone with both colour and time of flight depth cameras is used for automated grape yield estimation of Chardonnay grapes. A new technique is developed to automatically identify grape berries in the smartphone's depth maps. This utilises the distortion peaks in the depth map caused by diffused scattering of the light within each grape berry. This technique is then extended to allow unsupervised training of a YOLOv7 model for the detection of grape berries in the smartphone's colour images. A correlation coefficient (R 2) of 0.946 was achieved when comparing the count of grape berries observed in RGB images to those accurately identified by YOLO. Additionally, an average precision score of 0.970 was attained. Two techniques are then presented to automatically estimate the size of the grape berries and generate 3D models of grape bunches using both colour and depth information. • Grape yield estimation using a smartphone's colour and time of flight depth cameras. • Peaks in the depth map occur at berry locations due to diffuse scattering of light. • Grape berries 3D locations are automatically detected using these peaks. • Extended for unsupervised training of YOLO berry detection model for colour images. • Technique developed to estimate berry size from the depth and colour images. [ABSTRACT FROM AUTHOR]