Search

3D printing has revolutionized product design and manufacturing across various industries by enabling the creation of complex geometries with minimal waste. Despite its advancements, 3D printing still faces significant challenges, including spatial constraints and process control limitations. This paper introduces novel approaches to improve the functionality and precision of material extrusion 3D printers for the fused deposition modeling method, particularly for additional printing tasks such as printing on existing objects or continuing a print on a relocated object without prior knowledge of its position or even the environment is changed. We introduce a compensation system integrating a high-speed vision system for robot arms to address these challenges. Our system employs a three-step pose estimation process—fast point feature histograms (FPFH)-based, corner-based, and sub-pixel edge-based methods—to ensure high accuracy in restoring the position of printed pieces for additional printing tasks on a given object. Experimental results demonstrate substantial improvements in printing precision, with the system achieving sub-millimeter and sub-pixel accuracy. These advancements not only eliminate work area constraints but also enhance the adaptability and reliability of 3D printing processes. [ABSTRACT FROM AUTHOR]

Panoptic segmentation plays a crucial role in enabling robots to comprehend their surroundings, providing fine-grained scene understanding information for robots' intelligent tasks. Although existing methods have made some progress, they are prone to fail in areas with weak textures, small objects, etc. Inspired by biological vision research, we propose a cascaded contour-enhanced panoptic segmentation network called CCPSNet, attempting to enhance the discriminability of instances through structural knowledge. To acquire the scene structure, a cascade contour detection stream is designed, which extracts comprehensive scene contours using channel regulation structural perception module and coarse-to-fine cascade strategy. Furthermore, the contour-guided multi-scale feature enhancement stream is developed to boost the discrimination ability for small objects and weak textures. The stream integrates contour information and multi-scale context features through structural-aware feature modulation module and inverse aggregation technique. Experimental results show that our method improves accuracy on the Cityscapes (61.2 PQ) and COCO (43.5 PQ) datasets while also demonstrating robustness in challenging simulated real-world complex scenarios faced by robots, such as dirty cameras and rainy conditions. The proposed network promises to help the robot perceive the real scene. In future work, an unsupervised training strategy for the network could be explored to reduce the training cost. [ABSTRACT FROM AUTHOR]

This study proposes a method for determining the appropriate harvesting order for densely growing wood ear mushrooms by recognizing their growth stages and harvesting priorities from depth images obtained from a stereo camera. We aim to minimize crop damage and improve the quality of harvested crops during the harvesting of densely growing crops using a robot arm. The proposed two-stage method consists of two models—one of the models to recognize priority harvest regions, and the other model to identify individual wood ear mushroom regions and growth stages. The final harvesting order is determined based on the outputs of these models. The models were trained using simulated CGI data of wood ear mushroom growth. The experimental results show that the appropriate harvesting order can be outputted in 57.5% of the cases for the 40 sets of test data. The results show that it is possible to determine the harvesting order of dense wood ear mushrooms based solely on depth images. However, there is still room for improvement in operations in actual environments. Further work is needed to enhance the method's robustness and accuracy. [ABSTRACT FROM AUTHOR]

In order to rapidly identify and locate the weld seam initial point in robotic automated welding, we established a binocular vision system and proposed a weld seam initial point localization algorithm named WIPL-Net. Built upon the Fully Convolutional One-Stage object detection network, WIPL-Net introduces a lightweight ResNext as its backbone network and incorporates channel attention and enhanced feature fusion mechanisms to enhance feature detection and extraction capabilities. Subsequently, WIPL-Net is utilized to obtain the weld seam's initial point, and its three-dimensional coordinates are determined through trigonometric measurements. To further estimate the robot's posture at the initial point, we performed sparse three-dimensional reconstruction of the local region centered on the weld seam initial point based on You Only Look At Coefficients of Tensors instance segmentation and feature point matching. Finally, we conducted comparative experiments on WIPL-Net and conducted weld seam initial point localization experiments in real welding scenarios. The results demonstrate that our proposed method achieves a positioning error of less than 1.2 mm for the weld seam's initial point and a pose error of less than 10 degrees for the robot, meeting the requirement for real-time positioning of the weld seam's initial point. [ABSTRACT FROM AUTHOR]

Introduction: Accurately recognizing and understanding human motion actions presents a key challenge in the development of intelligent sports robots. Traditional methods often encounter significant drawbacks, such as high computational resource requirements and suboptimal real-time performance. To address these limitations, this study proposes a novel approach called Sports-ACtrans Net. Methods: In this approach, the Swin Transformer processes visual data to extract spatial features, while the Spatio-Temporal Graph Convolutional Network (ST-GCN) models human motion as graphs to handle skeleton data. By combining these outputs, a comprehensive representation of motion actions is created. Reinforcement learning is employed to optimize the action recognition process, framing it as a sequential decision-making problem. Deep Q-learning is utilized to learn the optimal policy, thereby enhancing the robot's ability to accurately recognize and engage in motion. Results and discussion: Experiments demonstrate significant improvements over state-of-the-art methods. This research advances the fields of neural computation, computer vision, and neuroscience, aiding in the development of intelligent robotic systems capable of understanding and participating in sports activities. [ABSTRACT FROM AUTHOR]

In response to the costly and error-prone manual satellite tracking on the International Space Station (ISS), this paper presents a deep neural network (DNN)-based robotic visual servoing solution to the automated tracking operation. This innovative approach directly addresses the critical issue of motion decoupling, which poses a significant challenge in current image moment-based visual servoing. The proposed method uses DNNs to estimate the manipulator's pose, resulting in a significant reduction of coupling effects, which enhances control performance and increases tracking precision. Real-time experimental tests are carried out using a 6-DOF Denso manipulator equipped with an RGB camera and an object, mimicking the targeting pin. The test results demonstrate a 32.04% reduction in pose error and a 21.67% improvement in velocity precision compared to conventional methods. These findings demonstrate that the method has the potential to improve efficiency and accuracy significantly in satellite target tracking and capturing. [ABSTRACT FROM AUTHOR]

Although grid maps help mobile robots navigate in indoor environments, some lack semantic information that would allow the robot to perform advanced autonomous tasks. In this paper, a semantic map production system is proposed to facilitate indoor mobile robot navigation tasks. The developed system is based on the employment of LiDAR technology and a vision-based system to obtain a semantic map with rich information, and it has been validated using the robot operating system (ROS) and you only look once (YOLO) v3 object detection model in simulation experiments conducted in indoor environments, adopting low-cost, -size, and -memory computers for increased accessibility. The obtained results are efficient in terms of object recognition accuracy, object localization error, and semantic map production precision, with an average map construction accuracy of 78.86%. [ABSTRACT FROM AUTHOR]

Multifunctionality, integration, and miniaturization are the mainstream trends in modern device development. However, optical display and depth perception, essential for innovative applications like intelligent driving and mixed reality, are typically achieved separately in complex and bulky optical systems. Metasurfaces, ultrathin artificial optical surfaces with strong light–matter interaction capabilities at the nanoscale, offer a versatile platform for creating multifunctional, highly integrated, and miniaturized devices by fully exploring spatial multiplexing and utilizing variables like location, polarization, and frequency. In this study, a multifunctional metasurface that integrates meta‐holography with Dammann gratings is presented. By engineering the phase profile, the device can display specific holographic images in the Fresnel region while simultaneously producing a high‐density spot cloud with over 37 000 spots covering the entire half space. As a proof‐of‐concept, a projection and reconstruction experiment is performed to elaborate its potential for retrieving three‐dimensional (3D) spatial information of real scenarios. The simultaneous generation of display and point cloud holds significant promise for applications in cutting‐edge imaging technologies such as intelligent driving and robot vision. [ABSTRACT FROM AUTHOR]

This article presents the research results in the application of image processing in determining the position, direction, and moving speed of different objects when they move in the free space in the field of view of measuring cameras. The research includes developing an algorithm to detect, identify, and locate objects and an algorithm to calculate the movement direction and instantaneous velocity of the object. Two robots with measurement systems were designed for testing, one carries the investigating object, and the other the camera. These robots can communicate with a computer system using a tele-wireless system. A program was also built for capturing images and analyzing the state of the model automatically. Experimental results show that the position, angle, and velocity of different objects can be captured well. The average error in determining the direction of movement is an average of 1.25°, and the error of the moving speed is less than 0.5 m/s. The research results provide a potential tool for designing robots for highly effective detection. The algorithm and the measurement system are simple and inexpensive but highly effective, and they can be used in the initial process of determining an object. A detailed description of the algorithm, robot system, and testing will be presented in this study. [ABSTRACT FROM AUTHOR]

Artificial visual memory systems have been of particular interest since the development of machine vision and bionic robots. Ordinarily, the conventional system architecture requires the complex integration of two functional modules, a photo‐sensor and a memory device, which greatly limits the operating efficiency and increases the extra energy consumption. Nonetheless, other simply configured optoelectronics memory devices generally face challenges of adaption in complex light environments. Here, a resistive switching (RS) perovskite‐based photomemristor is presented that mimics the retina function. The dual function of light perception and in situ storage are both achieved. In the dark condition, it exhibits impressive memory performance with a high ON/OFF ratio of 104, a long retention time of over 104 s, and a low operating voltage of 0.38 V. With illumination, it shows self‐powered, broadband photo‐detecting characteristics with responsivity of 70 mA W−1 and detectivity of 7.5 × 1010 Jones. More importantly, benefiting from the material dual‐phase configuration, the highly steady photo‐adjusted RS windows are achieved. Its light‐adaptive memory application in dynamic environments is further demonstrated using the mimicking retina for a machine eye. This work can provide a strategy for enhanced RS photomemristor and its application in changing and varied scenarios. [ABSTRACT FROM AUTHOR]

Landscape maintenance is essential for ensuring agricultural productivity, promoting sustainable land use, and preserving soil and ecosystem health. Pruning is a labor-intensive task among landscaping applications that often involves repetitive pruning operations. To address these limitations, this paper presents the development of a dual-arm holonomic robot (called the KOALA robot) for precision plant pruning. The robot utilizes a cross-functionality sensor fusion approach, combining light detection and ranging (LiDAR) sensor and depth camera data for plant recognition and isolating the data points that require pruning. The You Only Look Once v8 (YOLOv8) object detection model powers the plant detection algorithm, achieving a 98.5% pruning plant detection rate and a 95% pruning accuracy using camera, depth sensor, and LiDAR data. The fused data allows the robot to identify the target boxwood plants, assess the density of the pruning area, and optimize the pruning path. The robot operates at a pruning speed of 10–50 cm/s and has a maximum robot travel speed of 0.5 m/s, with the ability to perform up to 4 h of pruning. The robot's base can lift 400 kg, ensuring stability and versatility for multiple applications. The findings demonstrate the robot's potential to significantly enhance efficiency, reduce labor requirements, and improve landscape maintenance precision compared to those of traditional manual methods. This paves the way for further advancements in automating repetitive tasks within landscaping applications. [ABSTRACT FROM AUTHOR]

The use of computer vision in the industry has become fundamental, playing an essential role in areas such as quality control and inspection, object recognition/tracking, and automation. Despite this constant growth, robotic cell systems employing computer vision encounter significant challenges, such as a lack of flexibility to adapt to different tasks or types of objects, necessitating extensive adjustments each time a change is required. This highlights the importance of developing a system that can be easily reused and reconfigured to address these challenges. This paper introduces a versatile and adaptable framework that exploits Computer Vision and the Robot Operating System (ROS) to facilitate pick-and-place operations within robotic cells, offering a comprehensive solution for handling and sorting random-flow objects on conveyor belts. Designed to be easily configured and reconfigured, it accommodates ROS-compatible robotic arms and 3D vision systems, ensuring adaptability to different technological requirements and reducing deployment costs. Experimental results demonstrate the framework's high precision and accuracy in manipulating and sorting tested objects. Thus, this framework enhances the efficiency and flexibility of industrial robotic systems, making object manipulation more adaptable for unpredictable manufacturing environments. [ABSTRACT FROM AUTHOR]

Featured Application: The presented methodology can be used in all robotic assembly tasks where the camera position changes during operation, such as tasks where the camera is mounted on the robotic arm near the end-effector—for example, in fuse box assembly applications. The increase in positioning accuracy and repeatability allowed the integration of robots in assembly operations using guidance systems (structured applications) or video acquisition systems (unstructured applications). This paper proposes a procedure to determine the measuring plane using a 3D laser camera. To validate the procedure, the camera coordinates and orientation will be verified using robot coordinates. This procedure is an essential element for camera calibration and consists of developing a mathematical model using the least square method and planar regression. The mathematical model is considered necessary as a step towards optimizing the integration of robotic vision systems in assembly applications. A better calibrated camera has the potential to provide better recognition results, which are essential in this field. These improved results can then be used to increase the accuracy and repeatability of the robot. [ABSTRACT FROM AUTHOR]

Surgical robotics application in the field of minimally invasive surgery has developed rapidly and has been attracting increasingly more research attention in recent years. A common consensus has been reached that surgical procedures are to become less traumatic and with the implementation of more intelligence and higher autonomy, which is a serious challenge faced by the environmental sensing capabilities of robotic systems. One of the main sources of environmental information for robots are images, which are the basis of robot vision. In this review article, we divide clinical image into direct and indirect based on the object of information acquisition, and into continuous, intermittent continuous, and discontinuous according to the target-tracking frequency. The characteristics and applications of the existing surgical robots in each category are introduced based on these two dimensions. Our purpose in conducting this review was to analyze, summarize, and discuss the current evidence on the general rules on the application of image technologies for medical purposes. Our analysis gives insight and provides guidance conducive to the development of more advanced surgical robotics systems in the future. [ABSTRACT FROM AUTHOR]

Collaborative robotic manipulators can stop in case of a collision, according to the ISO/TS 15066 and ISO 10218-1 standards. However, in a human-robot collaboration scenario where the robot and the human share the workspace, a better solution with concerns to production and operator safety would be to perceive in advance the presence of an obstacle and be able to avoid it, thereby completing the task without halting the robot. In this paper, an obstacle avoidance algorithm is tested using a sensor system for real-time human detection; the operator represents a potential dynamic obstacle that can interfere with the robot motion. The sensor system consists of three RGB-D cameras. A custom software framework has been developed in Python exploiting machine learning tools for human skeleton detection. The coordinates of the human body joints relative to the manipulator base are used as input to the obstacle avoidance algorithm. The use of multiple sensors makes it possible to limit the occlusion problem; in addition, the choice of non-wearable sensors goes in the direction of better operator comfort. A series of experimental tests were performed to verify the accuracy of skeleton detection and the ability of the system to avoid obstacles in real time. The human motion caption system, in particular, was validated through a comparison with a commercial system based on wearable IMU sensors widely used and validated in motion capture. There is an NRMSE of 3.23 % for the RGB-D camera-based skeleton detection system, against an NRMSE of 12.23 % for the IMU wearable sensor system. Test results confirm that the system is able to avoid collisions with the human body under various conditions, static or dynamic, ensuring a minimum safety distance to any part of the manipulator. In 44 tests in which the operator moves around the robot with possible collisions (at a speed typical of manufacturing operations), the minimum operator-robot distance averaged 208 mm , being 200 mm the limit safety distance set by algorithm. [ABSTRACT FROM AUTHOR]

Copyright of Machine Tool & Hydraulics is the property of Guangzhou Mechanical Engineering Research Institute (GMERI) and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

In recent years, learning-based approaches have demonstrated significant promise in addressing intricate navigation tasks. Traditional methods for training deep neural network navigation policies rely on meticulously designed reward functions or extensive teleoperation datasets as navigation demonstrations. However, the former is often confined to simulated environments, and the latter demands substantial human labor, making it a time-consuming process. Our vision is for robots to autonomously learn navigation skills and adapt their behaviors to environmental changes without any human intervention. In this work, we discuss the self-supervised navigation problem and present Dynamic Graph Memory (DGMem), which facilitates training only with on-board observations. With the help of DGMem, agents can actively explore their surroundings, autonomously acquiring a comprehensive navigation policy in a data-efficient manner without external feedback. Our method is evaluated in photorealistic 3D indoor scenes, and empirical studies demonstrate the effectiveness of DGMem. [ABSTRACT FROM AUTHOR]

Robot semantic navigation has received significant attention recently, as it aims to achieve reliable mapping and navigation accuracy. Object detection tasks are vital in this endeavor, as a mobile robot needs to detect and recognize the objects in the area of interest to build an effective semantic map. To achieve this goal, this paper classifies and discusses recently developed object detection approaches and then presents the available vision datasets that can be employed in robot semantic navigation applications. In addition, this paper discusses several experimental studies that have validated the efficiency of object detection algorithms, including Faster R-CNN, YOLO v5, and YOLO v8. These studies also utilized a vision dataset to design and develop efficient robot semantic navigation systems, which is also discussed. According to several experiments conducted in a Fablab area, the YOLO v8 object classification model achieved the best results in terms of classification accuracy and processing speed. [ABSTRACT FROM AUTHOR]

Focused on the problems of cumbersome operation, low efficiency, and high cost in the traditional manual rebar binding process, we propose a mobile robot vision detection and path‐planning method for rebar binding to realize automated rebar binding by combining deep learning and path‐planning technology. A MobileNetV3‐SSD rebar binding crosspoints recognition model is built based on TensorFlow deep learning framework, and a crosspoints localization method combining control factor α and feature projection curve is introduced to achieve the localization of unbound crosspoints. In addition, A back‐and‐forth path‐planning algorithm with priority constraints combined with dead zone escape algorithm based on improved A* is proposed to achieve complete coverage path planning of the working area and path transfer of the dead zone. In the field test of the robot prototype, the classification accuracy and localization accuracy reached 94.40% and 90.49%, and the robot was able to reach complete coverage path planning successfully. The experimental results show that the visual detection method can achieve fast, noncontact and intelligent recognition of rebar binding crosspoints, which has good robustness and application value. At the same time, the proposed path‐planning method has higher efficiency in the execution of robot complete coverage path planning, and meets the basic requirements of path planning for rebar binding process. [ABSTRACT FROM AUTHOR]

As the third‐generation network succeeding artificial neural networks (ANNs), spiking neural networks (SNNs) offer high robustness and low energy consumption. Inspired by biological systems, the limitations of low robustness and high‐power consumption in deep reinforcement learning (DRL) are addressed by introducing SNNs. The Brain Q‐network (BrainQN) is proposed, which replaces the neurons in the classic Deep Q‐learning (DQN) algorithm with SNN neurons. BrainQN is trained using surrogate gradient learning (SGL) and ANN‐to‐SNN conversion methods. Robustness tests with input noise reveal BrainQN's superior performance, achieving an 82.14% increase in rewards under low noise and 71.74% under high noise compared to DQN. These findings highlight BrainQN's robustness and superior performance in noisy environments, supporting its application in complex scenarios. SGL‐trained BrainQN is more robust than ANN‐to‐SNN conversion under high noise. The differences in network output correlations between noisy and original inputs, along with training algorithm distinctions, explain this phenomenon. BrainQN successfully transitioned from a simulated Pong environment to a ball‐catching robot with dynamic vision sensors (DVS). On the neuromorphic chip PAICORE, it shows significant advantages in latency and power consumption compared to Jetson Xavier NX. [ABSTRACT FROM AUTHOR]

In swimming, the posture and technique of athletes are crucial for improving performance. However, traditional swimming coaches often struggle to capture and analyze athletes' movements in real-time, which limits the effectiveness of coaching. Therefore, this paper proposes RL-CWtrans Net: a robot vision-driven multimodal swimming training system that provides precise and real-time guidance and feedback to swimmers. The system utilizes the Swin-Transformer as a computer vision model to effectively extract the motion and posture features of swimmers. Additionally, with the help of the CLIP model, the system can understand natural language instructions and descriptions related to swimming. By integrating visual and textual features, the systemachieves amore comprehensive and accurate information representation. Finally, by employing reinforcement learning to train an intelligent agent, the system can provide personalized guidance and feedback based on multimodal inputs. Experimental results demonstrate significant advancements in accuracy and practicality for this multimodal robot swimming coaching system. The system is capable of capturing real-time movements and providing immediate feedback, thereby enhancing the effectiveness of swimming instruction. This technology holds promise. [ABSTRACT FROM AUTHOR]

Due to several complex factors such as the type, size, and shape of target object, vision-assisted robot grasping technology still faces serious challenges. In this research, a deep learning-based robot hand vision grasping algorithm was developed considering semi-structural environmental constraints. The proposed algorithm could build a deep learning network on the basis of the desired object, perform object recognition, category classification and position judgment, and complete robot hand-grasping tasks. The obtained experimental results demonstrated that the proposed algorithm effectively solved the problem of recognizing and classifying multi-category objects in a semi-structured environment, improving recognition rate and grasping rate and reducing collision rate. [ABSTRACT FROM AUTHOR]

In recent years, the integration of polarimetric imaging into robotic perception systems has increased significantly, driven by the accessibility of affordable polarimetric sensors. This technology complements traditional color imaging by capturing and analyzing the polarization characteristics of light. This additional information provides robots with valuable insights into object shape, material composition, and other properties, ultimately enabling more robust manipulation tasks. This review aims to provide a comprehensive analysis of the principles behind polarimetric imaging and its diverse applications within the field of robotic perception. By exploiting the polarization state of light, polarimetric imaging offers promising solutions to three key challenges in robot vision: Surface segmentation; depth estimation through polarization patterns; and 3D reconstruction using polarimetric data. This review emphasizes the practical value of polarimetric imaging in robotics by demonstrating its effectiveness in addressing real-world challenges. We then explore potential applications of this technology not only within the core robotics field but also in related areas. Through a comparative analysis, our goal is to elucidate the strengths and limitations of polarimetric imaging techniques. This analysis will contribute to a deeper understanding of its broad applicability across various domains within and beyond robotics. [ABSTRACT FROM AUTHOR]

The focus of this paper is on investigating the use of visual communication between two humanoid robots, in order to enhance the coordination of tasks between them. The problem continues to be an interesting and fruitful area of research from the days of using multiple robot manipulator arms for manufacturing as well as space robotics to current research in medical robotics. The approach here is to employ several off-the-shelf algorithms, software and hardware such as the NAO robot and support software, including Choregraphe, OpenCV to capture and process images, the SVM to classify objects in images, and the Python programming environment. Five robotic actions were studied and three modes. The experiments used one robot as the "viewer" and the second robot as the "subject" being analyzed. Results show that the visual communication system has an accuracy of 90% in correctly identifying the five movements. This research has shown an original solution, as a model that can enable robots to run in the complex service tasks consisting of multiple connected actions in a dynamic environment. This methodology can also let the intelligent operation of the robots serve in different scenes according to their actual requirements. This research focuses on enhancing the prototype robot vision function and development of additional value for consolidation manageable platform that increases service robots in the home environment of intelligent control capability. [ABSTRACT FROM AUTHOR]

The rapid development of artificial intelligence and remote sensing technologies is indispensable for modern agriculture. In orchard environments, challenges such as varying light conditions and shading complicate the tasks of intelligent picking robots. To enhance the recognition accuracy and efficiency of apple-picking robots, this study aimed to achieve high detection accuracy in complex orchard environments while reducing model computation and time consumption. This study utilized the CenterNet neural network as the detection framework, introducing gray-centered RGB color space vertical decomposition maps and employing grouped convolutions and depth-separable convolutions to design a lightweight feature extraction network, Light-Weight Net, comprising eight bottleneck structures. Based on the recognition results, the 3D coordinates of the picking point were determined within the camera coordinate system by using the transformation relationship between the image's physical coordinate system and the camera coordinate system, along with depth map distance information of the depth map. Experimental results obtained using a testbed with an orchard-picking robot indicated that the proposed model achieved an average precision (AP) of 96.80% on the test set, with real-time performance of 18.91 frames per second (FPS) and a model size of only 17.56 MB. In addition, the root-mean-square error of positioning accuracy in the orchard test was 4.405 mm, satisfying the high-precision positioning requirements of the picking robot vision system in complex orchard environments. [ABSTRACT FROM AUTHOR]

This research paper presents a comprehensive investigation into the utilization of color image processing technologies and deep learning algorithms in the development of a robot vision system specifically designed for 8-ball billiards. The sport of billiards, with its various games and ball arrangements, presents unique challenges for robotic vision systems. The proposed methodology addresses these challenges through two main components: object detection and ball pattern recognition. Initially, a robust algorithm is employed to detect the billiard balls using color space transformation and thresholding techniques. This is followed by determining the position of the billiard table through strategic cropping and isolation of the primary table area. The crucial phase involves the intricate task of recognizing ball patterns to differentiate between solid and striped balls. To achieve this, a modified convolutional neural network is utilized, leveraging the Xception network optimized by an innovative algorithm known as the Improved Chaos African Vulture Optimization (ICAVO) algorithm. The ICAVO algorithm enhances the Xception network's performance by efficiently exploring the solution space and avoiding local optima. The results of this study demonstrate a significant enhancement in recognition accuracy, with the Xception/ICAVO model achieving remarkable recognition rates for both solid and striped balls. This paves the way for the development of more sophisticated and efficient billiards robots. The implications of this research extend beyond 8-ball billiards, highlighting the potential for advanced robotic vision systems in various applications. The successful integration of color image processing, deep learning, and optimization algorithms shows the effectiveness of the proposed methodology. This research has far-reaching implications that go beyond just billiards. The cutting-edge robotic vision technology can be utilized for detecting and tracking objects in different sectors, transforming industrial automation and surveillance setups. By combining color image processing, deep learning, and optimization algorithms, the system proves its effectiveness and flexibility. The innovative approach sets the stage for creating advanced and productive robotic vision systems in various industries. [ABSTRACT FROM AUTHOR]

The current research background of digital robot visual defect detection focuses on the application of virtual artificial intelligence algorithms. Convolutional neural networks (CNNS) perform well in the field of image processing and can learn and extract image features, which provides a more refined analysis ability for the visual defect detection of digital robots. Therefore, this paper focuses on the application of convolutional neural networks in digital robot vision defect detection, and explores the main processes of data collection and preprocessing, feature extraction, model training and defect detection in detail. Finally, the results of simulation experiments are as follows: Compared with traditional methods that rely on edge detection or threshold segmentation, the optimized defect detection accuracy rate is increased by 11.4%, and the image detection speed is increased by about 33.9%. The research results have important practical significance for improving the quality control of industrial manufacturing, which can not only improve the quality of products, but also adapt to the constant changes of the production line. [ABSTRACT FROM AUTHOR]

The robot video question-answering system is an artificial intelligence application that integrates computer vision and natural language processing technologies. Recently, it has received widespread attention, especially with the rapid development of large language models (LLMs). The core technical challenge lies in the application of visual question answering (VQA). However, visual question answering currently faces several challenges. Firstly, the acquisition of human annotations is costly, and secondly, existing models require expensive retraining when replacing a particular module. We propose the VLM2LLM model, which significantly improves the performance of multimodal question-answering tasks by integrating visual-language matching and large-scale language models. Specifically, it overcomes the limitations of requiring massive computational resources for training and inference in previous models. Furthermore, it allows for the upgrading of our LLM version according to the latest research advancements and needs. The results demonstrate that the VLM2LLM model achieves the highest accuracy compared to other state-of-the-art models on three datasets: QAv2, A-OKVQA, and OK-VQA. We hope that the VLM2LLM model can drive advancements in the field of robot video question-answering and provide innovative solutions for a wider range of application domains. [ABSTRACT FROM AUTHOR]

In the realm of video salient object detection (VSOD), the majority of research has traditionally been centered on third‐person perspective videos. However, this focus overlooks the unique requirements of certain first‐person tasks, such as autonomous driving or robot vision. To bridge this gap, a novel dataset and a camera‐based VSOD model, CaMSD, specifically designed for egocentric videos, is introduced. First, the SalEgo dataset, comprising 17,400 fully annotated frames for video salient object detection, is presented. Second, a computational model that incorporates a camera movement module is proposed, designed to emulate the patterns observed when humans view videos. Additionally, to achieve precise segmentation of a single salient object during switches between salient objects, as opposed to simultaneously segmenting two objects, a saliency enhancement module based on the Squeeze and Excitation Block is incorporated. Experimental results show that the approach outperforms other state‐of‐the‐art methods in egocentric video salient object detection tasks. Dataset and codes can be found at https://github.com/hzhang1999/SalEgo. [ABSTRACT FROM AUTHOR]

The 3D imaging-based model reconstruction technology is being applied in various aspects of fruit planting management, picking, transportation, and promoting fruit industry automation development. The fruit model based on 3D imaging technology is closer to the external characteristics of the fruit itself. More accurate data is obtained when the model is imported into a computer for simulation and analysis for growth prediction, quality grading, and mechanical damage. The reconstructed fruit models can also be used to calibrate the parameters of the robot's vision system, optimize the grasping strategy of the picking machinery, and improve the identification sorting and picking accuracy. Some articles have been published on the principles of 3D imaging technology, and there are also numerous studies on the practical application of fruit models. However, no review article exists on applying 3D reconstruction and modeling of fruits using 3D imaging techniques. This paper will focus on the acquiring and processing of 3D data and analyzing the application of imaging principles and technologies such as triangulation, time-of-flight(TOF), interferometry, electromagnetic radiation(EMR), and ultrasound imaging(US). Also, this paper summarizes the application of fruit models for quality assessment, mechanical property study, and picking machinery. Finally, the latest progress in the application of fruit models is prospected. [ABSTRACT FROM AUTHOR]

In recent industrial settings, time-of-flight (ToF) cameras have become essential tools in various applications. These cameras provide high-performance 3D measurements without relying on ambient lighting; however, their performance can degrade due to environmental factors such as temperature, humidity, and distance to the target. This study proposes a novel method to enhance the pixel-level sensing accuracy of ToF cameras by obtaining precise depth data labels in real-world environments. By synchronizing 3D simulations with the actual ToF sensor viewpoints, accurate depth values were acquired and utilized to train AI algorithms, thereby improving ToF depth accuracy. This method was validated in industrial environments such as automobile manufacturing, where the introduction of 3D vision systems improved inspection accuracy compared to traditional 2D systems. Additionally, it was confirmed that ToF depth data can be used to correct positional errors in mobile robot manipulators. Experimental results showed that AI-based preprocessing effectively reduced noise and increased the precision of depth data compared to conventional methods. Consequently, ToF camera performance was enhanced, expanding their potential applications in industrial robotics and automated quality inspection. Future research will focus on developing real-time synchronization technology between ToF sensor data and simulation environments, as well as expanding the AI training dataset to achieve even higher accuracy. [ABSTRACT FROM AUTHOR]

Panoptic segmentation plays a crucial role in enabling robots to comprehend their surroundings, providing fine-grained scene understanding information for robots' intelligent tasks. Although existing methods have made some progress, they are prone to fail in areas with weak textures, small objects, etc. Inspired by biological vision research, we propose a cascaded contour-enhanced panoptic segmentation network called CCPSNet, attempting to enhance the discriminability of instances through structural knowledge. To acquire the scene structure, a cascade contour detection stream is designed, which extracts comprehensive scene contours using channel regulation structural perception module and coarse-to-fine cascade strategy. Furthermore, the contour-guided multi-scale feature enhancement stream is developed to boost the discrimination ability for small objects and weak textures. The stream integrates contour information and multi-scale context features through structural-aware feature modulation module and inverse aggregation technique. Experimental results show that our method improves accuracy on the Cityscapes (61.2 PQ) and COCO (43.5 PQ) datasets while also demonstrating robustness in challenging simulated real-world complex scenarios faced by robots, such as dirty cameras and rainy conditions. The proposed network promises to help the robot perceive the real scene. In future work, an unsupervised training strategy for the network could be explored to reduce the training cost.

AbstractHuman following is a very useful task in the robotics industry. With modern compact-sized robots, there is a demand for further investigated computer-vision solutions that can perform effectively on them. A well-known deep learning model along this line of thought is the MobileNet-Ssd, an object detection model renowned for its resource-constrained usage. Available in popular frameworks like TensorFlow and PyTorch, this model can be of great use in deployments on robotic applications. This research attempts to investigate the MobileNet-Ssd model in order to evaluate its suitability for stand-alone object detection on a Raspberry Pi. To determine the effect of input size on the model, the model’s performance has been investigated with speed in frames-per-second across different input sizes on both CPU and GPU-powered devices. To evaluate the model’s effectiveness in the human following task, a Raspberry Pi-based robot was designed leveraging the tracking-by-detection approach with TensorFlow-Lite. Furthermore, the model’s performance was evaluated using PyTorch while the model’s inputs were adjusted, and the results were compared to those of other state-of-the-art models. The investigation revealed that, despite its modest speeds, the model outperforms other noteworthy models in PyTorch and is an ideal choice when working with Raspberry Pi using TensorFlow-Lite.

In response to the issue of high tomato yield, low efficiency in harvesting tomatoes grown in greenhouses, and low recognition accuracy of nighttime harvesting robots, a design was developed and a robotic system was created specifically for nighttime greenhouse tomato harvesting. The robot employs a vision system and YOLOv5+HSV fusion algorithm to recognize and locate tomatoes. It then transmits this information to the robotic arm. By coordinating the visual system, the robotic arm, the end effector, and the lifting mechanism, the robot accurately picks ripe tomatoes. The robot was subjected to simulated field tests for visual recognition and harvesting, both during daytime and nighttime conditions. The results showed that the success rate of nighttime harvesting was slightly lower than during the daytime but remained at a relatively high level. The daytime harvesting success rate and the average time to pick a single fruit were 87.78% and 15.99 seconds, respectively. The nighttime harvesting success rate and the average time to pick a single fruit were 87.55% and 17.26 seconds, respectively. This approach effectively improves the recognition accuracy and harvesting speed of the harvesting robot, reducing damage to tomatoes during harvesting, and addresses the issues of supplementary lighting and image noise reduction for nighttime harvesting robots. [ABSTRACT FROM AUTHOR]

One of the significant application of robot vision technique that serves in biometric verification systems is Face Recognition (FR) technology. However, the effectiveness of the model is affected by disturbance from the real-world environment, including alterations in lighting, facial occlusion, and fluctuation in poses. Even though the recognition of faces has gained popularity due to the variety of applications, it still remains a complicated task to recognize because of an enormous variety of biometric data features. In the past few years, FR technology has seen a lot of development in this area of biometric and computer vision-oriented applications. The most important steps in creating an accurate FR system are the extraction of features and the categorization of these extracted features. The conventional method of feature extraction includes frequency domain features or the Eigenface technique. However, they are not robust to altering external factors like occlusion, illumination, and posture. Using deep learning techniques, a unique approach to address this problem in FR is suggested to recognize faces successfully. The images are aggregated in the initial phase. It is then uploaded to the Viola–Jones Face Detector tools to find the face. The detected face images are then subjected to the Adaptive Deep Convolutional Generative Adversarial Network (ADCGAN) to remove the occlusion from the facial images, where the parameters are subsequently optimized using the Modified Random Variable-based Galactic Swarm Optimizations (MRV-GSO) algorithm. The occlusion-removed face images are then given to the Multi-Cascaded Attentive and Adaptive Deep Learning Network (MCADN) model, where the outputs from the Dilated DenseNet and Residual Network (ResNet) are serially passed to the Bidirectional Long-Short Term Memory (Bi-LSTM) model forming the MCDAN. The MRV-GSO algorithm is performed to tune the hyperparameters in the MCADN model to produce the final recognized facial images. As a result of conducting several experimental studies, it is proved that the generated model outperformed standard approaches in terms of effective recognition rate. [ABSTRACT FROM AUTHOR]

Deep learning based stereo matching algorithms have been extensively researched in areas such as robot vision and autonomous driving due to their promising performance. However, these algorithms require a large amount of labeled data for training and encounter inadequate domain adaptability, which degraded their applicability and flexibility. This work addresses the two deficiencies and proposes a few-shot trained stereo matching model with high domain adaptability. In the model, stereo matching is formulated as the problem of dynamic optimization in the possible solution space, and a multi-scale matching cost computation method is proposed to obtain the possible solution space for the application scenes. Moreover, an adaptive recurrent 3D convolutional neural network is designed to determine the optimal solution from the possible solution space. Experimental results demonstrate that the proposed model outperforms the state-of-the-art stereo matching algorithms in terms of training requirements and domain adaptability. [ABSTRACT FROM AUTHOR]

Stones are one of the primary objects that impede the normal activity of underground pipelines. As human intervention is difficult inside a narrow underground pipe, a robot with a machine vision system is required. In order to remove the stones during regular robotic inspections, precise stone detection, segmentation, and measurement of their distance from the robot are needed. We applied Mask R-CNN to perform an instant segmentation of stones. The distance between the robot and the segmented stones was calculated using spatial information obtained from a lidar camera. Artificial light was used for both image acquisition and testing, as natural light is not available inside the underground pipe. ResNet101 was chosen as the foundation of the Mask R-CNN, and transfer learning was utilized to shorten the training time. The experimental results of our model showed that the average detection precision rate reached 92.0; the recall rate was 90.0%; and the F1 score rate reached 91.0%. The distance values were calculated efficiently with an error margin of 11.36 mm. Moreover, the Mask R-CNN-based stone detection model can detect asymmetrically shaped stones in complex background and lighting conditions. [ABSTRACT FROM AUTHOR]

A road multi‐object detection algorithm is one of the core algorithms for intelligent road cleaning robots relying on machine vision. Most existing object detection algorithms analyse all image regions and finally calculate the category and location of each object. However, it is not necessary to analyse all areas of the image when detecting objects on the road surface where the background changes little, and the number of objects is small because there will be a lot of invalid calculations. If we can perform targeted local analysis on images instead of analysing all image regions, it will improve the detection efficiency. Therefore, this paper proposes a multi‐object detection method using a binocular camera and a convolutional neural network (CNN) that effectively reduces invalid calculations during the detection and improves detection efficiency. In the developed method, the binocular vision image acquired by the binocular camera is stereo matched and equalized, while linear regression and coordinate transformation eliminate the angle of the camera pair concerning the road surface. Then, the coordinates of the regions of interest (ROI) is calculated in the left vision image and the features within the ROI is extracted from the corresponding CNN's feature map. Next, ROI pooling resizes the extracted feature maps of different sizes to the same size, which are then input to the fully connected layers to output the results. The proposed binocular network and faster R‐CNN (VGG16) are trained and tested on a dataset involving 1000 road waste images. The experimental results demonstrate that the developed binocular network improves the detection accuracy and speed by 28.56% and 78.39%, respectively, compared with faster R‐CNN (VGG16), providing a reliable basis for a machine vision‐based intelligent road cleaning robot. [ABSTRACT FROM AUTHOR]

Sorting unreelable inferior cocoons during the reeling process is essential for obtaining high-quality silk products. At present, silk reeling enterprises mainly rely on manual sorting, which is inefficient and labor-intensive. Automated sorting based on machine vision and sorting robots is a promising alternative. However, the accuracy and computational complexity of object detection are challenges for the practical application of automatic sorting, especially for small stains of inferior cocoons in images of densely distributed cocoons. To deal with this problem, an efficient fine-grained object detection network based on attention mechanism and multi-scale feature fusion, called AMMF-Net, is proposed for inferior silkworm cocoon recognition. In this model, fine-grained object features are key considerations to improve the detection accuracy. To efficiently extract fine-grained features of silkworm cocoon images, we designed an efficient hybrid feature extraction network (HFE-Net) that combines depth-wise separable convolution and Transformer as the backbone. It captures local and global information to extract fine-grained features of inferior silkworm cocoon images, improving the representation ability of the network. An efficient multi-scale feature fusion module (EMFF) is proposed as the neck of the object detection structure. It improves the typical down-sampling method of multi-scale feature fusion to avoid the loss of key information and achieve better performance. Our method is trained and evaluated on a dataset collected from multiple inferior cocoons. Extensive experiments validated the effectiveness and generalization performance of the HFE-Net network and the EMFF module, and the proposed AMMF-Net achieved the best detection results compared to other popular deep neural networks. [ABSTRACT FROM AUTHOR]

This paper presents a visual compass method utilizing global features, specifically spherical moments. One of the primary challenges faced by photometric methods employing global features is the variation in the image caused by the appearance and disappearance of regions within the camera's field of view as it moves. Additionally, modeling the impact of translational motion on the values of global features poses a significant challenge, as it is dependent on scene depths, particularly for non-planar scenes. To address these issues, this paper combines the utilization of image masks to mitigate abrupt changes in global feature values and the application of neural networks to tackle the modeling challenge posed by translational motion. By employing masks at various locations within the image, multiple estimations of rotation corresponding to the motion of each selected region can be obtained. Our contribution lies in offering a rapid method for implementing numerous masks on the image with real-time inference speed, rendering it suitable for embedded robot applications. Extensive experiments have been conducted on both real-world and synthetic datasets generated using Blender. The results obtained validate the accuracy, robustness, and real-time performance of the proposed method compared to a state-of-the-art method. [ABSTRACT FROM AUTHOR]

In order to address the challenge of non-destructive detection of tomato fruit ripeness in controlled environments, this study proposed a real-time instance segmentation method based on the edge device. This method combined the principles of phenotype robots and machine vision based on deep learning. A compact and remotely controllable phenotype detection robot was employed to acquire precise data on tomato ripeness. The video data were then processed by using an efficient backbone and the FeatFlowNet structure for feature extraction and analysis of key-frame to non-key-frame mapping from video data. To enhance the diversity of training datasets and the generalization of the model, an innovative approach was chosen by using random enhancement techniques. Besides, the PolyLoss optimization technique was applied to further improve the accuracy of the ripeness multi-class detection tasks. Through validation, the method of this study achieved real-time processing speeds of 90.1 fps (RTX 3070Ti) and 65.5 fps (RTX 2060 S), with an average detection accuracy of 97% compared to manually measured results. This is more accurate and efficient than other instance segmentation models according to actual testing in a greenhouse. Therefore, the results of this research can be deployed in edge devices and provide technical support for unmanned greenhouse monitoring devices or fruit-picking robots in facility environments. [ABSTRACT FROM AUTHOR]

Mobile service robots employ vision systems to discern objects in their workspaces for navigation or object detection. The lighting conditions of the surroundings affect a robot's ability to discern and navigate in its work environment. Robot inclusivity principles can be used to determine the suitability of a site's lighting condition for robot performance. This paper proposes a novel framework for autonomously auditing the Robot Inclusivity Index of indoor environments based on the lighting condition (RII-lux). The framework considers the factors of light intensity and the presence of glare to define the RII-Lux of a particular location in an environment. The auditing framework is implemented on a robot to autonomously generate a heatmap visually representing the variation in RII-Lux of an environment. The applicability of the proposed framework for generating true-to-life RII-Lux heatmaps has been validated through experimental results. [ABSTRACT FROM AUTHOR]

Wheeled robots play a crucial role in driving the autonomy and intelligence of robotics. However, they often encounter challenges such as tracking loss and poor real-time performance in low-texture environments. In response to these issues, this research proposes a real-time localization and mapping algorithm based on the fusion of multiple features, utilizing point, line, surface, and matrix decomposition characteristics. Building upon this foundation, the algorithm integrates multiple sensors to design a vision-based real-time localization and mapping algorithm for wheeled robots. The study concludes with experimental validation on a two-wheeled robot platform. The results indicated that the multi-feature fusion algorithm achieved the highest average accuracy in both conventional indoor datasets (84.57%) and sparse-feature indoor datasets (82.37%). In indoor scenarios, the vision-based algorithm integrating multiple sensors achieved an average accuracy of 85.4% with a processing time of 64.4 ms. In outdoor scenarios, the proposed algorithm exhibited a 14.51% accuracy improvement over a vision-based algorithm without closed-loop detection. In summary, the proposed method demonstrated outstanding accuracy and real-time performance, exhibiting favorable application effects across various practical scenarios. [ABSTRACT FROM AUTHOR]