Search

Owing to human labor shortages, the automation of labor-intensive manual waste-sorting is needed. The goal of automating waste-sorting is to replace the human role of robust detection and agile manipulation of waste items with robots. To achieve this, we propose three methods. First, we provide a combined manipulation method using graspless push-and-drop and pick-and-release manipulation. Second, we provide a robotic system that can automatically collect object images to quickly train a deep neural–network model. Third, we provide a method to mitigate the differences in the appearance of target objects from two scenes: one for dataset collection and the other for waste sorting in a recycling factory. If differences exist, the performance of a trained waste detector may decrease. We address differences in illumination and background by applying object scaling, histogram matching with histogram equalization, and background synthesis to the source target-object images. Via experiments in an indoor experimental workplace for waste-sorting, we confirm that the proposed methods enable quick collection of the training image sets for three classes of waste items (i.e., aluminum can, glass bottle, and plastic bottle) and detection with higher performance than the methods that do not consider the differences. We also confirm that the proposed method enables the robot quickly manipulate the objects.

Visual perception and modelling are essential tasks in many vision-dependent robotic tasks. This work presents a robotic vision system to perform fruit recognition, modelling, and environment modelling for autonomous apple harvesting. The fruit recognition applies a deep-learning model Dasnet to perform detection and segmentation on fruits, and segmentation on branches. Fruit modelling localises the centre and computes the grasp pose of each fruit based on Hough Transform. Environment modelling adopts Octrees to represent the occupied space within the working environment of the robot. The robot control computes the path and guide manipulator to pick the fruits based on the computed 3D model of the crop. The developed method is tested in both laboratory and orchard environments. Test results show that fruit recognition and modelling algorithm can accurately localise the fruits and compute the grasp pose in various situations. The Dasnet achieves 0.871 on F1 score of the fruit detection. Fruit modelling achieves 0.955 and 0.923 on the accuracy of the fruit centre estimation and grasp orientation, respectively. To illustrate the efficiency of the vision system in autonomous harvesting, a robotic harvesting experiment by using industry robotic arm in a controlled environment is conducted. Experimental results show that the proposed visual perception and modelling can efficiently guide the robotic arm to perform the detachment and success rate of harvesting is improved compared to the method which does not compute the grasp pose of fruits.

The problem discussed in this paper is how to create an integrated test automation setup, in which the test engineer does not have to use a pendant to communicate with the robotic arm and a separate test script to trigger on-device test automation for testing computer vision algorithms, such as motion-based image capture. There is no communication between the pendant to control the robotic arm and the test script that launches the on-device test automation for the motion-based image capture system executing on a mobile phone. Therefore, the test engineer needs to manually trigger the programs for the robot pendant to start the robotic arm, the automation, and launch times to keep them in sync. The solution to this problem is to design a test automation program using a middleware software (such as ORiN2 middleware software provided by Denso Robotics) to bypass the dependency on the robotic pendant, which must be manually started. The programmatic usage of the ORiN2 software also helps to establish a communication between the lab computer and a robot controller. The controller access object is used by the ORiN2 middleware software to communicate with the robotic controller of a Denso robot. It directly controls the robotic arm motion by accessing the programming variables. The image capture accuracy is calculated based on the number of snapshots and the similarity of the snapshots taken if the mobile phone under test is stationary and a motion is injected in the test subject. We compared the test results of the hardware-accelerated solution with a third-party solution, and found that the hardware accelerated solution is 12% more accurate for image capture accuracy at a medium panning speed. This paper discusses the integrated test automation framework for testing motion-based image capture system using a robotic arm.

Fiducial markers are commonly used to localize robots. Most existing systems use standard cameras to detect simple patterns on flat tags. Since depth is not directly sensed, the pose must be inferred from the tag geometry. These systems are low-cost and easy to implement on robotic systems, but their performances suffer in low-light conditions and on computationally constrained processors. We propose using 3D light detection and ranging (LiDAR) scanners to mitigate these issues. The reflectivity measurement provided by most LiDAR sensors provides a simple way to discern geometric differences on surfaces. We utilize this fact to create a custom “beacon” with reflective fiducials. Next, we design a high-performance segmentation and localization algorithm to find the 2D pose of a mobile robot. Our experiments proved that our system achieves an average euclidean error of less than 0.063 m at ranges of over 10 m while maintaining a runtime of under 3 ms on a basic single board computer. Additionally, our system is highly occlusion resistant. These results are confirmed with multiple field tests of the system.

The 3-D reconstruction involves the construction of a 3-D model from a set of images. The 3-D reconstruction has varied uses that include 3-D printing, the generation of 3-D models that can be shared through social media, and more. The 3-D reconstruction involves complex computations in mobile phones that must determine the pose estimation. The pose estimation involves the process of transforming a 2-D object into 3-D space. Once the pose estimation is done, then the mesh generation is performed using the graphics processing unit. This helps render the 3-D object. The competitive advantages of using hardware processors are to accelerate the intensive computation using graphics processors and digital signal processors. The stated problem that this technical paper addresses is the need for a reliable automated test for the 3-D reconstruction feature. The solution to this problem involved the design and development of an automated test system using a programmable robotic arm and rotor for precisely testing the quality of 3-D reconstruction features. The 3-D reconstruction testing involves using a robotic arm lab to accurately test the algorithmic integrity and end-to-end validation of the generated 3-D models. The robotic arm can move the hardware at different panning speeds, specific angles, fixed distances from the object, and more. The ability to reproduce the scanning at a fixed distance and the same panning speed helps to generate test results that can be benchmarked by different software builds. The 3-D reconstruction also requires a depth sensor to be mounted onto the device under examination. We use this robotic arm lab for functional, high performance, and stable validation of the 3-D reconstruction feature. This paper addresses the computer vision use case testing for 3-D reconstruction features and how we have used the robotic arm lab for automating these use cases.

Touch-to-track is a software feature that is used to track objects in embedded systems, such as mobile phones. The research question that is discussed in the paper is as follows: how can one design an automated test method for accurately testing object tracking algorithms? The challenge for the verification team was to design a method of test automation for testing this object tracking feature. The solution to the problem was to develop test automation algorithms using a robotic arm to accurately test this software feature. The robotic arm is used for executing test cases for tracking single or multiple objects in motion. To use the touch-to-track feature, the user selects an area in the camera preview by drawing a bounding box. Hough transformation and color segmentation are applied to accurately detect the bounding box drawn on every frame in which the object is detected. The area inside the bounding box is considered as the template. Template matching algorithms based on normalized cross-correlation, phase correlation, and speeded up robust features (SURF)-based feature extraction and matching are then applied to compare the template and original images. The tracking efficiency is calculated by dividing the total number of frames that contain the object being tracked by the total number of frames being tested. The tracking efficiencies achieved with the Hough-based bounding box detection algorithm followed by template matching algorithms based on normalized cross-correlation, phase correlation, and SURF-based feature extraction and matching are compared. These tracking efficiencies are calculated by comparing the ground truth to the tracking results for each frame. The tracking efficiency calculated for three objects is 87.7% with no colored background. The proposed object tracking solution with hardware acceleration is found to perform better than the third-party solution with respect to the latency in re-establishing tracking and the hand jitter scenario.

This paper outlines the numerous interdisciplinary activities required to implement two ultraviolet (UV) epoxy-based automated micro-optic assembly tools within a “smart factory” for high-power semiconductor laser manufacturing. After a brief overview of the motivation and approach, two examples of automated micro-optic assembly tools [e.g., fast-axis collimation (FAC) lens attach and mirror stripe alignment] for kilowatt-class laser-diode arrays and modules are given that demonstrate the breadth of automation activities required within a modern factory. For each system, the complete assembly sequence is outlined, and the steps that presented fully automated challenges are highlighted. In contrast, to the previous published results that focused on the details of the automated micro-optic alignment, the largest challenges primarily involved automation and repeatability of the UV-epoxy dispense and attach processes for these high-power semiconductor devices. These were solved by precise and stable mechanical design, customized lighting, and machine vision solutions. Finally, the complete integration of the tools into a “smart-factory” system is outlined. [ABSTRACT FROM AUTHOR]

Recent robotics applications require 3-D representations of the environments. In many cases, it is not feasible for a single robot to map the entire environment. In these cases, it is necessary for a team of robots to build maps independently and merge them into a single global map. In this paper, octree-based occupancy grids, which are currently the state-of-the-art 3-D map representation, are applied to the problem of multirobot mapping. Octrees allow large environments to be mapped efficiently, in terms of memory usage, while still providing sufficiently fine resolution where required. The main contribution of this work lies in the definition and validation of a system, which use map data from commonly mapped portions of the environment with registration techniques, such that maps are merged coherently despite measurement noise and error in the relative transformations between maps for experimental data sets. The system defined can then be used in a complete solution that is ported to mobile robots. The results demonstrate that octree occupancy grids are a suitable representation for multirobot 3-D mapping, but that the proposed techniques for improving erroneous transformation estimates between map frames allow multiple maps to be merged efficiently and robustly. [ABSTRACT FROM PUBLISHER]

The constraint force algorithm, as originally described by Fijany et al. (1995), calculates the forward dynamics of a system comprising N rigid bodies connected together in an unbranched chain with joints from a restricted class of joint types. It was designed for parallel calculation of the dynamics, and achieves O(log N) time complexity on O(N) processors. This paper presents a new formulation of the constraint force algorithm that corrects a major limitation in the original, and sheds new light on the relationship between this algorithm and other dynamics algorithms. The new version is applicable to systems with any type of joint, floating bases, and short branches off the main chain. It is obtained using a new technique for analysing constrained rigid-body systems by means of a change of basis in a dual system of vector spaces. This new technique is also described. [ABSTRACT FROM PUBLISHER]

A novel vision-based method for the circle pose determination is addressed. This method is based on two particular projected chords of a circle image. The first one is the projection of a circle chord which subtends the largest apex angle of the viewing cone for the circle image and the second one is the projection of a circle diameter whose back-projection plane bisects the above largest apex angle. This method is conceptually simple, since the circle center is the center point of the diameter chord and the circle orientation is given by the cross product of these two (directed) chords. We present theorems on the geometry of both the viewing cone and a re-projected circle image which are essential to the pose determination. We then give a pose determination method which is applicable under all viewing conditions. Experimental results illustrate the good performance of the method, when compared with other existing methods. [ABSTRACT FROM PUBLISHER]

A computer vision assisted semi-automatic virtual reality (VR) calibration technology has been developed that can accurately match a virtual environment of graphically simulated 3D models to the video images of the real task environment. We have developed a new 20-variable weighted least-squares algorithm that updates both camera and object models simultaneously for given two camera views of two mating objects. This simultaneous update enables accurate model matching even with rough, approximate initial camera calibrations. The developed semi-automatic VR calibration supports automated intermediate updates, eliminating nearly all operator interaction except for initial coarse matching. In our quasi-static supervisory telerobotic applications, intermediate VR calibrations are performed intermittently at a few robot stopping poses only, as a cost-effective and safer approach compared to real-time visual servoing. Extensive experimental results comparing alignment errors under various viewing conditions are presented. [ABSTRACT FROM PUBLISHER]

We present an integrated system in which an operator uses a simulated environment to program part-mating and contact tasks. Generation of models within this virtual environment is facilitated using a fast, occlusion tolerant, 3D grey-scale vision system which can recognize and accurately locate objects within the work site. A major goal of this work is to make robotic programming easy and intuitive for untrained users working with standard desktop hardware. Simulation offers the ease-of-use benefits of "programming by demonstration", coupled with the ability to create a programmer-friendly virtual environment. Within a simulated environment, it is also straightforward to track and interpret an operator's actions. The simulator models objects as polyhedra and implements full 3D contact dynamics. When a manipulation task is completed, local planning techniques are used to turn the virtual environment's motion sequence history into a set of robot motion commands capable of realizing the prescribed task. [ABSTRACT FROM PUBLISHER]

This paper addresses the problem of finding whether a given set of three-dimensional (3D) vectors (the object) can be brought to match a second set of vectors (the template) by means of an affine motion, minimizing a measure of the mismatch error and satisfying an assigned set of geometrical constraints. This problem is encountered in many applications of computer vision, robotics, and manufacturing processes, and has been tackled by several authors in the unconstrained case. Spherical, ellipsoidal and polyhedral constraints are here introduced in the problem, and a solution scheme based on an efficient convex optimization algorithm is proposed. An example of application of the proposed methodology to a manufacturing tolerancing problem is also provided. [ABSTRACT FROM PUBLISHER]

The 3-D reconstruction of visible polyhedral faces from a pair of general perspective views with the aid of a calibration plate is addressed. A polyhedron is placed on a planar calibration plate and two side views of both the polyhedron and the calibration plate are taken. Through proper arrangements we may assume that in the two views a number of polyhedral edges lying on the calibration plate and the whole calibration plate boundary are visible. We present an online camera calibration technique with the aid of the calibration plate and a two-stage process to find the vertex/edge correspondences without encountering the ambiguity problem of the conventional epipolar line technique. We then give a closed form solution to the 3-D polyhedral vertices visible in both images. We also describe other advantages of using our method for the 3-D polyhedron reconstruction. Experimental results show that the obtained 3-D polyhedral vertex coordinates are rather accurate. [ABSTRACT FROM PUBLISHER]