Search

Human-robot collaboration with traditional industrial robots is a cardinal step towards agile manufacturing and re-manufacturing processes. These processes require constant human presence, which results in lower operational efficiency based on current industrial collision avoidance systems. The work proposes a novel local and global sensing framework, which discusses a flexible sensor concept comprising a single 2D or 3D LiDAR while formulating occlusion due to the robot body. Moreover, this work extends the previous local global sensing methodology to incorporate local (co-moving) 3D sensors on the robot body. The local 3D camera faces toward the robot occlusion area, resulted from the robot body in front of a single global 3D LiDAR. Apart from the sensor concept, this work also proposes an efficient method to estimate sensitivity and reactivity of sensing and control sub-systems The proposed methodologies are tested with a heavy-duty industrial robot along with a 3D LiDAR and camera. The integrated local global sensing methods allow high robot speeds resulting in process efficiency while ensuring human safety and sensor flexibility.

Manual annotation for human action recognition with content semantics using 3D Point Cloud (3D-PC) in industrial environments consumes a lot of time and resources. This work aims to recognize, analyze, and model human actions to develop a framework for automatically extracting content semantics. Main Contributions of this work: 1. design a multi-layer structure of various DNN classifiers to detect and extract humans and dynamic objects using 3D-PC preciously, 2. empirical experiments with over 10 subjects for collecting datasets of human actions and activities in one industrial setting, 3. development of an intuitive GUI to verify human actions and its interaction activities with the environment, 4. design and implement a methodology for automatic sequence matching of human actions in 3D-PC. All these procedures are merged in the proposed framework and evaluated in one industrial Use-Case with flexible patch sizes. Comparing the new approach with standard methods has shown that the annotation process can be accelerated by 5.2 times through automation.

Das Softwaretool „SafeZone“ erlaubt die effiziente und zuverlässige Berechnung von Sicherheitsbereichen (Safe Bereichen) für komplexe schutzzaunlose Roboteranlagen, zum Beispiel im Karosseriebau und der Endmontage. Unter Berücksichtigung der realen Bewegungsbahnen und Geschwindigkeiten werden – im Gegensatz zum Status Quo – nur dort Gefahrenbereiche berechnet, wo auch Gefahren entstehen können. Das spart Platz und lässt Mensch sowie Roboter sicher näher zusammenrücken. The „SafeZone“ software tool enables the efficient and reliable calculation of safety areas (safe zones) for complex robot systems without protective fences, such as car body construction and final assembly. Considering real motion paths and speeds, safe zones are calculated – in contrast to the status quo – only where hazards can occur. This saves space and allows humans and robots to move closer together safely.

Most motion planners generate trajectories as low-level control inputs, such as joint torque or interpolation of joint angles, which cannot be deployed directly in most industrial robot control systems. Some industrial robot systems provide interfaces to execute planned trajectories by an additional control loop with low-level control inputs. However, there is a geometric and temporal deviation between the executed and the planned motions due to the inaccurate estimation of the inaccessible robot dynamic behavior and controller parameters in the planning phase. This deviation can lead to collisions or dangerous situations, especially in heavy-duty industrial robot applications where high-speed and long-distance motions are widely used. When deploying the planned robot motion, the actual robot motion needs to be iteratively checked and adjusted to avoid collisions caused by the deviation between the planned and the executed motions. This process takes a lot of time and engineering effort. Therefore, the state-of-the-art methods no longer meet the needs of today’s agile manufacturing for robotic systems that should rapidly plan and deploy new robot motions for different tasks. We present a data-driven motion planning approach using a neural network structure to simultaneously learn high-level motion commands and robot dynamics from acquired realistic collision-free trajectories. The trained neural network can generate trajectory in the form of high-level commands, such as Point-to-Point and Linear motion commands, which can be executed directly by the robot control system. The result carried out in various experimental scenarios has shown that the geometric and temporal deviation between the executed and the planned motions by the proposed approach has been significantly reduced, even if without access to the “black box” parameters of the robot. Furthermore, the proposed approach can generate new collision-free trajectories up to 10 times faster than benchmark motion planners.

Low cost 3D LiDAR complements cameras in perception algorithms for various robotic applications. Safe and efficient human robot collaboration requires easy and accurate extrinsic calibration of sensors with an industrial robot. This work presents an efficient and accurate method for extrinsic calibration between LiDAR, camera and an industrial robot. Open-box target mounted on robot enables parameter estimation by constraining sensor data to multiple planes, which constitute the target planner surface. The method enables extraction of eight correspondences between the sensors and the robot in each data sample. The method enables speedup in sensor setup and drastically reduces efforts required for data collection through automation. The results have been evaluated for a simulated and real environment.

In the era of Industry 4.0 and agile manufacturing, the conventional methodologies for risk assessment, risk reduction, and safety procedures may not fulfill the End-User requirements, especially the SMEs with their product diversity and changeable production lines and processes. This work proposes a novel approach for planning and implementing safe and flexible Human-Robot-Interaction (HRI) workspaces using multilayer HRI operation modes. The collaborative operation modes are grouped in different clusters and categorized at various levels systematically. In addition to that, this work proposes a safety-related finite-state machine for describing the transitions between these modes dynamically and properly. The proposed approach is integrated into a new dynamic risk assessment tool as a promising solution toward a new safety horizon in line with industry 4.0.

Industrial robots and cobots are widely deployed in most industrial sectors. However, robotic programming still needs a lot of time and effort in small batch sizes, and it demands specific expertise and special training, especially when various robotic platforms are required. Actual low-code or no-code robotic programming solutions are exorbitant and meager. This work proposes a novel approach for no-code robotic programming for end-users with adequate or no expertise in industrial robotic. The proposed method ensures intuitive and fast robotic programming by utilizing a finite state machine with three layers of natural interactions based on hand gesture, finger gesture, and voice recognition. The implemented system combines intelligent computer vision and voice control capabilities. Using a vision system, the human could transfer spatial information of a 3D point, lines, and trajectories using hand and finger gestures. The voice recognition system will assist the user in parametrizing robot parameters and interacting with the robot’s state machine. Furthermore, the proposed method will be validated and compared with state-of-the-art “Hand-Guiding” cobot devices within real-world experiments. The results obtained are auspicious, and indicate the capability of this novel approach for real-world deployment in an industrial context.

Compared to robots, human pathways are individually changeable. Particularly the human-robot-collaboration (HRC) with heavy-duty robots carry high risk potentials. To ensure safety and efficiency, the human variance in pathways should be determined in the planning of HRC. Hence this paper presents a method to investigate human pathway variations in a real study. In systematically selected tasks, human pathways are examined for 100 participants in manual and human-robot collaboration (HRC) scenarios. As a result, the variations of pathways are presented depending on various features: in nearly all cases the variance of women’s walking pathways is smaller than that of men.

Shorter product lifecycles, mass individualization and agile concepts for lot size1 require more flexible production. Constantchallenges may result in optimized processes. However, opimization effects not only on the production technologies but also on theoperators and their ergonomics. A simulation based ergonomic evaluation in such hybrid workspaces is tough and fruitless due toconstantly changing in the production processes and to the diversity of the human activities. An online ergonomics assessment andreal-time feedback ensures that employees improve the quality of their workplace and maintain healthy working conditions. This workproposes a method for agile production or traditional factory floors using a marker less camera-based system using real-time jointposition estimations as input and providing a continuous or summarized individual ergonomic feedback as output. Furthermore, firstimplementation results considering the requirements and overall system design for online ergonomic evaluation in agile productionenvironment are discussed.

Even by providing benefits addressing the main megatrend in today's manufacturing industries human-robot interaction (HRI) still lacks reasonably applied use cases, especially in small and medium-sized enterprises (SMEs). Most SMEs worry about complicated standards they have to consider when letting their employees work with a robot. Furthermore, they simply do not know where to start due to multiple promised benefits not precisely matching their requirements. Since they have individual starting points regarding their needs, available process data and documentation, it is expensive to obtain a customised evaluation of provided benefits. Thereby, the main challenge is to offer a consistent approach dealing with multiple requirements. The multi-layer concept of service modelling provides a possibility to resolve this contradiction. It aims for an overall goal by achieving predefined subgoals with individually matching methods. Thus, each manufacturing company can choose the most suitable methodology to analyse its production line. Regarding HRI it is possible to select workstations with the best fitting solution according to individual requirements. The selection will be quantified based on analyses about provided values of HRI in manufacturing. This study presents the service modelling approach for the evaluation of workstations that are best suited for the implementation of HRI in SME.

We present a general approach for realizing safe collaborative behavior and high-accuracy operations in industrial robots. For example, collaborative behavior is required in human robot collaboration, whereas high-accuracy is needed in robotic machining. The operation sequence of a robot with corresponding often-contradictory demands on safety functions, control mode and accuracy is specified in a task-oriented path planner. The task planner generates optimized reference trajectories based on an environmental model, kinematic and task-specific constraints. The resulting reference path is fed into an online operation management, which monitors the workspace, ensures safety, and realizes different online control strategies especially for the dynamic tasks such as human robot collaboration or high-precision operations. For the high-accuracy tasks, we implemented a model predictive controller using a mechatronic model for predicting future configurations of the robot based on the measured present configuration. The model-predictive control is demonstrated for the compensation of deviations due to external forces that acts on the end effector of the robot.

Implementation of safe and efficient human robot collaboration for agile production cells with heavy-duty industrial robots, having large stopping distances and large self-occlusion areas, is a challenging task. Collision avoidance is the main functionality required to realize this task. In fact, it requires accurate estimation of shortest distance between known (robot) and unknown (human or anything else) objects in a large area. This work proposes a selective fusion of global and local sensors, representing a large range 360° LiDAR and a small range RGB camera respectively, in the context of dynamic speed and separation monitoring. Safety functionality has been evaluated for collision detection between unknown dynamic object to manipulator joints. The system yields 29-40% efficiency compared to fenced system. Heavy-duty industrial robot and a controlled linear axis dummy is used for evaluating different robot and scenario configurations. Results suggest higher efficiency and safety when using local and global setup.

Unlike robots, humans do not have constant movements. Their pathways are individually changeable and influenced by circumstances. This paper presents a method to investigate human pathway variations in a real study. In systematically selected tasks, human pathways are examined for 100 participants in manual and human-robot collaboration (HRC) scenarios. As a result, the variations of pathways are presented depending on various features: e.g. in nearly all cases the variance of women’s walking pathways is smaller than that of men. VariPath database can be used in any planning process of manual or HRC scenarios to ensure safety and efficiency.

The need for cooperation between humans and industrial robots is in exponential increase, especially in production applications. However, human safety is the main concern, preventing any fenceless cooperation between humans and industrial robots. This paper presents elements of new strategy for ensuring human safety during various levels of interaction with heavy-load industrial robots. The proposed approach classifies the human–robot interaction (HRI) into four levels. In every level, different kinds of safety functions are developed and analyzed. An additional algorithm has been developed for classifying the dangerous during the interaction. The proposed approach is tested and analyzed on a HRI platform.

Importance of robots in industrial applications is a well-known fact. There are many tasks in industries, which can't be done alone by a robot or alone by a worker. Therefore, there is a need to establish a reliable and safe interaction environment between robots and workers. To do so, some information about the worker should be conveyed to the robot. This article focuses on industrial Human-Robot Interaction. For a safe and efficient Human-Robot Interaction, a robot needs to know about worker's position, posture and gesture. This paper proposes integration of an Inertial Measurement Unit (IMU) and a 3D camera as an image sensor to eliminate each other drawbacks and applies machine learning algorithm to detect postures and gestures of worker. Some experimental results during the interaction with heavy-duty robot will be presented

Actually, vision information can be used as sensory input in open-loop as well as in the closed-loop. However, the visual servoing approach is performed only inside the closed-loop robot control, because in the open-loop the vision sensor will represent the initial extraction of the features to generate directly the robot motion sequence and these features and motion could be off-line generated. On the contrast, closed-loop robot system uses the vision as real time sensor and it consists of two phases: tracking and control. Tracking provides a continuous estimation and update of features during the robot/object motion. Based on this information, a real time control loop will be generated. The main contribution of this work is a proposed visual servoing approach which will benefit from the images which are obtained by Kinect camera (RGB-D camera). The proposed visual servoing approach is called 4 × 2D visual servoing which combines the correspondent color and depth images to build two new images. Using these 4 images the control error signals will be calculated in order to track the objects. Firstly, this chapter will present all types of visual servoing then it will introduce the 4 × 2D visual servoing approach and the visible side coordinate system, after that it will illustrate the concept of the proposed approach and how the error signal will be calculated. In addition to that, this approach proposes a coordinate system which is called visible side coordinate system.

Human-Robot shared workspace is a dynamic and unstructured environment. In such environment, pre-programmed robot paths might cause collisions or production disruptions. In order to solve this problem, motion planning framework has been proposed that adapts the robot's movement (path and speed)according to the human movement or any other dynamic obstacles in real time. Firstly, it defines the safety distance between robot and human or other obstacles. During run-time the 3D-Smart-Sensors capture the current position of human and other dynamic/static obstacles in human-robot shared workspace. The proposed framework plans and optimizes collision free robot trajectories with consideration for safety distance, path length and executing time. Once a new optimal trajectory is found, the framework controls the robot to adjust its movement paths. Moreover, the proposed framework can adjust the robot velocity based on the 3D-Zone Model of human-robot shared workspace. Therefore, the robot can reach its goal quickly and safely in a dynamic and unstructured environment.

Even by providing benefits as a solution for increasing flexibility in manufacturing human-robot-interaction (HRI) still lacks reasonable industrial use cases, especially in small and mid-sized enterprises (SME). Most SME are confused due to the promised multiple benefits not matching their specific requirements. Furthermore, they do have very individual starting points regarding their needs, available process data and documentation. However, there had been analyses on different benefits of HRI for manufacturing and methods to identify individual motivation. The multi-layer approach of service modelling provides a methodology to fulfil an overall goal by achieving predefined subgoals with different methods. Thereby each manufacturing company can choose the most suitable methodology to analyze its production line. Regarding HRI it is possible to select workstations with the most suited benefits for the individual requirements. In this paper, a service modeling approach for the selection of workstations for an implementation of HRI is presented.

This paper deals with establishing of the electrical connection between a plug and a receptacle by a robot manipulator for the purpose of charging electrical vehicles. In general, the task of the robot for automatic charging of vehicles consists of two phases. In the first phase, the robot system defines the position of the charging receptacle of the vehicle using vision or infrared system. After that in the next phase, it starts to interact with the environment by connecting the charger plug to the charging receptacle (socket) of the vehicle. However, this phase is not always performed successfully, especially when the socket has complicated shape or consists of multi cores with different sizes. In this paper we will use robot force control to build up the connection. Additionally, an algorithm will proposed which improves the peg-in-hole task by generating spiral motion. The proposed algorithm has shown promising results performed on 5-pin industrial charger plug which is very hard to peg in the socket, even for the human, because it is secure and weatherproof (the plug should cover the whole socket cavity), moreover it has multi cores (5 pins) and it is provided with multiple notches to avoid mismatching between similar pins. In addition to that, the proposed algorithm has assumed that a small vision error could be occurred during estimating the initial position of vehicle's receptacle.

This paper presents an innovative human robot interaction (HRI) concept for increasing the automation level during the disassembly of unknown electric vehicle motors. The proposed system is based on a robot workstation which is able to cooperate and to learn from a supervisor by using vision and force sensors. The proposed algorithm consists of two phases: 1. Teaching phase: the robot can memorize cyclic subtasks based on safe and active physical interaction with a human. 2. Cognition phase: the robot can detect the screws and match them with the previously taught screws in order to define the type and position of the electric motor. Furthermore, the robot repeats the disassembly task independently and can modify its path according to the measured forces/moments.

In this investigation a unique platform has been developed for observing and evaluating various levels of interaction between humans and large industrial robots, starting from the low level of interaction and ending up with the most sophisticated physical interaction between human and robot. The proposed platform complies with the industrial requirements and represents a similar environment to the production lines. In addition, it respects the ISO standards concerning the human robot interaction (HRI) applications, e.g. ISO 10218-1/2:2011 and ISO/TS 15066.

Human robot shared workspace is usually divided into different zones, e.g. safe, warning, danger zones etc. In general, this kind of division could ensure the human safety during interaction with the industrial robots. However, the productivity, the efficiency and the diversity of the shared tasks have been overtaken. This work has proposed zone-based robot control based on 3D- advanced selection matrices. In every zone, every robot subspace could be vision-, force-, position or hybrid-controlled depending on the required tasks and the safety requirements. Using the proposed system, the industrial robots can perform various shared tasks with the human, including different control algorithms and safety procedures. The proposed control structure has been tested on different HRI scenarios.

Human safety is the main concern which prevents performing some tasks requiring physical interaction between human and robot. Therefore, the safety concept was previously based on eliminating contact between human and robots. This paper will propose a robot system which integrates different types of sensors to ensure human safety during the physical human robot interaction. The implemented sensors are vision, force and sensitive skin. Using vision system, the robot will be able to detect and recognize human face, loadfree human hand and any object carried by human hand (active hand). Furthermore, it will help the robot to define in which directions the force should be applied and what are the dangerous directions for human safety. The force sensor will help the robot to react to the motion of the human hand during the handing-over or assembling task. The sensitive skin will prevent any collision between the human and the robot arm. The proposed system is supported with a voice system for informing human about the actual status of the system.

This research work proposes a robotized workstation for automatic disassembly of electric vehicle motors. A novel image processing algorithm is proposed for autonomous detection of motor screws, to automate the task of motor disassembly. Instead of having a database of templates for matching, the screws are detected based on their characteristics with respect to its grayscale, depth and HSV values. Furthermore, with frame iterations, the accuracy of the system is increased mean while reducing its runtime. The algorithm is successfully tested and implemented and yields highly accurate results.

This paper will present a new proposed system which combines vision and force control in order to transfer model-free objects between human hand and robot hand. The proposed system will improve robot's facilities to interact with human, especially during handing-over tasks. The contributions of this work are: 1. The robot system will be able to detect all kinds of objects which are carried out by human hand without any a priori information about the model of the objects. 2. The robot system will track the object visually and grasp it with the help of force control. 3. In the proposed system, vision and force sensors are integrated in order to guarantee the safety issues, to guarantee the fulfillment of the grasping task and to react to the motion of human hand during the interaction phase. The proposed system is supported by experimental results which illustrate the capability of the proposed algorithms in different cases with different objects.

Most of robot applications which require interaction with human intend to hand over different objects between the both parties. In general, most of previous systems have supposed that the transfer task will be exclusively performed by human. However, how the task will be performed if the human is blind, elderly, disabled or concentrating on something else? In this case the transfer task should be exclusively controlled by robot, where the human could be considered as the weakest part. This paper will present vision algorithms which help the robot to recognize and to segment any object which is carried by the human hand in order to hand it over from the human hand even if the human is disabled, blind etc. The proposed vision algorithms are able to detect and to segment the carried objects with no idea about the model of the object in a very short cycle time, even if they are textureless, if the objects have the same color as the human skin and if the light conditions are changed strongly. The proposed algorithms are supported by successful results on different kinds of object.

In this paper the control of the well known inverted pendulum will be performed by using visual servoing algorithms. The pendulum will be mounted on the flange of an articulated robot arm and it will be observed by 2D-camera. The inclination angle of the pendulum will be computed in the real time with the help of image processing algorithms. In order to swing-up the pendulum, a special algorithm will be designed to generate the coordinated robot motions. Whereas, the balancing of the pendulum will be performed by a state space controller. Both algorithms will be implemented in the standard robot controller with a relatively high cycle time. Furthermore, the calculation of the inclination angle of the pendulum by the camera, which is integrated in the closed loop controller, instead of an angular transmitter will make the successful achievement of the task more difficult. All the proposed algorithms are verified successfully in the practical experiments.

Handing-over objects from robot to humans is essential step to perform different tasks especially those requiring physical interaction between the robot and the human, e.g. service robots can help elderly, blind and disabled people or human-robot teamwork could work together in factories. This work will propose a new robot system which combines visual servoing and force control in order to hand over model-free objects from undefined place to human hand. This work will present: 1. vision algorithms which help the robot system to detect and to segment the objects without any information about their model. 2. The possibility for visual tracking of human hand with the help of Kinect camera. 3. The importance of fusion vision and force control in order to ensure the safety during the human-robot interaction. 4. How the robot will deliver the objects to the human hand with the help of vision and force control. This work will be supported with experimental results.

This paper will propose an assistance robot system which is able to transfer model-free objects from/to human hand with the help of visual servoing and force control. The proposed robot system is fully automated, i.e. the handing-over task is performed exclusively by the robot and the human will be considered as the weakest party, e.g. elderly, disabled, blind, etc. The proposed system is supported with different real time vision algorithms to detect, to recognize and to track: 1. Any object located on flat surface or conveyor. 2. Any object carried by human hand. 3. The loadfree human hand. Furthermore, the proposed robot system has integrated vision and force feedback in order to: 1. Perform the handing-over task successfully starting from the free space motion until the full physical human-robot integration. 2. Guarantee the safety of the human and react to the motion of the human hand during the handing-over task. The proposed system has shown a great efficiency during the experiments.

Most of robot applications require robots to interact with environment, objects or even with human, which need to integrate vision and force information. Generally, there are five types of vision-force control: pure position control, pure force control, traded control, shared control and hybrid control. The important questions here are: How to define the most appropriate control mode for every part of different tasks and when the control system should switch from one control mode to the other. In this work an automatic decision system is suggested to define the most appropriate control mode for successive tasks and to choose the optimal structure of vision/force control without human intervention. This research will focus on the operations of library automation as real application for the proposed control system such as shelving, storage and sorting of inaccurately placed objects.

Sorting systems are required in many places with different types of applications such as in manufacturing industry, libraries, factories, warehouses, pharmacies, supermarkets etc. In this work it is suggested that an automated sorting system shelves and rearranges imprecisely placed objects using integrated vision/force robot control. The system will sort the objects not only depending on their shapes and forms but also using their alphabetic/numeric codes with the help of SIFT features. Vision system will find out the form, position, orientation and the alphabetic/numeric code of the object and it will define the most appropriated control mode for different cases of grasping. By combining vision and force feedback the grasping and transporting tasks of the objects will be performed. This research will focus on the control system and vision algorithms for library automation and book sorting as real application of this work.

Recently, vision and force control are operated in a more complex varied environments, which required a high demand for adaption, flexibility and fast performance. More information and features about the environment are required whenever the tasks are more complex and multifarious such as storage and retrieval tasks which are spread everywhere, for example in libraries, factories, warehouses, pharmacies, supermarkets and etc. In this work it is suggested a robot system to store and retrieve inaccurately placed objects according to their alphabetic/numeric codification system. The vision system will detect the objects' position and orientation and will analyze the alphabetic/numeric coding system using SIFT algorithm. After that the system will describe how it will grasp the objects by combining the vision and force feedback. This research will focus on the library automation as real application for this work.

Many automated manufacturing processes require robots to interact with environment and to perform force/torque interaction such as mechanical assembly. Importantly, impact forces occur while robot and environment are in contact. Robot manipulators and control systems can experience instability or poor control performance after impacting with an environment. In this paper we combine vision and force feedback in shared/traded vision-force control to reduce the impact forces and to increase the performance of robot, by calculating the distance between robot's end-effector and the environment and reducing the speed according to it. Experimental results are presented to illustrate the performance of robot in two cases with and without vision feedback and to compare these results with simulation.