Search

Object detection is crucial in diverse autonomous systems like surveillance, autonomous driving, and driver assistance, ensuring safety by recognizing pedestrians, vehicles, traffic lights, and signs. However, adverse weather conditions such as snow, fog, and rain pose a challenge, affecting detection accuracy and risking accidents and damage. This clearly demonstrates the need for robust object detection solutions that work in all weather conditions. We employed three strategies to enhance deep learningbased object detection in adverse weather: training on real world all-weather images, training on images with synthetic augmented weather noise, and integrating object detection with adverse weather image denoising. The synthetic weather noise is generated using analytical methods, GAN networks, and style-transfer networks. We compared the performance of these strategies by training object detection models using real-world all-weather images from the BDD100K dataset and, for assessment, employed unseen real-world adverse weather images. Adverse weather denoising methods were evaluated by denoising real-world adverse weather images, and the results of object detection denoised and original noisy images were compared. We found that the model trained using all-weather real-world images performed best, while the strategy of doing object detection on denoised images performed worst.

Social robots, owing to their embodied physical presence in human spaces and the ability to directly interact with the users and their environment, have a great potential to support children in various activities in education, healthcare and daily life. Child-Robot Interaction (CRI), as any domain involving children, inevitably faces the major challenge of designing generalized strategies to work with unique, turbulent and very diverse individuals. Addressing this challenging endeavor requires to combine the standpoint of the robot-centered perspective, i.e. what robots technically can and are best positioned to do, with that of the child-centered perspective, i.e. what children may gain from the robot and how the robot should act to best support them in reaching the goals of the interaction. This article aims to help researchers bridge the two perspectives and proposes to address the development of CRI scenarios with insights from child psychology and child development theories. To that end, we review the outcomes of the CRI studies, outline common trends and challenges, and identify two key factors from child psychology that impact child-robot interactions, especially in a long-term perspective: developmental stage and individual characteristics. For both of them we discuss prospective experiment designs which support building naturally engaging and sustainable interactions., Open Access funding enabled and organized by Projekt DEAL.

Children with visual impairments often struggle to fully participate in group activities due to limited access to visual cues. They have difculty perceiving what is happening, when, and how to act-leading to children with and without visual impairments being frustrated with the group activity, reducing mutual interactions. To address this, we created Touchibo, a tactile storyteller robot acting in a multisensory setting, encouraging touch-based interactions. Touchibo provides an inclusive space for group interaction as touch is a highly accessible modality in a mixed-visual ability context. In a study involving 107 children (37 with visual impairments), we compared Touchibo to an audio-only storyteller. Results indicate that Touchibo signifcantly improved children's individual and group participation perception, sparking touch-based interactions and the storyteller was more likable and helpful. Our study highlights touch-based robots' potential to enrich children's social interactions by prompting interpersonal touch, particularly in mixed-visual ability settings.

Robots and other intelligent systems navigating in complex dynamic environments should predict future actions and intentions of surrounding agents to reach their goals efficiently and avoid collisions. The dynamics of those agents strongly depends on their tasks, roles, or observable labels. Class-conditioned motion prediction is thus an appealing way to reduce forecast uncertainty and get more accurate predictions for heterogeneous agents. However, this is hardly explored in the prior art, especially for mobile robots and in limited data applications. In this paper, we analyse different class-conditioned trajectory prediction methods on two datasets. We propose a set of conditional pattern-based and efficient deep learning-based baselines, and evaluate their performance on robotics and outdoors datasets (TH & Ouml;R-MAGNI and Stanford Drone Dataset). Our experiments show that all methods improve accuracy in most of the settings when considering class labels. More importantly, we observe that there are significant differences when learning from imbalanced datasets, or in new environments where sufficient data is not available. In particular, we find that deep learning methods perform better on balanced datasets, but in applications with limited data, e.g., cold start of a robot in a new environment, or imbalanced classes, pattern-based methods may be preferable.

Research on robotic ostracism is still scarce and has only explored its effects on adult populations. Although the results revealed important carryover effects of robotic exclusion, there is no evidence yet that those results occur in child-robot interactions. This paper provides the first exploration of robotic ostracism with children. We conducted a study using the Robotic Cyberball Paradigm in a third-person perspective with a sample of 52 children aged between five to ten years old. The experimental study had two conditions: Exclusion and Inclusion. In the Exclusion condition, children observed a peer being excluded by two robots; while in the Inclusion condition, the observed peer interacted equally with the robots. Notably, even 5-year-old children could discern when robots excluded another child. Children who observed exclusion reported lower levels of belonging and control, and exhibited higher prosocial behaviour than those witnessing inclusion. However, no differences were found in children's meaningful existence, self-esteem, and physical proximity across conditions. Our user study provides important methodological considerations for applying the Robotic Cyberball Paradigm with children. The results extend previous literature on both robotic ostracism with adults and interpersonal ostracism with children. We finish discussing the broader implications of children observing ostracism in human-robot interactions., This work was supported by FCT projects UIDB/50009/2020 [35] and UIDB/50021/2020 [34], 2022.00816.CEECIND/CP1713/CT0013 [36], and the Portuguese Recovery and Resilience Program (PRR), IAPMEI/ANI/FCT under Agenda C645022399-00000057 (eGamesLab), through the scholarships BL195/2023 and BL270/2022_IST-ID, SFRH/BD/136212/2018, SFRH/BD/06452/2021. This work was also supported by EU-funded projects: DCitizens (GA 101079116), Hybrida (PTDC/CCI-INF/7366/2020), and Tailor (GA 952215).

Behavior trees (BTs) are increasingly popular in the robotics community. Yet in the growing body of published work on this topic, there is a lack of consensus on what to measure and how to quantify BTs when reporting results. This is not only due to the lack of standardized measures, but due to the sometimes ambiguous use of definitions to describe BT properties. This work provides a comprehensive overview of BT properties the community is interested in, how they relate to each other, the metrics currently used to measure BTs, and whether the metrics appropriately quantify those properties of interest. Finally, we provide the practitioner with a set of metrics to measure, as well as insights into the properties that can be derived from those metrics. By providing this holistic view of properties and their corresponding evaluation metrics, we hope to improve clarity when using BTs in robotics. This more systematic approach will make reported results more consistent and comparable when evaluating BTs., This work was partially supported by the Swedish Foundation for Strategic Research (SSF) (project ID19-0053), the Industrial Graduate School Collaborative AI & Robotics (CoAIRob), funded by the Swedish Knowledge Foundation under Grant Dnr:20190128, and by the European Union’s Horizon Europe Framework Programme under grant agreement No 101070596 (euROBIN).

Factor graphs are a very powerful graphical representation, used to model many problems in robotics. They are widely spread in the areas of Simultaneous Localization and Mapping (SLAM), computer vision, and localization. However, the physics of many real-world problems is better modeled through constraints, e.g., estimation in the presence of inconsistent measurements, or optimal control. Constraints handling is hard because the solution cannot be found by following the gradient descent direction as done by traditional factor graph solvers. The core idea of our method is to encapsulate the Augmented Lagrangian (AL) method in factors that can be integrated straightforwardly in existing factor graph solvers. Besides being a tool to unify different robotics areas, the modularity of factor graphs allows to easily combine multiple objectives and effectively exploiting the problem structure for efficiency. We show the generality of our approach by addressing three applications, arising from different areas: pose estimation, rotation synchronization and Model Predictive Control (MPC) of a pseudo-omnidirectional platform. We implemented our approach using C++ and ROS. Application results show that we can favorably compare against domain specific approaches.

To enable progress towards egocentric agents capable of understanding everyday tasks specified in natural language, we propose a benchmark and a synthetic dataset called Egocentric Task Verification (EgoTV). The goal in EgoTV is to verify the execution of tasks from egocentric videos based on the natural language description of these tasks. EgoTV contains pairs of videos and their task descriptions for multi-step tasks -- these tasks contain multiple sub-task decompositions, state changes, object interactions, and sub-task ordering constraints. In addition, EgoTV also provides abstracted task descriptions that contain only partial details about ways to accomplish a task. Consequently, EgoTV requires causal, temporal, and compositional reasoning of video and language modalities, which is missing in existing datasets. We also find that existing vision-language models struggle at such all-round reasoning needed for task verification in EgoTV Inspired by the needs of EgoTV, we propose a novel Neuro-Symbolic Grounding (NSG) approach that leverages symbolic representations to capture the compositional and temporal structure of tasks. We demonstrate NSG's capability towards task tracking and verification on our EgoTV dataset and a real-world dataset derived from CrossTask (CTV). We open-source the EgoTV and CTV datasets and the NSG model for future research on egocentric assistive agents.

This article presents an accurate, highly efficient, and learning-free method for large-scale odometry estimation using spinning radar, empirically found to generalize well across very diverse environments—outdoors, from urban to woodland, and indoors in warehouses and mines—without changing parameters. Our method integrates motion compensation within a sweep with one-to-many scan registration that minimizes distances between nearby oriented surface points and mitigates outliers with a robust loss function. Extending our previous approach conservative filtering for efficient and accurate radar odometry (CFEAR), we present an in-depth investigation on a wider range of datasets, quantifying the importance of filtering, resolution, registration cost and loss functions, keyframe history, and motion compensation. We present a new solving strategy and configuration that overcomes previous issues with sparsity and bias, and improves our state-of-the-art by 38%, thus, surprisingly, outperforming radar simultaneous localization and mapping (SLAM) and approaching lidar SLAM. The most accurate configuration achieves 1.09% error at 5 Hz on the Oxford benchmark, and the fastest achieves 1.79% error at 160 Hz.

The ability to follow the gaze of others deteriorates with age. This decline is typically tested with gaze cueing tasks, in which the time it takes to respond to targets on a screen is faster when they are preceded by a facial cue looking in the direction of the target (i.e., gaze cueing effect). It is unclear whether age-related differences in this effect occur with gaze cues other than the eyes, such as head orientation, and how these vary in function of the cue-target timing. Based on the perceived usefulness of social robots to assist older adults, we asked older and young adults to perform a gaze cueing task with the head of a NAO robot as the central cue. Crucially, the head was viewed from the back, and so its eye gaze was conveyed. In a control condition, the head was static and faced away from the participant. The stimulus onset asynchrony (SOA) between cue and target was 340 ms or 1000 ms. Both age groups showed a gaze cueing effect at both SOAs. Older participants showed a reduced facilitation effect (i.e., faster on congruent gazing trials than on neutral trials) at the 340-ms SOA compared to the 1000-ms SOA, and no differences between incongruent trials and neutral trials at the 340-ms SOA. Our results show that a robot with non-visible eyes can elicit gaze cueing effects. Age-related differences in the other effects are discussed regarding differences in processing time., Funding agency:Spanish Ministerio de Ciencia, RobWellproject RTI2018-095599-A-C22

This article studies group-wise point set registration and makes the following contributions: "FuzzyGReg", which is a new fuzzy cluster-based method to register multiple point sets jointly, and "FuzzyQA", which is the associated quality assessment to check registration accuracy automatically. Given a group of point sets, FuzzyGReg creates a model of fuzzy clusters and equally treats all the point sets as the elements of the fuzzy clusters. Then, the group-wise registration is turned into a fuzzy clustering problem. To resolve this problem, FuzzyGReg applies a fuzzy clustering algorithm to identify the parameters of the fuzzy clusters while jointly transforming all the point sets to achieve an alignment. Next, based on the identified fuzzy clusters, FuzzyQA calculates the spatial properties of the transformed point sets and then checks the alignment accuracy by comparing the similarity degrees of the spatial properties of the point sets. When a local misalignment is detected, a local re-alignment is performed to improve accuracy. The proposed method is cost-efficient and convenient to be implemented. In addition, it provides reliable quality assessments in the absence of ground truth and user intervention. In the experiments, different point sets are used to test the proposed method and make comparisons with state-of-the-art registration techniques. The experimental results demonstrate the effectiveness of our method.The code is available at https://gitsvn-nt.oru.se/qianfang.liao/FuzzyGRegWithQA

Robots can direct human attention using their eyes. However, it remains unclear whether it is the gaze or the low-level motion of the head rotation that drives attention. We isolated these components in a non-predictive gaze cueing task with a robot to explore how limited robotic signals orient attention. In each trial, the head of a NAO robot turned towards the left or right. To isolate the direction of rotation from its gaze, NAO was presented frontally and backward along blocks. Participants responded faster to targets on the gazed-at site, even when the eyes of the robot were not visible and the direction of rotation was opposed to that of the frontal condition. Our results showed that low-level motion did not orient attention, but the gaze direction of the robot did. These findings suggest that the robotic gaze is perceived as a social signal, similar to human gaze., Funding agency:Spanish Ministerio de Ciencia, Innovación y Universidades, RobWell project (No RTI2018-095599-A-C22)

Counterfactual Commonsense

For robots navigating in dynamic environments, exploiting and understanding uncertain human motion prediction is key to generate efficient, safe and legible actions. The robot may perform poorly and cause hindrances if it does not reason over possible, multi-modal future social interactions. With the goal of enhancing autonomous navigation in cluttered environments, we propose a novel formulation for nonlinear model predictive control including multi-modal predictions of human motion. As a result, our approach leads to less conservative, smooth and intuitive human-aware navigation with reduced risk of collisions, and shows a good balance between task efficiency, collision avoidance and human comfort. To show its effectiveness, we compare our approach against the state of the art in crowded simulated environments, and with real-world human motion data from the THOR dataset. This comparison shows that we are able to improve task efficiency, keep a larger distance to humans and significantly reduce the collision time, when navigating in cluttered dynamic environ-ments. Furthermore, the method is shown to work robustly with different state-of-the-art human motion predictors.

As the field of human-agent interaction (HAI) matures, it becomes essential to acknowledge and address the differing multidisciplinary approaches that support it to favor a common interdisciplinary understanding, thus adopting a broader methodological and epistemological perspective. The field of HAI recognizes that agents are not merely isolated technologies but embedded within society. Consequently, the advancement of HAI is not only part of an engineering problem but one that must be informed by diverse disciplines such as legal, philosophical, psychological, design, medical, and sociological, among others. The goal of this workshop is to provide a space where participants can explore the diverse methodologies that compose HAI, reflect on diverse research practices-with their strengths and limitations-, and provide a safe environment where participants can disseminate their research in clear and engaging terms, rather than technical, to foster interdisciplinary collaborations.

Robots are increasingly used in shared environments with humans, making effective communication a necessity for successful human-robot interaction. In our work, we study a crucial component: active communication of robot intent. Here, we present an anthropomorphic solution where a humanoid robot communicates the intent of its host robot acting as an "Anthropomorphic Robotic Mock Driver" (ARMoD). We evaluate this approach in two experiments in which participants work alongside a mobile robot on various tasks, while the ARMoD communicates a need for human attention, when required, or gives instructions to collaborate on a joint task. The experiments feature two interaction styles of the ARMoD: a verbal-only mode using only speech and a multimodal mode, additionally including robotic gaze and pointing gestures to support communication and register intent in space. Our results show that the multimodal interaction style, including head movements and eye gaze as well as pointing gestures, leads to more natural fixation behavior. Participants naturally identified and fixated longer on the areas relevant for intent communication, and reacted faster to instructions in collaborative tasks. Our research further indicates that the ARMoD intent communication improves engagement and social interaction with mobile robots in workplace settings.

Normal Distribution Transformation (NDT) registration is a fast, learning-free point cloud registration algorithm that works well in diverse environments. It uses the compact NDT representation to represent point clouds or maps as a spatial probability function that models the occupancy likelihood in an environment. However, because of the grid discretization in NDT maps, the global minima of the registration cost function do not always correlate to ground truth, particularly for rotational alignment. In this study, we examined the NDT registration cost function in-depth. We evaluated three modifications (Student-t likelihood function, inflated covariance/heavily broadened likelihood curve, and overlapping grid cells) that aim to reduce the negative impact of discretization in classical NDT registration. The first NDT modification improves likelihood estimates for matching the distributions of small population sizes; the second modification reduces discretization artifacts by broadening the likelihood tails through covariance inflation; and the third modification achieves continuity by creating the NDT representations with overlapping grid cells (without increasing the total number of cells). We used the Pomerleau Dataset evaluation protocol for our experiments and found significant improvements compared to the classic NDT D2D registration approach (27.7% success rate) using the registration cost functions "heavily broadened likelihood NDT" (HBL-NDT) (34.7% success rate) and "overlapping grid cells NDT" (OGC-NDT) (33.5% success rate). However, we could not observe a consistent improvement using the Student-t likelihood-based registration cost function (22.2% success rate) over the NDT P2D registration cost function (23.7% success rate). A comparative analysis with other state-of-art registration algorithms is also presented in this work. We found that HBL-NDT worked best for easy initial pose difficulties scenarios making it suitable for consecutive point cloud regi

Autonomous systems, that need to operate in human environments and interact with the users, rely on understanding and anticipating human activity and motion. Among the many factors which influence human motion, semantic attributes, such as the roles and ongoing activities of the detected people, provide a powerful cue on their future motion, actions, and intentions. In this work we adapt several popular deep learning models for trajectory prediction with labels corresponding to the roles of the people. To this end we use the novel THOR-Magni dataset, which captures human activity in industrial settings and includes the relevant semantic labels for people who navigate complex environments, interact with objects and robots, work alone and in groups. In qualitative and quantitative experiments we show that the role-conditioned LSTM, Transformer, GAN and VAE methods can effectively incorporate the semantic categories, better capture the underlying input distribution and therefore produce more accurate motion predictions in terms of Top-K ADE/FDE and log-likelihood metrics.

Focusing on failure to improve human-robot interactions represents a novel approach that calls into question human expectations of robots, as well as posing ethical and methodological challenges to researchers. Fictional representations of robots (still for many non-expert users the primary source of expectations and assumptions about robots) often emphasize the ways in which robots surpass/perfect humans, rather than portraying them as fallible. Thus, to encounter robots that come too close, drop items or stop suddenly starts to close the gap between fiction and reality. These kinds of failures - if mitigated by explanation or recovery procedures - have the potential to make the robot a little more relatable and human-like. However, studying failures in human-robot interaction requires producing potentially difficult or uncomfortable interactions in which robots failing to behave as expected may seem counterintuitive and unethical. In this space, interdisciplinary conversations are the key to untangling the multiple challenges and bringing themes of power and context into view. In this workshop, we invite researchers from across the disciplines to an interactive, interdisciplinary discussion around failure in social robotics. Topics for discussion include (but are not limited to) methodological and ethical challenges around studying failure in HRI, epistemological gaps in defining and understanding failure in HRI, sociocultural expectations around failure and users' responses.

Failures are natural and unavoidable events in any form of interaction, especially in human-robot interactions (HRI). Throughout the literature, the definition and classification of failures are diverse, depending on the source and application domain. However, the tolerance to the aftereffect of these failures is low in teleoperation due to its unstructured application domains. One such type of failure is called human induced interaction failure. This is an interesting and often overlooked failure type, due to the perspective that robots are designed always to obey the instructions given by the human operators. Regardless of the degree of automation that the robot is equipped with. But what if the instructions provided are faulty, dangerous, or misleading. This paper addresses the above mentioned research gap. It introduces a framework based on the concept of Intelligent Disobedience (ID), derived from guide dog training methods, to manage human induced interaction failures in teleoperation scenarios.

In the context of robotics, accurate ground-truth positioning is the cornerstone for the development of mapping and localization algorithms. In outdoor environments and over long distances, total stations provide accurate and precise measurements, that are unaffected by the usual factors that deteriorate the accuracy of Global Navigation Satellite System (GNSS). While a single robotic total station can track the position of a target in three Degrees Of Freedom (DOF), three robotic total stations and three targets are necessary to yield the full six DOF pose reference. Since it is crucial to express the position of targets in a common coordinate frame, we present a novel extrinsic calibration method of multiple robotic total stations with field deployment in mind. The proposed method does not require the manual collection of ground control points during the system setup, nor does it require tedious synchronous measurement on each robotic total station. Based on extensive experimental work, we compare our approach to the classical extrinsic calibration methods used in geomatics for surveying and demonstrate that our approach brings substantial time savings during the deployment. Tested on more than 30 km of trajectories, our new method increases the precision of the extrinsic calibration by 25 % compared to the best state-of-the-art method, which is the one taking manually static ground control points., This research was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) through the grant CRDPJ 527642-18 SNOW (Self-driving Navigation Optimized for Winter).

Robust SLAM in large-scale environments requires fault resilience and awareness at multiple stages, from sensing and odometry estimation to loop closure. In this work, we present TBV (Trust But Verify) Radar SLAM, a method for radar SLAM that introspectively verifies loop closure candidates. TBV Radar SLAM achieves a high correct-loop-retrieval rate by combining multiple place-recognition techniques: tightly coupled place similarity and odometry uncertainty search, creating loop descriptors from origin-shifted scans, and delaying loop selection until after verification. Robustness to false constraints is achieved by carefully verifying and selecting the most likely ones from multiple loop constraints. Importantly, the verification and selection are carried out after registration when additional sources of loop evidence can easily be computed. We integrate our loop retrieval and verification method with a robust odometry pipeline within a pose graph framework. By evaluation on public benchmarks we found that TBV Radar SLAM achieves 65% lower error than the previous state of the art. We also show that it generalizes across environments without needing to change any parameters. We provide the open-source implementation at https://github.com/dan11003/tbv_slam_public

A significant challenge in service robots is the semantic understanding of their surrounding areas. Traditional approaches addressed this problem by segmenting the environment into regions corresponding to full rooms that are assigned labels consistent with human perception, e.g. office or kitchen. However, different areas inside the same room can be used in different ways: Could the table and the chair in my kitchen become my office ? What is the category of that area now? office or kitchen? To adapt to these circumstances we propose a new paradigm where we intentionally relax the resulting labeling of place classifiers by allowing confusions, and by avoiding further filtering leading to clean full room classifications. Our hypothesis is that confusions can be beneficial to a service robot and, therefore, they can be kept and better exploited. Our approach creates a subdivision of the environment into different regions by maintaining the confusions which are due to the scene appearance or to the distribution of objects. In this paper, we present a proof of concept implemented in simulated and real scenarios, that improves efficiency in the robotic task of searching for objects by exploiting the confusions in place classifications., Funding agencies:Spanish Government RTI2018-095599-B-C21 RTI2018-095599-A-C22RoboCity2030-Madrid Robotics Digital Innovation Hub project, Spain S2018/NMT-4331

The increased popularity of Behavior Trees (BTs) in different fields of robotics requires efficient methods for learning BTs from data instead of tediously handcrafting them. Recent research in learning from demonstration reported encouraging results that this letter extends, improves and generalizes to arbitrary planning domains. We propose BT-Factor as a new method for learning expert knowledge by representing it in a BT. Execution traces of previously manually designed plans are used to generate a BT employing a combination of decision tree learning and logic factorization techniques originating from circuit design. We test BT-Factor in an industrially-relevant simulation environment from a mining scenario and compare it against a state-of-the-art BT learning method. The results show that our method generates compact BTs easy to interpret, and capable to capture accurately the relations that are implicit in the training data.

Registration of point cloud data containing both depth and color information is critical for a variety of applications, including in-field robotic plant manipulation, crop growth modeling, and autonomous navigation. However, current state-of-the-art registration methods often fail in challenging agricultural field conditions due to factors such as occlusions, plant density, and variable illumination. To address these issues, we propose the NDT-6D registration method, which is a color-based variation of the Normal Distribution Transform (NDT) registration approach for point clouds. Our method computes correspondences between pointclouds using both geometric and color information and minimizes the distance between these correspondences using only the three-dimensional (3D) geometric dimensions. We evaluate the method using the GRAPES3D data set collected with a commercial-grade RGB-D sensor mounted on a mobile platform in a vineyard. Results show that registration methods that only rely on depth information fail to provide quality registration for the tested data set. The proposed color-based variation outperforms state-of-the-art methods with a root mean square error (RMSE) of 1.1-1.6 cm for NDT-6D compared with 1.1-2.3 cm for other color-information-based methods and 1.2-13.7 cm for noncolor-information-based methods. The proposed method is shown to be robust against noises using the TUM RGBD data set by artificially adding noise present in an outdoor scenario. The relative pose error (RPE) increased similar to 14% for our method compared to an increase of similar to 75% for the best-performing registration method. The obtained average accuracy suggests that the NDT-6D registration methods can be used for in-field precision agriculture applications, for example, crop detection, size-based maturity estimation, and growth modeling.

In this paper, we propose a system that stems from the integration of an autonomous mobile robot with an IoT-based monitoring system to provide monitoring, assistance, and stimulation to older adults living alone in their own houses. The creation of an Internet of Robotics Things (IoRT) based on the interplay between pervasive smart objects and autonomous robotic systems is claimed to enable the creation of innovative services conceived for assisting the final user, especially in elderly care. The synergy between IoT and a Socially Assistive Robot (SAR) was conceived to offer robustness, reconfiguration, heterogeneity, and scalability, by bringing a strong added value to both the current SAR and IoT technologies. First, we propose a method to achieve the synergy and integration between the IoT system and the robot; then, we show how our method increases the performance and effectiveness of both to provide long-term support to the older adults. To do so, we present a case-study, where we focus on the detection of signs of the frailty syndrome, a set of vulnerabilities typically conveyed by a cognitive and physical decline in older people that concur in amplifying the risks of major diseases hindering the capabilities of independent living. Experimental evaluation is performed in both controlled settings and in a long-term real-world pilot study with 9 older adults in their own apartments, where the system was deployed autonomously for, on average, 12 weeks., Funding agency:PON project SI-ROBOTICS - Italian government

Multi-robot motion coordination is essential to make robots work safely and efficiently in a shared workspace, ensuring task completion while avoiding interference between each other's motions. Achieving online replanning and scaling to large numbers of robots are particularly challenging. In this paper, we present a motion coordination method for multiple robot manipulators with given targets. The approach separates the problem into a global coordination stage and a local trajectory replanning stage. Online reactivity and scalability are achieved by means of coordination in a reduced space and efficient trajectory replanning. We first introduce the method for systems with two robots, then extend it to systems with an arbitrary number of robots. The method determines whether kinematically-feasible and non-interfering trajectories leading all the robots to their targets exist. The approach generates time-efficient trajectories if solutions exist, or provides information for switching targets if solutions cannot be found. We show formally and empirically that the method has low computational overhead and scales quadratically with the number of robots. Experiments are conducted with up to three real 7-DOF robots and up to ten simulated robots. Note to Practitioners-In underground mining, a key process is that of tunneling, i.e., drilling and blasting rock to excavate tunnels that lead to sources of ore. Drilling is carried out by rigs equipped with multiple robotic arms. A key factor affecting the efficiency of tunneling is the time to completion of drilling operations. In current industrial practice, these operations are carried out manually by an operator steering the arms on the drill rig. Time to completion can be drastically reduced if the arms could operate concurrently, intelligently optimizing and coordinating their motions. However, existing methods for multi-arm motion coordination are inadequate, as they fail to cater to one or more of the following real-wor

Current intralogistics services require keeping up with e-commerce demands, reducing delivery times and waste, and increasing overall flexibility. As a consequence, the use of automated guided vehicles (AGVs) and, more recently, autonomous mobile robots (AMRs) for logistics operations is steadily increasing.

Robotic mapping provides spatial information for autonomous agents. Depending on the tasks they seek to enable, the maps created range from simple 2D representations of the environment geometry to complex, multilayered semantic maps. This survey article is about maps of dynamics (MoDs), which store semantic information about typical motion patterns in a given environment. Some MoDs use trajectories as input, and some can be built from short, disconnected observations of motion. Robots can use MoDs, for example, for global motion planning, improved localization, or human motion prediction. Accounting for the increasing importance of maps of dynamics, we present a comprehensive survey that organizes the knowledge accumulated in the field and identifies promising directions for future work. Specifically, we introduce field-specific vocabulary, summarize existing work according to a novel taxonomy, and describe possible applications and open research problems. We conclude that the field is mature enough, and we expect that maps of dynamics will be increasingly used to improve robot performance in real-world use cases. At the same time, the field is still in a phase of rapid development where novel contributions could significantly impact this research area., Funding agencies:Czech Ministry of Education by OP VVV CZ.02.1.01/0.0/0.0/16 019/0000765Business Finland 9249/31/2021

The generation of a robot program can be seen as a collection of sub-problems, where many combinations of some of these sub-problems are well studied. The performance of a robot program is strongly conditioned by the location of the tasks. However, the scope of previous methods does not include workspace layout design, likely missing high-quality solutions. In industrial applications, designing robot workspace layout is part of the commissioning. We broaden the scope and show how to model a dual-arm multi-tool robot assembly problem. Our model includes more robot programming sub-problems than previous methods, as well as workspace layout design. We propose a constraint programming formulation in MiniZinc that includes elements from scheduling and routing, extended with variable task locations. We evaluate the model on realistic assembly problems and workspaces, utilizing the dual-arm YuMi robot from ABB Ltd. We also evaluate redundant constraints and various formulations for avoiding arm-to-arm collisions. The best model variant quickly finds high-quality solutions for all problem instances. This demonstrates the potential of our approach as a valuable tool for a robot programmer.

Safety is a fundamental prerequisite that must be addressed before any interaction of robots with humans. Safety has been generally understood and studied as the physical safety of robots in human-robot interaction, whereas how humans perceive these robots has received less attention. Physical safety is a necessary condition for safe human-robot interaction. However, it is not a sufficient condition. A robot that is safe by hardware and software design can still be perceived as unsafe. This article focuses on perceived safety in human-robot interaction. We identified six factors that are closely related to perceived safety based on the literature and the insights obtained from our user studies. The identified factors are the context of robot use, comfort, experience and familiarity with robots, trust, the sense of control over the interaction, and transparent and predictable robot actions. We then made a literature review to identify the robot-related factors that influence perceived safety. Based the literature, we propose a taxonomy which includes human-related and robot-related factors. These factors can help researchers to quantify perceived safety of humans during their interactions with robots. The quantification of perceived safety can yield computational models that would allow mitigating psychological harm.

This article investigates the role that kinematic features play in human action similarity judgments. The results of three experiments with human participants are compared with the computational model that solves the same task. The chosen model has its roots in developmental robotics and performs action classification based on learned kinematic primitives. The comparative experimental results show that both model and human participants can reliably identify whether two actions are the same or not. Specifically, most of the given actions could be similarity judged based on very limited information from a single feature domain (velocity or spatial). Both velocity and spatial features were however necessary to reach a level of human performance on evaluated actions. The experimental results also show that human performance on an action identification task indicated that they clearly relied on kinematic information rather than on action semantics. The results show that both the model and human performance are highly accurate in an action similarity task based on kinematic-level features, which can provide an essential basis for classifying human actions., Funding agency:AFOSR FA8655-20-1-7035

We propose a hierarchical framework for collaborative intelligent systems. This framework organizes research challenges based on the nature of the collaborative activity and the information that must be shared, with each level building on capabilities provided by lower levels. We review research paradigms at each level, with a description of classical engineering-based approaches and modern alternatives based on machine learning, illustrated with a running example using a hypothetical personal service robot. We discuss cross-cutting issues that occur at all levels, focusing on the problem of communicating and sharing comprehension, the role of explanation and the social nature of collaboration. We conclude with a summary of research challenges and a discussion of the potential for economic and societal impact provided by technologies that enhance human abilities and empower people and society through collaboration with intelligent systems., Funding agency:MIAI Multidisciplinary AI Institute at the Universite Grenoble Alpes (MIAI@Grenoble Alpes) ANR-19-P3IA-0003

This article reports on the early stages of conception of a robotic control system based on Piaget's schemas theory. Beyond some initial experimental results, we question the scientific method used in developmental robotics (DevRob) and argue that it is premature to abstract away the functional architecture of the brain when so little is known about its mechanisms. Instead, we advocate for applying a method similar to the method used in model-based cognitive science, which consists in selecting plausible models using computational and physiological constraints. Previous study on schema-based robotics is analyzed through the critical lens of the proposed method, and a minimal system designed using this method is presented.

We position recent and emerging research in cognitive vision and perception addressing three key questions: (1) What kind of relational abstraction mechanisms are needed to perform (explainable) grounded inference --e.g., question-answering, qualitative generalisation, hypothetical reasoning-- relevant to embodied multimodal interaction? (2) How can such abstraction mechanisms be founded on behaviourally established cognitive human-factors emanating from naturalistic empirical observation? and (3) How to articulate behaviourally established abstraction mechanisms as formal declarative models suited for grounded knowledge representation and reasoning (KR) as part of large-scale hybrid AI and computational cognitive systems. We contextualise (1--3) in the backdrop of recent results at the interface of AI/KR, and Spatial Cognition and Computation. Our main purpose is to emphasise the importance of behavioural research based foundations for next-generation, human-centred AI, e.g., as relevant to applications in Autonomous Vehicles, Social and Industrial Robots, and Visuo-Auditory Media.

In this paper we present a multidisciplinary approach combining technical practices with sensory data to optimize cultivation practices for production of plants using sensory evaluation and further the how it affects nutritional content. We apply sensory evaluation of plants under mechanical stress, in this case robot cultivated basil. Plant stress is a research field studying plants' reactions to suboptimal conditions leading to effects on growth, crop yield, and resilience to harsh environmental conditions. Some of the effects induced by mechanical stress have been shown to be beneficial, both in futuristic commercial growing paradigms (e.g., vertical farming), as well as in altering the plant's nutritional content. This pilot study uses established sensory methods such as Liking, Just-About-Right (JAR) and Check-All-That-Apply (CATA) to study the sensory effect of mechanical stress on cropped basil induced by a specially developed robotic platform. Three different kinds of cropped basil were evaluated: (a) mechanically stressed-robot cultivated, (b) non-stressed -robot cultivated from the same cropping bed (reference); and (c) a commercially organic produced basil. We investigated liking, critical attributes, sensory profile, and the use of a semi-trained culinary panel to make any presumptions on consumer acceptance. The semi-trained panel consisted of 24 culinary students with experience of daily judging sensory aspects of specific food products and cultivated crops. The underlying goal is to assess potential market aspects related to novel mechanical cultivation systems. Results shows that basil cropped in a controlled robot cultivated platform resulted in significantly better liking compared to commercially organic produced basil. Results also showed that mechanical stress had not negatively affected the sensory aspects, suggesting that eventual health benefits eating stressed plants do not come at the expense of the sensory experience.

In this paper, we present a synopsis of the PlayGround project. Through neural-symbolic learning and reasoning, the PlayGround project assumes that high-level concepts and reasoning processes can be used to advance both symbol grounding and object affordance inference. However, a prerequisite for reasoning about objects and their affordances is integrated object representations that concurrently maintain symbolic values (e.g., high-level concepts), and sub-symbolic features (e.g., spatial aspects of objects). Integrated representations that, preferably, should be based upon neural-symbolic computation such that neural-symbolic models can, subsequently, be used for high-level reasoning processes. Nevertheless, reasoning processes for symbol grounding and affordance inference often require multiple inference steps. Taking inspiration from the cognitive prospects in simulation semantics, the PlayGround project further presumes that these reasoning processes can be simulated by neural rendering complementary to high-level reasoning processes.

Tracking deformable linear objects (DLOs) is a key element for applications where robots manipulate DLOs. However, the lack of distinctive features or appearance on the DLO and the object’s high-dimensional state space make tracking challenging and still an open question in robotics. In this paper, we propose a method for tracking the state of a DLO by applying a particle filter approach, where the posterior probability of each sample is estimated by a learned predictor. Our method can achieve accurate tracking even with no prerequisite segmentation which many related works require. Due to the differentiability of the posterior probability predictor, our method can leverage the gradients of posterior probabilities with respect to the latent states to improve the motion model in the particle filter. The preliminary experiments suggest that the proposed method can provide robust tracking results and the estimated DLO state converges quickly to the true state if the initial state is unknown.

Traditional approaches to manipulating the state of deformable linear objects (DLOs) - i.e., cables, ropes - rely on model-based planning. However, constructing an accurate dynamic model of a DLO is challenging due to the complexity of interactions and a high number of degrees of freedom. This renders the task of achieving a desired DLO shape particularly difficult and motivates the use of model-free alternatives, which while maintaining generality suffer from a high sample complexity. In this paper, we bridge the gap between these fundamentally different approaches and propose a framework that learns dynamic models of DLOs through trial-and-error interaction. Akin to model-based reinforcement learning (RL), we interleave learning and exploration to solve a 3D shape control task for a DLO. Our approach requires only a fraction of the interaction samples of the current state-of-the-art model-free RL alternatives to achieve superior shape control performance. Unlike offline model learning, our approach does not require expert knowledge for data collection, retains the ability to explore, and automatically selects relevant experience.

This paper deals with Multi-robot Trajectory Planning, that is, the problem of computing trajectories for multiple robots navigating in a shared space while minimizing for control energy. Approaches based on trajectory optimization can solve this problem optimally. However, such methods are hampered by complex robot dynamics and collision constraints that couple robot's decision variables. We propose a distributed multirobot optimization algorithm (DiMOpt) that addresses these issues by exploiting (1) consensus optimization strategies to tackle coupling collision constraints, and (2) a single-robot sequential convex programming method for efficiently handling non-convexities introduced by dynamics. We compare DiMOpt with a baseline centralized multi-robot sequential convex programming algorithm (SCP). We empirically demonstrate that DiMOpt scales well for large fleets of robots while computing solutions faster and with lower costs. Finally, DiMOpt is an iterative algorithm that finds feasible trajectories before converging to a locally optimal solution, and results suggest the quality of such fast initial solutions is comparable to a converged solution computed via SCP., Funding agency:KKS Synergy TeamRob

Tracking of deformable linear objects (DLOs) is important for many robotic applications. However, achieving robust and accurate tracking is challenging due to the lack of distinctive features or appearance on the DLO, the object's high-dimensional state space, and the presence of occlusion. In this letter, we propose a method for tracking the state of a DLO by applying a particle filter approach within a lower-dimensional state embedding learned by an autoencoder. The dimensionality reduction preserves state variation, while simultaneously enabling a particle filter to accurately track DLO state evolution with a practically feasible number of particles. Compared to previous works, our method requires neither running a high-fidelity physics simulation, nor manual designs of constraints and regularization. Without the assumption of knowing the initial DLO state, our method can achieve accurate tracking even under complex DLO motions and in the presence of severe occlusions.

Human motion trajectory prediction, an essential task for autonomous systems in many domains, has been on the rise in recent years. With a multitude of new methods proposed by different communities, the lack of standardized benchmarks and objective comparisons is increasingly becoming a major limitation to assess progress and guide further research. Existing benchmarks are limited in their scope and flexibility to conduct relevant experiments and to account for contextual cues of agents and environments. In this paper we present Atlas, a benchmark to systematically evaluate human motion trajectory prediction algorithms in a unified framework. Atlas offers data preprocessing functions, hyperparameter optimization, comes with popular datasets and has the flexibility to setup and conduct underexplored yet relevant experiments to analyze a method's accuracy and robustness. In an example application of Atlas, we compare five popular model- and learning-based predictors and find that, when properly applied, early physics-based approaches are still remarkably competitive. Such results confirm the necessity of benchmarks like Atlas.

The Green Area Factor(GYF) is an aggregate norm used as an index to quantify how much eco-efficient surface exists in a given area. Although the GYF is a single number, it expresses several different contributions of natural objects to the ecosystem. It is used as a planning tool to create and manage attractive urban environments ensuring the existence of required green/blue elements. Currently, the GYF model is gaining rapid attraction by different communities. However, calculating the GYF value is challenging as significant amount of manual effort is needed. In this study, we present a novel approach for automatic extraction of the GYF value from aerial imagery using semantic segmentation results. For model training and validation a set of RGB images captured by Drone imaging system is used. Each image is annotated into trees, grass, soil/open surface, building, and road. A modified U-net deep learning architecture is used for the segmentation of various objects by classifying each pixel into one of the semantic classes. From the segmented image we calculate the class-wise fractional area coverages that are used as input into the simplified GYF model called Sundbyberg for calculating the GYF value. Experimental results yield that the deep learning method provides about 92% mean IoU for test image segmentation and corresponding GYF value is 0.34.

We position ongoing research aimed at developing a general framework for structured spatio-temporal learning from multimodal human behavioural stimuli. The framework and its underlying general, modular methods serve as a model for the application of integrated (neural) visuo-auditory processing and (semantic) relational learning foundations for applications (primarily) in the behavioural sciences.

In robotics, deep learning models are used in many visual perception applications, including the tracking, detection and pose estimation of robotic manipulators. The state of the art methods however are conditioned on the availability of annotated training data, which may in practice be costly or even impossible to collect. Domain augmentation is one popular method to improve generalization to out-of-domain data by extending the training data set with predefined sources of variation, unrelated to the primary task. While this typically results in better performance on the target domain, it is not always clear that the trained models are capable to accurately separate the signals relevant to solving the task (e.g., appearance of an object of interest) from those associated with differences between the domains (e.g., lighting conditions). In this work we propose to improve the generalization capabilities of models trained with domain augmentation by formulating a secondary structured metric-space learning objective. We concentrate on one particularly challenging domain transfer task-visual state estimation for an articulated underground mining machine-and demonstrate the benefits of imposing structure on the encoding space. Our results indicate that the proposed method has the potential to transfer feature embeddings learned on the source domain, through a suitably designed augmentation procedure, and on to an unseen target domain.

Gaussian process (GP) implicit surface models provide environment and object representations which elegantly address noise and uncertainty while remaining sufficiently flexible to capture complex geometry. However, GP models quickly become intractable as the size of the observation set grows-a trait which is difficult to reconcile with the rate at which modern range sensors produce data. Furthermore, naive applications of GPs to implicit surface models allocate model resources uniformly, thus using precious resources to capture simple geometry. In contrast to prior work addressing these challenges though model sparsification, spatial partitioning, or ad-hoc filtering, we propose introducing model bias online through the GP's mean function. We achieve more accurate distance fields using smaller models by creating a distance field prior from features which are easy to extract and have analytic distance fields. In particular, we demonstrate this approach using linear features. We show the proposed distance field halves model size in a 2D mapping task using data from a SICK S300 sensor. When applied to a single 3D scene from the TUM RGB-D SLAM dataset, we achieve a fivefold reduction in model size. Our proposed prior results in more accurate GP implicit surfaces, while allowing existing models to function in larger environments or with larger spatial partitions due to reduced model size.

Stack-of-Tasks (SoT) control allows a robot to simultaneously fulfill a number of prioritized goals formulated in terms of (in)equality constraints in error space. Since this approach solves a sequence of Quadratic Programs (QP) at each time-step, without taking into account any temporal state evolution, it is suitable for dealing with local disturbances. However, its limitation lies in the handling of situations that require non-quadratic objectives to achieve a specific goal, as well as situations where countering the control disturbance would require a locally suboptimal action. Recent works address this shortcoming by exploiting Finite State Machines (FSMs) to compose the tasks in such a way that the robot does not get stuck in local minima. Nevertheless, the intrinsic trade-off between reactivity and modularity that characterizes FSMs makes them impractical for defining reactive behaviors in dynamic environments. In this letter, we combine the SoT control strategy with Behavior Trees (BTs), a task switching structure that addresses some of the limitations of the FSMs in terms of reactivity, modularity and re-usability. Experimental results on a Franka Emika Panda 7-DOF manipulator show the robustness of our framework, that allows the robot to benefit from the reactivity of both SoT and BTs., Funding agencies:Industrial Graduate School Collaborative AI & Robotics (CoAIRob)General Electric Dnr:20190128