Search

To facilitate human-robot interaction and gain human trust, a robot should recognize and adapt to changes in human behavior. This work documents different human behaviors observed while taking objects from an interactive robot in an experimental study, categorized across two dimensions: pull force applied and handedness. We also present the changes observed in human behavior upon repeated interaction with the robot to take various objects.

Robotic systems for manipulation tasks are increasingly expected to be easy to configure for new tasks or unpredictable environments, while keeping a transparent policy that is readable and verifiable by humans. We propose the method BEhavior TRee eXPansion with Large Language Models (BETR-XP-LLM) to dynamically and automatically expand and configure Behavior Trees as policies for robot control. The method utilizes an LLM to resolve errors outside the task planner's capabilities, both during planning and execution. We show that the method is able to solve a variety of tasks and failures and permanently update the policy to handle similar problems in the future., Comment: Submitted to ICRA 2025

Accurate recognition of human emotions is a crucial challenge in affective computing and human-robot interaction (HRI). Emotional states play a vital role in shaping behaviors, decisions, and social interactions. However, emotional expressions can be influenced by contextual factors, leading to misinterpretations if context is not considered. Multimodal fusion, combining modalities like facial expressions, speech, and physiological signals, has shown promise in improving affect recognition. This paper proposes a transformer-based multimodal fusion approach that leverages facial thermal data, facial action units, and textual context information for context-aware emotion recognition. We explore modality-specific encoders to learn tailored representations, which are then fused using additive fusion and processed by a shared transformer encoder to capture temporal dependencies and interactions. The proposed method is evaluated on a dataset collected from participants engaged in a tangible tabletop Pacman game designed to induce various affective states. Our results demonstrate the effectiveness of incorporating contextual information and multimodal fusion for affective state recognition.

Behavior Trees (BTs) were first conceived in the computer games industry as a tool to model agent behavior, but they received interest also in the robotics community as an alternative policy design to Finite State Machines (FSMs). The advantages of BTs over FSMs had been highlighted in many works, but there is no thorough practical comparison of the two designs. Such a comparison is particularly relevant in the robotic industry, where FSMs have been the state-of-the-art policy representation for robot control for many years. In this work we shed light on this matter by comparing how BTs and FSMs behave when controlling a robot in a mobile manipulation task. The comparison is made in terms of reactivity, modularity, readability, and design. We propose metrics for each of these properties, being aware that while some are tangible and objective, others are more subjective and implementation dependent. The practical comparison is performed in a simulation environment with validation on a real robot. We find that although the robot's behavior during task solving is independent on the policy representation, maintaining a BT rather than an FSM becomes easier as the task increases in complexity., Comment: Submitted to IEEE Transactions on Robotics (T-RO). arXiv admin note: text overlap with arXiv:2209.07392

In industrial applications Finite State Machines (FSMs) are often used to implement decision making policies for autonomous systems. In recent years, the use of Behavior Trees (BT) as an alternative policy representation has gained considerable attention. The benefits of using BTs over FSMs are modularity and reusability, enabling a system that is easy to extend and modify. However, there exists few published studies on successful implementations of BTs for industrial applications. This paper contributes with the lessons learned from implementing BTs in a complex industrial use case, where a robotic system assembles explosive charges and places them in holes on the rock face. The main result of the paper is that even if it is possible to model the entire system as a BT, combining BTs with FSMs can increase the readability and maintainability of the system. The benefit of such combination is remarked especially in the use case studied in this paper, where the full system cannot run autonomously but human supervision and feedback are needed.

Robotic systems for manipulation tasks are increasingly expected to be easy to configure for new tasks. While in the past, robot programs were often written statically and tuned manually, the current, faster transition times call for robust, modular and interpretable solutions that also allow a robotic system to learn how to perform a task. We propose the method Behavior-based Bayesian Optimization and Planning (BeBOP) that combines two approaches for generating behavior trees: we build the structure using a reactive planner and learn specific parameters with Bayesian optimization. The method is evaluated on a set of robotic manipulation benchmarks and is shown to outperform state-of-the-art reinforcement learning algorithms by being up to 46 times faster while simultaneously being less dependent on reward shaping. We also propose a modification to the uncertainty estimate for the random forest surrogate models that drastically improves the results., Comment: Submitted to ICRA 2024

Despite significant improvements in robot capabilities, they are likely to fail in human-robot collaborative tasks due to high unpredictability in human environments and varying human expectations. In this work, we explore the role of explanation of failures by a robot in a human-robot collaborative task. We present a user study incorporating common failures in collaborative tasks with human assistance to resolve the failure. In the study, a robot and a human work together to fill a shelf with objects. Upon encountering a failure, the robot explains the failure and the resolution to overcome the failure, either through handovers or humans completing the task. The study is conducted using different levels of robotic explanation based on the failure action, failure cause, and action history, and different strategies in providing the explanation over the course of repeated interaction. Our results show that the success in resolving the failures is not only a function of the level of explanation but also the type of failures. Furthermore, while novice users rate the robot higher overall in terms of their satisfaction with the explanation, their satisfaction is not only a function of the robot's explanation level at a certain round but also the prior information they received from the robot., Comment: Accepted and Presented at IEEE International Conference on Robot and Human Interactive Communication, IEEE RO-MAN 2023

The rapid rise of online enrollments in public universities has been fueled by a reliance on for-profit, third-party providers--especially online program managers. However, scholars know very little about the potential problems with this arrangement. We conduct a mixed methods analysis of 229 contracts between third-party providers and 117 two-year and four-year public universities in the US, data on the financing structure of third-party providers, and university online education webpages. We ask: What are the mechanisms through which third-party relationships with universities may be exploitative of students or the public universities that serve them? To what extent are potentially predatory processes linked to the private equity and venture capital financing structure of third-party providers? We highlight specific mechanisms that lead to five predatory processes: the targeting of marginalized students, extraction of revenue, privatization by obfuscation, for-profit creep, and university captivity. We demonstrate that contracts with private equity and venture capital financed third-party providers are more likely to include potentially problematic contract stipulations. We ground our findings in a growing body of work on "platform capitalism" and include recommendations for state universities, accreditors, and federal policy.

Handovers are basic yet sophisticated motor tasks performed seamlessly by humans. They are among the most common activities in our daily lives and social environments. This makes mastering the art of handovers critical for a social and collaborative robot. In this work, we present an experimental study that involved human-human handovers by 13 pairs, i.e., 26 participants. We record and explore multiple features of handovers amongst humans aimed at inspiring handovers amongst humans and robots. With this work, we further create and publish a novel data set of 8672 handovers, bringing together human motion and the forces involved. We further analyze the effect of object weight and the role of visual sensory input in human-human handovers, as well as possible design implications for robots. As a proof of concept, the data set was used for creating a human-inspired data-driven strategy for robotic grip release in handovers, which was demonstrated to result in better robot to human handovers., Comment: The data set of human-human handovers can be found at: https://github.com/paragkhanna1/dataset

Despite great advances in what robots can do, they still experience failures in human-robot collaborative tasks due to high randomness in unstructured human environments. Moreover, a human's unfamiliarity with a robot and its abilities can cause such failures to repeat. This makes the ability to failure explanation very important for a robot. In this work, we describe a user study that incorporated different robotic failures in a human-robot collaboration (HRC) task aimed at filling a shelf. We included different types of failures and repeated occurrences of such failures in a prolonged interaction between humans and robots. The failure resolution involved human intervention in form of human-robot bidirectional handovers. Through such studies, we aim to test different explanation types and explanation progression in the interaction and record humans., Comment: Contributed to the: "The Imperfectly Relatable Robot: An interdisciplinary workshop on the role of failure in HRI", ACM/IEEE International Conference on Human-Robot Interaction HRI 2023. Video can be found at: https://sites.google.com/view/hri-failure-ws/teaser-videos

Handovers frequently occur in our social environments, making it imperative for a collaborative robotic system to master the skill of handover. In this work, we aim to investigate the relationship between the grip force variation for a human giver and the sensed interaction force-torque in human-human handovers, utilizing a data-driven approach. A Long-Short Term Memory (LSTM) network was trained to use the interaction force-torque in a handover to predict the human grip force variation in advance. Further, we propose to utilize the trained network to cause human-like grip force variation for a robotic giver., Comment: Contributed to "Advances in Close Proximity Human-Robot Collaboration" Workshop in 2022 IEEE-RAS International Conference on Humanoid Robots (Humanoids 2022)

In modern industrial collaborative robotic applications, it is desirable to create robot programs automatically, intuitively, and time-efficiently. Moreover, robots need to be controlled by reactive policies to face the unpredictability of the environment they operate in. In this paper we propose a framework that combines a method that learns Behavior Trees (BTs) from demonstration with a method that evolves them with Genetic Programming (GP) for collaborative robotic applications. The main contribution of this paper is to show that by combining the two learning methods we obtain a method that allows non-expert users to semi-automatically, time-efficiently, and interactively generate BTs. We validate the framework with a series of manipulation experiments. The BT is fully learnt in simulation and then transferred to a real collaborative robot., Comment: Submitted to IEEE 19th Conference on Automation Science and Engineering (CASE) 2023

Behavior Trees are a task switching policy representation that can grant reactiveness and fault tolerance. Moreover, because of their structure and modularity, a variety of methods can be used to generate them automatically. In this short paper we introduce Behavior Trees in the context of robotic applications, with overview of autonomous synthesis methods., Comment: Accepted to the Workshop on Development and Design Pipelines - From first ideas to well-functioning robots, at the 2022 IEEE-RAS International Conference on Humanoid Robots (Humanoids 2022) November 28-30, Ginowan, Okinawa, Japan

In this paper we provide a practical demonstration of how the modularity in a Behavior Tree (BT) decreases the effort in programming a robot task when compared to a Finite State Machine (FSM). In recent years the way to represent a task plan to control an autonomous agent has been shifting from the standard FSM towards BTs. Many works in the literature have highlighted and proven the benefits of such design compared to standard approaches, especially in terms of modularity, reactivity and human readability. However, these works have often failed in providing a tangible comparison in the implementation of those policies and the programming effort required to modify them. This is a relevant aspect in many robotic applications, where the design choice is dictated both by the robustness of the policy and by the time required to program it. In this work, we compare backward chained BTs with a fault-tolerant design of FSMs by evaluating the cost to modify them. We validate the analysis with a set of experiments in a simulation environment where a mobile manipulator solves an item fetching task., Comment: Submitted to 2023 IEEE International Conference on Robotics and Automation (ICRA)

In recent years, robots are used in an increasing variety of tasks, especially by small- and medium- sized enterprises. These tasks are usually fast-changing, they have a collaborative scenario and happen in unpredictable environments with possible ambiguities. It is important to have methods capable of generating robot programs easily, that are made as general as possible by handling uncertainties. We present a system that integrates a method to learn Behavior Trees (BTs) from demonstration for pick and place tasks, with a framework that uses verbal interaction to ask follow-up clarification questions to resolve ambiguities. During the execution of a task, the system asks for user input when there is need to disambiguate an object in the scene, when the targets of the task are objects of a same type that are present in multiple instances. The integrated system is demonstrated on different scenarios of a pick and place task, with increasing level of ambiguities. The code used for this paper is made publicly available.

Recent work has broadened the scope of school effectiveness research to consider not only academic achievement but also other outcomes, especially college attendance. This literature has argued that high schools are an important determinant of college attendance, with some contending that high schools matter more for college attendance than for academic achievement. A separate branch of research has illustrated how place-based opportunities facilitate college attendance. We merge these two literatures by asking if schools' geographic context can explain apparent variation in effectiveness among Wisconsin high schools. We find that geographic context explains more than a quarter of the variance in traditional estimates of school effectiveness on college attendance, because factors like proximity to colleges are strongly associated with college attendance. Accounting for geography is therefore important in order not to overstate high schools' role in higher education outcomes. Results are based on multilevel models applied to rich administrative data on every Wisconsin public high school entrant between 2006 and 2011. [This is the online version of an article published in "Research in Higher Education" May 2020.]

Terrorism is a major problem worldwide, causing thousands of fatalities and billions of dollars in damage every year. Toward the end of better understanding and mitigating these attacks, we present a set of machine learning models that learn from localized news data in order to predict whether a terrorist attack will occur on a given calendar date and in a given state. The best model--a Random Forest that learns from a novel variable-length moving average representation of the feature space--achieves area under the receiver operating characteristic scores $> .667$ on four of the five states that were impacted most by terrorism between 2015 and 2018. Our key findings include that modeling terrorism as a set of independent events, rather than as a continuous process, is a fruitful approach--especially when the events are sparse and dissimilar. Additionally, our results highlight the need for localized models that account for differences between locations. From a machine learning perspective, we found that the Random Forest model outperformed several deep models on our multimodal, noisy, and imbalanced data set, thus demonstrating the efficacy of our novel feature representation method in such a context. We also show that its predictions are relatively robust to time gaps between attacks and observed characteristics of the attacks. Finally, we analyze factors that limit model performance, which include a noisy feature space and small amount of available data. These contributions provide an important foundation for the use of machine learning in efforts against terrorism in the United States and beyond.

Fast changing tasks in unpredictable, collaborative environments are typical for medium-small companies, where robotised applications are increasing. Thus, robot programs should be generated in short time with small effort, and the robot able to react dynamically to the environment. To address this we propose a method that combines context awareness and planning to learn Behavior Trees (BTs), a reactive policy representation that is becoming more popular in robotics and has been used successfully in many collaborative scenarios. Context awareness allows to infer from the demonstration the frames in which actions are executed and to capture relevant aspects of the task, while a planner is used to automatically generate the BT from the sequence of actions from the demonstration. The learned BT is shown to solve non-trivial manipulation tasks where learning the context is fundamental to achieve the goal. Moreover, we collected non-expert demonstrations to study the performances of the algorithm in industrial scenarios., Comment: Submitted to RO-MAN 2022

Industrial robots can solve very complex tasks in controlled environments, but modern applications require robots able to operate in unpredictable surroundings as well. An increasingly popular reactive policy architecture in robotics is Behavior Trees but as with other architectures, programming time still drives cost and limits flexibility. There are two main branches of algorithms to generate policies automatically, automated planning and machine learning, both with their own drawbacks. We propose a method for generating Behavior Trees using a Genetic Programming algorithm and combining the two branches by taking the result of an automated planner and inserting it into the population. Experimental results confirm that the proposed method of combining planning and learning performs well on a variety of robotic assembly problems and outperforms both of the base methods used separately. We also show that this type of high level learning of Behavior Trees can be transferred to a real system without further training., Comment: Submitted to IEEE/RSJ IROS 2021

Modern industrial applications require robots to be able to operate in unpredictable environments, and programs to be created with a minimal effort, as there may be frequent changes to the task. In this paper, we show that genetic programming can be effectively used to learn the structure of a behavior tree (BT) to solve a robotic task in an unpredictable environment. Moreover, we propose to use a simple simulator for the learning and demonstrate that the learned BTs can solve the same task in a realistic simulator, reaching convergence without the need for task specific heuristics. The learned solution is tolerant to faults, making our method appealing for real robotic applications.

The up-operators $u_i$ and down-operators $d_i$ (introduced as Schur operators by Fomin) act on partitions by adding/removing a box to/from the $i$th column if possible. It is well known that the $u_i$ alone satisfy the relations of the (local) plactic monoid, and the present authors recently showed that relations of degree at most 4 suffice to describe all relations between the up-operators. Here we characterize the algebra generated by the up- and down-operators together, showing that it can be presented using only quadratic relations., Comment: 13 pages

Behavior Trees (BTs) were invented as a tool to enable modular AI in computer games, but have received an increasing amount of attention in the robotics community in the last decade. With rising demands on agent AI complexity, game programmers found that the Finite State Machines (FSM) that they used scaled poorly and were difficult to extend, adapt and reuse. In BTs, the state transition logic is not dispersed across the individual states, but organized in a hierarchical tree structure, with the states as leaves. This has a significant effect on modularity, which in turn simplifies both synthesis and analysis by humans and algorithms alike. These advantages are needed not only in game AI design, but also in robotics, as is evident from the research being done. In this paper we present a comprehensive survey of the topic of BTs in Artificial Intelligence and Robotic applications. The existing literature is described and categorized based on methods, application areas and contributions, and the paper is concluded with a list of open research challenges.

Past research finds that the effect of socioeconomic origin on the probability of making educational transitions decreases over the educational career from primary to graduate school. Some have argued that this pattern of waning is the result of selective attrition, since those of modest social origins who make a given transition may have exceptional cognitive or noncognitive skills while more advantaged individuals may rely less heavily on these skills to continue their education. We study a sample of American 10th graders from 1980 to assess how much the pattern of waning effects is due to selective attrition along noncognitive skills for this cohort. We find that controlling for noncognitive skills does not make the effect of socioeconomic origin more stable across transitions. Still, socioeconomic advantage does not decline uniformly across transitions, and it appears most pronounced at the transition into college, whether accounting for noncognitive skills or not. Our results suggest that origins continue to drive educational attainment even among those who make it to postsecondary transitions. [This paper was published in "Research in Social Stratification and Mobility" v63 Article 100424 2019.]

In Wisconsin, racial disparities in K-12 achievement have taken center stage, and justifiably so: the black-white and Hispanic-white test score gap is wider in Wisconsin than in any other state (Wisconsin Department of Public Instruction, 2014). However, Wisconsin also sees inequality along economic lines and disparities in postsecondary outcomes, both of which warrant attention. If Wisconsin is like the rest of the nation, closing its gaps in academic achievement is not sufficient to equalize educational attainment. Nationally and in Wisconsin, economically disadvantaged high school graduates attend college, especially baccalaureate colleges, at much lower rates than their more advantaged peers. Schools play an important role in helping economically disadvantaged students go to college. In this report, the authors describe Wisconsin's economic disparities by postsecondary outcomes, assess the magnitude of between-school variation in school effects on economically disadvantaged students' baccalaureate college attendance, and show which school characteristics explain this variation. To determine which types of schools are more successful in sending economically disadvantaged students to baccalaureate colleges, this study uses a relatively dense set of student-level and school-level characteristics. The authors draw these data from the population of Wisconsin public school students who entered ninth grade for the first time between the 2006-07 and 2011-12 school years. Three principal questions guide this study: (1) How large are economic disparities in college attendance in Wisconsin?; (2) How much variation is there among high schools in the share of their low-income students who attend college, controlling for student characteristics?; and (3) Which high school characteristics explain this between-school variation? The questions are approached using the Wisconsin State Longitudinal Data System, the National Student Clearinghouse, and the National Center for Education Statistics' Common Core of Data. This report proceeds in six sections. First, the authors describe their data sources and measurement strategies. Second, they show the magnitude of economic disparities in postsecondary outcomes in Wisconsin. Third, they demonstrate that school effects vary widely. Fourth, they present estimates of how specific school-level characteristics influence low-income students' postsecondary outcomes, highlighting the importance of geographic characteristics such as location in a suburb, proximity to a University of Wisconsin (UW) 4-year campus, and the education level of adults in the district. Fifth, they examine facets of school organization. Finally, the authors offer concluding comments.

In its December 2019 edition, the \textit{Notices of the American Mathematical Society} published an essay critical of the use of diversity statements in academic hiring. The publication of this essay prompted many responses, including three public letters circulated within the mathematical sciences community. Each letter was signed by hundreds of people and was published online, also by the American Mathematical Society. We report on a study of the signatories' demographics, which we infer using a crowdsourcing approach. Letter A highlights diversity and social justice. The pool of signatories contains relatively more individuals inferred to be women and/or members of underrepresented ethnic groups. Moreover, this pool is diverse with respect to the levels of professional security and types of academic institutions represented. Letter B does not comment on diversity, but rather, asks for discussion and debate. This letter was signed by a strong majority of individuals inferred to be white men in professionally secure positions at highly research intensive universities. Letter C speaks out specifically against diversity statements, calling them "a mistake," and claiming that their usage during early stages of faculty hiring "diminishes mathematical achievement." Individuals who signed both Letters B and C, that is, signatories who both privilege debate and oppose diversity statements, are overwhelmingly inferred to be tenured white men at highly research intensive universities. Our empirical results are consistent with theories of power drawn from the social sciences., Comment: 21 pages, 2 tables, 2 figures; minor textual edits made to previous version

We study a representation of the (local) plactic monoid given by Schur operators $u_i$, which act on partitions by adding a box in column $i$ (if possible). In particular, we give a complete list of the relations that hold in the algebra of Schur operators.

Sampling is a fundamental aspect of any implementation of compressive sensing. Typically, the choice of sampling method is guided by the reconstruction basis. However, this approach can be problematic with respect to certain hardware constraints and is not responsive to domain-specific context. We propose a method for defining an order for a sampling basis that is optimal with respect to capturing variance in data, thus allowing for meaningful sensing at any desired level of compression. We focus on the Walsh-Hadamard sampling basis for its relevance to hardware constraints, but our approach applies to any sampling basis of interest. We illustrate the effectiveness of our method on the Physical Sciences Inc. Fabry-P\'{e}rot interferometer sensor multispectral dataset, the Johns Hopkins Applied Physics Lab FTIR-based longwave infrared sensor hyperspectral dataset, and a Colorado State University Swiss Ranger depth image dataset. The spectral datasets consist of simulant experiments, including releases of chemicals such as GAA and SF6. We combine our sampling and reconstruction with the adaptive coherence estimator (ACE) and bulk coherence for chemical detection and we incorporate an algorithmic threshold for ACE values to determine the presence or absence of a chemical. We compare results across sampling methods in this context. We have successful chemical detection at a compression rate of 90%. For all three datasets, we compare our sampling approach to standard orderings of sampling basis such as random, sequency, and an analog of sequency that we term `frequency.' In one instance, the peak signal to noise ratio was improved by over 30% across a test set of depth images., Comment: 15 pages

This work focuses on the problem of in-hand manipulation and regrasping of objects with parallel grippers. We propose Dexterous Manipulation Graph (DMG) as a representation on which we define planning for in-hand manipulation and regrasping. The DMG is a disconnected undirected graph that represents the possible motions of a finger along the object's surface. We formulate the in-hand manipulation and regrasping problem as a graph search problem from the initial to the final configuration. The resulting plan is a sequence of coordinated in-hand pushing and regrasping movements. We propose a dual-arm system for the execution of the sequence where both hands are used interchangeably. We demonstrate our approach on an ABB Yumi robot tasked with different grasp reconfigurations.

We propose the Dexterous Manipulation Graph as a tool to address in-hand manipulation and reposition an object inside a robot's end-effector. This graph is used to plan a sequence of manipulation primitives so to bring the object to the desired end pose. This sequence of primitives is translated into motions of the robot to move the object held by the end-effector. We use a dual arm robot with parallel grippers to test our method on a real system and show successful planning and execution of in-hand manipulation.

In this work we propose an approach to learn a robust policy for solving the pivoting task. Recently, several model-free continuous control algorithms were shown to learn successful policies without prior knowledge of the dynamics of the task. However, obtaining successful policies required thousands to millions of training episodes, limiting the applicability of these approaches to real hardware. We developed a training procedure that allows us to use a simple custom simulator to learn policies robust to the mismatch of simulation vs robot. In our experiments, we demonstrate that the policy learned in the simulator is able to pivot the object to the desired target angle on the real robot. We also show generalization to an object with different inertia, shape, mass and friction properties than those used during training. This result is a step towards making model-free reinforcement learning available for solving robotics tasks via pre-training in simulators that offer only an imprecise match to the real-world dynamics., Comment: (Rika Antonova and Silvia Cruciani contributed equally)