Search

We introduce STRING: Separable Translationally Invariant Position Encodings. STRING extends Rotary Position Encodings, a recently proposed and widely used algorithm in large language models, via a unifying theoretical framework. Importantly, STRING still provides exact translation invariance, including token coordinates of arbitrary dimensionality, whilst maintaining a low computational footprint. These properties are especially important in robotics, where efficient 3D token representation is key. We integrate STRING into Vision Transformers with RGB(-D) inputs (color plus optional depth), showing substantial gains, e.g. in open-vocabulary object detection and for robotics controllers. We complement our experiments with a rigorous mathematical analysis, proving the universality of our methods., Comment: Videos of STRING-based robotics controllers can be found here: https://sites.google.com/view/string-robotics

Multispectral pedestrian detection has gained significant attention in recent years, particularly in autonomous driving applications. To address the challenges posed by adversarial illumination conditions, the combination of thermal and visible images has demonstrated its advantages. However, existing fusion methods rely on the critical assumption that the RGB-Thermal (RGB-T) image pairs are fully overlapping. These assumptions often do not hold in real-world applications, where only partial overlap between images can occur due to sensors configuration. Moreover, sensor failure can cause loss of information in one modality. In this paper, we propose a novel module called the Hybrid Attention (HA) mechanism as our main contribution to mitigate performance degradation caused by partial overlap and sensor failure, i.e. when at least part of the scene is acquired by only one sensor. We propose an improved RGB-T fusion algorithm, robust against partial overlap and sensor failure encountered during inference in real-world applications. We also leverage a mobile-friendly backbone to cope with resource constraints in embedded systems. We conducted experiments by simulating various partial overlap and sensor failure scenarios to evaluate the performance of our proposed method. The results demonstrate that our approach outperforms state-of-the-art methods, showcasing its superiority in handling real-world challenges., Comment: Accepted for publication in IEEE Robotics and Automation Letters, October 2024

Achieving superior polymeric components through additive manufacturing (AM) relies on precise control of rheology. One key rheological property particularly relevant to AM is melt viscosity ($\eta$). Melt viscosity is influenced by polymer chemistry, molecular weight ($M_w$), polydispersity, induced shear rate ($\dot\gamma$), and processing temperature ($T$). The relationship of $\eta$ with $M_w$, $\dot\gamma$, and $T$ may be captured by parameterized equations. Several physical experiments are required to fit the parameters, so predicting $\eta$ of a new polymer material in unexplored physical domains is a laborious process. Here, we develop a Physics-Enforced Neural Network (PENN) model that predicts the empirical parameters and encodes the parametrized equations to calculate $\eta$ as a function of polymer chemistry, $M_w$, polydispersity, $\dot\gamma$, and $T$. We benchmark our PENN against physics-unaware Artificial Neural Network (ANN) and Gaussian Process Regression (GPR) models. Finally, we demonstrate that the PENN offers superior values of $\eta$ when extrapolating to unseen values of $M_w$, $\dot\gamma$, and $T$ for sparsely seen polymers.

User experience in data visualization is typically assessed through post-viewing self-reports, but these overlook the dynamic cognitive processes during interaction. This study explores the use of mind wandering -- a phenomenon where attention spontaneously shifts from a primary task to internal, task-related thoughts or unrelated distractions -- as a dynamic measure during visualization exploration. Participants reported mind wandering while viewing visualizations from a pre-labeled visualization database and then provided quantitative ratings of trust, engagement, and design quality, along with qualitative descriptions and short-term/long-term recall assessments. Results show that mind wandering negatively affects short-term visualization recall and various post-viewing measures, particularly for visualizations with little text annotation. Further, the type of mind wandering impacts engagement and emotional response. Mind wandering also functions as an intermediate process linking visualization design elements to post-viewing measures, influencing how viewers engage with and interpret visual information over time. Overall, this research underscores the importance of incorporating mind wandering as a dynamic measure in visualization design and evaluation, offering novel avenues for enhancing user engagement and comprehension., Comment: 11 pages, 11 figures, 2 tables. IEEE Vis 2024 Full Paper

Occupational exoskeletons promise to alleviate musculoskeletal injuries among industrial workers. Knowledge of the usability of the exoskeleton designs with respect to the user device interaction points, and the problems in design features, functions and parts, evaluated and rated using design principles is still limited. Further, the usability of exoskeletons when assembling, donning, doffing and disassembling them, tasks that can be considered pre and post use tasks are also critical to evaluate, especially from a device design standpoint. We conducted a heuristic evaluation of the usability of three popular exoskeletons, a back support device, a shoulder support device, and a sit stand exoskeleton when assembling, donning, doffing and disassembling them. Seven evaluators used Nielsen and Shneiderman usability heuristics to evaluate the devices. Results indicate that none of the three exoskeletons had any catastrophic usability problems, but all three had major usability problems including accommodating diverse users, the assembly, donning and doffing being a two person operation, poor documentation, a lack of sequence indicators during assembly of the devices, presence of safety hazards while donning and doffing the devices, and manual strength requirements. Further, the assembly task is the most difficult task resulting in the most violations of usability heuristics. The exoskeleton human factors research community should include diverse users in their evaluations and conduct usability, accessibility, and safety evaluations of these devices to provide design feedback to device designers.

Let A be a graph type and B an equivalence relation on a group $G$. Let $[g]$ be the equivalence class of $g$ with respect to the equivalence relation B. The B superA graph of $G$ is an undirected graph whose vertex set is $G$ and two distinct vertices $g, h \in G$ are adjacent if $[g] = [h]$ or there exist $x \in [g]$ and $y \in [h]$ such that $x$ and $y$ are adjacent in the A graph of $G$. In this paper, we compute spectrum of equality/conjugacy supercommuting graphs of dihedral/dicyclic groups and show that these graphs are not integral.

We investigate the two-dimensional behavior of colloidal patchy ellipsoids specifically designed to follow a two-step assembly process from the monomer state to mesoscopic liquid-crystal phases, via the formation of so-called bent-core units at the intermediate stage. Our model comprises a binary mixture of ellipses interacting via the Gay-Berne potential and decorated by surface patches, with the binary components being mirror-image variants of each other - referred to as left-handed and right-handed ellipses according to the position of their patches. The surface patches are designed so as in the first stage of the assembly the monomers form bent-cores units, i.e. V-shaped dimers with a specific bent angle. The Gay-Berne interactions, which act between the ellipses, drive the dimers to subsequently form the characteristic phase observed in bent-core liquid crystals. We numerically investigate -- by means of both Molecular Dynamics and Monte Carlo simulations -- the described two-step process: we first optimize a target bent-core unit and we then fully characterize its state diagram in temperature and density, defining the regions where the different liquid crystalline phases dominate.

We introduce Diffusion Augmented Agents (DAAG), a novel framework that leverages large language models, vision language models, and diffusion models to improve sample efficiency and transfer learning in reinforcement learning for embodied agents. DAAG hindsight relabels the agent's past experience by using diffusion models to transform videos in a temporally and geometrically consistent way to align with target instructions with a technique we call Hindsight Experience Augmentation. A large language model orchestrates this autonomous process without requiring human supervision, making it well-suited for lifelong learning scenarios. The framework reduces the amount of reward-labeled data needed to 1) finetune a vision language model that acts as a reward detector, and 2) train RL agents on new tasks. We demonstrate the sample efficiency gains of DAAG in simulated robotics environments involving manipulation and navigation. Our results show that DAAG improves learning of reward detectors, transferring past experience, and acquiring new tasks - key abilities for developing efficient lifelong learning agents. Supplementary material and visualizations are available on our website https://sites.google.com/view/diffusion-augmented-agents/, Comment: Published at 3rd Conference on Lifelong Learning Agents (CoLLAs), 2024

Occupational exoskeletons promise to reduce the incidence of musculoskeletal injuries; however, we do not know if their designs allow universal use by all workers. We also do not know how easy the tasks of assembling, donning, doffing, and disassembling exoskeletons are. The purpose of our study was to heuristically evaluate a back support, a shoulder support, a handgrip strength support, and a sit-stand exoskeleton for how well they are designed for universal use when assembling, donning, doffing, and disassembling the exoskeleton. Seven evaluators used universal design principles and associated criteria to independently evaluate and rate four exoskeletons when assembling, donning, doffing, and disassembling the devices. The rating scale was a Likert-type scale, where a rating of 1 represented not at all, and a rating of 5 represented an excellent design with respect to the universal design criteria for the task. The results indicate that providing perceptible information to the user, making the design equitable to use for a diverse set of users, making the design simple and intuitive to use with adequate feedback, and designing to prevent user errors, and when errors are made, allowing the user to recover quickly from the errors, were rated poorly. Assembling and donning tasks presented the most challenges.

The surge in AI usage demands innovative power reduction strategies. Novel Compute-in-Memory (CIM) architectures, leveraging advanced memory technologies, hold the potential for significantly lowering energy consumption by integrating storage with parallel Matrix-Vector-Multiplications (MVMs). This study addresses the 1T1R RRAM crossbar, a core component in numerous CIM architectures. We introduce an abstract model and a calibration methodology for estimating operational energy. Our tool condenses circuit-level behaviour into a few parameters, facilitating energy assessments for DNN workloads. Validation against low-level SPICE simulations demonstrates speedups of up to 1000x and energy estimations with errors below 1%., Comment: Pre-print of work presented at AICAS 2024. 5 pages, 6 figures

The U.S. Bureau of Labor Statistics projects that by the year 2029, the United States will lose a million jobs in the office and administrative support occupations because technology, automation, and artificial intelligence (AI) have the potential to substitute or replace the office and administrative functions performed by office workers. Despite the potential impact AI will have on office work and the important role office workers play in the American economy, we have limited knowledge of the state of the art research in office work at the intersection of emerging artificial intelligence technologies. In this study, we conducted a bibliometric analysis of the scholarly literature at the intersection of office work and artificial intelligence. We extracted literature sources from Compendex and Scopus databases and used VOSviewer for visualizing and quantifying our bibliometric analyses. Our findings from keywords analysis indicate that office automation, humans, human-computer interaction, and artificial intelligence occurred more frequently in the scholarly literature and had high link strengths. Keyword clusters from co-occurrence analysis indicate that intelligent buildings, robotics, and the internet of things are emerging topics in the office work domain. The two clusters related to ergonomics, worker characteristics, human performance, and safety indicate the types of human factors concerns that are more widely studied in office work settings. In summary, our findings on the state-of-the-art research in office work indicate that more studies have been conducted on smart buildings, robotics, and technology development for office work, compared to studies on office workers and their professional development., Comment: This work is being submitted to the IEEE for possible publication

We apply multi-agent deep reinforcement learning (RL) to train end-to-end robot soccer policies with fully onboard computation and sensing via egocentric RGB vision. This setting reflects many challenges of real-world robotics, including active perception, agile full-body control, and long-horizon planning in a dynamic, partially-observable, multi-agent domain. We rely on large-scale, simulation-based data generation to obtain complex behaviors from egocentric vision which can be successfully transferred to physical robots using low-cost sensors. To achieve adequate visual realism, our simulation combines rigid-body physics with learned, realistic rendering via multiple Neural Radiance Fields (NeRFs). We combine teacher-based multi-agent RL and cross-experiment data reuse to enable the discovery of sophisticated soccer strategies. We analyze active-perception behaviors including object tracking and ball seeking that emerge when simply optimizing perception-agnostic soccer play. The agents display equivalent levels of performance and agility as policies with access to privileged, ground-truth state. To our knowledge, this paper constitutes a first demonstration of end-to-end training for multi-agent robot soccer, mapping raw pixel observations to joint-level actions, that can be deployed in the real world. Videos of the game-play and analyses can be seen on our website https://sites.google.com/view/vision-soccer .

Transformers have emerged as the underpinning architecture for Large Language Models (LLMs). In generative language models, the inference process involves two primary phases: prompt processing and token generation. Token generation, which constitutes the majority of the computational workload, primarily entails vector-matrix multiplications and interactions with the Key-Value (KV) Cache. This phase is constrained by memory bandwidth due to the overhead of transferring weights and KV cache values from the memory system to the computing units. This memory bottleneck becomes particularly pronounced in applications that require long-context and extensive text generation, both of which are increasingly crucial for LLMs. This paper introduces "Keyformer", an innovative inference-time approach, to mitigate the challenges associated with KV cache size and memory bandwidth utilization. Keyformer leverages the observation that approximately 90% of the attention weight in generative inference focuses on a specific subset of tokens, referred to as "key" tokens. Keyformer retains only the key tokens in the KV cache by identifying these crucial tokens using a novel score function. This approach effectively reduces both the KV cache size and memory bandwidth usage without compromising model accuracy. We evaluate Keyformer's performance across three foundational models: GPT-J, Cerebras-GPT, and MPT, which employ various positional embedding algorithms. Our assessment encompasses a variety of tasks, with a particular emphasis on summarization and conversation tasks involving extended contexts. Keyformer's reduction of KV cache reduces inference latency by 2.1x and improves token generation throughput by 2.4x, while preserving the model's accuracy.

The loss of soil organic carbon (SOC) poses a severe danger to agricultural sustainability around the World. This review examines various farming practices and their impact on soil organic carbon storage. After a careful review of the literature, most of the research indicated that different farming practices, such as organic farming, cover crops, conservation tillage, and agroforestry, play vital roles in increasing the SOC content of the soil sustainably. Root exudation from cover crops increases microbial activity and helps break down complex organic compounds into organic carbon. Conservation tillage enhances the soil structure and maintains carbon storage without disturbing the soil. Agroforestry systems boost organic carbon input and fasten nutrient cycling because the trees and crops have symbiotic relationships. Intercropping and crop rotations have a role in changing the composition of plant residues and promoting carbon storage. There were many understanding on the complex interactions between soil organic carbon dynamics and agricultural practices. Based on the study, the paper reveals, the role of different agricultural practices like Carbon storage through cover crops, crop rotation, mulching Conservation tillage, conventional tillage, zero tillage and organic amendments in organic carbon storage in the soil for maximum crop yield to improve the economic condition of the cultivators.