Search

For most health or well-being interventions, the process of evaluation is distinct from the activity itself, both in terms of who is involved, and how the actual data is collected and analyzed. Tangible interaction affords the opportunity to combine direct and embodied collaboration with a holistic approach to data collection and evaluation. We demonstrate this potential by describing our experiences designing and using the Communal Loom, an artifact for art therapy that translates quantitative data to collectively woven artifacts., Comment: Presented at CHI 2022 Tangible Interaction for Supporting Well-being Workshop

We present TapeAgents, an agent framework built around a granular, structured log tape of the agent session that also plays the role of the session's resumable state. In TapeAgents we leverage tapes to facilitate all stages of the LLM Agent development lifecycle. The agent reasons by processing the tape and the LLM output to produce new thought and action steps and append them to the tape. The environment then reacts to the agent's actions by likewise appending observation steps to the tape. By virtue of this tape-centred design, TapeAgents can provide AI practitioners with holistic end-to-end support. At the development stage, tapes facilitate session persistence, agent auditing, and step-by-step debugging. Post-deployment, one can reuse tapes for evaluation, fine-tuning, and prompt-tuning; crucially, one can adapt tapes from other agents or use revised historical tapes. In this report, we explain the TapeAgents design in detail. We demonstrate possible applications of TapeAgents with several concrete examples of building monolithic agents and multi-agent teams, of optimizing agent prompts and finetuning the agent's LLM. We present tooling prototypes and report a case study where we use TapeAgents to finetune a Llama-3.1-8B form-filling assistant to perform as well as GPT-4o while being orders of magnitude cheaper. Lastly, our comparative analysis shows that TapeAgents's advantages over prior frameworks stem from our novel design of the LLM agent as a resumable, modular state machine with a structured configuration, that generates granular, structured logs and that can transform these logs into training text -- a unique combination of features absent in previous work.

In many settings, machine learning models may be used to inform decisions that impact individuals or entities who interact with the model. Such entities, or agents, may game model decisions by manipulating their inputs to the model to obtain better outcomes and maximize some utility. We consider a multi-agent setting where the goal is to identify the "worst offenders:" agents that are gaming most aggressively. However, identifying such agents is difficult without knowledge of their utility function. Thus, we introduce a framework in which each agent's tendency to game is parameterized via a scalar. We show that this gaming parameter is only partially identifiable. By recasting the problem as a causal effect estimation problem where different agents represent different "treatments," we prove that a ranking of all agents by their gaming parameters is identifiable. We present empirical results in a synthetic data study validating the usage of causal effect estimation for gaming detection and show in a case study of diagnosis coding behavior in the U.S. that our approach highlights features associated with gaming., Comment: 38 pages, 31 figures. NeurIPS 2024

Frontier AI safety policies highlight automation of AI research and development (R&D) by AI agents as an important capability to anticipate. However, there exist few evaluations for AI R&D capabilities, and none that are highly realistic and have a direct comparison to human performance. We introduce RE-Bench (Research Engineering Benchmark, v1), which consists of 7 challenging, open-ended ML research engineering environments and data from 71 8-hour attempts by 61 distinct human experts. We confirm that our experts make progress in the environments given 8 hours, with 82% of expert attempts achieving a non-zero score and 24% matching or exceeding our strong reference solutions. We compare humans to several public frontier models through best-of-k with varying time budgets and agent designs, and find that the best AI agents achieve a score 4x higher than human experts when both are given a total time budget of 2 hours per environment. However, humans currently display better returns to increasing time budgets, narrowly exceeding the top AI agent scores given an 8-hour budget, and achieving 2x the score of the top AI agent when both are given 32 total hours (across different attempts). Qualitatively, we find that modern AI agents possess significant expertise in many ML topics -- e.g. an agent wrote a faster custom Triton kernel than any of our human experts' -- and can generate and test solutions over ten times faster than humans, at much lower cost. We open-source the evaluation environments, human expert data, analysis code and agent trajectories to facilitate future research.

Eddy-resolving turbulence simulations require stochastic inflow conditions that accurately replicate the complex, multi-scale structures of turbulence. Traditional recycling-based methods rely on computationally expensive precursor simulations, while existing synthetic inflow generators often fail to reproduce realistic coherent structures of turbulence. Recent advances in deep learning (DL) have opened new possibilities for inflow turbulence generation, yet many DL-based methods rely on deterministic, autoregressive frameworks prone to error accumulation, resulting in poor robustness for long-term predictions. In this work, we present CoNFiLD-inlet, a novel DL-based inflow turbulence generator that integrates diffusion models with a conditional neural field (CNF)-encoded latent space to produce realistic, stochastic inflow turbulence. By parameterizing inflow conditions using Reynolds numbers, CoNFiLD-inlet generalizes effectively across a wide range of Reynolds numbers ($Re_\tau$ between $10^3$ and $10^4$) without requiring retraining or parameter tuning. Comprehensive validation through a priori and a posteriori tests in Direct Numerical Simulation (DNS) and Wall-Modeled Large Eddy Simulation (WMLES) demonstrates its high fidelity, robustness, and scalability, positioning it as an efficient and versatile solution for inflow turbulence synthesis., Comment: 27 pages, 10 figures

Sequential learning in a multi-agent resource constrained matching market has received significant interest in the past few years. We study decentralized learning in two-sided matching markets where the demand side (aka players or agents) competes for a `large' supply side (aka arms) with potentially time-varying preferences, to obtain a stable match. Despite a long line of work in the recent past, existing learning algorithms such as Explore-Then-Commit or Upper-Confidence-Bound remain inefficient for this problem. In particular, the per-agent regret achieved by these algorithms scales linearly with the number of arms, $K$. Motivated by the linear contextual bandit framework, we assume that for each agent an arm-mean can be represented by a linear function of a known feature vector and an unknown (agent-specific) parameter. Moreover, our setup captures the essence of a dynamic (non-stationary) matching market where the preferences over arms change over time. Our proposed algorithms achieve instance-dependent logarithmic regret, scaling independently of the number of arms, $K$.

CRAFT@Large (C@L) is an initiative launched by the MakerLAB at Cornell Tech to create an inclusive environment for the intercultural and intergenerational exchange of ideas through making. With our approach, we challenge the traditional definition of community outreach performed by academic makerspaces. Existing academic makerspaces often perform community engagement by only offering hourly, one-time workshops or by having community members provide a problem that is then used by students as a project assignment. These approaches position community members as occasional visitors and non-equal contributors, which not only conflict with the core values of co-creation but also limit the makerspaces' impact on connecting the universities and the communities. C@L explored an alternative approach in which we invited community members as long-term and equal co-makers into the academic makerspaces. In this article, we showcase two sets of collaborations that illustrate the continuity of people through co-making. We present how academic makerspaces can function as a hub that connects community members and partner organizations with the campus community in a long-term relationship.

Image-based diagnostic decision support systems (DDSS) utilizing deep learning have the potential to optimize clinical workflows. However, developing DDSS requires extensive datasets with expert annotations and is therefore costly. Leveraging report contents from radiological data bases with Natural Language Processing to annotate the corresponding image data promises to replace labor-intensive manual annotation. As mining "real world" databases can introduce label noise, noise-robust training losses are of great interest. However, current noise-robust losses do not consider noise estimations that can for example be derived based on the performance of the automatic label generator used. In this study, we expand the noise-robust Deep Abstaining Classifier (DAC) loss to an Informed Deep Abstaining Classifier (IDAC) loss by incorporating noise level estimations during training. Our findings demonstrate that IDAC enhances the noise robustness compared to DAC and several state-of-the-art loss functions. The results are obtained on various simulated noise levels using a public chest X-ray data set. These findings are reproduced on an in-house noisy data set, where labels were extracted from the clinical systems of the University Hospital Bonn by a text-based transformer. The IDAC can therefore be a valuable tool for researchers, companies or clinics aiming to develop accurate and reliable DDSS from routine clinical data., Comment: This preprint has no post-submission improvements or corrections. The Version of Record of this contribution is published in the Neural Information Processing, ICONIP 2024 Proceedings

Long-range attractive interactions between dark matter particles can significantly enhance their annihilation, particularly at low velocities. This ``Sommerfeld enhancement'' is typically computed by evaluating the deformation of the two-particle wavefunction due to the long-range potential, while ignoring the physics associated with the annihilation, and then scaling the appropriate annihilation matrix elements by factors that depend on the wavefunction in the limit where the particles approach zero relative separation. It has long been recognized that this approach is a valid approximation only in the limit where the annihilation rate is small, and breaks down in the regime where the enhanced annihilation rate approaches the unitarity bound, in which case ignoring the impact of the annihilation physics on the two-particle wavefunction cannot be justified and leads to apparent violations of unitarity. In the case where the physics relevant to annihilation occurs at a parametrically shorter distance scale (higher energy scale) compared with the long-range potential, we provide a simple prescription for correcting the Sommerfeld enhancement for the effects of the short-range physics, valid for all partial waves and for systems where multiple states are coupled by the long-range potential., Comment: 70 pages, 17 figures

Smart vehicles produce large amounts of data, much of which is sensitive and at risk of privacy breaches. As attackers increasingly exploit anonymised metadata within these datasets to profile drivers, it's important to find solutions that mitigate this information leakage without hindering innovation and ongoing research. Synthetic data has emerged as a promising tool to address these privacy concerns, as it allows for the replication of real-world data relationships while minimising the risk of revealing sensitive information. In this paper, we examine the use of synthetic data to tackle these challenges. We start by proposing a comprehensive taxonomy of 14 in-vehicle sensors, identifying potential attacks and categorising their vulnerability. We then focus on the most vulnerable signals, using the Passive Vehicular Sensor (PVS) dataset to generate synthetic data with a Tabular Variational Autoencoder (TVAE) model, which included over 1 million data points. Finally, we evaluate this against 3 core metrics: fidelity, utility, and privacy. Our results show that we achieved 90.1% statistical similarity and 78% classification accuracy when tested on its original intent while also preventing the profiling of the driver. The code can be found at https://github.com/krish-parikh/Synthetic-Data-Generation

This paper presents a computationally efficient model predictive control formulation that uses an integral Chebyshev collocation method to enable rapid operations of autonomous agents. By posing the finite-horizon optimal control problem and recursive re-evaluation of the optimal trajectories, minimization of the L2 norms of the state and control errors are transcribed into a quadratic program. Control and state variable constraints are parameterized using Chebyshev polynomials and are accommodated in the optimal trajectory generation programs to incorporate the actuator limits and keepout constraints. Differentiable collision detection of polytopes is leveraged for optimal collision avoidance. Results obtained from the collocation methods are benchmarked against the existing approaches on an edge computer to outline the performance improvements. Finally, collaborative control scenarios involving multi-agent space systems are considered to demonstrate the technical merits of the proposed work., Comment: This work has been submitted to the IEEE ACC 2025 for possible publication

The sensitivity of electron EDM experiments has been increasing at a rapid pace, and could yield indications of new physics in the coming decade. An intriguing possibility is that an EDM signal could be generated by new, electroweak-charged particles at the TeV scale that couple to the Higgs and contribute to the electron EDM at two-loop order via Barr-Zee diagrams. A high-energy muon collider could decisively search for new physics at this scale. In this work, we explore this complementarity between colliders and EDM experiments, and note that Barr-Zee diagrams from the aforementioned particles are closely related to vector-boson scattering processes at a muon collider. These loop corrections lead to kinematic features in the differential cross sections of these processes, dictated by the optical theorem. We demonstrate this connection in the context of the singlet-doublet and doublet-triplet extensions to the SM, explore the detectability of these features at a muon collider experiment, and discuss how these measurements can be used to ascertain the underlying model parameters., Comment: 22 pages + appendices, 9 figures

The scarcity of subspecialist medical expertise, particularly in rare, complex and life-threatening diseases, poses a significant challenge for healthcare delivery. This issue is particularly acute in cardiology where timely, accurate management determines outcomes. We explored the potential of AMIE (Articulate Medical Intelligence Explorer), a large language model (LLM)-based experimental AI system optimized for diagnostic dialogue, to potentially augment and support clinical decision-making in this challenging context. We curated a real-world dataset of 204 complex cases from a subspecialist cardiology practice, including results for electrocardiograms, echocardiograms, cardiac MRI, genetic tests, and cardiopulmonary stress tests. We developed a ten-domain evaluation rubric used by subspecialists to evaluate the quality of diagnosis and clinical management plans produced by general cardiologists or AMIE, the latter enhanced with web-search and self-critique capabilities. AMIE was rated superior to general cardiologists for 5 of the 10 domains (with preference ranging from 9% to 20%), and equivalent for the rest. Access to AMIE's response improved cardiologists' overall response quality in 63.7% of cases while lowering quality in just 3.4%. Cardiologists' responses with access to AMIE were superior to cardiologist responses without access to AMIE for all 10 domains. Qualitative examinations suggest AMIE and general cardiologist could complement each other, with AMIE thorough and sensitive, while general cardiologist concise and specific. Overall, our results suggest that specialized medical LLMs have the potential to augment general cardiologists' capabilities by bridging gaps in subspecialty expertise, though further research and validation are essential for wide clinical utility.

A Markov decision process-based state switching is devised, implemented, and analyzed for proximity operations of various autonomous vehicles. The framework contains a pose estimator along with a multi-state guidance algorithm. The unified pose estimator leverages the extended Kalman filter for the fusion of measurements from rate gyroscopes, monocular vision, and ultra-wideband radar sensors. It is also equipped with Mahalonobis distance-based outlier rejection and under-weighting of measurements for robust performance. The use of probabilistic Markov models to transition between various guidance modes is proposed to enable robust and efficient proximity operations. Finally, the framework is validated through an experimental analysis of the docking of two small satellites and the precision landing of an aerial vehicle., Comment: This work has been submitted to the IEEE ICRA 2025 for possible publication. Accompanying video : https://youtu.be/8-fetyf_SrM. arXiv admin note: text overlap with arXiv:2409.09665

Large Language Models (LLMs) are increasingly being used to generate text across various languages, for tasks such as translation, customer support, and education. Despite these advancements, LLMs show notable gender biases in English, which become even more pronounced when generating content in relatively underrepresented languages like Hindi. This study explores implicit gender biases in Hindi text generation and compares them to those in English. We developed Hindi datasets inspired by WinoBias to examine stereotypical patterns in responses from models like GPT-4o and Claude-3 sonnet. Our results reveal a significant gender bias of 87.8% in Hindi, compared to 33.4% in English GPT-4o generation, with Hindi responses frequently relying on gender stereotypes related to occupations, power hierarchies, and social class. This research underscores the variation in gender biases across languages and provides considerations for navigating these biases in generative AI systems.

A pose estimation technique based on error-state extended Kalman that fuses angular rates, accelerations, and relative range measurements is presented in this paper. An unconstrained dynamic model with kinematic coupling for a thrust-capable satellite is considered for the state propagation, and a pragmatic measurement model of the rate gyroscope, accelerometer, and an ultra-wideband radio are leveraged for the measurement update. The error-state extended Kalman filter framework is formulated for pose estimation, and its performance has been analyzed via several simulation scenarios. An application of the pose estimator for proximity operations and scaffolding formation of CubeSat deputies relative to their mother-ship is outlined. Finally, the performance of the error-state extended Kalman filter is demonstrated using experimental analysis consisting of a 3-DOF thrust cable satellite mock-up, rate gyroscope, accelerometer, and ultra-wideband radar modules., Comment: Presented at 46th Rocky Mountain AAS Guidance, Navigation and Control (GN&C) Conference, Breckenridge, CO

We analyse the stellar structure of a sample of dwarf ellipticals (dE) inhabiting various environments within the Virgo cluster. Integral-field observations with a high spectral resolution allow us to robustly determine their low velocity dispersions ($\sim25$ km s$^{-1}$) and higher-order kinematic moments out to the half-light radius. We find the dEs exhibit a diversity in ages with the younger dEs being less enhanced than the older, suggesting a complex star formation history for those dEs that recently entered Virgo while others have been quenched shortly after reionization. Orbit-superposition modeling allowed us to recover viewing angles, stellar mass-to-light ratios (with gradients), as well as the intrinsic orbit structure. We find that the angular momentum of the dEs is strongly suppressed compared to ordinary early-type galaxies and correlates with the environment. Flattened dEs are so because of a suppressed kinetic energy perpendicular to their equatorial plane. Combining population and dynamical modeling results, we find an age-dependent stellar initial mass function (IMF) or, alternatively, evidence for a more extended star formation history for those galaxies that have had higher initial mass and/or inhabited lower density environments. dEs appear to have a spatially homogeneous stellar structure but the state they were `frozen' in as they stopped forming stars varies dramatically according to their initial conditions., Comment: Accepted for publication in ApJ, 56 pages, 27 figures, 3 tables

A robust pose estimation algorithm based on an extended Kalman filter using measurements from accelerometers, rate gyroscopes, monocular vision and ultra-wideband radar is presented. The sensor fusion and pose estimation algorithm incorporates Mahalonobis distance-based outlier rejection and under-weighting of measurements for robust filter performance in the case of sudden range measurements led by the absence of measurements due to range limitations of radar transceivers. The estimator is further validated through an experimental analysis using low-cost radar, IMU and camera sensors. The pose estimate is utilized to perform proximity operations and docking of Transforming Proximity Operations and Docking Service (TPODS) satellite modules with a fixed target.

The LASR Laboratory is investigating the use of free-flying spacecraft modules in several on-orbit, servicing and manufacturing (OSAM) activities. Previous work consists of the system development and testing of the aforementioned thrust-capable modules. This study makes advancements to devise, implement and validate an algorithm for the estimation of inertial parameters of a rigid structure, to be maneuvered with the help of Transforming Proximity Operations and Docking Service (TPODS) satellite modules. The primary contribution of this activity is observability analysis to infer a conducive input sequence for estimating the inertial parameters. For the experimental validation of proposed estimation algorithm, real-time pose measurements are logged through the VICON motion capture system and the recorded data is utilized to assess the performance of the estimation algorithm to predict mass and moment of inertia of an isolated TPODS module., Comment: Presented at 2023 AAS/AIAA Astrodynamics Specialist Conference, Big Sky, MT

This paper presents a detailed system design and component selection for the Transforming Proximity Operations and Docking Service (TPODS) module, designed to gain custody of uncontrolled resident space objects (RSOs) via rendezvous and proximity operation (RPO). In addition to serving as a free-flying robotic manipulator to work with cooperative and uncooperative RSOs, the TPODS modules are engineered to have the ability to cooperate with one another to build scaffolding for more complex satellite servicing activities. The structural design of the prototype module is inspired by Tensegrity principles, minimizing the structural mass of the modules frame. The prototype TPODS module is fabricated using lightweight polycarbonate with an aluminum or carbon fiber frame. The inner shell that houses various electronic and pneumatic components is 3-D printed using ABS material. Four OpenMV H7 R1 cameras are used for the pose estimation of resident space objects (RSOs), including other TPODS modules. Compressed air supplied by an external source is used for the initial testing and can be replaced by module-mounted nitrogen pressure vessels for full on-board propulsion later. A Teensy 4.1 single-board computer is used as a central command unit that receives data from the four OpenMV cameras, and commands its thrusters based on the control logic., Comment: Presented at 45th Rocky Mountain AAS Guidance, Navigation and Control (GN&C) Conference, Breckenridge, CO

Transformer based Large Language Models (LLMs) have recently reached state of the art performance in Natural Language Processing (NLP) and Computer Vision (CV) domains. LLMs use the Multi-Headed Self-Attention (MHSA) mechanism to capture long-range global attention relationships among input words or image patches, drastically improving its performance over prior deep learning approaches. In this paper, we evaluate the performance of LLMs on the Cerebras Wafer Scale Engine (WSE). Cerebras WSE is a high performance computing system with 2.6 trillion transistors, 850,000 cores and 40 GB on-chip memory. Cerebras WSE's Sparse Linear Algebra Compute (SLAC) cores eliminates multiply-by-zeros operations and its 40 GB of on-chip memory is uniformly distributed among SLAC cores, enabling fast local access to model parameters. Moreover, Cerebras software configures routing between cores at runtime, optimizing communication overhead among cores. As LLMs are becoming more commonly used, new hardware architectures are needed to accelerate LLMs training and inference. We benchmark the effectiveness of this hardware architecture at accelerating LLMs training and inference. Additionally, we analyze if Cerebras WSE can scale the memory-wall associated with traditionally memory-bound compute tasks using its 20 PB/s high bandwidth memory. Furthermore, we examine the performance scalability of Cerebras WSE through a roofline model. By plotting performance metrics against computational intensity, we aim to assess their effectiveness at handling high compute-intensive LLMs training and inference tasks., Comment: IEEE HPEC 2024

We establish two spectral sequences in knot Floer homology associated to a directed strongly invertible knot K: one from the knot Floer homology of K to a two dimensional vector space, and one from the singular knot Floer homology of a singular knot associated to K to the knot Floer homology quotient knot of K. The first of these spectral sequences is used to define a numerical invariant of strongly invertible knots., Comment: 50 pages, 26 figures. Comments welcome!

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.

Depression is characterized by persistent sadness and loss of interest, significantly impairing daily functioning and now a widespread mental disorder. Traditional diagnostic methods rely on subjective assessments, necessitating objective approaches for accurate diagnosis. Our study investigates the use of facial action units (AUs) and emotions as biomarkers for depression. We analyzed facial expressions from video data of participants classified with or without depression. Our methodology involved detailed feature extraction, mean intensity comparisons of key AUs, and the application of time series classification models. Furthermore, we employed Principal Component Analysis (PCA) and various clustering algorithms to explore the variability in emotional expression patterns. Results indicate significant differences in the intensities of AUs associated with sadness and happiness between the groups, highlighting the potential of facial analysis in depression assessment.

A fundamental step in the patent application process is the determination of whether there exist prior patents that are novelty destroying. This step is routinely performed by both applicants and examiners, in order to assess the novelty of proposed inventions among the millions of applications filed annually. However, conducting this search is time and labor-intensive, as searchers must navigate complex legal and technical jargon while covering a large amount of legal claims. Automated approaches using information retrieval and machine learning approaches to detect novelty destroying patents present a promising avenue to streamline this process, yet research focusing on this space remains limited. In this paper, we introduce a novel data pipeline, ClaimCompare, designed to generate labeled patent claim datasets suitable for training IR and ML models to address this challenge of novelty destruction assessment. To the best of our knowledge, ClaimCompare is the first pipeline that can generate multiple novelty destroying patent datasets. To illustrate the practical relevance of this pipeline, we utilize it to construct a sample dataset comprising of over 27K patents in the electrochemical domain: 1,045 base patents from USPTO, each associated with 25 related patents labeled according to their novelty destruction towards the base patent. Subsequently, we conduct preliminary experiments showcasing the efficacy of this dataset in fine-tuning transformer models to identify novelty destroying patents, demonstrating 29.2% and 32.7% absolute improvement in MRR and P@1, respectively.

Identifying patient cohorts is fundamental to numerous healthcare tasks, including clinical trial recruitment and retrospective studies. Current cohort retrieval methods in healthcare organizations rely on automated queries of structured data combined with manual curation, which are time-consuming, labor-intensive, and often yield low-quality results. Recent advancements in large language models (LLMs) and information retrieval (IR) offer promising avenues to revolutionize these systems. Major challenges include managing extensive eligibility criteria and handling the longitudinal nature of unstructured Electronic Medical Records (EMRs) while ensuring that the solution remains cost-effective for real-world application. This paper introduces a new task, Automatic Cohort Retrieval (ACR), and evaluates the performance of LLMs and commercial, domain-specific neuro-symbolic approaches. We provide a benchmark task, a query dataset, an EMR dataset, and an evaluation framework. Our findings underscore the necessity for efficient, high-quality ACR systems capable of longitudinal reasoning across extensive patient databases.

Self-recording eating behaviors is a step towards a healthy lifestyle recommended by many health professionals. However, the current practice of manually recording eating activities using paper records or smartphone apps is often unsustainable and inaccurate. Smart glasses have emerged as a promising wearable form factor for tracking eating behaviors, but existing systems primarily identify when eating occurs without capturing details of the eating activities (E.g., what is being eaten). In this paper, we present EchoGuide, an application and system pipeline that leverages low-power active acoustic sensing to guide head-mounted cameras to capture egocentric videos, enabling efficient and detailed analysis of eating activities. By combining active acoustic sensing for eating detection with video captioning models and large-scale language models for retrieval augmentation, EchoGuide intelligently clips and analyzes videos to create concise, relevant activity records on eating. We evaluated EchoGuide with 9 participants in naturalistic settings involving eating activities, demonstrating high-quality summarization and significant reductions in video data needed, paving the way for practical, scalable eating activity tracking., Comment: Accepted at ISWC '24

INTRODUCTION: The pharmacological treatment of Major Depressive Disorder (MDD) relies on a trial-and-error approach. We introduce an artificial intelligence (AI) model aiming to personalize treatment and improve outcomes, which was deployed in the Artificial Intelligence in Depression Medication Enhancement (AIDME) Study. OBJECTIVES: 1) Develop a model capable of predicting probabilities of remission across multiple pharmacological treatments for adults with at least moderate major depression. 2) Validate model predictions and examine them for amplification of harmful biases. METHODS: Data from previous clinical trials of antidepressant medications were standardized into a common framework and included 9,042 adults with moderate to severe major depression. Feature selection retained 25 clinical and demographic variables. Using Bayesian optimization, a deep learning model was trained on the training set, refined using the validation set, and tested once on the held-out test set. RESULTS: In the evaluation on the held-out test set, the model demonstrated achieved an AUC of 0.65. The model outperformed a null model on the test set (p = 0.01). The model demonstrated clinical utility, achieving an absolute improvement in population remission rate in hypothetical and actual improvement testing. While the model did identify one drug (escitalopram) as generally outperforming the other drugs (consistent with the input data), there was otherwise significant variation in drug rankings. On bias testing, the model did not amplify potentially harmful biases. CONCLUSIONS: We demonstrate the first model capable of predicting outcomes for 10 different treatment options for patients with MDD, intended to be used at or near the start of treatment to personalize treatment. The model was put into clinical practice during the AIDME randomized controlled trial whose results are reported separately.

Recent advancements in Artificial Intelligence (AI) have largely been propelled by scaling. In Robotics, scaling is hindered by the lack of access to massive robot datasets. We advocate using realistic physical simulation as a means to scale environments, tasks, and datasets for robot learning methods. We present RoboCasa, a large-scale simulation framework for training generalist robots in everyday environments. RoboCasa features realistic and diverse scenes focusing on kitchen environments. We provide thousands of 3D assets across over 150 object categories and dozens of interactable furniture and appliances. We enrich the realism and diversity of our simulation with generative AI tools, such as object assets from text-to-3D models and environment textures from text-to-image models. We design a set of 100 tasks for systematic evaluation, including composite tasks generated by the guidance of large language models. To facilitate learning, we provide high-quality human demonstrations and integrate automated trajectory generation methods to substantially enlarge our datasets with minimal human burden. Our experiments show a clear scaling trend in using synthetically generated robot data for large-scale imitation learning and show great promise in harnessing simulation data in real-world tasks. Videos and open-source code are available at https://robocasa.ai/, Comment: RSS 2024

Designing faithful yet accurate AI models is challenging, particularly in the field of individual treatment effect estimation (ITE). ITE prediction models deployed in critical settings such as healthcare should ideally be (i) accurate, and (ii) provide faithful explanations. However, current solutions are inadequate: state-of-the-art black-box models do not supply explanations, post-hoc explainers for black-box models lack faithfulness guarantees, and self-interpretable models greatly compromise accuracy. To address these issues, we propose DISCRET, a self-interpretable ITE framework that synthesizes faithful, rule-based explanations for each sample. A key insight behind DISCRET is that explanations can serve dually as database queries to identify similar subgroups of samples. We provide a novel RL algorithm to efficiently synthesize these explanations from a large search space. We evaluate DISCRET on diverse tasks involving tabular, image, and text data. DISCRET outperforms the best self-interpretable models and has accuracy comparable to the best black-box models while providing faithful explanations. DISCRET is available at https://github.com/wuyinjun-1993/DISCRET-ICML2024., Comment: Accepted at ICML 2024. 22 pages, 5 figures

The recent introduction of OpenAI's text-to-video model Sora has sparked widespread public discourse across online communities. This study aims to uncover the dominant themes and narratives surrounding Sora by conducting topic modeling analysis on a corpus of 1,827 Reddit comments from five relevant subreddits (r/OpenAI, r/technology, r/singularity, r/vfx, and r/ChatGPT). The comments were collected over a two-month period following Sora's announcement in February 2024. After preprocessing the data, Latent Dirichlet Allocation (LDA) was employed to extract four key topics: 1) AI Impact and Trends in Sora Discussions, 2) Public Opinion and Concerns about Sora, 3) Artistic Expression and Video Creation with Sora, and 4) Sora's Applications in Media and Entertainment. Visualizations including word clouds, bar charts, and t-SNE clustering provided insights into the importance of topic keywords and the distribution of comments across topics. The results highlight prominent narratives around Sora's potential impact on industries and employment, public sentiment and ethical concerns, creative applications, and use cases in the media and entertainment sectors. While limited to Reddit data within a specific timeframe, this study offers a framework for understanding public perceptions of emerging generative AI technologies through online discourse analysis.

Finetuning large language models (LLMs) in federated learning (FL) settings has become increasingly important as it allows resource-constrained devices to finetune a model using private data. However, finetuning LLMs using backpropagation requires excessive memory (especially from intermediate activations) for resource-constrained devices. While Forward-mode Auto-Differentiation (AD) can significantly reduce memory footprint from activations, we observe that directly applying it to LLM finetuning results in slow convergence and poor accuracy. In this paper, we introduce Spry, an FL algorithm that splits trainable weights of an LLM among participating clients, such that each client computes gradients using forward-mode AD that are closer estimations of the true gradients. Spry achieves a low memory footprint, high accuracy, and fast convergence. We formally prove that the global gradients in Spry are unbiased estimators of true global gradients for homogeneous data distributions across clients, while heterogeneity increases bias of the estimates. We also derive Spry's convergence rate, showing that the gradients decrease inversely proportional to the number of FL rounds, indicating the convergence up to the limits of heterogeneity. Empirically, Spry reduces the memory footprint during training by 1.4-7.1x in contrast to backpropagation, while reaching comparable accuracy, across a wide range of language tasks, models, and FL settings. Spry reduces the convergence time by 1.2-20.3x and achieves 5.2-13.5% higher accuracy against zero-order methods. When finetuning Llama2-7B with LoRA, compared to the peak memory consumption of 33.9GB of backpropagation, Spry only consumes 6.2GB of peak memory. For OPT13B, the reduction is from 76.5GB to 10.8GB. Spry makes feasible previously impossible FL deployments on commodity edge devices. Our source code is available at https://github.com/Astuary/Spry., Comment: Accepted to NeurIPS 2024

Despite recent advances in diagnosis and treatment, atherosclerotic coronary artery diseases remain a leading cause of death worldwide. Various imaging modalities and metrics can detect lesions and predict patients at risk; however, identifying unstable lesions is still difficult. Current techniques cannot fully capture the complex morphology-modulated mechanical responses that affect plaque stability, leading to catastrophic failure and mute the benefit of device and drug interventions. Finite Element (FE) simulations utilizing intravascular imaging OCT (Optical Coherence Tomography) are effective in defining physiological stress distributions. However, creating 3D FE simulations of coronary arteries from OCT images is challenging to fully automate given OCT frame sparsity, limited material contrast, and restricted penetration depth. To address such limitations, we developed an algorithmic approach to automatically produce 3D FE-ready digital twins from labeled OCT images. The 3D models are anatomically faithful and recapitulate mechanically relevant tissue lesion components, automatically producing morphologies structurally similar to manually constructed models whilst including more minute details. A mesh convergence study highlighted the ability to reach stress and strain convergence with average errors of just 5.9% and 1.6% respectively in comparison to FE models with approximately twice the number of elements in areas of refinement. Such an automated procedure will enable analysis of large clinical cohorts at a previously unattainable scale and opens the possibility for in-silico methods for patient specific diagnoses and treatment planning for coronary artery disease.

The "massively-multilingual" training of multilingual models is known to limit their utility in any one language, and they perform particularly poorly on low-resource languages. However, there is evidence that low-resource languages can benefit from targeted multilinguality, where the model is trained on closely related languages. To test this approach more rigorously, we systematically study best practices for adapting a pre-trained model to a language family. Focusing on the Uralic family as a test case, we adapt XLM-R under various configurations to model 15 languages; we then evaluate the performance of each experimental setting on two downstream tasks and 11 evaluation languages. Our adapted models significantly outperform mono- and multilingual baselines. Furthermore, a regression analysis of hyperparameter effects reveals that adapted vocabulary size is relatively unimportant for low-resource languages, and that low-resource languages can be aggressively up-sampled during training at little detriment to performance in high-resource languages. These results introduce new best practices for performing language adaptation in a targeted setting.

We demonstrate the capabilities of an attention-based end-to-end approach for high-speed vision-based quadrotor obstacle avoidance in dense, cluttered environments, with comparison to various state-of-the-art learning architectures. Quadrotor unmanned aerial vehicles (UAVs) have tremendous maneuverability when flown fast; however, as flight speed increases, traditional model-based approaches to navigation via independent perception, mapping, planning, and control modules breaks down due to increased sensor noise, compounding errors, and increased processing latency. Thus, learning-based, end-to-end vision-to-control networks have shown to have great potential for online control of these fast robots through cluttered environments. We train and compare convolutional, U-Net, and recurrent architectures against vision transformer (ViT) models for depth image-to-control in high-fidelity simulation, observing that ViT models are more effective than others as quadrotor speeds increase and in generalization to unseen environments, while the addition of recurrence further improves performance while reducing quadrotor energy cost across all tested flight speeds. We assess performance at speeds of up to 7m/s in simulation and hardware. To the best of our knowledge, this is the first work to utilize vision transformers for end-to-end vision-based quadrotor control., Comment: 11 pages, 18 figures, 3 tables (with supplementary)

Recognizing driving behaviors is important for downstream tasks such as reasoning, planning, and navigation. Existing video recognition approaches work well for common behaviors (e.g. "drive straight", "brake", "turn left/right"). However, the performance is sub-par for underrepresented/rare behaviors typically found in tail of the behavior class distribution. To address this shortcoming, we propose Transfer-LMR, a modular training routine for improving the recognition performance across all driving behavior classes. We extensively evaluate our approach on METEOR and HDD datasets that contain rich yet heavy-tailed distribution of driving behaviors and span diverse traffic scenarios. The experimental results demonstrate the efficacy of our approach, especially for recognizing underrepresented/rare driving behaviors.