Search

Teaching robots to autonomously complete everyday tasks remains a challenge. Imitation Learning (IL) is a powerful approach that imbues robots with skills via demonstrations, but is limited by the labor-intensive process of collecting teleoperated robot data. Human videos offer a scalable alternative, but it remains difficult to directly train IL policies from them due to the lack of robot action labels. To address this, we propose to represent actions as short-horizon 2D trajectories on an image. These actions, or motion tracks, capture the predicted direction of motion for either human hands or robot end-effectors. We instantiate an IL policy called Motion Track Policy (MT-pi) which receives image observations and outputs motion tracks as actions. By leveraging this unified, cross-embodiment action space, MT-pi completes tasks with high success given just minutes of human video and limited additional robot demonstrations. At test time, we predict motion tracks from two camera views, recovering 6DoF trajectories via multi-view synthesis. MT-pi achieves an average success rate of 86.5% across 4 real-world tasks, outperforming state-of-the-art IL baselines which do not leverage human data or our action space by 40%, and generalizes to scenarios seen only in human videos. Code and videos are available on our website https://portal-cornell.github.io/motion_track_policy/.

Safe knife practices in the kitchen significantly reduce the risk of cuts, injuries, and serious accidents during food preparation. Using YOLOv7, an advanced object detection model, this study focuses on identifying safety risks during knife handling, particularly improper finger placement and blade contact with hand. The model's performance was evaluated using metrics such as precision, recall, mAP50, and mAP50-95. The results demonstrate that YOLOv7 achieved its best performance at epoch 31, with a mAP50-95 score of 0.7879, precision of 0.9063, and recall of 0.7503. These findings highlight YOLOv7's potential to accurately detect knife-related hazards, promoting the development of improved kitchen safety.

Software testing is a crucial but time-consuming aspect of software development, and recently, Large Language Models (LLMs) have gained popularity for automated test case generation. However, because LLMs are trained on vast amounts of open-source code, they often generate test cases that do not adhere to best practices and may even contain test smells (anti-patterns). To address this issue, we propose Reinforcement Learning from Static Quality Metrics (RLSQM), wherein we utilize Reinforcement Learning to generate high-quality unit tests based on static analysis-based quality metrics. First, we analyzed LLM-generated tests and show that LLMs frequently do generate undesirable test smells -- up to 37% of the time. Then, we implemented lightweight static analysis-based reward model and trained LLMs using this reward model to optimize for five code quality metrics. Our experimental results demonstrate that the RL-optimized Codex model consistently generated higher-quality test cases than the base LLM, improving quality metrics by up to 23%, and generated nearly 100% syntactically-correct code. RLSQM also outperformed GPT-4 on all code quality metrics, in spite of training a substantially cheaper Codex model. We provide insights into how reliably utilize RL to improve test generation quality and show that RLSQM is a significant step towards enhancing the overall efficiency and reliability of automated software testing. Our data are available at https://doi.org/10.6084/m9.figshare.25983166., Comment: This work was intended as a replacement of arXiv:2310.02368 and any subsequent updates will appear there

This work explores the YOLOv6 object detection model in depth, concentrating on its design framework, optimization techniques, and detection capabilities. YOLOv6's core elements consist of the EfficientRep Backbone for robust feature extraction and the Rep-PAN Neck for seamless feature aggregation, ensuring high-performance object detection. Evaluated on the COCO dataset, YOLOv6-N achieves 37.5\% AP at 1187 FPS on an NVIDIA Tesla T4 GPU. YOLOv6-S reaches 45.0\% AP at 484 FPS, outperforming models like PPYOLOE-S, YOLOv5-S, YOLOX-S, and YOLOv8-S in the same class. Moreover, YOLOv6-M and YOLOv6-L also show better accuracy (50.0\% and 52.8\%) while maintaining comparable inference speeds to other detectors. With an upgraded backbone and neck structure, YOLOv6-L6 delivers cutting-edge accuracy in real-time.

While imitation learning (IL) offers a promising framework for teaching robots various behaviors, learning complex tasks remains challenging. Existing IL policies struggle to generalize effectively across visual and spatial variations even for simple tasks. In this work, we introduce SPHINX: Salient Point-based Hybrid ImitatioN and eXecution, a flexible IL policy that leverages multimodal observations (point clouds and wrist images), along with a hybrid action space of low-frequency, sparse waypoints and high-frequency, dense end effector movements. Given 3D point cloud observations, SPHINX learns to infer task-relevant points within a point cloud, or salient points, which support spatial generalization by focusing on semantically meaningful features. These salient points serve as anchor points to predict waypoints for long-range movement, such as reaching target poses in free-space. Once near a salient point, SPHINX learns to switch to predicting dense end-effector movements given close-up wrist images for precise phases of a task. By exploiting the strengths of different input modalities and action representations for different manipulation phases, SPHINX tackles complex tasks in a sample-efficient, generalizable manner. Our method achieves 86.7% success across 4 real-world and 2 simulated tasks, outperforming the next best state-of-the-art IL baseline by 41.1% on average across 440 real world trials. SPHINX additionally generalizes to novel viewpoints, visual distractors, spatial arrangements, and execution speeds with a 1.7x speedup over the most competitive baseline. Our website (http://sphinx-manip.github.io) provides open-sourced code for data collection, training, and evaluation, along with supplementary videos.

In both 2016 and 2018, a census of the highly-endangered Grevy's zebra population was enabled by the Great Grevy's Rally (GGR), a citizen science event that produces population estimates via expert and algorithmic curation of volunteer-captured images. A complementary, scalable, and long-term Grevy's population monitoring approach involves deploying camera trap networks. However, in both scenarios, a substantial majority of zebra images are not usable for individual identification due to poor in-the-wild imaging conditions; camera trap images in particular present high rates of occlusion and high spatio-temporal similarity within image bursts. Our proposed filtering pipeline incorporates animal detection, species identification, viewpoint estimation, quality evaluation, and temporal subsampling to obtain individual crops suitable for re-ID, which are subsequently curated by the LCA decision management algorithm. Our method processed images taken during GGR-16 and GGR-18 in Meru County, Kenya, into 4,142 highly-comparable annotations, requiring only 120 contrastive human decisions to produce a population estimate within 4.6% of the ground-truth count. Our method also efficiently processed 8.9M unlabeled camera trap images from 70 cameras at the Mpala Research Centre in Laikipia County, Kenya over two years into 685 encounters of 173 individuals, requiring only 331 contrastive human decisions., Comment: 8 pages, 11 figures. Submitted to the IET Computer Vision Special Issue on Camera Traps, AI, and Ecology. Extended version of a workshop paper presented at Camera Traps, AI, and Ecology 2023

We present the results of microscopic investigations of antiferromagnetic CeNiGe$_3$, using neutron powder diffraction (NPD), inelastic neutron scattering (INS), and muon spin relaxation ($\mu$SR) measurements. CeNiGe$_3$ crystallizes in a centrosymmetric orthorhombic crystal structure (space group: $Cmmm$) and undergoes antiferromagnetic (AFM) ordering. The occurrence of long-range AFM ordering at $T_{\rm N} \approx 5.2$~K is confirmed by magnetic susceptibility, heat capacity, neutron diffraction, and $\mu$SR measurements. The NPD data characterize the AFM state with an incommensurate helical magnetic structure having a propagation vector $k$ = (0, 0.41, 1/2). In addition, INS measurements at 10~K identified two crystal electric field (CEF) excitations at 9.17~meV and 18.42~meV. We analyzed the INS data using a CEF model for an orthorhombic environment of Ce$^{3+}$ ($J=5/2$) and determined the CEF parameters and ground state wavefunctions of CeNiGe$_3$. Moreover, zero-field $\mu$SR data for CeNiGe$_3$ at $T< T_{\rm N}$ show long-range AFM ordering with three distinct oscillation frequencies corresponding to three different internal fields at the muon sites. The internal fields at the muon-stopping sites have been further investigated using density functional theory calculations., Comment: 13 pages, 9 figures

Understanding the structural growth of paediatric brains is a key step in the identification of various neuro-developmental disorders. However, our knowledge is limited by many factors, including the lack of automated image analysis tools, especially in Low and Middle Income Countries from the lack of high field MR images available. Low-field systems are being increasingly explored in these countries, and, therefore, there is a need to develop automated image analysis tools for these images. In this work, as a preliminary step, we consider two tasks: 1) automated quality assurance and 2) hippocampal segmentation, where we compare multiple approaches. For the automated quality assurance task a DenseNet combined with appearance-based transformations for synthesising artefacts produced the best performance, with a weighted accuracy of 82.3%. For the segmentation task, registration of an average atlas performed the best, with a final Dice score of 0.61. Our results show that although the images can provide understanding of large scale pathologies and gross scale anatomical development, there still remain barriers for their use for more granular analyses., Comment: MICCAI 2024 Low field pediatric brain magnetic resonance Image Segmentation and quality Assurance (LISA) Challenge

We present a semi-analytic model for a novel plasma-assisted chemical conversion pathway using triboplasmas generated in granular flows. Triboelectric charge relaxation is a well known phenomena where the potential generated from contact charging of particles exceeds the breakdown voltage of the background gas. In this work, we extend the triboelectric charge relaxation theory to include non equilibrium plasma energy and particle balance equations to predict the formation of dissociated and excited species that act as precursors to chemical conversion, for example in plasma-assisted ammonia synthesis. Our example case study with nitrogen background gas and teflon/aluminum tribomaterial system yielded high excited nitrogen species densities per collision that are comparable to current plasma-assisted conversion pathways. We also present a regime diagram for various gases where Paschen breakdown parameters are used to determine whether triboplasmas can be formed for a given effective work-function difference between two materials. Our sensitivity studies indicate particle velocity, particle radius, solids fraction and space charge effects play a critical role in overall plasma densities and excited species production.

We report the detailed investigation of the magnetic, transport, and magnetocaloric effects of GdSbSe by magnetic susceptibility $\chi(T)$, isothermal magnetization $M(H)$, resistivity $\rho(T, H)$, and heat capacity $C_p(T)$ measurements, crystallizing in the ZrSiS-type tetragonal crystal system with space group $P4/nmm$. Temperature-dependent magnetic susceptibility measurements revealed long-range antiferromagnetic ordering with two additional magnetic anomalies below N\'eel temperature ($T_N$ $\approx$ 8.6 K), corroborated through magnetocaloric and specific heat studies. Isothermal magnetization measurements unveil hidden metamagnetic signatures through a clear deviation from linearity. In addition, the enhanced value of the Sommerfeld coefficient ($\gamma$ = 152(5) mJ/ mol K$^2$) suggests strong electronic correlations in GdSbSe. The entropy of magnetization derived from magnetic isotherms unfolds the field-induced transition from Inverse magnetocaloric Effect (IMCE) to Conventional MCE. The detailed transport properties indicate a semimetallic behavior, strongly coupled with magnetic order. Deviations from Kohler's rule and non-linear Hall resistivity anticipate the possibility of Dirac-like dispersion with non-trivial characteristics., Comment: 10 pages, 9 Figures

Superconducting coaxial cables represent critical communication channels for interconnecting superconducting quantum processors. Here, we report mechanically-intermixed indium joins to aluminum coaxial cables for low loss quantum interconnects. We describe an ABCD matrix formalism to characterize the total resonator internal quality factor ($Q_i$) and any contact ($R_{cont}$) or shunt resistance ($R_{shunt}$) associated with the mechanically-intermixed indium joins. We present four resonator test systems incorporating three indium join methods over the typical frequency range of interest (3-5.5GHz) at temperatures below $20mK$. We measure high internal quality factor aluminum cables ($Q_i = 1.55 \pm 0.37 x 10^6$) through a push-to-connect indium join of the outer conductor that capacitively couples the inner conductor for reflection measurements. We then characterize the total internal quality factors of modes of a cable resonator with a push-to-connect superconducting cable-splice at the midpoint to find mean $Q_i = 1.40 x 10^6$ and $Q_i = 9.39 x 10^5$ for even and odd-modes respectively and use an ABCD matrix model of the system to extract $R_{cont} = 6x10^{-4} \Omega$ for the indium join of the inner conductor. Finally, we demonstrate indium press-mold cable-to-chip connections where the cable-to-chip join is placed at a current node and voltage node through varying on-chip waveguide lengths with mean $Q_i = 1.24 x 10^6$ and $Q_i = 1.07 x 10^6$ respectively to extract $R_{cont} = 8.5x10^{-4} \Omega$ and $R_{shunt} = 1.3x10^7 \Omega$ for the interface. With these techniques, we demonstrate a set of low-loss methods to join superconducting cables for future quantum, Comment: 6 pages, 5 figures

Data-flow analyses like points-to analysis can vastly improve the precision of other analyses, and help perform powerful code optimizations. However, whole-program points-to analysis of large programs tend to be expensive - both in terms of time and memory. Consequently, many compilers (both static and JIT) and program-analysis tools tend to employ faster - but more conservative - points-to analysis to improve usability. As an alternative to such trading of precision for performance, various techniques have been proposed to perform precise yet expensive fixed-point points-to analyses ahead of time in a static analyzer, store the results, and then transmit them to independent compilation/program-analysis stages that may need them. However, an underlying concern of safety affects all such techniques - can a compiler (or program analysis tool) trust the points-to analysis results generated by another compiler/tool? In this work, we address this issue of trust, while keeping the issues of performance efficiency in mind. We propose ART: Analysis-results Representation Template - a novel scheme to efficiently and concisely encode results of flow-sensitive, context-insensitive points-to analysis computed by a static analyzer for use in any independent system that may benefit from such a highly precise points-to analysis. Our scheme has two components: (i) a producer that can statically perform expensive points-to analysis and encode the same concisely. (ii) a consumer that, on receiving such encoded results, can regenerate the points-to analysis results encoded by the artwork if it is deemed safe. We demonstrate the usage of ART by implementing a producer (in Soot) and two consumers (in Soot and the Eclipse OpenJ9 JIT compiler). We evaluate our implementation over various benchmarks from the DaCapo and SPECjvm2008 suites.

Vehicle detection is an important task in the management of traffic and automatic vehicles. This study provides a comparative analysis of five YOLOv5 variants, YOLOv5n6s, YOLOv5s6s, YOLOv5m6s, YOLOv5l6s, and YOLOv5x6s, for vehicle detection in various environments. The research focuses on evaluating the effectiveness of these models in detecting different types of vehicles, such as Car, Bus, Truck, Bicycle, and Motorcycle, under varying conditions including lighting, occlusion, and weather. Performance metrics such as precision, recall, F1-score, and mean Average Precision are utilized to assess the accuracy and reliability of each model. YOLOv5n6s demonstrated a strong balance between precision and recall, particularly in detecting Cars. YOLOv5s6s and YOLOv5m6s showed improvements in recall, enhancing their ability to detect all relevant objects. YOLOv5l6s, with its larger capacity, provided robust performance, especially in detecting Cars, but not good with identifying Motorcycles and Bicycles. YOLOv5x6s was effective in recognizing Buses and Cars but faced challenges with Motorcycle class.

One of the requirements of network slicing in 5G networks is RAN (radio access network) scheduling with rate guarantees. We study a three-time-scale algorithm for maximum sum utility scheduling, with minimum rate constraints. As usual, the scheduler computes an index for each UE in each slot, and schedules the UE with the maximum index. This is at the fastest, natural time-scale of channel fading. The next time-scale is of the exponentially weighted moving average (EWMA) rate update. The slowest time scale in our algorithm is an "index-bias" update by a stochastic approximation algorithm, with a step-size smaller than the EWMA. The index-biases are related to Lagrange multipliers, and bias the slot indices of the UEs with rate guarantees, promoting their more frequent scheduling. We obtain a pair of coupled ordinary differential equations (o.d.e.) such that the unique stable points of the two o.d.e.s are the primal and dual solutions of the constrained utility optimization problem. The UE rate and index-bias iterations track the asymptotic behaviour of the o.d.e. system for small step-sizes of the two slower time-scale iterations. Simulations show that, by running the index-bias iteration at a slower time-scale than the EWMA iteration and using the EWMA throughput itself in the index-bias update, the UE rates stabilize close to the optimum operating point on the rate region boundary, and the index-biases have small fluctuations around the optimum Lagrange multipliers. We compare our results with a prior two-time-scale algorithm and show improved performance., Comment: Technical Report

The Segment Anything Model (SAM), introduced to the computer vision community by Meta in April 2023, is a groundbreaking tool that allows automated segmentation of objects in images based on prompts such as text, clicks, or bounding boxes. SAM excels in zero-shot performance, segmenting unseen objects without additional training, stimulated by a large dataset of over one billion image masks. SAM 2 expands this functionality to video, leveraging memory from preceding and subsequent frames to generate accurate segmentation across entire videos, enabling near real-time performance. This comparison shows how SAM has evolved to meet the growing need for precise and efficient segmentation in various applications. The study suggests that future advancements in models like SAM will be crucial for improving computer vision technology.

A semi-streaming algorithm in dynamic graph streams processes any $n$-vertex graph by making one or multiple passes over a stream of insertions and deletions to edges of the graph and using $O(n \cdot \mbox{polylog}(n))$ space. Semi-streaming algorithms for dynamic streams were first obtained in the seminal work of Ahn, Guha, and McGregor in 2012, alongside the introduction of the graph sketching technique, which remains the de facto way of designing algorithms in this model and a highly popular technique for designing graph algorithms in general. We settle the pass complexity of approximating maximum matchings in dynamic streams via semi-streaming algorithms by improving the state-of-the-art in both upper and lower bounds. We present a randomized sketching based semi-streaming algorithm for $O(1)$-approximation of maximum matching in dynamic streams using $O(\log\log{n})$ passes. The approximation ratio of this algorithm can be improved to $(1+\epsilon)$ for any fixed $\epsilon > 0$ even on weighted graphs using standard techniques. This exponentially improves upon several $O(\log{n})$ pass algorithms developed for this problem since the introduction of the dynamic graph streaming model. In addition, we prove that any semi-streaming algorithm (not only sketching based) for $O(1)$-approximation of maximum matching in dynamic streams requires $\Omega(\log\log{n})$ passes. This presents the first multi-pass lower bound for this problem, which is already also optimal, settling a longstanding open question in this area., Comment: 87 pages, 13 figures

Knife safety in the kitchen is essential for preventing accidents or injuries with an emphasis on proper handling, maintenance, and storage methods. This research presents a comparative analysis of three YOLO models, YOLOv5, YOLOv8, and YOLOv10, to detect the hazards involved in handling knife, concentrating mainly on ensuring fingers are curled while holding items to be cut and that hands should only be in contact with knife handle avoiding the blade. Precision, recall, F-score, and normalized confusion matrix are used to evaluate the performance of the models. The results indicate that YOLOv5 performed better than the other two models in identifying the hazard of ensuring hands only touch the blade, while YOLOv8 excelled in detecting the hazard of curled fingers while holding items. YOLOv5 and YOLOv8 performed almost identically in recognizing classes such as hand, knife, and vegetable, whereas YOLOv5, YOLOv8, and YOLOv10 accurately identified the cutting board. This paper provides insights into the advantages and shortcomings of these models in real-world settings. Moreover, by detailing the optimization of YOLO architectures for safe knife handling, this study promotes the development of increased accuracy and efficiency in safety surveillance systems.

Robot-assisted feeding has the potential to improve the quality of life for individuals with mobility limitations who are unable to feed themselves independently. However, there exists a large gap between the homogeneous, curated plates existing feeding systems can handle, and truly in-the-wild meals. Feeding realistic plates is immensely challenging due to the sheer range of food items that a robot may encounter, each requiring specialized manipulation strategies which must be sequenced over a long horizon to feed an entire meal. An assistive feeding system should not only be able to sequence different strategies efficiently in order to feed an entire meal, but also be mindful of user preferences given the personalized nature of the task. We address this with FLAIR, a system for long-horizon feeding which leverages the commonsense and few-shot reasoning capabilities of foundation models, along with a library of parameterized skills, to plan and execute user-preferred and efficient bite sequences. In real-world evaluations across 6 realistic plates, we find that FLAIR can effectively tap into a varied library of skills for efficient food pickup, while adhering to the diverse preferences of 42 participants without mobility limitations as evaluated in a user study. We demonstrate the seamless integration of FLAIR with existing bite transfer methods [19, 28], and deploy it across 2 institutions and 3 robots, illustrating its adaptability. Finally, we illustrate the real-world efficacy of our system by successfully feeding a care recipient with severe mobility limitations. Supplementary materials and videos can be found at: https://emprise.cs.cornell.edu/flair ., Comment: RSS 2024

Special relativity allows massless particles to have states of different integer (or half-integer) helicities that mix under boosts, much like the spin-states of a massive particle. Such massless particles are known as continuous spin particles (CSPs), a term coined by Wigner, and they are notable for their infinite tower of spin polarizations. The mixing under boosts is controlled by a spin-scale $\rho$ with units of momentum. Normally, we assume $\rho=0$. The interactions of CSPs are known to satisfy certain simple properties, one of which is that the $\rho \rightarrow 0$ limit generically recovers familiar interactions of massless scalars, photons, or gravitons, with all other polarizations decoupling in this limit. Thus, one can ask if the photon of the Standard Model is a CSP at small but non-zero $\rho$. One concern about this possibility -- originally raised by Wigner -- is that the infinite tower of polarizations could pose problems for thermodynamics. To address this question, we study the thermal evolution of a CSP photon gas coupled to isothermal matter, across CSP helicity modes and phase space. We find that the structure of the interactions dictated by Lorentz symmetry imply well behaved thermodynamics. When the CSP photon's interactions to charged matter are turned on, the primary $h=\pm 1$ helicity modes thermalize quickly, while the other modes require increasingly long time-scales to thermalize, set by powers of $T/\rho$. In familiar thermal systems, the CSP photon behaves like the QED photon with small $\rho$- and time- dependent corrections to its effective relativistic degrees of freedom. Sizable departures from familiar thermal behavior arise at energy scales comparable to $\rho$ and could have testable experimental consequences., Comment: 13 pages + 4 appendix pages, 7 figures

The abilities of modern large language models (LLMs) in solving natural language processing, complex reasoning, sentiment analysis and other tasks have been extraordinary which has prompted their extensive adoption. Unfortunately, these abilities come with very high memory and computational costs which precludes the use of LLMs on most hardware platforms. To mitigate this, we propose an effective method of finding Pareto-optimal network architectures based on LLaMA2-7B using one-shot NAS. In particular, we fine-tune LLaMA2-7B only once and then apply genetic algorithm-based search to find smaller, less computationally complex network architectures. We show that, for certain standard benchmark tasks, the pre-trained LLaMA2-7B network is unnecessarily large and complex. More specifically, we demonstrate a 1.5x reduction in model size and 1.3x speedup in throughput for certain tasks with negligible drop in accuracy. In addition to finding smaller, higher-performing network architectures, our method does so more effectively and efficiently than certain pruning or sparsification techniques. Finally, we demonstrate how quantization is complementary to our method and that the size and complexity of the networks we find can be further decreased using quantization. We believe that our work provides a way to automatically create LLMs which can be used on less expensive and more readily available hardware platforms.

We investigate the problem of pixelwise correspondence for deformable objects, namely cloth and rope, by comparing both classical and learning-based methods. We choose cloth and rope because they are traditionally some of the most difficult deformable objects to analytically model with their large configuration space, and they are meaningful in the context of robotic tasks like cloth folding, rope knot-tying, T-shirt folding, curtain closing, etc. The correspondence problem is heavily motivated in robotics, with wide-ranging applications including semantic grasping, object tracking, and manipulation policies built on top of correspondences. We present an exhaustive survey of existing classical methods for doing correspondence via feature-matching, including SIFT, SURF, and ORB, and two recently published learning-based methods including TimeCycle and Dense Object Nets. We make three main contributions: (1) a framework for simulating and rendering synthetic images of deformable objects, with qualitative results demonstrating transfer between our simulated and real domains (2) a new learning-based correspondence method extending Dense Object Nets, and (3) a standardized comparison across state-of-the-art correspondence methods. Our proposed method provides a flexible, general formulation for learning temporally and spatially continuous correspondences for nonrigid (and rigid) objects. We report root mean squared error statistics for all methods and find that Dense Object Nets outperforms baseline classical methods for correspondence, and our proposed extension of Dense Object Nets performs similarly.

Large language models (LLMs) has experienced exponential growth, they demonstrate remarkable performance across various tasks. Notwithstanding, contemporary research primarily centers on enhancing the size and quality of pretraining data, still utilizing the next token prediction task on autoregressive transformer model structure. The efficacy of this task in truly facilitating the model's comprehension of code logic remains questionable, we speculate that it still interprets code as mere text, while human emphasizes the underlying logical knowledge. In order to prove it, we introduce a new task, "Logically Equivalent Code Selection," which necessitates the selection of logically equivalent code from a candidate set, given a query code. Our experimental findings indicate that current LLMs underperform in this task, since they understand code by unordered bag of keywords. To ameliorate their performance, we propose an advanced pretraining task, "Next Token Prediction+". This task aims to modify the sentence embedding distribution of the LLM without sacrificing its generative capabilities. Our experimental results reveal that following this pretraining, both Code Llama and StarCoder, the prevalent code domain pretraining models, display significant improvements on our logically equivalent code selection task and the code completion task.

We study the CHAIN communication problem introduced by Cormode et al. [ICALP 2019]. It is a generalization of the well-studied INDEX problem. For $k\geq 1$, in CHAIN$_{n,k}$, there are $k$ instances of INDEX, all with the same answer. They are shared between $k+1$ players as follows. Player 1 has the first string $X^1 \in \{0,1\}^n$, player 2 has the first index $\sigma^1 \in [n]$ and the second string $X^2 \in \{0,1\}^n$, player 3 has the second index $\sigma^2 \in [n]$ along with the third string $X^3 \in \{0,1\}^n$, and so on. Player $k+1$ has the last index $\sigma^k \in [n]$. The communication is one way from each player to the next, starting from player 1 to player 2, then from player 2 to player 3 and so on. Player $k+1$, after receiving the message from player $k$, has to output a single bit which is the answer to all $k$ instances of INDEX. It was proved that the CHAIN$_{n,k}$ problem requires $\Omega(n/k^2)$ communication by Cormode et al., and they used it to prove streaming lower bounds for approximation of maximum independent sets. Subsequently, it was used by Feldman et al. [STOC 2020] to prove lower bounds for streaming submodular maximization. However, these works do not get optimal bounds on the communication complexity of CHAIN$_{n,k}$, and in fact, it was conjectured by Cormode et al. that $\Omega(n)$ bits are necessary, for any $k$. As our main result, we prove the optimal lower bound of $\Omega(n)$ for CHAIN$_{n,k}$. This settles the open conjecture of Cormode et al. in the affirmative. The key technique is to use information theoretic tools to analyze protocols over the Jensen-Shannon divergence measure, as opposed to total variation distance. As a corollary, we get an improved lower bound for approximation of maximum independent set in vertex arrival streams through a reduction from CHAIN directly., Comment: 24 pages, 3 figures; Fixed a flaw in the previous argument

The landscape of software development has witnessed a paradigm shift with the advent of AI-powered assistants, exemplified by GitHub Copilot. However, existing solutions are not leveraging all the potential capabilities available in an IDE such as building, testing, executing code, git operations, etc. Therefore, they are constrained by their limited capabilities, primarily focusing on suggesting code snippets and file manipulation within a chat-based interface. To fill this gap, we present AutoDev, a fully automated AI-driven software development framework, designed for autonomous planning and execution of intricate software engineering tasks. AutoDev enables users to define complex software engineering objectives, which are assigned to AutoDev's autonomous AI Agents to achieve. These AI agents can perform diverse operations on a codebase, including file editing, retrieval, build processes, execution, testing, and git operations. They also have access to files, compiler output, build and testing logs, static analysis tools, and more. This enables the AI Agents to execute tasks in a fully automated manner with a comprehensive understanding of the contextual information required. Furthermore, AutoDev establishes a secure development environment by confining all operations within Docker containers. This framework incorporates guardrails to ensure user privacy and file security, allowing users to define specific permitted or restricted commands and operations within AutoDev. In our evaluation, we tested AutoDev on the HumanEval dataset, obtaining promising results with 91.5% and 87.8% of Pass@1 for code generation and test generation respectively, demonstrating its effectiveness in automating software engineering tasks while maintaining a secure and user-controlled development environment.

Natural language and images are commonly used as goal representations in goal-conditioned imitation learning (IL). However, natural language can be ambiguous and images can be over-specified. In this work, we propose hand-drawn sketches as a modality for goal specification in visual imitation learning. Sketches are easy for users to provide on the fly like language, but similar to images they can also help a downstream policy to be spatially-aware and even go beyond images to disambiguate task-relevant from task-irrelevant objects. We present RT-Sketch, a goal-conditioned policy for manipulation that takes a hand-drawn sketch of the desired scene as input, and outputs actions. We train RT-Sketch on a dataset of paired trajectories and corresponding synthetically generated goal sketches. We evaluate this approach on six manipulation skills involving tabletop object rearrangements on an articulated countertop. Experimentally we find that RT-Sketch is able to perform on a similar level to image or language-conditioned agents in straightforward settings, while achieving greater robustness when language goals are ambiguous or visual distractors are present. Additionally, we show that RT-Sketch has the capacity to interpret and act upon sketches with varied levels of specificity, ranging from minimal line drawings to detailed, colored drawings. For supplementary material and videos, please refer to our website: http://rt-sketch.github.io.

Mobile networks have increased spectral efficiency through advanced multiplexing strategies that are coordinated by base stations (BS) in licensed spectrum. However, external interference on clients leads to significant performance degradation during dynamic (unlicensed) spectrum access (DSA). We introduce the notion of network tomography for DSA, whereby clients are transformed into spectrum sensors, whose joint access statistics are measured and used to account for interfering sources. Albeit promising, performing such tomography naively incurs an impractical overhead that scales exponentially with the multiplexing order of the strategies deployed -- which will only continue to grow with 5G/6G technologies. To this end, we propose a novel, scalable network tomography framework called NeTo-X that estimates joint client access statistics with just linear overhead, and forms a blue-print of the interference, thus enabling efficient DSA for future networks. NeTo-X's design incorporates intelligent algorithms that leverage multi-channel diversity and the spatial locality of interference impact on clients to accurately estimate the desired interference statistics from just pair-wise measurements of its clients. The merits of its framework are showcased in the context of resource management and jammer localization applications, where its performance significantly outperforms baseline approaches and closely approximates optimal performance at a scalable overhead., Comment: Accepted to IEEE INFOCOM 2024. 11 pages, 11 figures

The integration of Large Language Models (LLMs) into Development Environments (IDEs) has become a focal point in modern software development. LLMs such as OpenAI GPT-3.5/4 and Code Llama offer the potential to significantly augment developer productivity by serving as intelligent, chat-driven programming assistants. However, utilizing LLMs out of the box is unlikely to be optimal for any given scenario. Rather, each system requires the LLM to be honed to its set of heuristics to ensure the best performance. In this paper, we introduce the Copilot evaluation harness: a set of data and tools for evaluating LLM-guided IDE interactions, covering various programming scenarios and languages. We propose our metrics as a more robust and information-dense evaluation than previous state of the art evaluation systems. We design and compute both static and execution based success metrics for scenarios encompassing a wide range of developer tasks, including code generation from natural language (generate), documentation generation from code (doc), test case generation (test), bug-fixing (fix), and workspace understanding and query resolution (workspace). These success metrics are designed to evaluate the performance of LLMs within a given IDE and its respective parameter space. Our learnings from evaluating three common LLMs using these metrics can inform the development and validation of future scenarios in LLM guided IDEs.

To understand its evolution and the effects of its eruptive events, the Sun is permanently monitored by multiple satellite missions. The optically-thin emission of the solar plasma and the limited number of viewpoints make it challenging to reconstruct the geometry and structure of the solar atmosphere; however, this information is the missing link to understand the Sun as it is: a three-dimensional evolving star. We present a method that enables a complete 3D representation of the uppermost solar layer (corona) observed in extreme ultraviolet (EUV) light. We use a deep learning approach for 3D scene representation that accounts for radiative transfer, to map the entire solar atmosphere from three simultaneous observations. We demonstrate that our approach provides unprecedented reconstructions of the solar poles, and directly enables height estimates of coronal structures, solar filaments, coronal hole profiles, and coronal mass ejections. We validate the approach using model-generated synthetic EUV images, finding that our method accurately captures the 3D geometry of the Sun even from a limited number of 32 ecliptic viewpoints ($|\text{latitude}| \leq 7^\circ$). We quantify uncertainties of our model using an ensemble approach that allows us to estimate the model performance in absence of a ground-truth. Our method enables a novel view of our closest star, and is a breakthrough technology for the efficient use of multi-instrument datasets, which paves the way for future cluster missions.

Gas-particle flows are commonly simulated through two-fluid model at industrial-scale. However, these simulations need very fine grid to have accurate flow predictions, which is prohibitively demanding in terms of computational resources. To circumvent this problem, the filtered two-fluid model has been developed, where large-scale flow field is numerically resolved and small-scale fluctuations are accounted for through subgrid-scale modeling. In this study, we have performed fine-grid two-fluid simulations of dilute gas-particle flows in periodic domains and applied explicit filtering to generate datasets. Then, these datasets have been used to develop artificial neural network (ANN) models for closures such as the filtered drag force and solid phase stress for the filtered two-fluid model. The set of input variables for the subgrid drag force ANN model that has been found previously to work well for dense flow regimes is found to work as well for the dilute regime. In addition, we present a Galilean invariant tensor basis neural network (TBNN) model for the filtered solid phase stress which can capture nicely the anisotropic nature of the solid phase stress arising from subgrid-scale velocity fluctuations. Finally, the predictions provided by this new TBNN model are compared with those obtained from a simple eddy-viscosity ANN model.

In the semi-streaming model for processing massive graphs, an algorithm makes multiple passes over the edges of a given $n$-vertex graph and is tasked with computing the solution to a problem using $O(n \cdot \text{polylog}(n))$ space. Semi-streaming algorithms for Maximal Independent Set (MIS) that run in $O(\log\log{n})$ passes have been known for almost a decade, however, the best lower bounds can only rule out single-pass algorithms. We close this large gap by proving that the current algorithms are optimal: Any semi-streaming algorithm for finding an MIS with constant probability of success requires $\Omega(\log\log{n})$ passes. This settles the complexity of this fundamental problem in the semi-streaming model, and constitutes one of the first optimal multi-pass lower bounds in this model. We establish our result by proving an optimal round vs communication tradeoff for the (multi-party) communication complexity of MIS. The key ingredient of this result is a new technique, called hierarchical embedding, for performing round elimination: we show how to pack many but small hard $(r-1)$-round instances of the problem into a single $r$-round instance, in a way that enforces any $r$-round protocol to effectively solve all these $(r-1)$-round instances also. These embeddings are obtained via a novel application of results from extremal graph theory -- in particular dense graphs with many disjoint unique shortest paths -- together with a newly designed graph product, and are analyzed via information-theoretic tools such as direct-sum and message compression arguments., Comment: 60 pages, 14 figures

Recent one-shot Neural Architecture Search algorithms rely on training a hardware-agnostic super-network tailored to a specific task and then extracting efficient sub-networks for different hardware platforms. Popular approaches separate the training of super-networks from the search for sub-networks, often employing predictors to alleviate the computational overhead associated with search. Additionally, certain methods also incorporate the quantization policy within the search space. However, while the quantization policy search for convolutional neural networks is well studied, the extension of these methods to transformers and especially foundation models remains under-explored. In this paper, we demonstrate that by using multi-objective search algorithms paired with lightly trained predictors, we can efficiently search for both the sub-network architecture and the corresponding quantization policy and outperform their respective baselines across different performance objectives such as accuracy, model size, and latency. Specifically, we demonstrate that our approach performs well across both uni-modal (ViT and BERT) and multi-modal (BEiT-3) transformer-based architectures as well as convolutional architectures (ResNet). For certain networks, we demonstrate an improvement of up to $4.80x$ and $3.44x$ for latency and model size respectively, without degradation in accuracy compared to the fully quantized INT8 baselines.

Instruction tuning, a specialized technique to enhance large language model (LLM) performance via instruction datasets, relies heavily on the quality of employed data. Existing quality improvement methods alter instruction data through dataset expansion or curation. However, the expansion method risks data redundancy, potentially compromising LLM performance, while the curation approach confines the LLM's potential to the original dataset. Our aim is to surpass the original data quality without encountering these shortcomings. To achieve this, we propose LIFT (LLM Instruction Fusion Transfer), a novel and versatile paradigm designed to elevate the instruction quality to new heights. LIFT strategically broadens data distribution to encompass more high-quality subspaces and eliminates redundancy, concentrating on high-quality segments across overall data subspaces. Experimental results demonstrate that, even with a limited quantity of high-quality instruction data selected by our paradigm, LLMs not only consistently uphold robust performance across various tasks but also surpass some state-of-the-art results, highlighting the significant improvement in instruction quality achieved by our paradigm.

We present a Human Body model based IDentification system (HMID) system that is jointly trained for shape, pose and biometric identification. HMID is based on the Human Mesh Recovery (HMR) network and we propose additional losses to improve and stabilize shape estimation and biometric identification while maintaining the pose and shape output. We show that when our HMID network is trained using additional shape and pose losses, it shows a significant improvement in biometric identification performance when compared to an identical model that does not use such losses. The HMID model uses raw images instead of silhouettes and is able to perform robust recognition on images collected at range and altitude as many anthropometric properties are reasonably invariant to clothing, view and range. We show results on the USF dataset as well as the BRIAR dataset which includes probes with both clothing and view changes. Our approach (using body model losses) shows a significant improvement in Rank20 accuracy and True Accuracy Rate on the BRIAR evaluation dataset., Comment: to be published in IEEE International Joint Conference on Biometrics, Ljubljana, Slovenia 2023

We investigate the effect of alloying at the 3d transition metal site of a rare-earth-transition metal oxide, by considering NdFe0.5Cr0.5O3 alloy with two equal and random distribution of 3d ions, Cr and Fe, interacting with an early 4f rare earth ion, Nd. Employing temperature- and field-dependent magnetization measurements, temperature-dependent x-ray diffraction, neutron powder diffraction, and Raman spectroscopy, we characterize its structural and magnetic properties. Our study reveals bipolar magnetic switching (arising from negative magnetization) and magnetocaloric effect which underline the potential of the studied alloy in device application. The neutron diffraction study shows the absence of spin reorientation transition over the entire temperature range of 1.5-320 K, although both parent compounds exhibit spin orientation transition. We discuss the microscopic origin of this curious behavior. The neutron diffraction results also reveal the ordering of Nd spins at an unusually high temperature of about 40 K, which is corroborated by Raman measurements., Comment: 49 pages 18 figures