Search

In this paper, we introduce a novel unbiased regression loss for DETR-based detectors. The conventional $L_{1}$ regression loss tends to bias towards larger boxes, as they disproportionately contribute more towards the overall loss compared to smaller boxes. Consequently, the detection performance for small objects suffers. To alleviate this bias, the proposed new unbiased loss, termed Sized $L_{1}$ loss, normalizes the size of all boxes based on their individual width and height. Our experiments demonstrate consistent improvements in both fully-supervised and semi-supervised settings using the MS-COCO benchmark dataset.

Each request in LLM inference goes through two phases: compute-bound prefill and memory-bandwidth-bound decode. To improve GPU utilization, recent systems use hybrid batching that combines the prefill and decode phases of different requests into the same batch. Hybrid batching works well for linear operations as it amortizes the cost of loading model weights from HBM. However, attention computation in hybrid batches remains inefficient because existing attention kernels are optimized for either prefill or decode. In this paper, we present POD-Attention -- the first GPU kernel that efficiently computes attention for hybrid batches. POD-Attention aims to maximize the utilization of both compute and memory bandwidth by carefully allocating the GPU's resources such that prefill and decode operations happen concurrently on the same multiprocessor. We integrate POD-Attention in a state-of-the-art LLM inference scheduler Sarathi-Serve. POD-Attention speeds up attention computation by up to 75% (mean 28%) and increases LLM serving throughput by up to 22% in offline inference. In online inference, POD-Attention enables lower time-to-first-token (TTFT), time-between-tokens (TBT), and request execution latency versus Sarathi-Serve.

In recent years, Large Language Models (LLMs) have demonstrated high reasoning capabilities, drawing attention for their applications as agents in various decision-making processes. One notably promising application of LLM agents is robotic manipulation. Recent research has shown that LLMs can generate text planning or control code for robots, providing substantial flexibility and interaction capabilities. However, these methods still face challenges in terms of flexibility and applicability across different environments, limiting their ability to adapt autonomously. Current approaches typically fall into two categories: those relying on environment-specific policy training, which restricts their transferability, and those generating code actions based on fixed prompts, which leads to diminished performance when confronted with new environments. These limitations significantly constrain the generalizability of agents in robotic manipulation. To address these limitations, we propose a novel method called EnvBridge. This approach involves the retention and transfer of successful robot control codes from source environments to target environments. EnvBridge enhances the agent's adaptability and performance across diverse settings by leveraging insights from multiple environments. Notably, our approach alleviates environmental constraints, offering a more flexible and generalizable solution for robotic manipulation tasks. We validated the effectiveness of our method using robotic manipulation benchmarks: RLBench, MetaWorld, and CALVIN. Our experiments demonstrate that LLM agents can successfully leverage diverse knowledge sources to solve complex tasks. Consequently, our approach significantly enhances the adaptability and robustness of robotic manipulation agents in planning across diverse environments.

`Extreme Classification'' (or XC) is the task of annotating data points (queries) with relevant labels (documents), from an extremely large set of $L$ possible labels, arising in search and recommendations. The most successful deep learning paradigm that has emerged over the last decade or so for XC is to embed the queries (and labels) using a deep encoder (e.g. DistilBERT), and use linear classifiers on top of the query embeddings. This architecture is of appeal because it enables millisecond-time inference using approximate nearest neighbor search (ANNS). The key question is how do we design training algorithms that are accurate as well as scale to $O(100M)$ labels on a limited number of GPUs. State-of-the-art XC techniques that demonstrate high accuracies (e.g., DEXML, Ren\'ee, DEXA) on standard datasets have per-epoch training time that scales as $O(L)$ or employ expensive negative sampling strategies, which are prohibitive in XC scenarios. In this work, we develop an accurate and scalable XC algorithm ASTRA with two key observations: (a) building ANNS index on the classifier vectors and retrieving hard negatives using the classifiers aligns the negative sampling strategy to the loss function optimized; (b) keeping the ANNS indices current as the classifiers change through the epochs is prohibitively expensive while using stale negatives (refreshed periodically) results in poor accuracy; to remedy this, we propose a negative sampling strategy that uses a mixture of importance sampling and uniform sampling. By extensive evaluation on standard XC as well as proprietary datasets with 120M labels, we demonstrate that ASTRA achieves SOTA precision, while reducing training time by 4x-15x relative to the second best.

This study reports a comprehensive environmental scan of the generative AI (GenAI) infrastructure in the national network for clinical and translational science across 36 institutions supported by the Clinical and Translational Science Award (CTSA) Program led by the National Center for Advancing Translational Sciences (NCATS) of the National Institutes of Health (NIH) at the United States. With the rapid advancement of GenAI technologies, including large language models (LLMs), healthcare institutions face unprecedented opportunities and challenges. This research explores the current status of GenAI integration, focusing on stakeholder roles, governance structures, and ethical considerations by administering a survey among leaders of health institutions (i.e., representing academic medical centers and health systems) to assess the institutional readiness and approach towards GenAI adoption. Key findings indicate a diverse range of institutional strategies, with most organizations in the experimental phase of GenAI deployment. The study highlights significant variations in governance models, with a strong preference for centralized decision-making but notable gaps in workforce training and ethical oversight. Moreover, the results underscore the need for a more coordinated approach to GenAI governance, emphasizing collaboration among senior leaders, clinicians, information technology staff, and researchers. Our analysis also reveals concerns regarding GenAI bias, data security, and stakeholder trust, which must be addressed to ensure the ethical and effective implementation of GenAI technologies. This study offers valuable insights into the challenges and opportunities of GenAI integration in healthcare, providing a roadmap for institutions aiming to leverage GenAI for improved quality of care and operational efficiency.

The ion-lipid interface in Langmuir monolayers of Dipalmitoylphosphatidylcholine (DPPC) on pure water and 10 mM solutions of Na+ and K+ at different [K+]/[Na+] (a), atom/atom ratios, were studied initially by Surface Pressure (p) versus Specific Molecular Area (A) isotherms. The values of a were chosen as 0 (no K+), 0.43 ([K+]:[Na+] = 30:70) and 1.0 ([K+]:[Na+] = 50:50) These monolayers were studied through X-Ray Reflectivity (XRR) and Near Edge X-ray Absorption Fine Structure (NEXAFS) spectroscopy at the O K-edge. The two-dimensional rigidity of the monolayer was found to increase with Na+ ions with respect to the pristine monolayer but fall drastically and non-linearly below the pristine value with introduction of the K+ ions, as a was increased. Analysis of the XRR profiles provided the thickness, average electron density (aed) and the interfacial roughness of the phosphatidylcholine head group and the two hydrocarbon tails of the monolayers on Si (001), from which the angle (f) between the head and the tails was determined. This was also follow the same as former one. From NEXAFS, it was found that a linear increase in the cation ratio towards K led to a nonlinear variation in the P=O bond energy and a weakening of the P-O bond energy, the latter becoming more pronounced with K ions, consistent with Fajans rule. Also a split in the C=O p-bond peak was observed at a = 1.0. These results cannot be explained with the model of a uniform electric field due to the cations, which would fall linearly with increase in the K+ proportion, and rather suggest a structured field due a spatial variation in charge density in an interfacial layer of high ion concentration assembled by the counterionic attraction of the phosphatidylcholine head groups. Our results have important implications for the cell membrane, where such mixtures at high concentrations constitute the norm.

We investigate a series of galaxy properties computed using the merger trees and environmental histories from dark matter only cosmological simulations, using the predictive semi-recurrent neural network outlined in Chittenden and Tojeiro (2023), and using stochastic improvements presented in our companion paper: Behera, Tojeiro and Chittenden (2024). We apply these methods to the dark matter only runs of the IllustrisTNG simulations to understand the effects of baryon removal, and to the gigaparsec-volume pure dark matter simulation Uchuu, to understand the effects of the lower resolution or alternative metrics for halo properties. We find that the machine learning model recovers accurate summary statistics derived from the predicted star formation and stellar metallicity histories, and correspondent spectroscopy and photometry. However, the inaccuracies of the model's application to dark simulations are substantial for low mass and slowly growing haloes. For these objects, the halo mass accretion rate is exaggerated due to the lack of stellar feedback, yet the formation of the halo can be severely limited by the absence of low mass progenitors in a low resolution simulation. Furthermore, differences in the structure and environment of higher mass haloes results in an overabundance of red, quenched galaxies. These results signify progress towards a machine learning model which builds high fidelity mocks based on a physical interpretation of the galaxy-halo connection, yet they illustrate the need to account for differences in halo properties and the resolution of the simulation., Comment: 21 pages, 17 figures, submitted to MNRAS alongside companion paper: arXiv:2409.16548

We present a novel methodology to improve neural network (NN) predictions of galaxy formation histories by incorporating semi-stochastic corrections to account for short-timescale variability. Our paper addresses limitations in existing models that capture broad trends in galaxy evolution, but fail to reproduce the bursty nature of star formation and chemical enrichment, resulting in inaccurate predictions of key observables such as stellar masses, optical spectra, and colour distributions. We introduce a simple technique to add stochastic components by utilizing the power spectra of galaxy formation histories. We justify our stochastic approach by studying the correlation between the phases of the halo mass assembly and star-formation histories in the IllustrisTNG simulation, and we find that they are correlated only on timescales longer than 6 Gyr, with a strong dependence on galaxy type. Building on NNs developed in Chittenden & Tojeiro (2023), trained on hydrodynamical simulations from the IllustrisTNG project, which predict time-resolved star formation and stellar metallicity histories for central and satellite galaxies based solely on the properties and evolution of their dark matter halos and environments, this approach successfully recovers realistic variability in galaxy properties at short timescales. It significantly improves the accuracy of predicted stellar masses, metallicities, spectra, and colour distributions and provides a powerful framework for generating large, realistic mock galaxy catalogs, while also enhancing our understanding of the complex interplay between galaxy evolution and dark matter halo assembly., Comment: 18 pages, 20 figures, submitted to MNRAS alongside companion paper - arXiv:2409.16079

Serving large language models (LLMs) in production can incur substantial costs, which has prompted recent advances in inference system optimizations. Today, these systems are evaluated against conventional latency and throughput metrics (eg. TTFT, TBT, Normalised Latency and TPOT). However, these metrics fail to fully capture the nuances of LLM inference, leading to an incomplete assessment of user-facing performance crucial for real-time applications such as chat and translation. In this paper, we first identify the pitfalls of current performance metrics in evaluating LLM inference systems. We then propose Etalon, a comprehensive performance evaluation framework that includes fluidity-index -- a novel metric designed to reflect the intricacies of the LLM inference process and its impact on real-time user experience. Finally, we evaluate various existing open-source platforms and model-as-a-service offerings using Etalon, discussing their strengths and weaknesses. Etalon is available at https://github.com/project-etalon/etalon.

The correct interpretation of breast density is important in the assessment of breast cancer risk. AI has been shown capable of accurately predicting breast density, however, due to the differences in imaging characteristics across mammography systems, models built using data from one system do not generalize well to other systems. Though federated learning (FL) has emerged as a way to improve the generalizability of AI without the need to share data, the best way to preserve features from all training data during FL is an active area of research. To explore FL methodology, the breast density classification FL challenge was hosted in partnership with the American College of Radiology, Harvard Medical School's Mass General Brigham, University of Colorado, NVIDIA, and the National Institutes of Health National Cancer Institute. Challenge participants were able to submit docker containers capable of implementing FL on three simulated medical facilities, each containing a unique large mammography dataset. The breast density FL challenge ran from June 15 to September 5, 2022, attracting seven finalists from around the world. The winning FL submission reached a linear kappa score of 0.653 on the challenge test data and 0.413 on an external testing dataset, scoring comparably to a model trained on the same data in a central location., Comment: 16 pages, 9 figures

Optimizing the deployment of Large language models (LLMs) is expensive today since it requires experimentally running an application workload against an LLM implementation while exploring large configuration space formed by system knobs such as parallelization strategies, batching techniques, and scheduling policies. To address this challenge, we present Vidur - a large-scale, high-fidelity, easily-extensible simulation framework for LLM inference performance. Vidur models the performance of LLM operators using a combination of experimental profiling and predictive modeling, and evaluates the end-to-end inference performance for different workloads by estimating several metrics of interest such as latency and throughput. We validate the fidelity of Vidur on several LLMs and show that it estimates inference latency with less than 9% error across the range. Further, we present Vidur-Search, a configuration search tool that helps optimize LLM deployment. Vidur-Search uses Vidur to automatically identify the most cost-effective deployment configuration that meets application performance constraints. For example, Vidur-Search finds the best deployment configuration for LLaMA2-70B in one hour on a CPU machine, in contrast to a deployment-based exploration which would require 42K GPU hours - costing ~218K dollars. Source code for Vidur is available at https://github.com/microsoft/vidur.

Efficient management of GPU memory is essential for high throughput LLM inference. Prior systems used to reserve KV-cache memory ahead-of-time that resulted in wasted capacity due to internal fragmentation. Inspired by demand paging, vLLM proposed PagedAttention to enable dynamic memory allocation for KV-cache. This approach eliminates fragmentation and improves serving throughout. However, to be able to allocate physical memory dynamically, PagedAttention changes the layout of KV-cache from contiguous virtual memory to non-contiguous virtual memory. As a consequence, one needs to rewrite the attention kernels to support paging, and implement a memory manager in the serving framework. This results in both performance and programming overheads, as well as portability challenges in adopting state-of-the-art attention kernels. In this paper, we propose vAttention, a new approach for dynamic KV-cache memory management. In contrast to PagedAttention, vAttention stores KV-cache in contiguous virtual memory and leverages OS support for on-demand allocation of physical memory. vAttention thus enables one to use state-of-the art attention kernels out-of-the-box by adding support for dynamic allocation of physical memory without having to re-write their code. We implement vAttention in the vLLM serving stack to show that it also helps improve decode throughput by up to 1.99x over vLLM, and the end-to-end serving throughput by up to 1.22x and 1.29x, compared to using the state-of-the-art PagedAttention based kernels of FlashAttention and FlashInfer., Comment: 14 pages, 13 figures, 10 tables

The interactions between tumor cells and the tumor microenvironment (TME) dictate therapeutic efficacy of radiation and many systemic therapies in breast cancer. However, to date, there is not a widely available method to reproducibly measure tumor and immune phenotypes for each patient's tumor. Given this unmet clinical need, we applied multiple instance learning (MIL) algorithms to assess activity of ten biologically relevant pathways from the hematoxylin and eosin (H&E) slide of primary breast tumors. We employed different feature extraction approaches and state-of-the-art model architectures. Using binary classification, our models attained area under the receiver operating characteristic (AUROC) scores above 0.70 for nearly all gene expression pathways and on some cases, exceeded 0.80. Attention maps suggest that our trained models recognize biologically relevant spatial patterns of cell sub-populations from H&E. These efforts represent a first step towards developing computational H&E biomarkers that reflect facets of the TME and hold promise for augmenting precision oncology., Comment: Paper accepted at the First Workshop on Imageomics (Imageomics-AAAI-24) - Discovering Biological Knowledge from Images using AI (https://sites.google.com/vt.edu/imageomics-aaai-24/home), held as part of the 38th Annual AAAI Conference on Artificial Intelligence (https://aaai.org/aaai-conference/)

We report the electrochemical sodium-ion kiinetics and distribution of relaxation time (DRT) analysis of a newly designed mixed polyanionic NaFe$_{1.6}$V$_{0.4}$(PO$_{4}$)(SO$_{4}$)$_{2}$@CNT composite as a cathode. The specific capacity of 104 mAhg$^{-1}$ is observed at 0.1~C with the average working voltage of $\sim$3~V. Intriguingly, a remarkable rate capability and reversibility are demonstrated up to very high current rate of 25~C. The long cycling test up to 10~C shows high capacity retention even after 2000 cycles. The detailed analysis of galvanostatic intermittent titration technique (GITT) and cyclic voltammetry (CV) data reveal the diffusion coefficient of 10$^{-8}$--10$^{-11}$ cm$^{2}$s$^{-1}$. We find excellent stability in the thermal testing between 25--55$^\circ$C temperatures and 80\% capacity retention up to 100 cycles at 5~C. Further, we analyse the individual electrochemical processes in the time domain using the novel DRT technique at different temperatures. The {\it ex-situ} investigation shows the stable and reversible structure, morphology and electronic states of the long cycled cathode material. More importantly, we demonstrate relatively high specific energy of $\approx$155 Wh kg$^{-1}$ (considering the total active material loading of both the electrodes) at 0.2~C for full cell battery having excellent rate capability up to 10~C and long cyclic stability at 1~C., Comment: submitted

Each LLM serving request goes through two phases. The first is prefill which processes the entire input prompt and produces the first output token and the second is decode which generates the rest of output tokens, one-at-a-time. Prefill iterations have high latency but saturate GPU compute due to parallel processing of the input prompt. In contrast, decode iterations have low latency but also low compute utilization because a decode iteration processes only a single token per request. This makes batching highly effective for decodes and consequently for overall throughput. However, batching multiple requests leads to an interleaving of prefill and decode iterations which makes it challenging to achieve both high throughput and low latency. We introduce an efficient LLM inference scheduler, Sarathi-Serve, to address this throughput-latency tradeoff. Sarathi-Serve introduces chunked-prefills which splits a prefill request into near equal sized chunks and creates stall-free schedules that adds new requests in a batch without pausing ongoing decodes. Stall-free scheduling unlocks the opportunity to improve throughput with large batch sizes while minimizing the effect of batching on latency. Furthermore, uniform batches in Sarathi-Serve ameliorate the imbalance between iterations resulting in minimal pipeline bubbles. Our techniques yield significant improvements in inference performance across models and hardware under tail latency constraints. For Mistral-7B on single A100 GPUs, we achieve 2.6x higher serving capacity and up to 3.7x higher serving capacity for the Yi-34B model on two A100 GPUs as compared to vLLM. When used with pipeline parallelism on Falcon-180B, Sarathi-Serve provides up to 5.6x gain in the end-to-end serving capacity. The source code for Sarathi-Serve is available at https://github.com/microsoft/sarathi-serve.

Owing to recent advancements, Large Language Models (LLMs) can now be deployed as agents for increasingly complex decision-making applications in areas including robotics, gaming, and API integration. However, reflecting past experiences in current decision-making processes, an innate human behavior, continues to pose significant challenges. Addressing this, we propose Retrieval-Augmented Planning (RAP) framework, designed to dynamically leverage past experiences corresponding to the current situation and context, thereby enhancing agents' planning capabilities. RAP distinguishes itself by being versatile: it excels in both text-only and multimodal environments, making it suitable for a wide range of tasks. Empirical evaluations demonstrate RAP's effectiveness, where it achieves SOTA performance in textual scenarios and notably enhances multimodal LLM agents' performance for embodied tasks. These results highlight RAP's potential in advancing the functionality and applicability of LLM agents in complex, real-world applications.

In this study, we present a comprehensive exploration of formation of different phases in lipid molecules using a coarse-grained implicit solvent model, where each lipid molecule is represented as a rigid, three-bead rod-like structure. Our study not only successfully replicates the spontaneous self-assembly of lipid bilayers but also elucidates the intricate phase transitions between the gel phase, liquid phase, and the elusive ripple phase(Pbeta). Specifically, we uncover the pivotal role of molecular rigidity in promoting the formation of the ripple phase. The significance of our findings lies in their potential to reshape our understanding of lipid bilayer dynamics and phase transitions. By shedding light on the ripple phase, a phase previously difficult to simulate convincingly, the insights gained from this study have the potential to guide future investigations into the behavior of biological membranes and their role in cellular processes., Comment: 7 pages

We investigate how well a simple leading order perturbation theory model of the bispectrum can fit the BAO feature in the measured bispectrum monopole of galaxies. Previous works showed that perturbative models of galaxy bispectrum start failing at the wavenumbers of k ~ 0.1 Mpc/h. We show that when the BAO feature in the bispectrum is separated it can be successfully modeled up to much higher wavenumbers. We validate our modeling on GLAM simulations that were run with and without the BAO feature in the initial conditions. We also quantify the amount of systematic error due to BAO template being offset from the true cosmology. We find that the systematic errors do not exceed 0.3 per cent for reasonable deviations from the true cosmology., Comment: Published version

Conversational AIs, or chatbots, mimic human speech when conversing. Smart assistants facilitate the automation of several tasks that needed human intervention earlier. Because of their accuracy, absence of dependence on human resources, and accessibility around the clock, chatbots can be employed in vehicles too. Due to people's propensity to divert their attention away from the task of driving while engaging in other activities like calling, playing music, navigation, and getting updates on the weather forecast and latest news, road safety has declined and accidents have increased as a result. It would be advantageous to automate these tasks using voice commands rather than carrying them out manually. This paper focuses on the development of a voice-based smart assistance application for vehicles based on the RASA framework. The smart assistant provides functionalities like navigation, communication via calls, getting weather forecasts and the latest news updates, and music that are completely voice-based in nature., Comment: 6 pages, 4 figures, accepted at IEEE International Conference on Computational Intelligence, Networks and Security ICCINS-2023

Fair face recognition is all about learning invariant feature that generalizes to unseen faces in any demographic group. Unfortunately, face datasets inevitably capture the imbalanced demographic attributes that are ubiquitous in real-world observations, and the model learns biased feature that generalizes poorly in the minority group. We point out that the bias arises due to the confounding demographic attributes, which mislead the model to capture the spurious demographic-specific feature. The confounding effect can only be removed by causal intervention, which requires the confounder annotations. However, such annotations can be prohibitively expensive due to the diversity of the demographic attributes. To tackle this, we propose to generate diverse data partitions iteratively in an unsupervised fashion. Each data partition acts as a self-annotated confounder, enabling our Invariant Feature Regularization (INV-REG) to deconfound. INV-REG is orthogonal to existing methods, and combining INV-REG with two strong baselines (Arcface and CIFP) leads to new state-of-the-art that improves face recognition on a variety of demographic groups. Code is available at https://github.com/PanasonicConnect/InvReg., Comment: Accepted by International Conference on Computer Vision (ICCV) 2023

We present Monte Carlo simulations under constant NVT conditions on a minimal three beads coarse grained implicit solvent model of lipid molecules, with the hydrophilic head represented by one bead and the hydrophobic tail represented by two beads. We consider two lipids, one with the head and tail bead sizes equal and the other with the tail beads smaller than the head. When cooled to the ambient temperature from an initial isotropic phase at high temperature, the first lipid transforms spontaneously to a lamellar phase while the second lipid transforms to a micellar phase, showing the crucial role of the head and tail size ratio on lipid phases., Comment: conference paper