Search

Connected Autonomous Vehicles (CAVs) benefit from Vehicle-to-Everything (V2X) communication, which enables the exchange of sensor data to achieve Collaborative Perception (CP). To reduce cumulative errors in perception modules and mitigate the visual occlusion, this paper introduces a new task, Collaborative Joint Perception and Prediction (Co-P&P), and provides a conceptual framework for its implementation to improve motion prediction of surrounding objects, thereby enhancing vehicle awareness in complex traffic scenarios. The framework consists of two decoupled core modules, Collaborative Scene Completion (CSC) and Joint Perception and Prediction (P&P) module, which simplify practical deployment and enhance scalability. Additionally, we outline the challenges in Co-P&P and discuss future directions for this research area., Comment: 8 pages, 4 figures, accepted by conference VEHITS2025

We present 50-fs, single-shot measurements of the x-ray thermal diffuse scattering (TDS) from copper foils that have been shocked via nanosecond laser-ablation up to pressures above 135~GPa. We hence deduce the x-ray Debye-Waller (DW) factor, providing a temperature measurement. The targets were laser-shocked with the DiPOLE 100-X laser at the High Energy Density (HED) endstation of the European X-ray Free-Electron Laser (EuXFEL). Single x-ray pulses, with a photon energy of 18 keV, were scattered from the samples and recorded on Varex detectors. Despite the targets being highly textured (as evinced by large variations in the elastic scattering), and with such texture changing upon compression, the absolute intensity of the azimuthally averaged inelastic TDS between the Bragg peaks is largely insensitive to these changes, and, allowing for both Compton scattering and the low-level scattering from a sacrificial ablator layer, provides a reliable measurement of $T/\Theta_D^2$, where $\Theta_D$ is the Debye temperature. We compare our results with the predictions of the SESAME 3336 and LEOS 290 equations of state for copper, and find good agreement within experimental errors. We thus demonstrate that single-shot temperature measurements of dynamically compressed materials can be made via thermal diffuse scattering of XFEL radation., Comment: 17 pages, 9 figures in main article; 10 pages, 5 figures in supplementary material

The increasingly large numbers of multiple images in cluster-scale gravitational lenses have allowed for tighter constraints on the mass distributions of these systems. Most lens models have progressed alongside this increase in image number. The general assumption is that these improvements would result in lens models converging to a common solution, suggesting that models are approaching the true mass distribution. To test whether or not this is occurring, we examine a sample of lens models of MACS J0416.1$-$2403 containing varying number of images as input. Splitting the sample into two bins (those including $<150$ and $>150$ images), we quantify the similarity of models in each bin using three comparison metrics, two of which are novel: Median Percent Difference, Frechet Distance, and Wasserstein Distance. In addition to quantifying similarity, the Frechet distance metric seems to also be an indicator of the mass sheet degeneracy. Each metric indicates that models with a greater number of input images are no more similar between one another than models with fewer input images. This suggests that lens models are neither converging nor diverging to a common solution for this system, regardless of method. With this result, we suggest that future models more carefully investigate lensing degeneracies and anomalous mass clumps (mass features significantly displaced from baryonic counterparts) to rigorously evaluate their model's validity. We also recommend further study into alternative, underutilized lens model priors (e.g. flux ratios) as an additional input constraint to image positions in hopes of breaking existing degeneracies., Comment: 10 pages, 7 figures. To be submitted to the Open Journal of Astrophysics. Comments welcome!

Building a generalist model for user interface (UI) understanding is challenging due to various foundational issues, such as platform diversity, resolution variation, and data limitation. In this paper, we introduce Ferret-UI 2, a multimodal large language model (MLLM) designed for universal UI understanding across a wide range of platforms, including iPhone, Android, iPad, Webpage, and AppleTV. Building on the foundation of Ferret-UI, Ferret-UI 2 introduces three key innovations: support for multiple platform types, high-resolution perception through adaptive scaling, and advanced task training data generation powered by GPT-4o with set-of-mark visual prompting. These advancements enable Ferret-UI 2 to perform complex, user-centered interactions, making it highly versatile and adaptable for the expanding diversity of platform ecosystems. Extensive empirical experiments on referring, grounding, user-centric advanced tasks (comprising 9 subtasks $\times$ 5 platforms), GUIDE next-action prediction dataset, and GUI-World multi-platform benchmark demonstrate that Ferret-UI 2 significantly outperforms Ferret-UI, and also shows strong cross-platform transfer capabilities., Comment: Accepted to ICLR 2025

Animal vocalization denoising is a task similar to human speech enhancement, which is relatively well-studied. In contrast to the latter, it comprises a higher diversity of sound production mechanisms and recording environments, and this higher diversity is a challenge for existing models. Adding to the challenge and in contrast to speech, we lack large and diverse datasets comprising clean vocalizations. As a solution we use as training data pseudo-clean targets, i.e. pre-denoised vocalizations, and segments of background noise without a vocalization. We propose a train set derived from bioacoustics datasets and repositories representing diverse species, acoustic environments, geographic regions. Additionally, we introduce a non-overlapping benchmark set comprising clean vocalizations from different taxa and noise samples. We show that that denoising models (demucs, CleanUNet) trained on pseudo-clean targets obtained with speech enhancement models achieve competitive results on the benchmarking set. We publish data, code, libraries, and demos at https://mariusmiron.com/research/biodenoising., Comment: 5 pages, 2 tables

Contrastive Language-Image Pre-training (CLIP) has been a celebrated method for training vision encoders to generate image/text representations facilitating various applications. Recently, CLIP has been widely adopted as the vision backbone of multimodal large language models (MLLMs) to connect image inputs for language interactions. The success of CLIP as a vision-language foundation model relies on aligning web-crawled noisy text annotations at image levels. Nevertheless, such criteria may become insufficient for downstream tasks in need of fine-grained vision representations, especially when region-level understanding is demanding for MLLMs. In this paper, we improve the localization capability of CLIP with several advances. We propose a pre-training method called Contrastive Localized Language-Image Pre-training (CLOC) by complementing CLIP with region-text contrastive loss and modules. We formulate a new concept, promptable embeddings, of which the encoder produces image embeddings easy to transform into region representations given spatial hints. To support large-scale pre-training, we design a visually-enriched and spatially-localized captioning framework to effectively generate region-text pseudo-labels at scale. By scaling up to billions of annotated images, CLOC enables high-quality regional embeddings for image region recognition and retrieval tasks, and can be a drop-in replacement of CLIP to enhance MLLMs, especially on referring and grounding tasks., Comment: Preprint

We present MM1.5, a new family of multimodal large language models (MLLMs) designed to enhance capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning. Building upon the MM1 architecture, MM1.5 adopts a data-centric approach to model training, systematically exploring the impact of diverse data mixtures across the entire model training lifecycle. This includes high-quality OCR data and synthetic captions for continual pre-training, as well as an optimized visual instruction-tuning data mixture for supervised fine-tuning. Our models range from 1B to 30B parameters, encompassing both dense and mixture-of-experts (MoE) variants, and demonstrate that careful data curation and training strategies can yield strong performance even at small scales (1B and 3B). Additionally, we introduce two specialized variants: MM1.5-Video, designed for video understanding, and MM1.5-UI, tailored for mobile UI understanding. Through extensive empirical studies and ablations, we provide detailed insights into the training processes and decisions that inform our final designs, offering valuable guidance for future research in MLLM development.

Nevanlinna functions are meromorphic functions with a finite number of asymptotic values and no critical values. In [KK2] it was proved that if the orbits of all the asymptotic values accumulate on a compact set on which the function acts as a repeller, then the function acts ergodically on its Julia set. In [CJK4] we proved the action of the function on its Julia set is still ergodic if some, but not all of the asymptotic values land on infinity, and the remaining ones land on a compact repeller. In this paper, we complete the characterization of ergodicity for Nevanlinna functions by proving that if all the asymptotic values land on infinity, then the Julia set is the whole sphere and the action of the map there is non-ergodic., Comment: 18 pages, 4 figures

The Abstraction and Reasoning Corpus (ARC) is a visual program synthesis benchmark designed to test challenging out-of-distribution generalization in humans and machines. Since 2019, limited progress has been observed on the challenge using existing artificial intelligence methods. Comparing human and machine performance is important for the validity of the benchmark. While previous work explored how well humans can solve tasks from the ARC benchmark, they either did so using only a subset of tasks from the original dataset, or from variants of ARC, and therefore only provided a tentative estimate of human performance. In this work, we obtain a more robust estimate of human performance by evaluating 1729 humans on the full set of 400 training and 400 evaluation tasks from the original ARC problem set. We estimate that average human performance lies between 73.3% and 77.2% correct with a reported empirical average of 76.2% on the training set, and between 55.9% and 68.9% correct with a reported empirical average of 64.2% on the public evaluation set. However, we also find that 790 out of the 800 tasks were solvable by at least one person in three attempts, suggesting that the vast majority of the publicly available ARC tasks are in principle solvable by typical crowd-workers recruited over the internet. Notably, while these numbers are slightly lower than earlier estimates, human performance still greatly exceeds current state-of-the-art approaches for solving ARC. To facilitate research on ARC, we publicly release our dataset, called H-ARC (human-ARC), which includes all of the submissions and action traces from human participants., Comment: 12 pages, 7 figures

Jahn-Teller distortions of transition-metal coordination environments link orbital occupancies to structure. In the solid state, such distortions can be strongly correlated through the propagation of strain and/or through orbital interactions. Cooperative Jahn-Teller (CJT) order of this kind affects the electronic, magnetic, and structural properties of the materials in which it occurs. Conventionally studied in dense ceramics, CJT order also occurs in hybrid materials, albeit the underlying phenomenology is not well established. Here we use synchrotron powder X-ray diffraction measurements to determine links between composition and Jahn-Teller effects in a series of Prussian blue analogue (PBA) families. We develop a simple model based on the dual considerations of strain and crystal-field stabilisation that can account for the stability, extent of CJT order, and crystallite strain measured experimentally. This model shows how PBA compositions might be tuned to control the emergence and nature of CJT effects. A key implication of the model is that PBAs harbour their own complex interplay between compositional, structural, orbital, and magnetic degrees of freedom related to that of strongly-correlated functional materials such as the manganite perovskites.

Though vision transformers (ViTs) have achieved state-of-the-art performance in a variety of settings, they exhibit surprising failures when performing tasks involving visual relations. This begs the question: how do ViTs attempt to perform tasks that require computing visual relations between objects? Prior efforts to interpret ViTs tend to focus on characterizing relevant low-level visual features. In contrast, we adopt methods from mechanistic interpretability to study the higher-level visual algorithms that ViTs use to perform abstract visual reasoning. We present a case study of a fundamental, yet surprisingly difficult, relational reasoning task: judging whether two visual entities are the same or different. We find that pretrained ViTs fine-tuned on this task often exhibit two qualitatively different stages of processing despite having no obvious inductive biases to do so: 1) a perceptual stage wherein local object features are extracted and stored in a disentangled representation, and 2) a relational stage wherein object representations are compared. In the second stage, we find evidence that ViTs can learn to represent somewhat abstract visual relations, a capability that has long been considered out of reach for artificial neural networks. Finally, we demonstrate that failures at either stage can prevent a model from learning a generalizable solution to our fairly simple tasks. By understanding ViTs in terms of discrete processing stages, one can more precisely diagnose and rectify shortcomings of existing and future models.

The transition-metal dicyanides M(CN)$_2$ (M = Zn, Cd) are amongst the most important negative thermal expansion (NTE) materials known, favoured for the magnitude, isotropy, and thermal persistence of the NTE behaviour they show. The conventional picture of the NTE mechanism in this family is one of correlated rotations and translations of M(C/N)$_4$ polyhedra acting to draw the diamondoid network of M--CN--M linkages in on itself. An implication of this mechanism is increased transverse vibrational motion of C and N atoms relative to the isotropic displacements of M atoms, which act as anchors. Here, we use a combination of neutron total scattering measurements and \emph{ab initio} calculations to reassess the vibrational behaviour of the M(CN)$_2$ family. We find that M, C, and N atoms all exhibit similar degrees of local thermal motion, such that the cyanide linkages behave as pseudo-springs connecting M$\ldots$M pairs. This interpretation leads us to uncover a `hidden' dispersion in the M(CN)$_2$ phonon dispersions, closely related to that of diamond and silicon themselves. By virtue of this mapping, a simple geometric model based on the competing energy scales of network stretching and flexing -- long applied to interpret NTE modes in C and Si -- turns out to capture the key NTE physics of M(CN)$_2$, especially at low temperatures. Our study highlights the potential insight gained by coarse-graining the complex lattice dynamics of framework materials in terms of what we call `framework modes' -- the correlated distortions of the underlying network structure itself.

Effective resistance is a distance between vertices of a graph that is both theoretically interesting and useful in applications. We study a variant of effective resistance called the biharmonic distance. While the effective resistance measures how well-connected two vertices are, we prove several theoretical results supporting the idea that the biharmonic distance measures how important an edge is to the global topology of the graph. Our theoretical results connect the biharmonic distance to well-known measures of connectivity of a graph like its total resistance and sparsity. Based on these results, we introduce two clustering algorithms using the biharmonic distance. Finally, we introduce a further generalization of the biharmonic distance that we call the $k$-harmonic distance. We empirically study the utility of biharmonic and $k$-harmonic distance for edge centrality and graph clustering., Comment: Accepted to ICML 2024. In v2, we correct an error in the definition of electrical flows and, accordingly, the proofs of Lemma 2.2 and Theorem 4.1

Recent advancements in multimodal large language models (MLLMs) have been noteworthy, yet, these general-domain MLLMs often fall short in their ability to comprehend and interact effectively with user interface (UI) screens. In this paper, we present Ferret-UI, a new MLLM tailored for enhanced understanding of mobile UI screens, equipped with referring, grounding, and reasoning capabilities. Given that UI screens typically exhibit a more elongated aspect ratio and contain smaller objects of interest (e.g., icons, texts) than natural images, we incorporate "any resolution" on top of Ferret to magnify details and leverage enhanced visual features. Specifically, each screen is divided into 2 sub-images based on the original aspect ratio (i.e., horizontal division for portrait screens and vertical division for landscape screens). Both sub-images are encoded separately before being sent to LLMs. We meticulously gather training samples from an extensive range of elementary UI tasks, such as icon recognition, find text, and widget listing. These samples are formatted for instruction-following with region annotations to facilitate precise referring and grounding. To augment the model's reasoning ability, we further compile a dataset for advanced tasks, including detailed description, perception/interaction conversations, and function inference. After training on the curated datasets, Ferret-UI exhibits outstanding comprehension of UI screens and the capability to execute open-ended instructions. For model evaluation, we establish a comprehensive benchmark encompassing all the aforementioned tasks. Ferret-UI excels not only beyond most open-source UI MLLMs, but also surpasses GPT-4V on all the elementary UI tasks.

Radio Frequency (RF) device fingerprinting has been recognized as a potential technology for enabling automated wireless device identification and classification. However, it faces a key challenge due to the domain shift that could arise from variations in the channel conditions and environmental settings, potentially degrading the accuracy of RF-based device classification when testing and training data is collected in different domains. This paper introduces a novel solution that leverages contrastive learning to mitigate this domain shift problem. Contrastive learning, a state-of-the-art self-supervised learning approach from deep learning, learns a distance metric such that positive pairs are closer (i.e. more similar) in the learned metric space than negative pairs. When applied to RF fingerprinting, our model treats RF signals from the same transmission as positive pairs and those from different transmissions as negative pairs. Through experiments on wireless and wired RF datasets collected over several days, we demonstrate that our contrastive learning approach captures domain-invariant features, diminishing the effects of domain-specific variations. Our results show large and consistent improvements in accuracy (10.8\% to 27.8\%) over baseline models, thus underscoring the effectiveness of contrastive learning in improving device classification under domain shift., Comment: 6 pages, 5 figures, accepted by 2024 IEEE International Conference on Communications (ICC)

Conventional deep learning models deal with images one-by-one, requiring costly and time-consuming expert labeling in the field of medical imaging, and domain-specific restriction limits model generalizability. Visual in-context learning (ICL) is a new and exciting area of research in computer vision. Unlike conventional deep learning, ICL emphasizes the model's ability to adapt to new tasks based on given examples quickly. Inspired by MAE-VQGAN, we proposed a new simple visual ICL method called SimICL, combining visual ICL pairing images with masked image modeling (MIM) designed for self-supervised learning. We validated our method on bony structures segmentation in a wrist ultrasound (US) dataset with limited annotations, where the clinical objective was to segment bony structures to help with further fracture detection. We used a test set containing 3822 images from 18 patients for bony region segmentation. SimICL achieved an remarkably high Dice coeffient (DC) of 0.96 and Jaccard Index (IoU) of 0.92, surpassing state-of-the-art segmentation and visual ICL models (a maximum DC 0.86 and IoU 0.76), with SimICL DC and IoU increasing up to 0.10 and 0.16. This remarkably high agreement with limited manual annotations indicates SimICL could be used for training AI models even on small US datasets. This could dramatically decrease the human expert time required for image labeling compared to conventional approaches, and enhance the real-world use of AI assistance in US image analysis.

Large language models (LLMs) have attracted significant attention for potential applications in digital health, while their application in mental health is subject to ongoing debate. This systematic review aims to evaluate the usage of LLMs in mental health, focusing on their strengths and limitations in early screening, digital interventions, and clinical applications. Adhering to PRISMA guidelines, we searched PubMed, IEEE Xplore, Scopus, JMIR, and ACM using keywords: 'mental health OR mental illness OR mental disorder OR psychiatry' AND 'large language models'. We included articles published between January 1, 2017, and April 30, 2024, excluding non-English articles. 30 articles were evaluated, which included research on mental health conditions and suicidal ideation detection through text (n=15), usage of LLMs for mental health conversational agents (CAs) (n=7), and other applications and evaluations of LLMs in mental health (n=18). LLMs exhibit substantial effectiveness in detecting mental health issues and providing accessible, de-stigmatized eHealth services. However, the current risks associated with the clinical use might surpass their benefits. The study identifies several significant issues: the lack of multilingual datasets annotated by experts, concerns about the accuracy and reliability of the content generated, challenges in interpretability due to the 'black box' nature of LLMs, and persistent ethical dilemmas. These include the lack of a clear ethical framework, concerns about data privacy, and the potential for over-reliance on LLMs by both therapists and patients, which could compromise traditional medical practice. Despite these issues, the rapid development of LLMs underscores their potential as new clinical aids, emphasizing the need for continued research and development in this area.

Recent advances in machine learning, specifically transformer architecture, have led to significant advancements in commercial domains. These powerful models have demonstrated superior capability to learn complex relationships and often generalize better to new data and problems. This paper presents a novel transformer-powered approach for enhancing prediction accuracy in multi-modal output scenarios, where sparse experimental data is supplemented with simulation data. The proposed approach integrates transformer-based architecture with a novel graph-based hyper-parameter optimization technique. The resulting system not only effectively reduces simulation bias, but also achieves superior prediction accuracy compared to the prior method. We demonstrate the efficacy of our approach on inertial confinement fusion experiments, where only 10 shots of real-world data are available, as well as synthetic versions of these experiments., Comment: MLST

This study introduces Ascle, a pioneering natural language processing (NLP) toolkit designed for medical text generation. Ascle is tailored for biomedical researchers and healthcare professionals with an easy-to-use, all-in-one solution that requires minimal programming expertise. For the first time, Ascle evaluates and provides interfaces for the latest pre-trained language models, encompassing four advanced and challenging generative functions: question-answering, text summarization, text simplification, and machine translation. In addition, Ascle integrates 12 essential NLP functions, along with query and search capabilities for clinical databases. The toolkit, its models, and associated data are publicly available via https://github.com/Yale-LILY/MedGen., Comment: 5 figures, 4 tables

Although deep neural networks can achieve human-level performance on many object recognition benchmarks, prior work suggests that these same models fail to learn simple abstract relations, such as determining whether two objects are the same or different. Much of this prior work focuses on training convolutional neural networks to classify images of two same or two different abstract shapes, testing generalization on within-distribution stimuli. In this article, we comprehensively study whether deep neural networks can acquire and generalize same-different relations both within and out-of-distribution using a variety of architectures, forms of pretraining, and fine-tuning datasets. We find that certain pretrained transformers can learn a same-different relation that generalizes with near perfect accuracy to out-of-distribution stimuli. Furthermore, we find that fine-tuning on abstract shapes that lack texture or color provides the strongest out-of-distribution generalization. Our results suggest that, with the right approach, deep neural networks can learn generalizable same-different visual relations.

Material functionality can be strongly determined by structure extending only over nanoscale distances. The pair distribution function presents an opportunity to shift structural studies beyond idealized crystal models and investigate structure over varying length scales. Applying this method with ultrafast time resolution has the potential to similarly disrupt the study of structural dynamics and phase transitions. Here, we demonstrate such a measurement of CuIr$_{2}$S$_{4}$ optically pumped from its low temperature Ir-dimerized phase. Dimers are optically suppressed without spatial correlation, generating a structure whose level of disorder depends strongly on length scale. The re-development of structural ordering over tens of picoseconds is directly tracked over both space and time as a transient state is approached. This measurement demonstrates both the crucial role of local structure and disorder in non-equilibrium processes and the feasibility of accessing this information with state-of-the-art XFEL facilities., Comment: 14 page manuscript with 4 figures. 13 page Supplementary with 17 figures. 27 pages and 21 figures in total

We study Nevanlinna functions f that are transcendental meromorphic functions having N asymptotic values and no critical values. In [KK] it was proved that if the orbits of all the asymptotic values have accumulation sets that are compact and on which f is a repeller, then f acts ergodically on its Julia set. In this paper, we prove that if some, but not all of the asymptotic values have this property, while the others are prepoles, the same holds true. This is the first paper to consider this mixed case., Comment: 17 pages

Supervised deep learning offers great promise to automate analysis of medical images from segmentation to diagnosis. However, their performance highly relies on the quality and quantity of the data annotation. Meanwhile, curating large annotated datasets for medical images requires a high level of expertise, which is time-consuming and expensive. Recently, to quench the thirst for large data sets with high-quality annotation, self-supervised learning (SSL) methods using unlabeled domain-specific data, have attracted attention. Therefore, designing an SSL method that relies on minimal quantities of labeled data has far-reaching significance in medical images. This paper investigates the feasibility of deploying the Masked Autoencoder for SSL (SSL-MAE) of TransUNet, for segmenting bony regions from children's wrist ultrasound scans. We found that changing the embedding and loss function in SSL-MAE can produce better downstream results compared to the original SSL-MAE. In addition, we determined that only pretraining TransUNet embedding and encoder with SSL-MAE does not work as well as TransUNet without SSL-MAE pretraining on downstream segmentation tasks.

A critical challenge in developing scalable error-corrected quantum systems is the accumulation of errors while performing operations and measurements. One promising approach is to design a system where errors can be detected and converted into erasures. Such a system utilizing erasure qubits are known to have relaxed requirements for quantum error correction. A recent proposal aims to do this using a dual-rail encoding with superconducting cavities. However, experimental characterization and demonstration of a dual-rail cavity qubit has not yet been realized. In this work, we implement such a dual-rail cavity qubit; we demonstrate a projective logical measurement with integrated erasure detection and use it to measure dual-rail qubit idling errors. We measure logical state preparation and measurement errors at the $0.01\%$-level and detect over $99\%$ of cavity decay events as erasures. We use the precision of this new measurement protocol to distinguish different types of errors in this system, finding that while decay errors occur with probability $\sim 0.2\%$ per microsecond, phase errors occur 6 times less frequently and bit flips occur at least 140 times less frequently. These findings represent the first confirmation of the expected error hierarchy necessary to concatenate dual-rail erasure qubits into a highly efficient erasure code.