Search

Multimodal foundation models, such as Gemini and ChatGPT, have revolutionized human-machine interactions by seamlessly integrating various forms of data. Developing a universal spoken language model that comprehends a wide range of natural language instructions is critical for bridging communication gaps and facilitating more intuitive interactions. However, the absence of a comprehensive evaluation benchmark poses a significant challenge. We present Dynamic-SUPERB Phase-2, an open and evolving benchmark for the comprehensive evaluation of instruction-based universal speech models. Building upon the first generation, this second version incorporates 125 new tasks contributed collaboratively by the global research community, expanding the benchmark to a total of 180 tasks, making it the largest benchmark for speech and audio evaluation. While the first generation of Dynamic-SUPERB was limited to classification tasks, Dynamic-SUPERB Phase-2 broadens its evaluation capabilities by introducing a wide array of novel and diverse tasks, including regression and sequence generation, across speech, music, and environmental audio. Evaluation results indicate that none of the models performed well universally. SALMONN-13B excelled in English ASR, while WavLLM demonstrated high accuracy in emotion recognition, but current models still require further innovations to handle a broader range of tasks. We will soon open-source all task data and the evaluation pipeline.

Handling test-time unknown degradation is the major challenge in Blind Image Restoration (BIR), necessitating high model generalization. An effective strategy is to incorporate prior knowledge, either from human input or generative model. In this paper, we introduce Instant-reference Image Restoration (InstantIR), a novel diffusion-based BIR method which dynamically adjusts generation condition during inference. We first extract a compact representation of the input via a pre-trained vision encoder. At each generation step, this representation is used to decode current diffusion latent and instantiate it in the generative prior. The degraded image is then encoded with this reference, providing robust generation condition. We observe the variance of generative references fluctuate with degradation intensity, which we further leverage as an indicator for developing a sampling algorithm adaptive to input quality. Extensive experiments demonstrate InstantIR achieves state-of-the-art performance and offering outstanding visual quality. Through modulating generative references with textual description, InstantIR can restore extreme degradation and additionally feature creative restoration.

Imagine searching a collection of coins for quarters ($0.25$), dimes ($0.10$), nickels ($0.05$), and pennies ($0.01$)-a hybrid foraging task where observers look for multiple instances of multiple target types. In such tasks, how do target values and their prevalence influence foraging and eye movement behaviors (e.g., should you prioritize rare quarters or common nickels)? To explore this, we conducted human psychophysics experiments, revealing that humans are proficient reward foragers. Their eye fixations are drawn to regions with higher average rewards, fixation durations are longer on more valuable targets, and their cumulative rewards exceed chance, approaching the upper bound of optimal foragers. To probe these decision-making processes of humans, we developed a transformer-based Visual Forager (VF) model trained via reinforcement learning. Our VF model takes a series of targets, their corresponding values, and the search image as inputs, processes the images using foveated vision, and produces a sequence of eye movements along with decisions on whether to collect each fixated item. Our model outperforms all baselines, achieves cumulative rewards comparable to those of humans, and approximates human foraging behavior in eye movements and foraging biases within time-limited environments. Furthermore, stress tests on out-of-distribution tasks with novel targets, unseen values, and varying set sizes demonstrate the VF model's effective generalization. Our work offers valuable insights into the relationship between eye movements and decision-making, with our model serving as a powerful tool for further exploration of this connection. All data, code, and models will be made publicly available.

The Indian summer monsoon (ISM) and associated monsoon intraseasonal oscillations (MISOs) influence the billions of people living in the Indian subcontinent. This study explores the role of autoconversion parameterization in microphysical schemes for the simulation of MISO with the coupled climate model, e.g., the Climate Forecast System version 2 (CFSv2), by conducting sensitivity experiments in two resolutions (~100 km and ~38 km). Results reveal that the modified autoconversion parameterization better simulates the active-break spells of the ISM rainfall. The main improvements include the contrasting features of rainfall over land and ocean and the MISO index, representing MISO periodicity. The improvements are qualitatively and quantitatively more significant in the higher-resolution simulations, particularly regarding rainfall spatial patterns over the Indian subcontinent during active spells. The MISO monitoring index in the revised CFSv2 also shows improvement compared to the control run. This study concludes that proper autoconversion parameterization in the coupled climate model can lead to enhanced representation of active/break spells and sub-seasonal variability of ISM.

This paper evaluates the experimental generation of internal solitary waves (ISWs) in a miscible two-layer system with a free surface using a jet-array wavemaker (JAW). Unlike traditional gate-release experiments, the JAW system generates ISWs by forcing a prescribed vertical distribution of mass flux. Experiments examine three different layer-depth ratios, with ISW amplitudes up to the maximum allowed by the extended Korteweg-de Vries (eKdV) solution. Phase speeds and wave profiles are captured via planar laser-induced fluorescence and the velocity field is measured synchronously using particle imaging velocimetry. Measured properties are directly compared with the eKdV predictions. As expected, small- and intermediate-amplitude waves match well with the corresponding eKdV solutions, with errors in amplitude and phase speed below 10%. For large waves with amplitudes approaching the maximum allowed by the eKdV solution, the phase speed and the velocity profiles resemble the eKdV solution while the wave profiles are distorted following the trough. This can potentially be attributed to Kelvin-Helmholtz instabilities forming at the pycnocline. Larger errors are generally observed when the local Richardson number at the JAW inlet exceeds the threshold for instability., Comment: 25 pages, 11 figures

We introduce MVSplat360, a feed-forward approach for 360{\deg} novel view synthesis (NVS) of diverse real-world scenes, using only sparse observations. This setting is inherently ill-posed due to minimal overlap among input views and insufficient visual information provided, making it challenging for conventional methods to achieve high-quality results. Our MVSplat360 addresses this by effectively combining geometry-aware 3D reconstruction with temporally consistent video generation. Specifically, it refactors a feed-forward 3D Gaussian Splatting (3DGS) model to render features directly into the latent space of a pre-trained Stable Video Diffusion (SVD) model, where these features then act as pose and visual cues to guide the denoising process and produce photorealistic 3D-consistent views. Our model is end-to-end trainable and supports rendering arbitrary views with as few as 5 sparse input views. To evaluate MVSplat360's performance, we introduce a new benchmark using the challenging DL3DV-10K dataset, where MVSplat360 achieves superior visual quality compared to state-of-the-art methods on wide-sweeping or even 360{\deg} NVS tasks. Experiments on the existing benchmark RealEstate10K also confirm the effectiveness of our model. The video results are available on our project page: https://donydchen.github.io/mvsplat360., Comment: NeurIPS 2024, Project page: https://donydchen.github.io/mvsplat360, Code: https://github.com/donydchen/mvsplat360

Student mental health is a sensitive issue that necessitates special attention. A primary concern is the student-to-counselor ratio, which surpasses the recommended standard of 250:1 in most universities. This imbalance results in extended waiting periods for in-person consultations, which cause suboptimal treatment. Significant efforts have been directed toward developing mental health dialogue systems utilizing the existing open-source mental health-related datasets. However, currently available datasets either discuss general topics or various strategies that may not be viable for direct application due to numerous ethical constraints inherent in this research domain. To address this issue, this paper introduces a specialized mental health dataset that emphasizes the active listening strategy employed in conversation for counseling, also named as ConvCounsel. This dataset comprises both speech and text data, which can facilitate the development of a reliable pipeline for mental health dialogue systems. To demonstrate the utility of the proposed dataset, this paper also presents the NYCUKA, a spoken mental health dialogue system that is designed by using the ConvCounsel dataset. The results show the merit of using this dataset., Comment: Accepted at O-COCOSDA 2024, Won Best Student Paper Award

Automated detection of chemical plumes presents a segmentation challenge. The segmentation problem for gas plumes is difficult due to the diffusive nature of the cloud. The advantage of considering hyperspectral images in the gas plume detection problem over the conventional RGB imagery is the presence of non-visual data, allowing for a richer representation of information. In this paper we present an effective method of visualizing hyperspectral video sequences containing chemical plumes and investigate the effectiveness of segmentation techniques on these post-processed videos. Our approach uses a combination of dimension reduction and histogram equalization to prepare the hyperspectral videos for segmentation. First, Principal Components Analysis (PCA) is used to reduce the dimension of the entire video sequence. This is done by projecting each pixel onto the first few Principal Components resulting in a type of spectral filter. Next, a Midway method for histogram equalization is used. These methods redistribute the intensity values in order to reduce flicker between frames. This properly prepares these high-dimensional video sequences for more traditional segmentation techniques. We compare the ability of various clustering techniques to properly segment the chemical plume. These include K-means, spectral clustering, and the Ginzburg-Landau functional.

For a basic classical Lie superalgebra $\mathfrak s$, let $\mathfrak g$ be the central extension of the Takiff superalgebra $\mathfrak s\otimes\Lambda(\theta)$, where $\theta$ is an odd indeterminate. We study the category of $\mathfrak g$-Whittaker modules associated with a nilcharacter $\chi$ of $\mathfrak g$ and show that it is equivalent to the category of $\mathfrak s$-Whittaker modules associated with a nilcharacter of $\mathfrak s$ determined by $\chi$. In the case when $\chi$ is regular, we obtain, as an application, an equivalence between the categories of modules over the supersymmetric finite $W$-algebras associated to the odd principal nilpotent element at non-critical levels and the category of the modules over the principal finite $W$-superalgebra associated to $\mathfrak s$. Here, a supersymmetric finite $W$-algebra is conjecturally the Zhu algebra of a supersymmetric affine $W$-algebra. This allows us to classify and construct irreducible representations of a principal finite supersymmetric $W$-algebra., Comment: 27 pages

Synthetic data augmentation via large language models (LLMs) allows researchers to leverage additional training data, thus enhancing the performance of downstream tasks, especially when real-world data is scarce. However, the generated data can deviate from the real-world data, and this misalignment can bring deficient outcomes while applying the trained model to applications. Therefore, we proposed efficient weighted-loss approaches to align synthetic data with real-world distribution by emphasizing high-quality and diversified data generated by LLMs with using merely a little real-world data. We empirically assessed the effectiveness of our method on multiple text classification tasks, and the results showed leveraging our approaches on a BERT-level model robustly outperformed standard cross-entropy and other data weighting approaches, providing potential solutions to effectively leveraging synthetic data from any suitable data generator for model training., Comment: 12 pages, 7 figures

We report Magnetospheric Multiscale observations of oxygen ions (O+) during a coronal mass ejection in April 2023 when the solar wind was sub-Alfv\'enic and Alfv\'en wings formed. For the first time, O+ characteristics are studied at the contact region between the unshocked solar wind and the magnetosphere. The O+ ions show energies between 100s eV and ~30 keV. The possible sources are the ring current, the warm plasma cloak, and the ionosphere. The O+ ions exhibit bi-directional streaming along newly-formed closed field lines (CFLs), and dominantly anti-parallel on earlier-formed CFLs. Escaping O+ ions in the unshocked solar wind are observed. During the recovery phase, the O+ pitch-angle distribution associated with flux tubes shows dispersion, indicating potential loss to the solar wind. Our results show escaping as well as trapped O+ ions in the region where a magnetic cloud, an Alfv\'en wing, and magnetospheric field lines are mixed.

Photon-mediated dipole-dipole interactions arise from atom-light interactions, which are universal and prevalent in a wide range of open quantum systems. This pairwise and long-range spin-exchange interaction results from multiple light scattering among the atoms. A recent surge of interests and progresses in both experiments and theories promises this core mechanism of collective interactions as a resource to study quantum science and to envision next-generation applications in quantum technology. Here we summarize recent developments in both theories and experiments, where we introduce several central theoretical approaches and focus on cooperative light scattering from small sample of free-space atoms, an atom-waveguide coupled interface that hosts the waveguide QED, and topological quantum optical platforms. The aim of this review is to manifest the essential and distinct features of collective dynamics induced by resonant dipole-dipole interactions and to reveal unprecedented opportunities in enhancing the performance or offering new applications in light manipulations, quantum metrology, quantum computations, and light harvesting innovations., Comment: 11 figures, a review article

In this and a companion paper [arXiv:2410.09277], we give a general and comprehensive theory for nonequilibrium (NEQ) network forces and flows (Caliber Force Theory, CFT). It follows the "Two Laws" structure of Equilibrium Thermodynamics, where a First Law asserts conservation constraints and a Second Law is a variational predictor of a system's status, that, when taken together, give a rich set of prediction tools. The novel results for network flows here are that: (1) CFT defines forces, in this case for the three independent dynamic quantities: node distribution, edge traffic, and cycle flux, allowing for treating complex nets with conflicting constraints, actions, costs and benefits. (2) CFT is not limited to approximations and assumptions about nearness to equilibria, Local Detailed Balance or heat baths. (3) CFT shows that Kirchhoff current and voltage laws are an incomplete set; we find that a third such law applies when fluctuations are important.

We derive a closed-form achievable rate for entanglement-assisted classical communication over a lossy thermal-noise bosonic channel, where the entanglement is in the form of a Two-Mode Squeezed Vacuum (TMSV) modulation restricted to Phase Shift Keying (PSK). The achievable rate is non-asymptotic in terms of the mean signal photon number, mean noise photon number, and transmissivity defining the communication channel, which provides insights into the interplay of these physical parameters and bridges recent experimental demonstrations of entanglement-assisted communications with the coding theorems used in information-theoretic proofs. The key challenge we address is deriving an analytical bound for the von Neumann entropy of the non-Gaussian mixed state resulting from the phase modulation of one arm of a TMSV. Our approach hinges on two key observations: 1) as the size of the PSK modulation increases, the resulting mixed state converges in trace distance to a diagonal state in the Fock basis; 2) the Fock-basis representation of the diagonal state involves hypergeometric functions that can be appropriately bounded to offer a tractable lower bound for the Holevo information.

The magnetic cloud within an interplanetary coronal mass ejection (ICME) is characterized by high magnetic field intensities. In this study, we investigate the interaction of a magnetic cloud carrying a density structure with the Earth's bow shock during the ICME event on 24 April 2023. Elevated abundances of cold protons and heavier ions, namely alpha particles and singly charged helium ions, associated with the prominence plasma are observed within this structure. The plasma downstream of the bow shock exhibits an irregular compression pattern which could be due to the presence of heavy ions. Heavy ions carry a significant fraction of the upstream flow energy; however, due to their different charge per mass ratio and rigidity, they are less scattered by the electromagnetic and electrostatic waves at the shock. We find that downstream of the shock, while the thermal ion energy is only a small fraction of the background magnetic energy density, nevertheless increased ion fluxes reduce the characteristic wave speeds in the that region. As such, we observe a transition state of an unstable bow shock layer across which the plasma flow is super Alfv\'enic in both upstream and downstream regions. Our findings help with understanding the intense space weather impacts of such events.

Reinforcement learning combined with sim-to-real transfer offers a general framework for developing locomotion controllers for legged robots. To facilitate successful deployment in the real world, smoothing techniques, such as low-pass filters and smoothness rewards, are often employed to develop policies with smooth behaviors. However, because these techniques are non-differentiable and usually require tedious tuning of a large set of hyperparameters, they tend to require extensive manual tuning for each robotic platform. To address this challenge and establish a general technique for enforcing smooth behaviors, we propose a simple and effective method that imposes a Lipschitz constraint on a learned policy, which we refer to as Lipschitz-Constrained Policies (LCP). We show that the Lipschitz constraint can be implemented in the form of a gradient penalty, which provides a differentiable objective that can be easily incorporated with automatic differentiation frameworks. We demonstrate that LCP effectively replaces the need for smoothing rewards or low-pass filters and can be easily integrated into training frameworks for many distinct humanoid robots. We extensively evaluate LCP in both simulation and real-world humanoid robots, producing smooth and robust locomotion controllers. All simulation and deployment code, along with complete checkpoints, is available on our project page: https://lipschitz-constrained-policy.github.io., Comment: 8 pages

The COVID-19 pandemic has profoundly impacted global economies and healthcare systems, revealing critical vulnerabilities in both. In response, our study introduces a groundbreaking method for the detection of SARS-CoV-2 cDNA, leveraging Luminescence resonance energy transfer (LRET) between upconversion nanoparticles (UCNPs) and gold nanoparticles (AuNPs) to achieve an unprecedented detection limit of 242 femtomolar (fM). This innovative sensing platform utilizes UCNPs conjugated with one primer and AuNPs with another, targeting the 5' and 3' ends of the SARS-CoV-2 cDNA, respectively, enabling precise differentiation of mismatched DNA sequences and significantly enhancing detection specificity. Through rigorous experimental analysis, we established a quenching efficiency range from 10.4\% to 73.6\%, with an optimal midpoint of 42\%, thereby demonstrating the superior sensitivity of our method. By comparing the quenching efficiency of mismatched DNAs to the target DNA, we identified an optimal DNA:UCNP:AuNP ratio that ensures accurate detection. Our comparative analysis with existing SARS-CoV-2 detection methods revealed that our approach not only provides a lower detection limit but also offers higher specificity and potential for rapid, on-site testing. This study demonstrates the superior sensitivity and specificity of using UCNPs and AuNPs for SARS-CoV-2 cDNA detection, offering a significant advancement in rapid, accessible diagnostic technologies. Our method, characterized by its low detection limit and high precision, represents a critical step forward in managing current and future viral outbreaks, contributing to the enhancement of global healthcare responsiveness and infectious disease control., Comment: 14 pages, 5 figs

Extreme multi-label classification (XMC) aims to identify relevant subsets from numerous labels. Among the various approaches for XMC, tree-based linear models are effective due to their superior efficiency and simplicity. However, the space complexity of tree-based methods is not well-studied. Many past works assume that storing the model is not affordable and apply techniques such as pruning to save space, which may lead to performance loss. In this work, we conduct both theoretical and empirical analyses on the space to store a tree model under the assumption of sparse data, a condition frequently met in text data. We found that, some features may be unused when training binary classifiers in a tree method, resulting in zero values in the weight vectors. Hence, storing only non-zero elements can greatly save space. Our experimental results indicate that tree models can achieve up to a 95% reduction in storage space compared to the standard one-vs-rest method for multi-label text classification. Our research provides a simple procedure to estimate the size of a tree model before training any classifier in the tree nodes. Then, if the model size is already acceptable, this approach can help avoid modifying the model through weight pruning or other techniques., Comment: EMNLP 2024

Non-EQuilibrium (NEQ) statistical physics has not had the same depth of rigor and generality of foundational grounding as that of EQuilibrium (EQ) statistical physics, where forces and fluctuational response functions are derived from potentials such as $1/T = \partial S/\partial U$ and where conjugate variables are related through Maxwell's Relations. Here, we show that the dynamical counterparts to the First and Second Laws for equilibria -- maximizing path entropies, not state entropies, for the latter -- give the full set of corresponding relations for NEQ. Network flows parse into three functional independent components: node populations, edge traffic and cycle flux. We also generalize to "cost-benefit" relations beyond just work and heat dissipation.

We establish a super duality as an equivalence between Whittaker module categories over a pair of classical Lie algebra and Lie superalgebra in the infinite-rank limit. Building on this result and utilizing the Losev-Shu-Xiao decomposition, we obtain a super duality which is an equivalence between module categories over a pair of finite $W$-algebras and $W$-superalgebras at the infinite-rank limit., Comment: 35 pages

Adverse weather image restoration aims to remove unwanted degraded artifacts, such as haze, rain, and snow, caused by adverse weather conditions. Existing methods achieve remarkable results for addressing single-weather conditions. However, they face challenges when encountering unpredictable weather conditions, which often happen in real-world scenarios. Although different weather conditions exhibit different degradation patterns, they share common characteristics that are highly related and complementary, such as occlusions caused by degradation patterns, color distortion, and contrast attenuation due to the scattering of atmospheric particles. Therefore, we focus on leveraging common knowledge across multiple weather conditions to restore images in a unified manner. In this paper, we propose a Triplet Attention Network (TANet) to efficiently and effectively address all-in-one adverse weather image restoration. TANet consists of Triplet Attention Block (TAB) that incorporates three types of attention mechanisms: Local Pixel-wise Attention (LPA) and Global Strip-wise Attention (GSA) to address occlusions caused by non-uniform degradation patterns, and Global Distribution Attention (GDA) to address color distortion and contrast attenuation caused by atmospheric phenomena. By leveraging common knowledge shared across different weather conditions, TANet successfully addresses multiple weather conditions in a unified manner. Experimental results show that TANet efficiently and effectively achieves state-of-the-art performance in all-in-one adverse weather image restoration. The source code is available at https://github.com/xhuachris/TANet-ACCV-2024., Comment: 17 pages (ACCV 2024)

This paper explores the problem of commonsense-level vision-knowledge conflict in Multimodal Large Language Models (MLLMs), where visual information contradicts model's internal commonsense knowledge (see Figure 1). To study this issue, we introduce an automated pipeline, augmented with human-in-the-loop quality control, to establish a benchmark aimed at simulating and assessing the conflicts in MLLMs. Utilizing this pipeline, we have crafted a diagnostic benchmark comprising 374 original images and 1,122 high-quality question-answer (QA) pairs. This benchmark covers two types of conflict target and three question difficulty levels, providing a thorough assessment tool. Through this benchmark, we evaluate the conflict-resolution capabilities of nine representative MLLMs across various model families and find a noticeable over-reliance on textual queries. Drawing on these findings, we propose a novel prompting strategy, "Focus-on-Vision" (FoV), which markedly enhances MLLMs' ability to favor visual data over conflicting textual knowledge. Our detailed analysis and the newly proposed strategy significantly advance the understanding and mitigating of vision-knowledge conflicts in MLLMs. The data and code are made publicly available.

To address the challenge of short-term object pose tracking in dynamic environments with monocular RGB input, we introduce a large-scale synthetic dataset OmniPose6D, crafted to mirror the diversity of real-world conditions. We additionally present a benchmarking framework for a comprehensive comparison of pose tracking algorithms. We propose a pipeline featuring an uncertainty-aware keypoint refinement network, employing probabilistic modeling to refine pose estimation. Comparative evaluations demonstrate that our approach achieves performance superior to existing baselines on real datasets, underscoring the effectiveness of our synthetic dataset and refinement technique in enhancing tracking precision in dynamic contexts. Our contributions set a new precedent for the development and assessment of object pose tracking methodologies in complex scenes., Comment: 13 pages, 9 figures

Text-based image generation models, such as Stable Diffusion and DALL-E 3, hold significant potential in content creation and publishing workflows, making them the focus in recent years. Despite their remarkable capability to generate diverse and vivid images, considerable efforts are being made to prevent the generation of harmful content, such as abusive, violent, or pornographic material. To assess the safety of existing models, we introduce a novel jailbreaking method called Chain-of-Jailbreak (CoJ) attack, which compromises image generation models through a step-by-step editing process. Specifically, for malicious queries that cannot bypass the safeguards with a single prompt, we intentionally decompose the query into multiple sub-queries. The image generation models are then prompted to generate and iteratively edit images based on these sub-queries. To evaluate the effectiveness of our CoJ attack method, we constructed a comprehensive dataset, CoJ-Bench, encompassing nine safety scenarios, three types of editing operations, and three editing elements. Experiments on four widely-used image generation services provided by GPT-4V, GPT-4o, Gemini 1.5 and Gemini 1.5 Pro, demonstrate that our CoJ attack method can successfully bypass the safeguards of models for over 60% cases, which significantly outperforms other jailbreaking methods (i.e., 14%). Further, to enhance these models' safety against our CoJ attack method, we also propose an effective prompting-based method, Think Twice Prompting, that can successfully defend over 95% of CoJ attack. We release our dataset and code to facilitate the AI safety research.

Animal vocalization denoising is a task similar to human speech enhancement, a well-studied field of research. In contrast to the latter, it is applied to a higher diversity of sound production mechanisms and recording environments, and this higher diversity is a challenge for existing models. Adding to the challenge and in contrast to speech, we lack large and diverse datasets comprising clean vocalizations. As a solution we use as training data pseudo-clean targets, i.e. pre-denoised vocalizations, and segments of background noise without a vocalization. We propose a train set derived from bioacoustics datasets and repositories representing diverse species, acoustic environments, geographic regions. Additionally, we introduce a non-overlapping benchmark set comprising clean vocalizations from different taxa and noise samples. We show that that denoising models (demucs, CleanUNet) trained on pseudo-clean targets obtained with speech enhancement models achieve competitive results on the benchmarking set. We publish data, code, libraries, and demos https://mariusmiron.com/research/biodenoising., Comment: 5 pages, 2 tables

We present a comprehensive theoretical investigation of the strain-modulated excitonic properties of uni-axially strained transition-metal dichalcogenide monolayers (TMD-MLs) by solving the Bethe-Salpeter equation (BSE) established on the basis of first principles. We show that imposing an uni-axial strain onto a MoS_$2$ monolayers leads to the diversified band dispersions of the bright exciton (BX), gray exciton (GX), and dark exciton (DX) states, as a consequence of the competitive interplay between strain-induced valley drift (VD) and momentum-dependent electron-hole exchange interaction (EHEI). While the band dispersions of BX doublet in the light-accessible small reciprocal area remain almost unchanged against strain, the band dispersion of DX is reshaped by an increasing uni-axial strain from a parabola to a Mexican-hat-like profile, featured with unusual sign-reversal of the heavy effective mass and strain-activated brightness. In contrast, the effective mass of GX is drastically lightened by uni-axial strain and remains always positive. We show that the strain-diversified exciton band dispersions leads to the distinct exciton diffusivities and angle-resolved optical patterns of BX, GX, and DX in a strained TMD-ML, suggesting the feasibility of {\it spatially} resolving spinallowed and -forbidden excitons in exciton transport experiments and angle-resolved optical spectroscopies., Comment: 18pages, 5 figures

Moving mirrors as analogue sources of Hawking radiation from black holes have been explored extensively, less so with cosmological particle creation (CPC), even though the analogy between dynamical Casimir effect (DCE) and CPC based on the mechanism of parametric amplification of quantum field fluctuations has also been known for a long time. This `perspective' essay intends to convey some of the rigor and thoroughness of quantum field theory in curved spacetime, which serves as the theoretical foundation of CPC, to DCE, which enjoys a variety of active experimental explorations. We have selected out seven issues of relevance to address, starting from the naively simple ones, e.g., why should one be bothered with `curved' spacetime when performing a laboratory experiment in ostensibly flat space, to foundational theoretical ones, such as the frequent appearance of nonlocal dissipation in the system dynamics induced by colored noises in its field environment, the existence of quantum Lenz law and fluctuation-dissipation relations in the backreaction effects of DCE emission on the moving atom/mirror or the source, and the construction of a microphysics model to account for the dynamical responses of a mirror or medium. The strengthening of theoretical ground for DCE is useful not only for improving conceptual clarity but needed for the development of proof of concept type of future experimental designs for DCE. Results from DCE experiments in turn will enrich our understanding of quantum field effects in the early universe because they are, in the spirit of analogue gravity, our best hopes for the verification of these fundamental processes., Comment: 48 pages

Children often suffer wrist trauma in daily life, while they usually need radiologists to analyze and interpret X-ray images before surgical treatment by surgeons. The development of deep learning has enabled neural networks to serve as computer-assisted diagnosis (CAD) tools to help doctors and experts in medical image diagnostics. Since YOLOv8 model has obtained the satisfactory success in object detection tasks, it has been applied to various fracture detection. This work introduces four variants of Feature Contexts Excitation-YOLOv8 (FCE-YOLOv8) model, each incorporating a different FCE module (i.e., modules of Squeeze-and-Excitation (SE), Global Context (GC), Gather-Excite (GE), and Gaussian Context Transformer (GCT)) to enhance the model performance. Experimental results on GRAZPEDWRI-DX dataset demonstrate that our proposed YOLOv8+GC-M3 model improves the mAP@50 value from 65.78% to 66.32%, outperforming the state-of-the-art (SOTA) model while reducing inference time. Furthermore, our proposed YOLOv8+SE-M3 model achieves the highest mAP@50 value of 67.07%, exceeding the SOTA performance. The implementation of this work is available at https://github.com/RuiyangJu/FCE-YOLOv8., Comment: arXiv admin note: text overlap with arXiv:2407.03163

We consider an autonomous navigation robot that can accept human commands through natural language to provide services in an indoor environment. These natural language commands may include time, position, object, and action components. However, we observe that the positional components within such commands usually refer to objects in the environment that may contain different levels of positional ambiguity. For example, the command "Go to the chair!" may be ambiguous when there are multiple chairs of the same type in a room. In order to disambiguate these commands, we employ a large language model and a large vision-language model to conduct multiple turns of conversations with the user. We propose a two-level approach that utilizes a vision-language model to map the meanings in natural language to a unique object ID in images and then performs another mapping from the unique object ID to a 3D depth map, thereby allowing the robot to navigate from its current position to the target position. To the best of our knowledge, this is the first work linking foundation models to the positional ambiguity issue.

Goal-oriented planning, or anticipating a series of actions that transition an agent from its current state to a predefined objective, is crucial for developing intelligent assistants aiding users in daily procedural tasks. The problem presents significant challenges due to the need for comprehensive knowledge of temporal and hierarchical task structures, as well as strong capabilities in reasoning and planning. To achieve this, prior work typically relies on extensive training on the target dataset, which often results in significant dataset bias and a lack of generalization to unseen tasks. In this work, we introduce VidAssist, an integrated framework designed for zero/few-shot goal-oriented planning in instructional videos. VidAssist leverages large language models (LLMs) as both the knowledge base and the assessment tool for generating and evaluating action plans, thus overcoming the challenges of acquiring procedural knowledge from small-scale, low-diversity datasets. Moreover, VidAssist employs a breadth-first search algorithm for optimal plan generation, in which a composite of value functions designed for goal-oriented planning is utilized to assess the predicted actions at each step. Extensive experiments demonstrate that VidAssist offers a unified framework for different goal-oriented planning setups, e.g., visual planning for assistance (VPA) and procedural planning (PP), and achieves remarkable performance in zero-shot and few-shot setups. Specifically, our few-shot model outperforms the prior fully supervised state-of-the-art method by +7.7% in VPA and +4.81% PP task on the COIN dataset while predicting 4 future actions. Code, and models are publicly available at https://sites.google.com/view/vidassist., Comment: Accepted by ECCV 2024 (Oral)

Wrist trauma and even fractures occur frequently in daily life, particularly among children who account for a significant proportion of fracture cases. Before performing surgery, surgeons often request patients to undergo X-ray imaging first, and prepare for the surgery based on the analysis of the X-ray images. With the development of neural networks, You Only Look Once (YOLO) series models have been widely used in fracture detection for Computer-Assisted Diagnosis, where the YOLOv8 model has obtained the satisfactory results. Applying the attention modules to neural networks is one of the effective methods to improve the model performance. This paper proposes YOLOv8-ResCBAM, which incorporates Convolutional Block Attention Module integrated with resblock (ResCBAM) into the original YOLOv8 network architecture. The experimental results on the GRAZPEDWRI-DX dataset demonstrate that the mean Average Precision calculated at Intersection over Union threshold of 0.5 (mAP 50) of the proposed model increased from 63.6% of the original YOLOv8 model to 65.8%, which achieves the state-of-the-art performance. The implementation code is available at https://github.com/RuiyangJu/Fracture_Detection_Improved_YOLOv8., Comment: Accepted by ICONIP 2024. arXiv admin note: substantial text overlap with arXiv:2402.09329

Today's most advanced multimodal models remain proprietary. The strongest open-weight models rely heavily on synthetic data from proprietary VLMs to achieve good performance, effectively distilling these closed models into open ones. As a result, the community is still missing foundational knowledge about how to build performant VLMs from scratch. We present Molmo, a new family of VLMs that are state-of-the-art in their class of openness. Our key innovation is a novel, highly detailed image caption dataset collected entirely from human annotators using speech-based descriptions. To enable a wide array of user interactions, we also introduce a diverse dataset mixture for fine-tuning that includes in-the-wild Q&A and innovative 2D pointing data. The success of our approach relies on careful choices for the model architecture details, a well-tuned training pipeline, and, most critically, the quality of our newly collected datasets, all of which will be released. The best-in-class 72B model within the Molmo family not only outperforms others in the class of open weight and data models but also compares favorably against proprietary systems like GPT-4o, Claude 3.5, and Gemini 1.5 on both academic benchmarks and human evaluation. We will be releasing all of our model weights, captioning and fine-tuning data, and source code in the near future. Select model weights, inference code, and demo are available at https://molmo.allenai.org.

Magnetic reconnection changes the magnetic field topology and facilitates the energy and particle exchange at magnetospheric boundaries such as the Earth's magnetopause. The flow shear perpendicular to the reconnecting plane prevails at the flank magnetopause under southward interplanetary magnetic field (IMF) conditions. However, the effect of the out-of-plane flow shear on asymmetric reconnection is an open question. In this study, we utilize kinetic simulations to investigate the impact of the out-of-plane flow shear on asymmetric reconnection. By systematically varying the flow shear strength, we analyze the flow shear effects on the reconnection rate, the diffusion region structure, and the energy conversion rate. We find that the reconnection rate increases with the upstream out-of-plane flow shear, and for the same upstream conditions, it is higher at the dusk side than at the dawn side. The diffusion region is squeezed in the outflow direction due to magnetic pressure which is proportional to the square of the Alfv\'en Mach number of the shear flow. The out-of-plane flow shear increases the energy conversion rate J \cdot E', and for the same upstream conditions, the magnitude of J \cdot E' is larger at the dusk side than at the dawn side. This study reveals that out-of-plane flow shear not only enhances the reconnection rate but also significantly boosts energy conversion, with more pronounced effects on the dusk-side flank than on the dawn-side flank. These insights pave the way for better understanding the solar wind-magnetosphere interactions., Comment: 20 pages, 8 figures

This paper focuses on the stability of both local and global error bounds for a proper lower semicontinuous convex function defined on a Banach space. Without relying on any dual space information, we first provide precise estimates of error bound moduli using directional derivatives. For a given proper lower semicontinuous convex function on a Banach space, we prove that the stability of local error bounds under small perturbations is equivalent to the directional derivative at a reference point having a non-zero minimum over the unit sphere. Additionally, the stability of global error bounds is shown to be equivalent to the infimum of the directional derivatives, at all points on the boundary of the solution set, being bounded away from zero over some neighborhood of the unit sphere., Comment: arXiv admin note: substantial text overlap with arXiv:2302.02279

Feeding the growing human population sustainably amidst climate change is one of the most important challenges in the 21st century. Current practices often lead to the overuse of agronomic inputs, such as synthetic fertilizers and water, resulting in environmental contamination and diminishing returns on crop productivity. The complexity of agricultural systems, involving plant-environment interactions and human management, presents significant scientific and technical challenges for developing sustainable practices. Addressing these challenges necessitates transdisciplinary research, involving intense collaboration among fields such as plant science, engineering, computer science, and social sciences. Here, we present five case studies from two research centers demonstrating successful transdisciplinary approaches toward more sustainable water and fertilizer use. These case studies span multiple scales. Starting from whole-plant signaling, we explore how reporter plants can transform our understanding of plant communication and enable efficient application of water and fertilizers. We then show how new fertilizer technologies could increase the availability of phosphorus in the soil. To accelerate advancements in breeding new cultivars, we discuss robotic technologies for high-throughput plant screening in different environments at a population scale. At the ecosystem scale, we investigate phosphorus recovery from aquatic systems and methods to minimize phosphorus leaching. Finally, as agricultural outputs affect all people, we show how to integrate stakeholder perspectives and needs into the research. With these case studies, we hope to encourage the scientific community to adopt transdisciplinary research and promote cross-training among biologists, engineers, and social scientists to drive discovery and innovation in advancing sustainable agricultural systems.

Audio-visual pre-trained models have gained substantial attention recently and demonstrated superior performance on various audio-visual tasks. This study investigates whether pre-trained audio-visual models demonstrate non-arbitrary associations between sounds and visual representations$\unicode{x2013}$known as sound symbolism$\unicode{x2013}$which is also observed in humans. We developed a specialized dataset with synthesized images and audio samples and assessed these models using a non-parametric approach in a zero-shot setting. Our findings reveal a significant correlation between the models' outputs and established patterns of sound symbolism, particularly in models trained on speech data. These results suggest that such models can capture sound-meaning connections akin to human language processing, providing insights into both cognitive architectures and machine learning strategies., Comment: SLT 2024

We present a Morphology-Informed Heterogeneous Graph Neural Network (MI-HGNN) for learning-based contact perception. The architecture and connectivity of the MI-HGNN are constructed from the robot morphology, in which nodes and edges are robot joints and links, respectively. By incorporating the morphology-informed constraints into a neural network, we improve a learning-based approach using model-based knowledge. We apply the proposed MI-HGNN to two contact perception problems, and conduct extensive experiments using both real-world and simulated data collected using two quadruped robots. Our experiments demonstrate the superiority of our method in terms of effectiveness, generalization ability, model efficiency, and sample efficiency. Our MI-HGNN improved the performance of a state-of-the-art model that leverages robot morphological symmetry by 8.4% with only 0.21% of its parameters. Although MI-HGNN is applied to contact perception problems for legged robots in this work, it can be seamlessly applied to other types of multi-body dynamical systems and has the potential to improve other robot learning frameworks. Our code is made publicly available at https://github.com/lunarlab-gatech/Morphology-Informed-HGNN., Comment: 6 pages, 5 figures; This work has been submitted to the IEEE for possible publication

Clinical diagnosis of stuttering requires an assessment by a licensed speech-language pathologist. However, this process is time-consuming and requires clinicians with training and experience in stuttering and fluency disorders. Unfortunately, only a small percentage of speech-language pathologists report being comfortable working with individuals who stutter, which is inadequate to accommodate for the 80 million individuals who stutter worldwide. Developing machine learning models for detecting stuttered speech would enable universal and automated screening for stuttering, enabling speech pathologists to identify and follow up with patients who are most likely to be diagnosed with a stuttering speech disorder. Previous research in this area has predominantly focused on utterance-level detection, which is not sufficient for clinical settings where word-level annotation of stuttering is the norm. In this study, we curated a stuttered speech dataset with word-level annotations and introduced a word-level stuttering speech detection model leveraging self-supervised speech models. Our evaluation demonstrates that our model surpasses previous approaches in word-level stuttering speech detection. Additionally, we conducted an extensive ablation analysis of our method, providing insight into the most important aspects of adapting self-supervised speech models for stuttered speech detection., Comment: Accepted by IEEE SLT 2024

Large Vision-Language-Action (VLA) models, leveraging powerful pre trained Vision-Language Models (VLMs) backends, have shown promise in robotic control due to their impressive generalization ability. However, the success comes at a cost. Their reliance on VLM backends with billions of parameters leads to high computational costs and inference latency, limiting the testing scenarios to mainly quasi-static tasks and hindering performance in dynamic tasks requiring rapid interactions. To address these limitations, this paper proposes HiRT, a Hierarchical Robot Transformer framework that enables flexible frequency and performance trade-off. HiRT keeps VLMs running at low frequencies to capture temporarily invariant features while enabling real-time interaction through a high-frequency vision-based policy guided by the slowly updated features. Experiment results in both simulation and real-world settings demonstrate significant improvements over baseline methods. Empirically, in static tasks, we double the control frequency and achieve comparable success rates. Additionally, on novel real-world dynamic ma nipulation tasks which are challenging for previous VLA models, HiRT improves the success rate from 48% to 75%.

Multicomponent phase separation is a routine occurrence in both living and synthetic systems. Thermodynamics provides a straightforward path to determine the phase boundaries that characterize these transitions for systems at equilibrium. The prevalence of phase separation in complex systems outside the confines of equilibrium motivates the need for a genuinely nonequilibrium theory of multicomponent phase coexistence. Here, we develop a mechanical theory for coexistence that casts coexistence criteria into the familiar form of equality of state functions. Our theory generalizes traditional equilibrium notions such as the species chemical potential and thermodynamic pressure to systems out of equilibrium. Crucially, while these notions may not be identifiable for all nonequilibrium systems, we numerically verify their existence for a variety of systems by introducing the phenomenological Multicomponent Active Model B+. Our work establishes an initial framework for understanding multicomponent coexistence that we hope can serve as the basis for a comprehensive theory for high-dimensional nonequilibrium phase transitions.

Organ transplantation serves as the primary therapeutic strategy for end-stage organ failures. However, allograft rejection is a common complication of organ transplantation. Histological assessment is essential for the timely detection and diagnosis of transplant rejection and remains the gold standard. Nevertheless, the traditional histochemical staining process is time-consuming, costly, and labor-intensive. Here, we present a panel of virtual staining neural networks for lung and heart transplant biopsies, which digitally convert autofluorescence microscopic images of label-free tissue sections into their brightfield histologically stained counterparts, bypassing the traditional histochemical staining process. Specifically, we virtually generated Hematoxylin and Eosin (H&E), Masson's Trichrome (MT), and Elastic Verhoeff-Van Gieson (EVG) stains for label-free transplant lung tissue, along with H&E and MT stains for label-free transplant heart tissue. Subsequent blind evaluations conducted by three board-certified pathologists have confirmed that the virtual staining networks consistently produce high-quality histology images with high color uniformity, closely resembling their well-stained histochemical counterparts across various tissue features. The use of virtually stained images for the evaluation of transplant biopsies achieved comparable diagnostic outcomes to those obtained via traditional histochemical staining, with a concordance rate of 82.4% for lung samples and 91.7% for heart samples. Moreover, virtual staining models create multiple stains from the same autofluorescence input, eliminating structural mismatches observed between adjacent sections stained in the traditional workflow, while also saving tissue, expert time, and staining costs., Comment: 21 Pages, 5 Figures

Open quantum systems are susceptible to losses in information, energy, and particles due to their surrounding environment. One novel strategy to mitigate these losses is to transform them into advantages for quantum technologies through tailored non-Hermitian quantum systems. In this work, we theoretically propose a fast generation of multipartite entanglement in non-Hermitian qubits. Our findings reveal that weakly coupled non-Hermitian qubits can accelerate multiparty entanglement generation by thousands of times compared to Hermitian qubits, in particular when approaching the $2^n$-th order exceptional points of $n$ qubits in the ${\cal P}{\cal T}-$ symmetric regime. Furthermore, we show that Hermitian qubits can generate GHZ states with a high fidelity more than $0.9995$ in a timescale comparable to that of non-Hermitian qubits, but at the expense of intense driving and large coupling constant. Our approach is scalable to a large number of qubits, presenting a promising pathway for advancing quantum technologies through the non-Hermiticity and higher-order exceptional points in many-body quantum systems., Comment: 5 figures