Search

We introduce ReXTime, a benchmark designed to rigorously test AI models' ability to perform temporal reasoning within video events. Specifically, ReXTime focuses on reasoning across time, i.e. human-like understanding when the question and its corresponding answer occur in different video segments. This form of reasoning, requiring advanced understanding of cause-and-effect relationships across video segments, poses significant challenges to even the frontier multimodal large language models. To facilitate this evaluation, we develop an automated pipeline for generating temporal reasoning question-answer pairs, significantly reducing the need for labor-intensive manual annotations. Our benchmark includes 921 carefully vetted validation samples and 2,143 test samples, each manually curated for accuracy and relevance. Evaluation results show that while frontier large language models outperform academic models, they still lag behind human performance by a significant 14.3% accuracy gap. Additionally, our pipeline creates a training dataset of 9,695 machine generated samples without manual effort, which empirical studies suggest can enhance the across-time reasoning via fine-tuning., Comment: Project page: https://rextime.github.io/

Undergraduate students of physics in Tibet have great difficulty learning quantum mechanics (QM). We attempt to use PER-based methods to help Tibetan students learn QM. In this preliminary study, we incorporate Mathematica in a QM course at Tibet University and record students' learning experiences. Tibetan students tend to have subjective feelings of learning Mathematica, whereas Han students (majority) are more focused on the operational techniques of Mathematica. The results also suggest that both Tibetan students and Han students show limited improvement in time-independent Schrodinger equations after learning QM with Mathematica. Further effort is needed to improve the academic literacy skills of physics students in Tibet.

Traditional diagnosis of chronic diseases involves in-person consultations with physicians to identify the disease. However, there is a lack of research focused on predicting and developing application systems using clinical notes and blood test values. We collected five years of Electronic Health Records (EHRs) from Taiwan's hospital database between 2017 and 2021 as an AI database. Furthermore, we developed an EHR-based chronic disease prediction platform utilizing Large Language Multimodal Models (LLMMs), successfully integrating with frontend web and mobile applications for prediction. This prediction platform can also connect to the hospital's backend database, providing physicians with real-time risk assessment diagnostics. The demonstration link can be found at https://www.youtube.com/watch?v=oqmL9DEDFgA.

Large Multimodal Models (LMMs) excel at comprehending human instructions and demonstrate remarkable results across a broad spectrum of tasks. Reinforcement Learning from Human Feedback (RLHF) and AI Feedback (RLAIF) further refine LLMs by aligning them with specific preferences. These methods primarily use ranking-based feedback for entire generations. With advanced AI models (Teacher), such as GPT-4 and Claude 3 Opus, we can request various types of detailed feedback that are expensive for humans to provide. We propose a two-stage algorithm ARES that Alternates REinforcement Learning (RL) and Supervised Fine-Tuning (SFT). First, we request the Teacher to score how much each sentence contributes to solving the problem in a Chain-of-Thought (CoT). This sentence-level feedback allows us to consider individual valuable segments, providing more granular rewards for the RL procedure. Second, we ask the Teacher to correct the wrong reasoning after the RL stage. The RL procedure requires massive efforts for hyperparameter tuning and often generates errors like repetitive words and incomplete sentences. With the correction feedback, we stabilize the RL fine-tuned model through SFT. We conduct experiments on multi-model dataset ScienceQA and A-OKVQA to demonstrate the effectiveness of our proposal. ARES rationale reasoning achieves around 70% win rate against baseline models judged by GPT-4o. Additionally, we observe that the improved rationale reasoning leads to a 2.5% increase in inference answer accuracy on average for the multi-modal datasets.

The application of the Internet in the field of education is becoming more and more popular, and a large amount of educational data is generated in the process. How to effectively use these data has always been a key issue in the field of educational data mining. In this work, a machine learning model based on Long Short-Term Memory Network (LSTM) was used to conduct an in-depth analysis of educational big data to evaluate student performance. The LSTM model efficiently processes time series data, allowing us to capture time-dependent and long-term trends in students' learning activities. This approach is particularly useful for analyzing student progress, engagement, and other behavioral patterns to support personalized education. In an experimental analysis, we verified the effectiveness of the deep learning method in predicting student performance by comparing the performance of different models. Strict cross-validation techniques are used to ensure the accuracy and generalization of experimental results., Comment: Education Big Data, Performance Analysis, Machine Learning, Long Short-Term Memory Network

Language models (LMs) exhibit impressive performance and generalization capabilities. However, LMs struggle with the persistent challenge of catastrophic forgetting, which undermines their long-term sustainability in continual learning (CL). Existing approaches usually address the issue by incorporating old task data or task-wise inductive bias into LMs. However, old data and accurate task information are often unavailable or costly to collect, hindering the availability of current CL approaches for LMs. To address this limitation, we introduce $\textbf{MIGU}$ ($\textbf{M}$agn$\textbf{I}$tude-based $\textbf{G}$radient $\textbf{U}$pdating for continual learning), a rehearsal-free and task-label-free method that only updates the model parameters with large magnitudes of output in LMs' linear layers. MIGU is based on our observation that the L1-normalized magnitude distribution of the output in LMs' linear layers is different when the LM models deal with different task data. By imposing this simple constraint on the gradient update process, we can leverage the inherent behaviors of LMs, thereby unlocking their innate CL abilities. Our experiments demonstrate that MIGU is universally applicable to all three LM architectures (T5, RoBERTa, and Llama2), delivering state-of-the-art or on-par performance across continual finetuning and continual pre-training settings on four CL benchmarks. For example, MIGU brings a 15.2% average accuracy improvement over conventional parameter-efficient finetuning baselines in a 15-task CL benchmark. MIGU can also seamlessly integrate with all three existing CL types to further enhance performance. Code is available at \href{https://github.com/wenyudu/MIGU}{this https URL}., Comment: preprint, 19 pages

Mixed-anion compounds, which incorporate multiple types of anions into materials, displays tailored crystal structures and physical/chemical properties, garnering immense interests in various applications such as batteries, catalysis, photovoltaics, and thermoelectrics. However, detailed studies regarding correlations between crystal structure, chemical bonding, and thermal/vibrational properties are rare for these compounds, which limits the exploration of mixed-anion compounds for associated thermal applications. In this work, we investigate the lattice dynamics and thermal transport properties of the metal chalcohalides, CuBiSCl2. A high-purity polycrystalline CuBiSCl2 sample, successfully synthesized via modified solid-state synthetic method, exhibits a low lattice thermal conductivity of 0.9-0.6 W m-1 K-1 from 300 to 573 K. By combining various experimental techniques including 3D electron diffraction with theoretical calculations, we elucidate the origin of low lattice thermal conductivity in CuBiSCl2. The stereo-chemical activity of the 6s2 lone pair of Bi3+ favors an asymmetric environment with neighboring anions involving both short and long bond lengths. This particularity often implies weak bonding, low structure dimensionality, and strong anharmonicity, leading to low lattice thermal conductivity. In addition, the strong two-fold linear S-Cu-S coordination with weak Cu -- Cl interactions induces large anisotropic vibration of Cu or structural disorder, which enables strong phonon-phonon scattering and decreases lattice thermal conductivity. The investigations into lattice dynamics and thermal transport properties of CuBiSCl2 broadens the scope of the existing mixed-anion compounds suitable for the associated thermal applications, offering a new avenue for the search of low thermal conductivity materials in low-cost mixed-anion compounds.

Motivated by settings such as medical treatments or aircraft maintenance, we consider a scheduling problem with jobs that consist of two operations, a test and a processing part. The time required to execute the test is known in advance while the time required to execute the processing part becomes known only upon completion of the test. We use competitive analysis to study algorithms for minimizing the sum of completion times for $n$ given jobs on a single machine. As our main result, we prove using a novel analysis technique that the natural $1$-SORT algorithm has competitive ratio at most 1.861. For the special case of uniform test times, we show that a simple threshold-based algorithm has competitive ratio at most 1.585. We also prove a lower bound that shows that no deterministic algorithm can be better than $\sqrt{2}$-competitive even in the case of uniform test times.

Recent advances in Large Language Models (LLMs) have shown inspiring achievements in constructing autonomous agents that rely on language descriptions as inputs. However, it remains unclear how well LLMs can function as few-shot or zero-shot embodied agents in dynamic interactive environments. To address this gap, we introduce LangSuitE, a versatile and simulation-free testbed featuring 6 representative embodied tasks in textual embodied worlds. Compared with previous LLM-based testbeds, LangSuitE (i) offers adaptability to diverse environments without multiple simulation engines, (ii) evaluates agents' capacity to develop ``internalized world knowledge'' with embodied observations, and (iii) allows easy customization of communication and action strategies. To address the embodiment challenge, we devise a novel chain-of-thought (CoT) schema, EmMem, which summarizes embodied states w.r.t. history information. Comprehensive benchmark results illustrate challenges and insights of embodied planning. LangSuitE represents a significant step toward building embodied generalists in the context of language models.

Gravitational echoes can be used to probe the structure of spacetime. In this paper, we investigate the gravitational echoes in different braneworld models in five-dimensional spacetime. We derive the gravitational perturbation equations of these models, and obtain the time-dependent evolution equations of the extra-dimensional and radial components. Using a Gaussian wave packet as initial data, we study the time evolution of the gravitational perturbation. By monitoring the evolution of the Gaussian wave packet, the gravitational echoes are observed whether the wave packet is generated from inside or outside the braneworld. Furthermore, we can restrict the parameters of the braneworld by calculating the strength of the first gravitational echo and using the current gravitational wave data., Comment: 29 pages, 19 figures, 1 table

Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input. This phenomenon has been known as the lost-in-the-middle problem. In this work, we make three contributions. First, we set out to understand the factors that cause this phenomenon. In doing so, we establish a connection between lost-in-the-middle to LLMs' intrinsic attention bias: LLMs exhibit a U-shaped attention bias where the tokens at the beginning and at the end of its input receive higher attention, regardless of their relevance. Second, we mitigate this positional bias through a calibration mechanism, found-in-the-middle, that allows the model to attend to contexts faithfully according to their relevance, even though when they are in the middle. Third, we show found-in-the-middle not only achieves better performance in locating relevant information within a long context, but also eventually leads to improved retrieval-augmented generation (RAG) performance across various tasks, outperforming existing methods by up to 15 percentage points. These findings open up future directions in understanding LLM attention bias and its potential consequences., Comment: ACL Findings 2024

The Keck Planet Imager and Characterizer (KPIC), a series of upgrades to the Keck II Adaptive Optics System and Instrument Suite, aims to demonstrate high-resolution spectroscopy of faint exoplanets that are spatially resolved from their host stars. In this paper, we measure KPIC's sensitivity to companions as a function of separation (i.e., the contrast curve) using on-sky data collected over four years of operation. We show that KPIC is able to reach contrasts of $1.3 \times 10^{-4}$ at 90 mas and $9.2 \times 10^{-6}$ at 420 mas separation from the star, and that KPIC can reach planet-level sensitivities at angular separations within the inner working angle of coronagraphic instruments such as GPI and SPHERE. KPIC is also able to achieve more extreme contrasts than other medium-/high-resolution spectrographs that are not as optimized for high-contrast performance. We decompose the KPIC performance budget into individual noise terms and discuss limiting factors. The fringing that results from combining a high-contrast imaging system with a high-resolution spectrograph is identified as an important source of systematic noise. After mitigation and correction, KPIC is able to reach within a factor of 2 of the photon noise limit at separations < 200 mas. At large separations, KPIC is limited by the background noise performance of NIRSPEC., Comment: 16 pages, 6 figures, submitted to the proceedings of SPIE Astronomical Telescopes + Instrumentation 2024, 13096-69

A comprehensive guidance view for cardiac interventional surgery can be provided by the real-time fusion of the intraoperative 2D images and preoperative 3D volume based on the ultrasound frame-to-volume registration. However, cardiac ultrasound images are characterized by a low signal-to-noise ratio and small differences between adjacent frames, coupled with significant dimension variations between 2D frames and 3D volumes to be registered, resulting in real-time and accurate cardiac ultrasound frame-to-volume registration being a very challenging task. This paper introduces a lightweight end-to-end Cardiac Ultrasound frame-to-volume Registration network, termed CU-Reg. Specifically, the proposed model leverages epicardium prompt-guided anatomical clues to reinforce the interaction of 2D sparse and 3D dense features, followed by a voxel-wise local-global aggregation of enhanced features, thereby boosting the cross-dimensional matching effectiveness of low-quality ultrasound modalities. We further embed an inter-frame discriminative regularization term within the hybrid supervised learning to increase the distinction between adjacent slices in the same ultrasound volume to ensure registration stability. Experimental results on the reprocessed CAMUS dataset demonstrate that our CU-Reg surpasses existing methods in terms of registration accuracy and efficiency, meeting the guidance requirements of clinical cardiac interventional surgery., Comment: This paper has been accepted by MICCAI 2024

Mobile robots operating in crowded environments require the ability to navigate among humans and surrounding obstacles efficiently while adhering to safety standards and socially compliant mannerisms. This scale of the robot navigation problem may be classified as both a local path planning and trajectory optimization problem. This work presents an array of force sensors that act as a tactile layer to complement the use of a LiDAR for the purpose of inducing awareness of contact with any surrounding objects within immediate vicinity of a mobile robot undetected by LiDARs. By incorporating the tactile layer, the robot can take more risks in its movements and possibly go right up to an obstacle or wall, and gently squeeze past it. In addition, we built up a simulation platform via Pybullet which integrates Robot Operating System (ROS) and reinforcement learning (RL) together. A touch-aware neural network model was trained on it to create an RL-based local path planner for dynamic obstacle avoidance. Our proposed method was demonstrated successfully on an omni-directional mobile robot who was able to navigate in a crowded environment with high agility and versatility in movement, while not being overly sensitive to nearby obstacles-not-in-contact.

We demonstrate sub-recoil Sisyphus cooling using the long-lived $^{3}\mathrm{P}_{0}$ clock state in alkaline-earth-like ytterbium. A 1388 nm optical standing wave nearly resonant with the $^{3}\textrm{P}_{0}$$\,\rightarrow$$\,^{3}\textrm{D}_{1}$ transition creates a spatially periodic light shift of the $^{3}\textrm{P}_{0}$ clock state. Following excitation on the ultranarrow clock transition, we observe Sisyphus cooling in this potential, as the light shift is correlated with excitation to $^{3}\textrm{D}_{1}$ and subsequent spontaneous decay to the $^{1}\textrm{S}_{0}$ ground state. We observe that cooling enhances the loading efficiency of atoms into a 759 nm magic-wavelength one-dimensional (1D) optical lattice, as compared to standard Doppler cooling on the $^{1}\textrm{S}_{0}$$\,\rightarrow\,$$^{3}\textrm{P}_{1}$ transition. Sisyphus cooling yields temperatures below 200 nK in the weakly confined, transverse dimensions of the 1D optical lattice. These lower temperatures improve optical lattice clocks by facilitating the use of shallow lattices with reduced light shifts, while retaining large atom numbers to reduce the quantum projection noise. This Sisyphus cooling can be pulsed or continuous and is applicable to a range of quantum metrology applications., Comment: 8 pages, 6 figures

Coronal mass ejections (CMEs) and their driven shocks are a major source of large geomagnetic storms due to their large and long-lasting, southward component of magnetic field in the sheath and the flux rope (e.g., magnetic cloud). Predicting the strength and arrival time of southward fields accurately thus plays a key role in space weather predictions. To address this problem, we have developed a new model, which combines the global three-dimensional, time-dependent, magnetohydrodynamic (MHD), data-driven model (G3DMHD) and a self-contained magnetic flux-rope model [1]. As a demonstration and validation, here we simulate the evolution of a Sun-Earth-directed CME that erupted on 2012-July-12. The computational domain spans from 2.5 solar radii (Rs) from the surface of the Sun, where the flux rope is injected, to 245 Rs. We compare the time profiles of the simulated MHD parameters (Density, velocity, temperature, and magnetic field) with in situ solar wind observations acquired at ~1 AU by the Wind spacecraft and the result is encouraging. The model successfully reproduces the shock, sheath, and flux rope similar to those observed by Wind., Comment: in press, Phys Astron Int J. 2024; 8(I):1-3.DOI:10.15406/paij.2024.08.00324

Ensembling multiple models has always been an effective approach to push the limits of existing performance and is widely used in classification tasks by simply averaging the classification probability vectors from multiple classifiers to achieve better accuracy. However, in the thriving open-source Large Language Model (LLM) community, ensembling methods are rare and typically limited to ensembling the full-text outputs of LLMs, such as selecting the best output using a ranker, which leads to underutilization of token-level probability information. In this paper, we treat the Generation of each token by LLMs as a Classification (GaC) for ensembling. This approach fully exploits the probability information at each generation step and better prevents LLMs from producing early incorrect tokens that lead to snowballing errors. In experiments, we ensemble state-of-the-art LLMs on several benchmarks, including exams, mathematics and reasoning, and observe that our method breaks the existing community performance ceiling. Furthermore, we observed that most of the tokens in the answer are simple and do not affect the correctness of the final answer. Therefore, we also experimented with ensembling only key tokens, and the results showed better performance with lower latency across benchmarks.

As a fundamental structural feature, the symmetry of materials determines the exotic quantum properties in transition metal dichalcogenides (TMDs) with charge density wave (CDW). Breaking the inversion symmetry, the Janus structure, an artificially constructed lattice, provides an opportunity to tune the CDW states and the related properties. However, limited by the difficulties in atomic-level fabrication and material stability, the experimental visualization of the CDW states in 2D TMDs with Janus structure is still rare. Here, using surface selenization of VTe2, we fabricated monolayer Janus VTeSe. With scanning tunneling microscopy, an unusual root13-root13 CDW state with threefold rotational symmetry breaking was observed and characterized. Combined with theoretical calculations, we find this CDW state can be attributed to the charge modulation in the Janus VTeSe, beyond the conventional electron-phonon coupling. Our findings provide a promising platform for studying the CDW states and artificially tuning the electronic properties toward the applications.

We present the first X-ray spectropolarimetric results for Cygnus X-1 in its soft state from a campaign of five IXPE observations conducted during 2023 May-June. Companion multiwavelength data during the campaign are likewise shown. The 2-8 keV X-rays exhibit a net polarization degree PD=1.99%+/-0.13% (68% confidence). The polarization signal is found to increase with energy across IXPE's 2-8 keV bandpass. The polarized X-rays exhibit an energy-independent polarization angle of PA=-25.7+/-1.8 deg. East of North (68% confidence). This is consistent with being aligned to Cyg X-1's AU-scale compact radio jet and its pc-scale radio lobes. In comparison to earlier hard-state observations, the soft state exhibits a factor of 2 lower polarization degree, but a similar trend with energy and a similar (also energy-independent) position angle. When scaling by the natural unit of the disk temperature, we find the appearance of a consistent trendline in the polarization degree between soft and hard states. Our favored polarimetric model indicates Cyg X-1's spin is likely high (a* above ~0.96). The substantial X-ray polarization in Cyg X-1's soft state is most readily explained as resulting from a large portion of X-rays emitted from the disk returning and reflecting off the disk surface, generating a high polarization degree and a polarization direction parallel to the black hole spin axis and radio jet. In IXPE's bandpass, the polarization signal is dominated by the returning reflection emission. This constitutes polarimetric evidence for strong gravitational lensing of X-rays close to the black hole., Comment: 20 pages, accepted for publication in ApJL

We present a careful calibration of the exciton fraction of polaritons in high-$Q$ ($\sim 300,000$), long-lifetime ($\sim 300$ ps), GaAs/AlGaAs microcavities.This is a crucial parameter for many-body theories which include the polariton-polariton interactions.It is much harder to establish this number in high-$Q$ structures compared to low-$Q$ structures, because the upper polariton is nearly invisible in high-$Q$ cavities.We present a combination of photoluminescence, photoluminescence excitation, and reflectivity measurements to highly constrain the fit model, and compare the results of this model to the results from low-$Q$ structures.We present a fitted curve of exciton fraction as a function of the lower polariton energy for multiple samples which have been used in prior experiments., Comment: 16 pages, 11 figures

We propose using mechanistic interpretability -- techniques for reverse engineering model weights into human-interpretable algorithms -- to derive and compactly prove formal guarantees on model performance. We prototype this approach by formally proving lower bounds on the accuracy of 151 small transformers trained on a Max-of-$K$ task. We create 102 different computer-assisted proof strategies and assess their length and tightness of bound on each of our models. Using quantitative metrics, we find that shorter proofs seem to require and provide more mechanistic understanding. Moreover, we find that more faithful mechanistic understanding leads to tighter performance bounds. We confirm these connections by qualitatively examining a subset of our proofs. Finally, we identify compounding structureless noise as a key challenge for using mechanistic interpretability to generate compact proofs on model performance., Comment: accepted to ICML 2024 Workshop on Mechanistic Interpretability (Spotlight)

We construct a four-dimensional version of the equation of state (EoS) model NEOS, NEOS-4D, as a function of the temperature and chemical potentials of baryon, electric charge, and strangeness for the hot and dense QCD matter created in relativistic nuclear collisions. This EoS enables multiple conserved charge current evolution in a relativistic fluid. Input from Lattice QCD simulations and a hadron resonance gas model is considered for constructing the equation of state. We investigate its applicability to the relativistic hydrodynamic description of nuclear collisions and present a method for efficient numerical implementation., Comment: 11 pages, 9 figures

Iron nanoparticles are among the most promising low-dimensional materials in terms of applications. This particularity is attributable to the magnetic properties of these nanoparticles, which exhibit different allotropes as a function of temperature. In this work, we sought to characterise at the atomic scale how their structural and magnetic transformations can be affected by the size. To achieve this objective, we developed a tight-binding model incorporating a magnetic contribution via a Stoner term implemented in a Monte Carlo code to relax the structure and the magnetic state. Using our approach, we show that magnetism is strongly reinforced by the surface, which leads to increase the Curie temperature as the size of the particle decreases contrary to the solid-solid transition temperature. Our work thus provides a deep understanding at the atomic scale of the key factors that determines the structural and magnetic properties of Fe nanoparticles, shedding more light on their unique character which is crucial for further applications.

Large Language Models (LLMs) have demonstrated effectiveness not only in language tasks but also in video reasoning. This paper introduces a novel dataset, Tropes in Movies (TiM), designed as a testbed for exploring two critical yet previously overlooked video reasoning skills: (1) Abstract Perception: understanding and tokenizing abstract concepts in videos, and (2) Long-range Compositional Reasoning: planning and integrating intermediate reasoning steps for understanding long-range videos with numerous frames. Utilizing tropes from movie storytelling, TiM evaluates the reasoning capabilities of state-of-the-art LLM-based approaches. Our experiments show that current methods, including Captioner-Reasoner, Large Multimodal Model Instruction Fine-tuning, and Visual Programming, only marginally outperform a random baseline when tackling the challenges of Abstract Perception and Long-range Compositional Reasoning. To address these deficiencies, we propose Face-Enhanced Viper of Role Interactions (FEVoRI) and Context Query Reduction (ConQueR), which enhance Visual Programming by fostering role interaction awareness and progressively refining movie contexts and trope queries during reasoning processes, significantly improving performance by 15 F1 points. However, this performance still lags behind human levels (40 vs. 65 F1). Additionally, we introduce a new protocol to evaluate the necessity of Abstract Perception and Long-range Compositional Reasoning for task resolution. This is done by analyzing the code generated through Visual Programming using an Abstract Syntax Tree (AST), thereby confirming the increased complexity of TiM. The dataset and code are available at: https://ander1119.github.io/TiM, Comment: Project page: https://ander1119.github.io/TiM

The low-energy collective excitations of semiconductors and insulators often couple strongly to light, allowing them to be probed optically. We argue here that in fractional Chern insulators intra-band collective excitations are dark in the sense that they couple anomalously weakly to light. This conclusion is based on a relationship between ideal quantum geometry and the structure factor of a Chern band, and on a classical plasma analogy motivated by the vortexibility property of ideal Chern bands., Comment: Main text: 5 pages, 3 figures, Supmat: 5 pages, 1 figure; Comments are welcome

We simulate quench dynamics of the Su-Schrieffer-Heeger (SSH) chain on the IBM quantum computers, calculating the R\'enyi entanglement entropy, the twist order parameter and the Berry phase. The latter two quantities can be deduced from a slow-twist operator defined in the Lieb-Schultz-Mattis theorem. The R\'enyi entropy is obtained using a recently developed randomized measurement scheme. The twist order parameter and the Berry phase are measured without the need for additional gates or ancilla qubits. We consider quench protocols in which a trivial initial state evolves dynamically in time under the topological SSH Hamiltonian in the fully dimerized limit (the flat-band limit). During these quenches, there are persistent and periodic oscillations in the time evolution of both entanglement entropy and twist order parameter. Through the implementation of error mitigation techniques using a global depolarizing ansatz and postselection, our simulations on the IBM devices yield results that closely match exact solutions., Comment: 9 pages, 11 figures

Wavefront shaping can tailor multipath interference to control multiple scattering of waves in complex optical systems. However, full-wave simulations that capture multiple scattering are computationally demanding given the large system size and the large number of input channels. Recently, an "augmented partial factorization" (APF) method was proposed to significantly speed-up such full-wave simulations. In this tutorial, we illustrate how to perform wavefront shaping simulations with the APF method using the open-source frequency-domain electromagnetic scattering solver MESTI. We present the foundational concepts and then walk through four examples: computing the scattering matrix of a slab with random permittivities, open high-transmission channels through disorder, focusing inside disorder with phase conjugation, and reflection matrix computation in a spatial focused-beam basis. The goal is to lower the barrier for researchers to use simulations to explore the rich phenomena enabled by wavefront shaping.

Composed Image Retrieval (CIR) aims to retrieve a target image based on a reference image and conditioning text, enabling controllable searches. Due to the expensive dataset construction cost for CIR triplets, a zero-shot (ZS) CIR setting has been actively studied to eliminate the need for human-collected triplet datasets. The mainstream of ZS-CIR employs an efficient projection module that projects a CLIP image embedding to the CLIP text token embedding space, while fixing the CLIP encoders. Using the projected image embedding, these methods generate image-text composed features by using the pre-trained text encoder. However, their CLIP image and text encoders suffer from the task discrepancy between the pre-training task (text $\leftrightarrow$ image) and the target CIR task (image + text $\leftrightarrow$ image). Conceptually, we need expensive triplet samples to reduce the discrepancy, but we use cheap text triplets instead and update the text encoder. To that end, we introduce the Reducing Task Discrepancy of text encoders for Composed Image Retrieval (RTD), a plug-and-play training scheme for the text encoder that enhances its capability using a novel target-anchored text contrastive learning. We also propose two additional techniques to improve the proposed learning scheme: a hard negatives-based refined batch sampling strategy and a sophisticated concatenation scheme. Integrating RTD into the state-of-the-art projection-based ZS-CIR methods significantly improves performance across various datasets and backbones, demonstrating its efficiency and generalizability., Comment: 17 pages

To mitigate the suboptimal nature of graph structure, Graph Structure Learning (GSL) has emerged as a promising approach to improve graph structure and boost performance in downstream tasks. Despite the proposal of numerous GSL methods, the progresses in this field mostly concentrated on node-level tasks, while graph-level tasks (e.g., graph classification) remain largely unexplored. Notably, applying node-level GSL to graph classification is non-trivial due to the lack of find-grained guidance for intricate structure learning. Inspired by the vital role of subgraph in graph classification, in this paper we explore the potential of subgraph structure learning for graph classification by tackling the challenges of key subgraph selection and structure optimization. We propose a novel Motif-driven Subgraph Structure Learning method for Graph Classification (MOSGSL). Specifically, MOSGSL incorporates a subgraph structure learning module which can adaptively select important subgraphs. A motif-driven structure guidance module is further introduced to capture key subgraph-level structural patterns (motifs) and facilitate personalized structure learning. Extensive experiments demonstrate a significant and consistent improvement over baselines, as well as its flexibility and generalizability for various backbones and learning procedures., Comment: 16 pages, 8 figures

The interrelationship between a material's structure and its properties lies at the heart of materials-related research. Finding how the changes of one affect the other is of primary importance in theoretical and computational materials studies. In this work, based on Hershfield nonequilibrium quantum statistics and the mean-field approach with steady-state density functional theory, we derive a first-principles method to calculate nonequilibrium effects induced forces acting on atoms, enabling structure optimizations and molecular dynamics simulations for molecular junctions under external biases. By applying the method to a few molecular devices, we found that in general, the external bias can induce profound nonequilibrium effects on both electronic/transport properties and the geometric structure of these devices, and consequent changes in electronic properties and geometric structure are closely interrelated. Particularly, when the bias voltage is above 1.0 V, significant structure phase changes could occur, causing dramatic changes in I-V characteristics and vibrational spectra. These findings greatly broaden our understanding of quantum electronic devices and provide a new avenue for discovering novel transport phenomena at molecular scale., Comment: 17 pages, 11 figures

Brain tumor segmentation remains a significant challenge, particularly in the context of multi-modal magnetic resonance imaging (MRI) where missing modality images are common in clinical settings, leading to reduced segmentation accuracy. To address this issue, we propose a novel strategy, which is called masked predicted pre-training, enabling robust feature learning from incomplete modality data. Additionally, in the fine-tuning phase, we utilize a knowledge distillation technique to align features between complete and missing modality data, simultaneously enhancing model robustness. Notably, we leverage the Holder pseudo-divergence instead of the KLD for distillation loss, offering improve mathematical interpretability and properties. Extensive experiments on the BRATS2018 and BRATS2020 datasets demonstrate significant performance enhancements compared to existing state-of-the-art methods.

Representations from pre-trained speech foundation models (SFMs) have shown impressive performance in many downstream tasks. However, the potential benefits of incorporating pre-trained SFM representations into speaker voice similarity assessment have not been thoroughly investigated. In this paper, we propose SVSNet+, a model that integrates pre-trained SFM representations to improve performance in assessing speaker voice similarity. Experimental results on the Voice Conversion Challenge 2018 and 2020 datasets show that SVSNet+ incorporating WavLM representations shows significant improvements compared to baseline models. In addition, while fine-tuning WavLM with a small dataset of the downstream task does not improve performance, using the same dataset to learn a weighted-sum representation of WavLM can substantially improve performance. Furthermore, when WavLM is replaced by other SFMs, SVSNet+ still outperforms the baseline models and exhibits strong generalization ability., Comment: Accepted to INTERSPEECH 2024

Large audio-language models (LALMs) enhance traditional large language models by integrating audio perception capabilities, allowing them to tackle audio-related tasks. Previous research has primarily focused on assessing the performance of LALMs across various tasks, yet overlooking their reliability, particularly concerning issues like object hallucination. In our study, we introduce methods to assess the extent of object hallucination of publicly available LALMs. Our findings reveal that LALMs are comparable to specialized audio captioning models in their understanding of audio content, but struggle to answer discriminative questions, specifically those requiring the identification of the presence of particular object sounds within an audio clip. This limitation highlights a critical weakness in current LALMs: their inadequate understanding of discriminative queries. Moreover, we explore the potential of prompt engineering to enhance LALMs' performance on discriminative questions., Comment: Accepted to Interspeech 2024

Despite the recent successes of multi-agent reinforcement learning (MARL) algorithms, efficiently adapting to co-players in mixed-motive environments remains a significant challenge. One feasible approach is to hierarchically model co-players' behavior based on inferring their characteristics. However, these methods often encounter difficulties in efficient reasoning and utilization of inferred information. To address these issues, we propose Hierarchical Opponent modeling and Planning (HOP), a novel multi-agent decision-making algorithm that enables few-shot adaptation to unseen policies in mixed-motive environments. HOP is hierarchically composed of two modules: an opponent modeling module that infers others' goals and learns corresponding goal-conditioned policies, and a planning module that employs Monte Carlo Tree Search (MCTS) to identify the best response. Our approach improves efficiency by updating beliefs about others' goals both across and within episodes and by using information from the opponent modeling module to guide planning. Experimental results demonstrate that in mixed-motive environments, HOP exhibits superior few-shot adaptation capabilities when interacting with various unseen agents, and excels in self-play scenarios. Furthermore, the emergence of social intelligence during our experiments underscores the potential of our approach in complex multi-agent environments., Comment: Accepted at ICML 2024

We study the mean field limit of a rank-based model with common noise, which arises as an extension to models for the market capitalization of firms in stochastic portfolio theory. We show that, under certain conditions on the drift and diffusion coefficients, the empirical cumulative distribution function converges to the solution of a stochastic PDE. A key step in the proof, which is of independent interest, is to show that any solution to an associated martingale problem is also a pathwise entropy solution to the stochastic PDE, a notion introduced in a recent series of papers [32, 33, 19, 16, 17].

A striking prediction from the random matrix theory in mesoscopic physics is the existence of "open channels": waves that can use multipath interference to achieve perfect transmission across an opaque disordered medium even in the multiple-scattering regime. Realization of such open channels requires a coherent control of the complete incident wavefront. To date, the open channels have only been demonstrated in scalar two-dimensional (2D) structures, both experimentally and with numerical studies. Here, we utilize a recently proposed "augmented partial factorization" full-wave simulation method to compute the scattering matrix from 3D vectorial Maxwell's equations and demonstrate the existence of open channels in 3D disordered media. We examine the spatial profile of such open channels, demonstrate the existence of a bimodal transmission eigenvalue distribution with full control, and study the effects of incomplete polarization control and of a finite illumination area. This study confirms the validity of the random matrix theory in vectorial systems. The simulation framework provides full access to the complex multi-channel wave transport in 3D disordered systems, filling the gap left by experimental capabilities.

We show how to efficiently sample the joint posterior of two non-covariant experiments with a large set of nuisance parameters. Specifically, we train an ensemble of normalizing flows to learn the the posterior distribution of both experiments. Once trained, we can use the flows to draw $\mathcal{O} (10^9)$ samples from the joint posterior of two independent cosmological measurements in seconds -- saving up to $\mathcal{O}(1)$ ton of $\text{CO}_2$ per Monte Carlo run. Using this new technique we find joint constraints between the Dark Energy Survey $3 \times 2$ point measurement, South Pole Telescope and Planck CMB lensing and a BOSS direct fit full shape analyses, for the first time. We find $\Omega_{\rm m} = 0.32^{+0.01}_{-0.01}$ and $S_8 = 0.79 ^ {+0.01}_ {-0.01}$. We release a public package called {\tt CombineHarvesterFlow} (https://github.com/pltaylor16/CombineHarvesterFlow) which performs these calculations., Comment: 10 pages, 6 figures. To be submitted to PRD

Navigating through a physical environment to reach a desired location involves a complex interplay of cognitive, sensory, and motor functions. When navigating with others, experiencing a degree of behavioral and cognitive synchronization is both natural and ubiquitous. This synchronization facilitates a harmonious effort toward achieving a common goal, reflecting how individuals instinctively align their actions and thoughts in collaborative settings. Collaborative spatial tasks, which are crucial in daily and professional settings, require coordinated navigation and problem-solving skills. This study explores the neural mechanisms underlying such tasks by using hyperscanning electroencephalography (EEG) technology to examine brain dynamics in dyadic route planning within a virtual reality setting. By analyzing intra- and inter-brain couplings across delta, theta, alpha, beta, and gamma EEG bands using both functional and effective connectivity measures, we identified significant neural synchronization patterns associated with collaborative task performance in both leaders and followers. Functional intra-brain connectivity analyses revealed distinct neural engagement across EEG frequency bands, with increased delta couplings observed in both leaders and followers. Theta connectivity was particularly enhanced in followers, whereas the alpha band exhibited divergent patterns that indicate role-specific neural strategies. Inter-brain analysis revealed increased delta causality between interacting members but decreased theta and gamma couplings from followers to leaders. Additionally, inter-brain analysis indicated decreased couplings in faster-performing dyads, especially in theta bands. These insights enhance our understanding of the neural mechanisms driving collaborative spatial navigation and demonstrate the effectiveness of hyperscanning in studying complex brain-to-brain interactions.

Volumetric rendering based methods, like NeRF, excel in HDR view synthesis from RAWimages, especially for nighttime scenes. While, they suffer from long training times and cannot perform real-time rendering due to dense sampling requirements. The advent of 3D Gaussian Splatting (3DGS) enables real-time rendering and faster training. However, implementing RAW image-based view synthesis directly using 3DGS is challenging due to its inherent drawbacks: 1) in nighttime scenes, extremely low SNR leads to poor structure-from-motion (SfM) estimation in distant views; 2) the limited representation capacity of spherical harmonics (SH) function is unsuitable for RAW linear color space; and 3) inaccurate scene structure hampers downstream tasks such as refocusing. To address these issues, we propose LE3D (Lighting Every darkness with 3DGS). Our method proposes Cone Scatter Initialization to enrich the estimation of SfM, and replaces SH with a Color MLP to represent the RAW linear color space. Additionally, we introduce depth distortion and near-far regularizations to improve the accuracy of scene structure for downstream tasks. These designs enable LE3D to perform real-time novel view synthesis, HDR rendering, refocusing, and tone-mapping changes. Compared to previous volumetric rendering based methods, LE3D reduces training time to 1% and improves rendering speed by up to 4,000 times for 2K resolution images in terms of FPS. Code and viewer can be found in https://github.com/Srameo/LE3D .

Quantum computational fluid dynamics (QCFD) offers a promising alternative to classical computational fluid dynamics (CFD) by leveraging quantum algorithms for higher efficiency. This paper introduces a comprehensive QCFD method, including an iterative method "Iterative-QLS" that suppresses error in quantum linear solver, and a subspace method to scale the solution to a larger size. We implement our method on a superconducting quantum computer, demonstrating successful simulations of steady Poiseuille flow and unsteady acoustic wave propagation. The Poiseuille flow simulation achieved a relative error of less than $0.2\%$, and the unsteady acoustic wave simulation solved a 5043-dimensional matrix. We emphasize the utilization of the quantum-classical hybrid approach in applications of near-term quantum computers. By adapting to quantum hardware constraints and offering scalable solutions for large-scale CFD problems, our method paves the way for practical applications of near-term quantum computers in computational science., Comment: 31 pages, 10 figures

In the extended thermodynamics of black holes, there exists a thermodynamical pressure and its conjugate volume. Extensive studies have been conducted on the phase structure of numerous black holes, which have demonstrated striking similarities to the phase structure of various ordinary matter systems. A comparison of the thermodynamic properties of spherically symmetric AdS black holes with those of ordinary thermodynamic systems reveals that the isovolumetric heat capacity of the former is zero, whereas that of the latter is non-zero. It is a subject of interest for the intrinsic reason for this discrepancy. The effective thermodynamic quantities, based on the context of the boundary between the black hole horizon and the cosmological horizon in dS spacetime, as well as the interaction between the two horizons, are presented. The heat capacity in the Reissner-N\"{o}rdstrom-de Sitter (RN-dS) spacetime is then investigated, and it is demonstrated that the behavior of the heat capacity in the RN-dS spacetime is analogous to that of Schottky specific heat. Treating the two horizon interfaces in the RN-dS spacetime as two distinct energy levels in a two-energy system, and the thermodynamic properties in the RN-dS spacetime are discussed with the method of studying the thermodynamic properties in an ordinary two-energy system, to elucidate the intrinsic reasons for the occurrence of Schottky specific heat in the RN-dS spacetime. The heat capacity observed in the RN-dS spacetime is not only consistent with that of the Schottky specific heat described by the effective thermodynamic quantities in the RN-dS spacetime, but also with that of the heat capacity in an ordinary two-energy-level system. The quantum properties in the RN-dS spacetime are reflected, thereby providing a new avenue for further in-depth study of the quantum properties of black holes and dS spacetime.

Generative text-to-image models enable us to synthesize unlimited amounts of images in a controllable manner, spurring many recent efforts to train vision models with synthetic data. However, every synthetic image ultimately originates from the upstream data used to train the generator. What additional value does the intermediate generator provide over directly training on relevant parts of the upstream data? Grounding this question in the setting of image classification,a we compare finetuning on task-relevant, targeted synthetic data generated by Stable Diffusion -- a generative model trained on the LAION-2B dataset -- against finetuning on targeted real images retrieved directly from LAION-2B. We show that while synthetic data can benefit some downstream tasks, it is universally matched or outperformed by real data from our simple retrieval baseline. Our analysis suggests that this underperformance is partially due to generator artifacts and inaccurate task-relevant visual details in the synthetic images. Overall, we argue that retrieval is a critical baseline to consider when training with synthetic data -- a baseline that current methods do not yet surpass. We release code, data, and models at https://github.com/scottgeng00/unmet-promise., Comment: Correspondence to sgeng at cs dot washington dot edu. RK and PWK equally advised the project

The rapid growth of Speech Emotion Recognition (SER) has diverse global applications, from improving human-computer interactions to aiding mental health diagnostics. However, SER models might contain social bias toward gender, leading to unfair outcomes. This study analyzes gender bias in SER models trained with Self-Supervised Learning (SSL) at scale, exploring factors influencing it. SSL-based SER models are chosen for their cutting-edge performance. Our research pioneering research gender bias in SER from both upstream model and data perspectives. Our findings reveal that females exhibit slightly higher overall SER performance than males. Modified CPC and XLS-R, two well-known SSL models, notably exhibit significant bias. Moreover, models trained with Mandarin datasets display a pronounced bias toward valence. Lastly, we find that gender-wise emotion distribution differences in training data significantly affect gender bias, while upstream model representation has a limited impact., Comment: Accepted by INTERSPEECH 2024

Traditional reinforcement learning control for quadruped robots often relies on privileged information, demanding meticulous selection and precise estimation, thereby imposing constraints on the development process. This work proposes a Self-learning Latent Representation (SLR) method, which achieves high-performance control policy learning without the need for privileged information. To enhance the credibility of our proposed method's evaluation, SLR is compared with open-source code repositories of state-of-the-art algorithms, retaining the original authors' configuration parameters. Across four repositories, SLR consistently outperforms the reference results. Ultimately, the trained policy and encoder empower the quadruped robot to navigate steps, climb stairs, ascend rocks, and traverse various challenging terrains. Robot experiment videos are at https://11chens.github.io/SLR/

Real-space imaging of three-dimensional atomic structures is a critical yet challenging task in materials science. Although scanning transmission electron microscopy has achieved sub-angstrom lateral resolution through techniques like electron ptychography1,2, depth resolution remains limited to only 2 to 3 nanometers with a single projection setup3,4. Attaining better depth resolution typically necessitates large sample tilt angles and many projections, as seen in atomic electron tomography5,6. Here, we develop a new algorithm based on multislice electron ptychography which couples only a few projections at small tilt angles, but is sufficient to improve the depth resolution by more than threefold to the sub-nanometer scale, and potentially to the atomic level. This technique maintains high resolving power for both light and heavy atoms, and significantly improves the visibility of single dopants. We are thus able to experimentally detect dilute substitutional praseodymium dopants in a brownmillerite oxide, Ca2Co2O5, in three dimensions and observe the accompanying lattice distortion. This technique requires only a moderate level of data acquisition or processing, and can be seamlessly integrated into electron microscopes equipped with conventional components., Comment: 27 pages, 5 figures, 10 supplementary figures

With the development of large-scale Language Models (LLM), fine-tuning pre-trained LLM has become a mainstream paradigm for solving downstream tasks of natural language processing. However, training a language model in the legal field requires a large number of legal documents so that the language model can learn legal terminology and the particularity of the format of legal documents. The typical NLP approaches usually rely on many manually annotated data sets for training. However, in the legal field application, it is difficult to obtain a large number of manually annotated data sets, which restricts the typical method applied to the task of drafting legal documents. The experimental results of this paper show that not only can we leverage a large number of annotation-free legal documents without Chinese word segmentation to fine-tune a large-scale language model, but more importantly, it can fine-tune a pre-trained LLM on the local computer to achieve the generating legal document drafts task, and at the same time achieve the protection of information privacy and to improve information security issues., Comment: 12th International Conference on Software Engineering & Trends (SE 2024), April 27 ~ 28, 2024, Copenhagen, Denmark Volume Editors : David C. Wyld, Dhinaharan Nagamalai (Eds) ISBN : 978-1-923107-24-3

We show that a pseudo-Nambu-Goldstone boson (pNGB) with an initial misalignment angle can drive successful spontaneous baryogenesis, and become a good dark matter candidate if the corresponding global symmetry is non-restored at high temperatures. Considering a dimension-five explicit breaking operator, we find that the pNGB starts its motion with a sliding across rapidly decreasing potential barriers during which the baryon asymmetry is generated and frozen, and later it oscillates as dark matter. It is predicted that the pNGB mass and decay constant are around $5\,{\rm eV}$ and $3\times10^6\,{\rm GeV}$, respectively, while the radial mode has a light mass $O(10)\,{\rm MeV}$ and a small mixing $O(10^{-4})$ with the Higgs boson. Applied to the Majoron in the type-I seesaw model, the heaviest right-handed neutrino is required to be as light as $100\,{\rm GeV}$. These predictions can be tested at kaon experiments, heavy neutral lepton searches, LHC, and future colliders., Comment: 5 pages, 2 figures with supplemental material, v2: discussion on the isocurvature perturbation constraint and references added

Decoding images from non-invasive electroencephalographic (EEG) signals has been a grand challenge in understanding how the human brain process visual information in real-world scenarios. To cope with the issues of signal-to-noise ratio and nonstationarity, this paper introduces a MUltimodal Similarity-keeping contrastivE learning (MUSE) framework for zero-shot EEG-based image classification. We develop a series of multivariate time-series encoders tailored for EEG signals and assess the efficacy of regularized contrastive EEG-Image pretraining using an extensive visual EEG dataset. Our method achieves state-of-the-art performance, with a top-1 accuracy of 19.3% and a top-5 accuracy of 48.8% in 200-way zero-shot image classification. Furthermore, we visualize neural patterns via model interpretation, shedding light on the visual processing dynamics in the human brain. The code repository for this work is available at: https://github.com/ChiShengChen/MUSE_EEG., Comment: 19 pages, 14 figures

With the booming of Large Language Models (LLMs), prompt-learning has become a promising method mainly researched in various research areas. Recently, many attempts based on prompt-learning have been made to improve the performance of text classification. However, most of these methods are based on heuristic Chain-of-Thought (CoT), and tend to be more complex but less efficient. In this paper, we rethink the LLM-based text classification methodology, propose a simple and effective transfer learning strategy, namely LLMEmbed, to address this classical but challenging task. To illustrate, we first study how to properly extract and fuse the text embeddings via various lightweight LLMs at different network depths to improve their robustness and discrimination, then adapt such embeddings to train the classifier. We perform extensive experiments on publicly available datasets, and the results show that LLMEmbed achieves strong performance while enjoys low training overhead using lightweight LLM backbones compared to recent methods based on larger LLMs, i.e. GPT-3, and sophisticated prompt-based strategies. Our LLMEmbed achieves adequate accuracy on publicly available benchmarks without any fine-tuning while merely use 4% model parameters, 1.8% electricity consumption and 1.5% runtime compared to its counterparts. Code is available at: https://github.com/ChunLiu-cs/LLMEmbed-ACL2024., Comment: ACL 2024 main conference

In contextual optimization, a decision-maker observes historical samples of uncertain variables and associated concurrent covariates, without knowing their joint distribution. Given an additional covariate observation, the goal is to choose a decision that minimizes some operational costs. A prevalent issue here is covariate shift, where the marginal distribution of the new covariate differs from historical samples, leading to decision performance variations with nonparametric or parametric estimators. To address this, we propose a distributionally robust approach that uses an ambiguity set by the intersection of two Wasserstein balls, each centered on typical nonparametric or parametric distribution estimators. Computationally, we establish the tractable reformulation of this distributionally robust optimization problem. Statistically, we provide guarantees for our Wasserstein ball intersection approach under covariate shift by analyzing the measure concentration of the estimators. Furthermore, to reduce computational complexity, we employ a surrogate objective that maintains similar generalization guarantees. Through synthetic and empirical case studies on income prediction and portfolio optimization, we demonstrate the strong empirical performance of our proposed models.