Search

Diffractive deep neural network (D2NN), known for its high speed, low power consumption, and strong parallelism, has been widely applied across various fields, including pattern recognition, image processing, and image transmission. However, existing network architectures primarily focus on data representation within the original domain, with limited exploration of the latent space, thereby restricting the information mining capabilities and multifunctional integration of D2NNs. Here, we propose an all-optical autoencoder (OAE) framework that can encode the input wavefield into a prior shape distribution in the latent space and decode the encoded pattern back to the original wavefield. By leveraging the non-reciprocal property of D2NN, the OAE models function as encoders in one direction of wave propagation and as decoders in the opposite direction. We further apply the models to three key areas: image denoising, noise-resistant reconfigurable image classification, and image generation. Proof-of-concept experiments have been conducted to validate numerical simulations. Our OAE framework fully exploits the potential of latent space representations, enabling a single set of diffractive processors to simultaneously achieve image reconstruction, representation, and generation. It can be viewed as both a counterpart and an extension of the electronic autoencoder model. This work not only offers fresh insights into the design of optical generative models but also paves the way for developing and applying multifunctional, highly integrated, and general optical intelligent systems., Comment: 21 pages, 7 figure

Super-Resolution Ultrasound (SRUS) imaging through localising and tracking microbubbles, also known as Ultrasound Localisation Microscopy (ULM), has demonstrated significant potential for reconstructing microvasculature and flows with sub-diffraction resolution in clinical diagnostics. However, imaging organs with large tissue movements, such as those caused by respiration, presents substantial challenges. Existing methods often require breath holding to maintain accumulation accuracy, which limits data acquisition time and ULM image saturation. To improve image quality in the presence of large tissue movements, this study introduces an approach integrating high-frame-rate ultrasound with online precise robotic probe control. Tested on a microvasculature phantom with translation motions up to 20 mm, twice the aperture size of the matrix array used, our method achieved real-time tracking of the moving phantom and imaging volume rate at 85 Hz, keeping majority of the target volume in the imaging field of view. ULM images of the moving cross channels in the phantom were successfully reconstructed in post-processing, demonstrating the feasibility of super-resolution imaging under large tissue motions. This represents a significant step towards ULM imaging of organs with large motion.

Precision breast cancer (BC) risk assessment is crucial for developing individualized screening and prevention. Despite the promising potential of recent mammogram (MG) based deep learning models in predicting BC risk, they mostly overlook the 'time-to-future-event' ordering among patients and exhibit limited explorations into how they track history changes in breast tissue, thereby limiting their clinical application. In this work, we propose a novel method, named OA-BreaCR, to precisely model the ordinal relationship of the time to and between BC events while incorporating longitudinal breast tissue changes in a more explainable manner. We validate our method on public EMBED and inhouse datasets, comparing with existing BC risk prediction and time prediction methods. Our ordinal learning method OA-BreaCR outperforms existing methods in both BC risk and time-to-future-event prediction tasks. Additionally, ordinal heatmap visualizations show the model's attention over time. Our findings underscore the importance of interpretable and precise risk assessment for enhancing BC screening and prevention efforts. The code will be accessible to the public.

With the increasing integration of inverter-based resources into the power grid, there has been a notable reduction in system inertia, potentially compromising frequency stability. To assess the suitability of existing area inertia estimation techniques for real-world power systems, this paper presents a rigorous comparative analysis of system identification, measurement reconstruction, and electromechanical oscillation-based area inertia estimation methodologies, specifically applied to the large-scale and multi-area WECC 240-bus power system. Comprehensive results show that the system identification-based approach exhibits superior robustness and accuracy relative to its counterparts.

We introduce \texttt{cymyc}, a high-performance Python library for numerical investigation of the geometry of a large class of string compactification manifolds and their associated moduli spaces. We develop a well-defined geometric ansatz to numerically model tensor fields of arbitrary degree on a large class of Calabi-Yau manifolds. \texttt{cymyc} includes a machine learning component which incorporates this ansatz to model tensor fields of interest on these spaces by finding an approximate solution to the system of partial differential equations they should satisfy., Comment: 35 pages, 12 figures

Navigating efficiently to an object in an unexplored environment is a critical skill for general-purpose intelligent robots. Recent approaches to this object goal navigation problem have embraced a modular strategy, integrating classical exploration algorithms-notably frontier exploration-with a learned semantic mapping/exploration module. This paper introduces a novel informative path planning and 3D object probability mapping approach. The mapping module computes the probability of the object of interest through semantic segmentation and a Bayes filter. Additionally, it stores probabilities for common objects, which semantically guides the exploration based on common sense priors from a large language model. The planner terminates when the current viewpoint captures enough voxels identified with high confidence as the object of interest. Although our planner follows a zero-shot approach, it achieves state-of-the-art performance as measured by the Success weighted by Path Length (SPL) and Soft SPL in the Habitat ObjectNav Challenge 2023, outperforming other works by more than 20%. Furthermore, we validate its effectiveness on real robots. Project webpage: https://ippon-paper.github.io/

This paper presents a new look at the neural network (NN) robustness problem, from the point of view of graph theory analysis, specifically graph curvature. Graph curvature (e.g., Ricci curvature) has been used to analyze system dynamics and identify bottlenecks in many domains, including road traffic analysis and internet routing. We define the notion of neural Ricci curvature and use it to identify bottleneck NN edges that are heavily used to ``transport data" to the NN outputs. We provide an evaluation on MNIST that illustrates that such edges indeed occur more frequently for inputs where NNs are less robust. These results will serve as the basis for an alternative method of robust training, by minimizing the number of bottleneck edges.

Vision-language models, such as CLIP, have shown impressive generalization capacities when using appropriate text descriptions. While optimizing prompts on downstream labeled data has proven effective in improving performance, these methods entail labor costs for annotations and are limited by their quality. Additionally, since CLIP is pre-trained on highly imbalanced Web-scale data, it suffers from inherent label bias that leads to suboptimal performance. To tackle the above challenges, we propose a label-Free prompt distribution learning and bias correction framework, dubbed as **Frolic**, which boosts zero-shot performance without the need for labeled data. Specifically, our Frolic learns distributions over prompt prototypes to capture diverse visual representations and adaptively fuses these with the original CLIP through confidence matching. This fused model is further enhanced by correcting label bias via a label-free logit adjustment. Notably, our method is not only training-free but also circumvents the necessity for hyper-parameter tuning. Extensive experimental results across 16 datasets demonstrate the efficacy of our approach, particularly outperforming the state-of-the-art by an average of $2.6\%$ on 10 datasets with CLIP ViT-B/16 and achieving an average margin of $1.5\%$ on ImageNet and its five distribution shifts with CLIP ViT-B/16. Codes are available in https://github.com/zhuhsingyuu/Frolic., Comment: NeurIPS 2024 Spotlight

Despite significant advances in deepfake detection, handling varying image quality, especially due to different compressions on online social networks (OSNs), remains challenging. Current methods succeed by leveraging correlations between paired images, whether raw or compressed. However, in open-world scenarios, paired data is scarce, with compressed images readily available but corresponding raw versions difficult to obtain. This imbalance, where unpaired data vastly outnumbers paired data, often leads to reduced detection performance, as existing methods struggle without corresponding raw images. To overcome this issue, we propose a novel approach named the open-world deepfake detection network (ODDN), which comprises two core modules: open-world data aggregation (ODA) and compression-discard gradient correction (CGC). ODA effectively aggregates correlations between compressed and raw samples through both fine-grained and coarse-grained analyses for paired and unpaired data, respectively. CGC incorporates a compression-discard gradient correction to further enhance performance across diverse compression methods in OSN. This technique optimizes the training gradient to ensure the model remains insensitive to compression variations. Extensive experiments conducted on 17 popular deepfake datasets demonstrate the superiority of the ODDN over SOTA baselines., Comment: 9 pages, 4 figures

This work is a continuation of the program initiated in [arXiv:2311.18302]. We show how one can define novel gauge-theoretic (holomorphic) Floer homologies of seven, six, and five-manifolds, from the physics of a topologically-twisted 8d $\mathcal{N}=1$ gauge theory on a Spin$(7)$-manifold via its supersymmetric quantum mechanics interpretation. They are associated with $G_2$ instanton, Donaldson-Thomas, and Haydys-Witten configurations on the seven, six, and five-manifolds, respectively. We also show how one can define hyperk\"ahler Floer homologies specified by hypercontact three-manifolds, and symplectic Floer homologies of instanton moduli spaces. In turn, this will allow us to derive Atiyah-Floer type dualities between the various gauge-theoretic Floer homologies and symplectic intersection Floer homologies of instanton moduli spaces. Via a 2d gauged Landau-Ginzburg model interpretation of the 8d theory, one can derive novel Fukaya-Seidel type $A_\infty$-categories that categorify Donaldson-Thomas, Haydys-Witten, and Vafa-Witten configurations on six, five, and four-manifolds, respectively, where an Atiyah-Floer type correspondence for the Donaldson-Thomas case can be established. Last but not least, topological invariance of the theory suggests a relation amongst these Floer homologies and Fukaya-Seidel type $A_\infty$-categories for certain Spin$(7)$-manifolds. Our work therefore furnishes purely physical proofs and generalizations of the conjectures by Donaldson-Thomas [2], Donaldson-Segal [3], Cherkis [4], Hohloch-Noetzel-Salamon [5], Salamon [6], Haydys [7], and Bousseau [8], and more., Comment: 82 pp

We introduce the Principled Reasoning and Acting (PRAct) framework, a novel method for learning and enforcing action principles from trajectory data. Central to our approach is the use of text gradients from a reflection and optimization engine to derive these action principles. To adapt action principles to specific task requirements, we propose a new optimization framework, Reflective Principle Optimization (RPO). After execution, RPO employs a reflector to critique current action principles and an optimizer to update them accordingly. We develop the RPO framework under two scenarios: Reward-RPO, which uses environmental rewards for reflection, and Self-RPO, which conducts self-reflection without external rewards. Additionally, two RPO methods, RPO-Traj and RPO-Batch, is introduced to adapt to different settings. Experimental results across four environments demonstrate that the PRAct agent, leveraging the RPO framework, effectively learns and applies action principles to enhance performance., Comment: Accepted to SIG CoNLL 2024

In this work, we examine the effectiveness of an uncertainty quantification framework known as Evidential Deep Learning applied in the context of biomedical image segmentation. This class of models involves assigning Dirichlet distributions as priors for segmentation labels, and enables a few distinct definitions of model uncertainties. Using the cardiac and prostate MRI images available in the Medical Segmentation Decathlon for validation, we found that Evidential Deep Learning models with U-Net backbones generally yielded superior correlations between prediction errors and uncertainties relative to the conventional baseline equipped with Shannon entropy measure, Monte-Carlo Dropout and Deep Ensemble methods. We also examined these models' effectiveness in active learning, finding that relative to the standard Shannon entropy-based sampling, they yielded higher point-biserial uncertainty-error correlations while attaining similar performances in Dice-Sorensen coefficients. These superior features of EDL models render them well-suited for segmentation tasks that warrant a critical sensitivity in detecting large model errors., Comment: 15 pages

In this paper, we investigate a class of inviscid generalized surface quasi-geostrophic (SQG) equations on the half-plane with a rigid boundary. Compared to the Biot-Savart law in the vorticity form of the 2D Euler equation, the velocity formula here includes an additional Fourier multiplier operator $m(\Lambda)$. When $m(\Lambda) = \Lambda^\alpha$, where $\Lambda = (-\Delta)^{1/2}$ and $\alpha\in (0,2)$, the equation reduces to the well-known $\alpha$-SQG equation. Finite-time singularity formation for patch solutions to the $\alpha$-SQG equation was famously discovered by Kiselev, Ryzhik, Yao, and Zlato\v{s} [Ann. Math., 184 (2016), pp. 909-948]. We establish finite-time singularity formation for patch solutions to the generalized SQG equations under the Osgood condition \[\int_2^\infty \frac{1}{r (\log r) m(r)} dr < \infty\] along with some additional mild conditions. Notably, our result fills the gap between the globally well-posed 2D Euler equation ($\alpha = 0$) and the $\alpha$-SQG equation ($\alpha > 0$). Furthermore, in line with Elgindi's global regularity results for 2D Loglog-Euler type equations [Arch. Rat. Mech. Anal., 211 (2014), pp. 965-990], our findings suggest that the Osgood condition serves as a sharp threshold that distinguishes global regularity and finite-time singularity in these models. In addition, we generalize the local regularity and finite-time singularity results for patch solutions to the $\alpha$-SQG equation, as established by Gancedo and Patel [Ann. PDE, 7 (2021), no. 1, Art. no. 4], extending them to cases where $m(r)$ behaves like $r^\alpha$ near infinity but does not have an explicit formulation., Comment: 62 pages, 4 figures

We study the following distance realization problem. Given a quasi-metric $D$ on a set $T$ of terminals, does there exist a directed Okamura-Seymour graph that realizes $D$ as the (directed) shortest-path distance metric on $T$? We show that, if we are further given the circular ordering of terminals lying on the boundary, then Monge property is a sufficient and necessary condition. This generalizes previous results for undirected Okamura-Seymour instances. With the circular ordering, we give a greedy algorithm for constructing a directed Okamura-Seymour instance that realizes the input quasi-metric. The algorithm takes the dual perspective concerning flows and routings, and is based on a new way of analyzing graph structures, by viewing graphs as \emph{paths and their intersections}. We believe this new understanding is of independent interest and will prove useful in other problems in graph theory and graph algorithms. We also design an efficient algorithm for finding such a circular ordering that makes $D$ satisfy Monge property, if one exists. Combined with our result above, this gives an efficient algorithm for the distance realization problem.

Stacking van der Waals magnets holds promise for creating new hybrid materials with properties that do not exist in bulk materials. Here we investigate orthogonally twisted stacks of the van der Waals antiferromagnet CrSBr, aiming to exploit an extreme misalignment of magnetic anisotropy across the twisted interface.Using nitrogen-vacancy centre microscopy, we construct vector maps of the magnetisation, and track their evolution under an external field, in a range of twisted compensated and uncompensated configurations differing by the number of layers. We show that twisted stacking consistently modifies the local magnetic switching behaviour of constituent flakes, and that these modifications are spatially non-uniform. In the case of compensated component flakes (even number of layers), we demonstrate that the combination of dipolar coupling and stacking-induced strain can reduce the switching field by over an order of magnitude. Conversely, in uncompensated component flakes (odd number of layers), we observe indications of a non-zero interlayer exchange interaction between twisted flakes during magnetization reversal, which can persistently modify magnetic order. This work highlights the importance of spatial imaging in investigating stacking-induced magnetic effects, particularly in the case of twistronics where spatial variation is expected and can be conflated with structural imperfections., Comment: 6 main text pages, 4 figures plus 8 supplementary information pages, 9 figures

In this paper, we study convex elements in a (twisted) Weyl group introduced by Ivanov and the first named author. We show that each conjugacy class of the twisted Weyl group contains a convex element, and moreover, the Steinberg cross-sections exist for all convex elements. This result strictly enlarges the cases of Steinberg cross-sections from a new perspective, and will play an essential role in the study of higher Deligne-Lusztig representations., Comment: 11 pages

Web development involves turning UI designs into functional webpages, which can be difficult for both beginners and experienced developers due to the complexity of HTML's hierarchical structures and styles. While Large Language Models (LLMs) have shown promise in generating source code, two major challenges persist in UI-to-HTML code generation: (1) effectively representing HTML's hierarchical structure for LLMs, and (2) bridging the gap between the visual nature of UI designs and the text-based format of HTML code. To tackle these challenges, we introduce Waffle, a new fine-tuning strategy that uses a structure-aware attention mechanism to improve LLMs' understanding of HTML's structure and a contrastive fine-tuning approach to align LLMs' understanding of UI images and HTML code. Models fine-tuned with Waffle show up to 9.00 pp (percentage point) higher HTML match, 0.0982 higher CW-SSIM, 32.99 higher CLIP, and 27.12 pp higher LLEM on our new benchmark WebSight-Test and an existing benchmark Design2Code, outperforming current fine-tuning methods.

Metal additive manufacturing (AM) opens the possibility for spatial control of as-fabricated microstructure and properties. However, since the solid state diffusional transformations that drive microstructure outcomes are governed by nonlinear ODEs in terms of temperature, which is itself governed by PDEs over the entire part domain, solving for the system inputs needed to achieve desired microstructure distributions has proven difficult. In this work, we present a trajectory optimization approach for spatial control of microstructure in metal AM, which we demonstrate by controlling the hardness of a low-alloy steel in electron beam powder bed fusion (EB-PBF). To this end, we present models for thermal and microstructural dynamics. Next, we use experimental data to identify the parameters of the microstructure transformation dynamics. We then pose spatial microstructure control as a finite-horizon optimal control problem. The optimal power field trajectory is computed using an augmented Lagrangian differential dynamic programming (AL-DDP) method with GPU acceleration. The resulting time-varying power fields are then realized on an EB-PBF machine through an approximation scheme. Measurements of the resultant hardness shows that the optimized power field trajectory is able to closely produce the desired hardness distribution., Comment: 6 pages, 6 figures

This paper presents the first measurement of $\psi{(2S)}$ and $\chi_{c1}(3872)$ meson production within fully reconstructed jets. Each quarkonium state (tag) is reconstructed via its decay to the $J/\psi$($\rightarrow\mu^+\mu^-$)$\pi^+\pi^-$ final state in the forward region using proton-proton collision data collected by the LHCb experiment at the center-of-mass-energy of $13 \text{TeV}$ in 2016, corresponding to an integrated luminosity of $1.64 \text{fb}^{-1}$. The fragmentation function, presented as the ratio of the quarkonium-tag transverse momentum to the full jet transverse momentum ($p_{\mathrm{T}}(\text{tag})/p_{\mathrm{T}}(\text{jet})$), is measured differentially in $p_{\mathrm{T}}(\text{jet})$ and $p_{\mathrm{T}}(\text{tag})$ bins. The distributions are separated into promptly produced quarkonia from proton-proton collisions and quarkonia produced from displaced $b$-hadron decays. While the displaced quarkonia fragmentation functions are in general well described by parton-shower predictions, the prompt quarkonium distributions differ significantly from fixed-order non-relativistic QCD (NRQCD) predictions followed by a QCD parton shower., Comment: All figures and tables, along with machine-readable versions and any supplementary material and additional information, are available at https://lbfence.cern.ch/alcm/public/analysis/full-details/1618/ (LHCb public pages)

Performative prediction (PP) is a framework that captures distribution shifts that occur during the training of machine learning models due to their deployment. As the trained model is used, its generated data could cause the model to evolve, leading to deviations from the original data distribution. The impact of such model-induced distribution shifts in the federated learning (FL) setup remains unexplored despite being increasingly likely to transpire in real-life use cases. Although Jin et al. (2024) recently extended PP to FL in a straightforward manner, the resulting model only converges to a performative stable point, which may be far from optimal. The methods in Izzo et al. (2021); Miller et al. (2021) can find a performative optimal point in centralized settings, but they require the performative risk to be convex and the training data to be noiseless, assumptions often violated in realistic FL systems. This paper overcomes all of these shortcomings and proposes Performative robust optimal Federated Learning (ProFL), an algorithm that finds performative optimal points in FL from noisy and contaminated data. We present the convergence analysis under the Polyak-Lojasiewicz condition, which applies to non-convex objectives. Extensive experiments on multiple datasets validate our proposed algorithms' efficiency., Comment: 27 pages with Appendix, 18 figures. The paper has been submitted and is currently under review

The discovery of a charge density wave (CDW) in the antiferromagnetic kagome metal FeGe has prompted interest in the interplay between kagome physics, CDW, and magnetism. However, a crucial aspect for understanding these emergent phenomena-the precise CDW structure-remains ambiguous. Recent studies have assumed uniformly distributed Ge dimers, but this assumption is problematic. The predicted band structure based on this model exhibits an abrupt disappearance of a Ge-$p$ band in the Fermi surface, contradicting experimental observations from angle-resolved photoemission spectroscopy (ARPES). In this study, we propose that a CDW phase with disordered Ge dimers can reconcile theoretical predictions with ARPES results. This model reproduces the observed CDW gaps while preserving the Ge-$p$ band. Depending on experimental conditions, Ge dimers can be randomly distributed or exhibit phase separation from pristine regions. Our findings reveal the crucial role of Ge dimer disorder in the FeGe CDW and suggest potential implications of this disorder for other properties, such as magnetism and transport, in this system., Comment: Supplementary included

Long-Context Question Answering (LCQA), a challenging task, aims to reason over long-context documents to yield accurate answers to questions. Existing long-context Large Language Models (LLMs) for LCQA often struggle with the "lost in the middle" issue. Retrieval-Augmented Generation (RAG) mitigates this issue by providing external factual evidence. However, its chunking strategy disrupts the global long-context information, and its low-quality retrieval in long contexts hinders LLMs from identifying effective factual details due to substantial noise. To this end, we propose LongRAG, a general, dual-perspective, and robust LLM-based RAG system paradigm for LCQA to enhance RAG's understanding of complex long-context knowledge (i.e., global information and factual details). We design LongRAG as a plug-and-play paradigm, facilitating adaptation to various domains and LLMs. Extensive experiments on three multi-hop datasets demonstrate that LongRAG significantly outperforms long-context LLMs (up by 6.94%), advanced RAG (up by 6.16%), and Vanilla RAG (up by 17.25%). Furthermore, we conduct quantitative ablation studies and multi-dimensional analyses, highlighting the effectiveness of the system's components and fine-tuning strategies. Data and code are available at https://github.com/QingFei1/LongRAG., Comment: EMNLP 2024 Main

Machine-learning-based phase pickers have been successfully leveraged to build high-resolution earthquake catalogs using seismic data on land. However, their performance when applied to ocean bottom seismic (OBS) data remains to be evaluated. In this study, we first adopt three machine-learning-based phase pickers - EQTransformer, Pickblue, and OBSTansformer - to build three earthquake catalogs for the 350-km-long Blanco oceanic transform fault (BTF) based on a year-long OBS deployment. We then systematically compare these catalogs with an existing catalog which utilized a traditional workflow. Results indicate that the Pickblue-based catalog documents more events and/or provides better-constrained locations than the other catalogs. The different performances of the three phase pickers suggest that detailed assessment of catalogs built using automatic workflows is necessary to prevent misinterpretations, especially when applied to regions without training samples. The Pickblue-based catalog reveals seismicity gaps in three extensional segments of BTF which likely represent aseismic slip zones affected by seawater infiltration. Furthermore, most earthquakes are shallower than the 600-degree isotherm predicted by a half-space conductive cooling model, except for the Blanco Ridge segment which has hosted 80% of the Mw > 6.0 earthquakes along BTF since 1976. These Blanco Ridge deep earthquake clusters can be explained by hydrothermal cooling or the serpentinization of mantle peridotite due to seawater infiltration along conduits created by the deeper ruptures of large earthquakes. Our analyses also demonstrate the importance of careful examination of automatically produced earthquake catalogs since mislocated events can lead to very different interpretations of fault slip modes from seismicity distribution., Comment: 38 pages and 16 figures

The self-attention mechanism traditionally relies on the softmax operator, necessitating positional embeddings like RoPE, or position biases to account for token order. But current methods using still face length generalisation challenges. We propose an alternative attention mechanism based on the stick-breaking process: For each token before the current, we determine a break point $\beta_{i,j}$, which represents the proportion of the remaining stick to allocate to the current token. We repeat the process until the stick is fully allocated, resulting in a sequence of attention weights. This process naturally incorporates recency bias, which has linguistic motivations for grammar parsing (Shen et. al., 2017). We study the implications of replacing the conventional softmax-based attention mechanism with stick-breaking attention. We then discuss implementation of numerically stable stick-breaking attention and adapt Flash Attention to accommodate this mechanism. When used as a drop-in replacement for current softmax+RoPE attention systems, we find that stick-breaking attention performs competitively with current methods on length generalisation and downstream tasks. Stick-breaking also performs well at length generalisation, allowing a model trained with $2^{11}$ context window to perform well at $2^{14}$ with perplexity improvements.

Full-duplex spoken dialogue systems significantly advance over traditional turn-based dialogue systems, as they allow simultaneous bidirectional communication, closely mirroring human-human interactions. However, achieving low latency and natural interactions in full-duplex dialogue systems remains a significant challenge, especially considering human conversation dynamics such as interruptions, backchannels, and overlapping speech. In this paper, we introduce a novel End-to-End GPT-based model OmniFlatten for full-duplex conversation, capable of effectively modeling the complex behaviors inherent to natural conversations with low latency. To achieve full-duplex communication capabilities, we propose a multi-stage post-training scheme that progressively adapts a text-based large language model (LLM) backbone into a speech-text dialogue LLM, capable of generating text and speech in real time, without modifying the architecture of the backbone LLM. The training process comprises three stages: modality alignment, half-duplex dialogue learning, and full-duplex dialogue learning. Throughout all training stages, we standardize the data using a flattening operation, which allows us to unify the training methods and the model architecture across different modalities and tasks. Our approach offers a straightforward modeling technique and a promising research direction for developing efficient and natural end-to-end full-duplex spoken dialogue systems. Audio samples of dialogues generated by OmniFlatten can be found at this web site (https://omniflatten.github.io/)., Comment: Work in progress

Forming single-photon emitters (SPEs) in insulating hexagonal boron nitride (hBN) has sparked wide interests in the quantum photonics. Despite significant progress, it remains challenging to deterministically create SPEs at precise locations with a specific type of element for creating defects. In this study, we present a straightforward approach to generate site-deterministic carbon-functionalized quantum emitters in hBN by harnessing ultrasonic nanoindentation. The obtained SPEs are high-quality and can be scaled up to large arrays in a single fabrication step. Comprehensive experimental analyses reveal that the insertion of carbon atoms into the hBN lattice is the source of the robust quantum emission. Complementary theoretical studies suggest possible candidates for the structural origin of the defects based on our experimental results. This rapid and scalable nanoindentation method provides a new way to create SPE arrays with specific types of atoms, enabling the comprehensive investigation of the origins and mechanics of SPE formations in two-dimensional (2D) materials and beyond.

The XLZD collaboration is developing a two-phase xenon time projection chamber with an active mass of 60 to 80 t capable of probing the remaining WIMP-nucleon interaction parameter space down to the so-called neutrino fog. In this work we show that, based on the performance of currently operating detectors using the same technology and a realistic reduction of radioactivity in detector materials, such an experiment will also be able to competitively search for neutrinoless double beta decay in $^{136}$Xe using a natural-abundance xenon target. XLZD can reach a 3$\sigma$ discovery potential half-life of 5.7$\times$10$^{27}$ yr (and a 90% CL exclusion of 1.3$\times$10$^{28}$ yr) with 10 years of data taking, corresponding to a Majorana mass range of 7.3-31.3 meV (4.8-20.5 meV). XLZD will thus exclude the inverted neutrino mass ordering parameter space and will start to probe the normal ordering region for most of the nuclear matrix elements commonly considered by the community., Comment: 29 pages, 7 figures

Under the premise that quantum gravity becomes non-negligible, higher-order corrections of non-commutative geometry dominate. In this paper, we studied the thermodynamics of high-order corrections for Schwarzschild-AdS black hole with Lorentz distribution in the framework of non-commutative geometry. Our results indicate that when high-order corrections dominate, the thermodynamic behavior of Schwarzschild-AdS black hole in non-commutative geometry will gradually approach that of ordinary Schwarzschild-AdS black hole. In addition, we also studied the Joule-Thomson effect of Schwarzschild-AdS black hole under high-order corrections.

Real-world noise removal is crucial in low-level computer vision. Due to the remarkable generation capabilities of diffusion models, recent attention has shifted towards leveraging diffusion priors for image restoration tasks. However, existing diffusion priors-based methods either consider simple noise types or rely on approximate posterior estimation, limiting their effectiveness in addressing structured and signal-dependent noise commonly found in real-world images. In this paper, we build upon diffusion priors and propose adaptive likelihood estimation and MAP inference during the reverse diffusion process to tackle real-world noise. We introduce an independent, non-identically distributed likelihood combined with the noise precision (inverse variance) prior and dynamically infer the precision posterior using variational Bayes during the generation process. Meanwhile, we rectify the estimated noise variance through local Gaussian convolution. The final denoised image is obtained by propagating intermediate MAP solutions that balance the updated likelihood and diffusion prior. Additionally, we explore the local diffusion prior inherent in low-resolution diffusion models, enabling direct handling of high-resolution noisy images. Extensive experiments and analyses on diverse real-world datasets demonstrate the effectiveness of our method. Code is available at https://github.com/HUST-Tan/DiffusionVI., Comment: Accepted by NeurIPS2024 as Spotlight

AI holds promise for transforming scientific processes, including hypothesis generation. Prior work on hypothesis generation can be broadly categorized into theory-driven and data-driven approaches. While both have proven effective in generating novel and plausible hypotheses, it remains an open question whether they can complement each other. To address this, we develop the first method that combines literature-based insights with data to perform LLM-powered hypothesis generation. We apply our method on five different datasets and demonstrate that integrating literature and data outperforms other baselines (8.97\% over few-shot, 15.75\% over literature-based alone, and 3.37\% over data-driven alone). Additionally, we conduct the first human evaluation to assess the utility of LLM-generated hypotheses in assisting human decision-making on two challenging tasks: deception detection and AI generated content detection. Our results show that human accuracy improves significantly by 7.44\% and 14.19\% on these tasks, respectively. These findings suggest that integrating literature-based and data-driven approaches provides a comprehensive and nuanced framework for hypothesis generation and could open new avenues for scientific inquiry., Comment: 30 pages, 7 figures, code link: https://github.com/ChicagoHAI/hypothesis-generation

This paper focuses on creating synthetic data to improve the quality of image captions. Existing works typically have two shortcomings. First, they caption images from scratch, ignoring existing alt-text metadata, and second, lack transparency if the captioners' training data (e.g. GPT) is unknown. In this paper, we study a principled approach Altogether based on the key idea to edit and re-align existing alt-texts associated with the images. To generate training data, we perform human annotation where annotators start with the existing alt-text and re-align it to the image content in multiple rounds, consequently constructing captions with rich visual concepts. This differs from prior work that carries out human annotation as a one-time description task solely based on images and annotator knowledge. We train a captioner on this data that generalizes the process of re-aligning alt-texts at scale. Our results show our Altogether approach leads to richer image captions that also improve text-to-image generation and zero-shot image classification tasks., Comment: accepted by EMNLP 2024; MetaCLIPv2

We propose the Large View Synthesis Model (LVSM), a novel transformer-based approach for scalable and generalizable novel view synthesis from sparse-view inputs. We introduce two architectures: (1) an encoder-decoder LVSM, which encodes input image tokens into a fixed number of 1D latent tokens, functioning as a fully learned scene representation, and decodes novel-view images from them; and (2) a decoder-only LVSM, which directly maps input images to novel-view outputs, completely eliminating intermediate scene representations. Both models bypass the 3D inductive biases used in previous methods -- from 3D representations (e.g., NeRF, 3DGS) to network designs (e.g., epipolar projections, plane sweeps) -- addressing novel view synthesis with a fully data-driven approach. While the encoder-decoder model offers faster inference due to its independent latent representation, the decoder-only LVSM achieves superior quality, scalability, and zero-shot generalization, outperforming previous state-of-the-art methods by 1.5 to 3.5 dB PSNR. Comprehensive evaluations across multiple datasets demonstrate that both LVSM variants achieve state-of-the-art novel view synthesis quality. Notably, our models surpass all previous methods even with reduced computational resources (1-2 GPUs). Please see our website for more details: https://haian-jin.github.io/projects/LVSM/ ., Comment: project page: https://haian-jin.github.io/projects/LVSM/

Designing reward functions is a core component of reinforcement learning but can be challenging for truly complex behavior. Reinforcement Learning from Human Feedback (RLHF) has been used to alleviate this challenge by replacing a hand-coded reward function with a reward function learned from preferences. However, it can be exceedingly inefficient to learn these rewards as they are often learned tabula rasa. We investigate whether Large Language Models (LLMs) can reduce this query inefficiency by converting an iterative series of human preferences into code representing the rewards. We propose In-Context Preference Learning (ICPL), a method that uses the grounding of an LLM to accelerate learning reward functions from preferences. ICPL takes the environment context and task description, synthesizes a set of reward functions, and then repeatedly updates the reward functions using human rankings of videos of the resultant policies. Using synthetic preferences, we demonstrate that ICPL is orders of magnitude more efficient than RLHF and is even competitive with methods that use ground-truth reward functions instead of preferences. Finally, we perform a series of human preference-learning trials and observe that ICPL extends beyond synthetic settings and can work effectively with humans-in-the-loop. Additional information and videos are provided at https://sites.google.com/view/few-shot-icpl/home.

Building on the success of large language models (LLMs), recent advancements such as GPT-4o have enabled real-time speech interactions through LLM-based voice assistants, offering a significantly improved user experience compared to traditional text-based interactions. However, the absence of benchmarks designed to evaluate these speech interaction capabilities has hindered progress of LLM-based voice assistants development. Current evaluations focus primarily on automatic speech recognition (ASR) or general knowledge evaluation with clean speeches, neglecting the more intricate, real-world scenarios that involve diverse speaker characteristics, environmental and content factors. To address this, we introduce VoiceBench, the first benchmark designed to provide a multi-faceted evaluation of LLM-based voice assistants. VoiceBench also includes both real and synthetic spoken instructions that incorporate the above three key real-world variations. Extensive experiments reveal the limitations of current LLM-based voice assistant models and offer valuable insights for future research and development in this field., Comment: Work in progress. Data is available at https://github.com/MatthewCYM/VoiceBench

In this study, we present the pulse profile of the unique and the second brightest gamma-ray burst GRB 230307A, and analyze its temporal behavior using a joint GECAM--Fermi/GBM time-resolved spectral analysis. The utilization of GECAM data is advantageous as it successfully captured significant data during the pile-up period of the Fermi/GBM. We investigate the evolution of its flux, photon fluence, photon flux, peak energy, and the corresponding hardness-intensity and hardness-flux correlations. The findings within the first 27 seconds exhibit consistent patterns reported previously, providing valuable insights for comparing observations with predictions from the synchrotron radiation model invoking an expanding shell. Beyond the initial 27 seconds, we observe a notable transition in the emitted radiation, attributed to high latitude emission (HLE), influenced by the geometric properties of the shells and the relativistic Doppler effects. By modeling the data within the framework of the large-radius internal shock model, we discuss the required parameters as well as the limitations of the model. We conclude that a more complicated synchrotron emission model is needed to fully describe the observational data of GRB 230307A., Comment: Accepted for publication in The Astrophysical Journal (ApJ)

This report describes the experimental strategy and technologies for a next-generation xenon observatory sensitive to dark matter and neutrino physics. The detector will have an active liquid xenon target mass of 60-80 tonnes and is proposed by the XENON-LUX-ZEPLIN-DARWIN (XLZD) collaboration. The design is based on the mature liquid xenon time projection chamber technology of the current-generation experiments, LZ and XENONnT. A baseline design and opportunities for further optimization of the individual detector components are discussed. The experiment envisaged here has the capability to explore parameter space for Weakly Interacting Massive Particle (WIMP) dark matter down to the neutrino fog, with a 3$\sigma$ evidence potential for the spin-independent WIMP-nucleon cross sections as low as $3\times10^{-49}\rm cm^2$ (at 40 GeV/c$^2$ WIMP mass). The observatory is also projected to have a 3$\sigma$ observation potential of neutrinoless double-beta decay of $^{136}$Xe at a half-life of up to $5.7\times 10^{27}$ years. Additionally, it is sensitive to astrophysical neutrinos from the atmosphere, sun, and galactic supernovae., Comment: 32 pages, 14 figures

Multimodal Large Language Models (MLLMs) utilize multimodal contexts consisting of text, images, or videos to solve various multimodal tasks. However, we find that changing the order of multimodal input can cause the model's performance to fluctuate between advanced performance and random guessing. This phenomenon exists in both single-modality (text-only or image-only) and mixed-modality (image-text-pair) contexts. Furthermore, we demonstrate that popular MLLMs pay special attention to certain multimodal context positions, particularly the beginning and end. Leveraging this special attention, we place key video frames and important image/text content in special positions within the context and submit them to the MLLM for inference. This method results in average performance gains of 14.7% for video-caption matching and 17.8% for visual question answering tasks. Additionally, we propose a new metric, Position-Invariant Accuracy (PIA), to address order bias in MLLM evaluation. Our research findings contribute to a better understanding of Multi-Modal In-Context Learning (MMICL) and provide practical strategies for enhancing MLLM performance without increasing computational costs.

Stroke-based Rendering (SBR) aims to decompose an input image into a sequence of parameterized strokes, which can be rendered into a painting that resembles the input image. Recently, Neural Painting methods that utilize deep learning and reinforcement learning models to predict the stroke sequences have been developed, but suffer from longer inference time or unstable training. To address these issues, we propose AttentionPainter, an efficient and adaptive model for single-step neural painting. First, we propose a novel scalable stroke predictor, which predicts a large number of stroke parameters within a single forward process, instead of the iterative prediction of previous Reinforcement Learning or auto-regressive methods, which makes AttentionPainter faster than previous neural painting methods. To further increase the training efficiency, we propose a Fast Stroke Stacking algorithm, which brings 13 times acceleration for training. Moreover, we propose Stroke-density Loss, which encourages the model to use small strokes for detailed information, to help improve the reconstruction quality. Finally, we propose a new stroke diffusion model for both conditional and unconditional stroke-based generation, which denoises in the stroke parameter space and facilitates stroke-based inpainting and editing applications helpful for human artists design. Extensive experiments show that AttentionPainter outperforms the state-of-the-art neural painting methods.

Gravitational waves (GWs) offer a new observational window into the universe, providing insights into compact objects and cosmic structures. Gravitational lensing, commonly studied in electromagnetic waves, also affects GWs, introducing magnification, time delays, and multiple images. While existing studies focus on static lenses, many astrophysical lenses are dynamic, with time-varying mass distributions such as moving stars or orbiting binaries. We develop a general theoretical framework to describe non-static lenses and demonstrate how they modulate GW signals, inducing unique time-varying amplitude modulations and spectral broadening. By examining uniformly moving and orbiting binary lenses, we show that these modulations provide new observational signatures, enhancing our understanding of lensing objects and the dynamics of the universe. Our findings have important implications for GW astronomy, offering novel ways to probe lens dynamics and improve the interpretations of GW signals., Comment: 20 pages, 8 figures

We report our experience formally modelling and verifying CXL.cache, the inter-device cache coherence protocol of the Compute Express Link standard. We have used the Isabelle proof assistant to create a formal model for CXL.cache based on the prose English specification. This led to us identifying and proposing fixes to several problems we identified as unclear, ambiguous or inaccurate, some of which could lead to incoherence if left unfixed. Nearly all our issues and proposed fixes have been confirmed and tentatively accepted by the CXL consortium for adoption, save for one which is still under discussion. To validate the faithfulness of our model we performed scenario verification of essential restrictions such as "Snoop-pushes-GO", and produced a fully mechanised proof of a coherence property of the model. The considerable size of this proof, comprising tens of thousands of lemmas, prompted us to develop new proof automation tools, which we have made available for other Isabelle users working with similarly cumbersome proofs., Comment: 12 pages

Due to the rising awareness of privacy protection and the voluminous scale of speech data, it is becoming infeasible for Automatic Speech Recognition (ASR) system developers to train the acoustic model with complete data as before. For example, the data may be owned by different curators, and it is not allowed to share with others. In this paper, we propose a novel paradigm to solve salient problems plaguing the ASR field. In the first stage, multiple acoustic models are trained based upon different subsets of the complete speech data, while in the second phase, two novel algorithms are utilized to generate a high-quality acoustic model based upon those trained on data subsets. We first propose the Genetic Merge Algorithm (GMA), which is a highly specialized algorithm for optimizing acoustic models but suffers from low efficiency. We further propose the SGD-Based Optimizational Merge Algorithm (SOMA), which effectively alleviates the efficiency bottleneck of GMA and maintains superior model accuracy. Extensive experiments on public data show that the proposed methods can significantly outperform the state-of-the-art. Furthermore, we introduce Shapley Value to estimate the contribution score of the trained models, which is useful for evaluating the effectiveness of the data and providing fair incentives to their curators.

Protein structures are important for understanding their functions and interactions. Currently, many protein structure prediction methods are enriching the structure database. Discriminating the origin of structures is crucial for distinguishing between experimentally resolved and computationally predicted structures, evaluating the reliability of prediction methods, and guiding downstream biological studies. Building on works in structure prediction, We developed a structure-sensitive supervised deep learning model, Crystal vs Predicted Evaluator for Protein Structure (CPE-Pro), to represent and discriminate the origin of protein structures. CPE-Pro learns the structural information of proteins and captures inter-structural differences to achieve accurate traceability on four data classes, and is expected to be extended to more. Simultaneously, we utilized Foldseek to encode protein structures into "structure-sequences" and trained a protein Structural Sequence Language Model, SSLM. Preliminary experiments demonstrated that, compared to large-scale protein language models pre-trained on vast amounts of amino acid sequences, the "structure-sequence" enables the language model to learn more informative protein features, enhancing and optimizing structural representations. We have provided the code, model weights, and all related materials on https://github.com/GouWenrui/CPE-Pro-main.git.

Deep neural networks are ubiquitously adopted in many applications, such as computer vision, natural language processing, and graph analytics. However, well-trained neural networks can make prediction errors after deployment as the world changes. \textit{Model editing} involves updating the base model to correct prediction errors with less accessible training data and computational resources. Despite recent advances in model editors in computer vision and natural language processing, editable training in graph neural networks (GNNs) is rarely explored. The challenge with editable GNN training lies in the inherent information aggregation across neighbors, which can lead model editors to affect the predictions of other nodes unintentionally. In this paper, we first observe the gradient of cross-entropy loss for the target node and training nodes with significant inconsistency, which indicates that directly fine-tuning the base model using the loss on the target node deteriorates the performance on training nodes. Motivated by the gradient inconsistency observation, we propose a simple yet effective \underline{G}radient \underline{R}ewiring method for \underline{E}ditable graph neural network training, named \textbf{GRE}. Specifically, we first store the anchor gradient of the loss on training nodes to preserve the locality. Subsequently, we rewire the gradient of the loss on the target node to preserve performance on the training node using anchor gradient. Experiments demonstrate the effectiveness of GRE on various model architectures and graph datasets in terms of multiple editing situations. The source code is available at \url{https://github.com/zhimengj0326/Gradient_rewiring_editing}, Comment: NeurIPS 2024

Concept Bottleneck Models (CBMs) have been proposed as a compromise between white-box and black-box models, aiming to achieve interpretability without sacrificing accuracy. The standard training procedure for CBMs is to predefine a candidate set of human-interpretable concepts, extract their values from the training data, and identify a sparse subset as inputs to a transparent prediction model. However, such approaches are often hampered by the tradeoff between enumerating a sufficiently large set of concepts to include those that are truly relevant versus controlling the cost of obtaining concept extractions. This work investigates a novel approach that sidesteps these challenges: BC-LLM iteratively searches over a potentially infinite set of concepts within a Bayesian framework, in which Large Language Models (LLMs) serve as both a concept extraction mechanism and prior. BC-LLM is broadly applicable and multi-modal. Despite imperfections in LLMs, we prove that BC-LLM can provide rigorous statistical inference and uncertainty quantification. In experiments, it outperforms comparator methods including black-box models, converges more rapidly towards relevant concepts and away from spuriously correlated ones, and is more robust to out-of-distribution samples.

Large generative models (LMs) are increasingly being considered for high-stakes decision-making. This work considers how such models compare to humans and predictive AI models on a specific case of recidivism prediction. We combine three datasets -- COMPAS predictive AI risk scores, human recidivism judgements, and photos -- into a dataset on which we study the properties of several state-of-the-art, multimodal LMs. Beyond accuracy and bias, we focus on studying human-LM alignment on the task of recidivism prediction. We investigate if these models can be steered towards human decisions, the impact of adding photos, and whether anti-discimination prompting is effective. We find that LMs can be steered to outperform humans and COMPAS using in context-learning. We find anti-discrimination prompting to have unintended effects, causing some models to inhibit themselves and significantly reduce their number of positive predictions.

The advancements in cloud-based Large Languages Models (LLMs) have revolutionized AI-assisted programming. However, their integration into certain local development environments like ones within the Apple software ecosystem (e.g., iOS apps, macOS) remains challenging due to computational demands and sandboxed constraints. This paper presents CAMP, a multi-model AI-assisted programming framework that consists of a local model that employs Retrieval-Augmented Generation (RAG) to retrieve contextual information from the codebase to facilitate context-aware prompt construction thus optimizing the performance of the cloud model, empowering LLMs' capabilities in local Integrated Development Environments (IDEs). The methodology is actualized in Copilot for Xcode, an AI-assisted programming tool crafted for Xcode that employs the RAG module to address software constraints and enables diverse generative programming tasks, including automatic code completion, documentation, error detection, and intelligent user-agent interaction. The results from objective experiments on generated code quality and subjective experiments on user adoption collectively demonstrate the pilot success of the proposed system and mark its significant contributions to the realm of AI-assisted programming., Comment: 12 pages, 3 figures, 4 tables

Colorectal cancer is the most common cancer in Singapore and the third most common cancer worldwide. Blood in a person's stool is a symptom of this disease, and it is usually detected by the faecal occult blood test (FOBT). However, the FOBT presents several limitations - the collection process for the stool samples is tedious and unpleasant, the waiting period for results is about 2 weeks and costs are involved. In this research, we propose a simple-to-use, fast and cost-free alternative - a stool recognition neural network that determines if there is blood in one's stool (which indicates a possible risk of colorectal cancer) from an image of it. As this is a new classification task, there was limited data available, hindering classifier performance. Hence, various Generative Adversarial Networks (GANs) (DiffAugment StyleGAN2, DCGAN, Conditional GAN) were trained to generate images of high fidelity to supplement the dataset. Subsequently, images generated by the GAN with the most realistic images (DiffAugment StyleGAN2) were concatenated to the classifier's training batch on-the-fly, improving accuracy to 94%. This model was then deployed to a mobile app - Poolice, where users can take a photo of their stool and obtain instantaneous results if there is blood in their stool, prompting those who do to seek medical advice. As "early detection saves lives", we hope our app built on our stool recognition neural network can help people detect colorectal cancer earlier, so they can seek treatment and have higher chances of survival., Comment: 21 pages, 28 figures