Search

Recently, medical image synthesis gains more and more popularity, along with the rapid development of generative models. Medical image synthesis aims to generate an unacquired image modality, often from other observed data modalities. Synthesized images can be used for clinical diagnostic assistance, data augmentation for model training and validation or image quality improving. In the meanwhile, the flow-based models are among the successful generative models for the ability of generating realistic and high-quality synthetic images. However, most flow-based models require to calculate flow ordinary different equation (ODE) evolution steps in transfer process, for which the performances are significantly limited by heavy computation time due to a large number of time iterations. In this paper, we propose a novel flow-based model, namely Discrete Process Matching (DPM) to accomplish the bi-modality image transfer tasks. Different to other flow matching based models, we propose to utilize both forward and backward ODE flow and enhance the consistency on the intermediate images of few discrete time steps, resulting in a transfer process with much less iteration steps while maintaining high-quality generations for both modalities. Our experiments on three datasets of MRI T1/T2 and CT/MRI demonstrate that DPM outperforms other state-of-the-art flow-based methods for bi-modality image synthesis, achieving higher image quality with less computation time cost., Comment: There are illegal data and models which is not allow for publish now. I need to withdraw the paper and revise it to other public examples

Complex 3D scene understanding has gained increasing attention, with scene encoding strategies playing a crucial role in this success. However, the optimal scene encoding strategies for various scenarios remain unclear, particularly compared to their image-based counterparts. To address this issue, we present a comprehensive study that probes various visual encoding models for 3D scene understanding, identifying the strengths and limitations of each model across different scenarios. Our evaluation spans seven vision foundation encoders, including image-based, video-based, and 3D foundation models. We evaluate these models in four tasks: Vision-Language Scene Reasoning, Visual Grounding, Segmentation, and Registration, each focusing on different aspects of scene understanding. Our evaluations yield key findings: DINOv2 demonstrates superior performance, video models excel in object-level tasks, diffusion models benefit geometric tasks, and language-pretrained models show unexpected limitations in language-related tasks. These insights challenge some conventional understandings, provide novel perspectives on leveraging visual foundation models, and highlight the need for more flexible encoder selection in future vision-language and scene-understanding tasks., Comment: Project page: https://yunzeman.github.io/lexicon3d , Github: https://github.com/YunzeMan/Lexicon3D

Since the advent of ChatGPT, Large Language Models (LLMs) have excelled in various tasks but remain as black-box systems. Consequently, the reasoning bottlenecks of LLMs are mainly influenced by their internal architecture. As a result, many researchers have begun exploring the potential internal mechanisms of LLMs, with most studies focusing on attention heads. Our survey aims to shed light on the internal reasoning processes of LLMs by concentrating on the underlying mechanisms of attention heads. We first distill the human thought process into a four-stage framework: Knowledge Recalling, In-Context Identification, Latent Reasoning, and Expression Preparation. Using this framework, we systematically review existing research to identify and categorize the functions of specific attention heads. Furthermore, we summarize the experimental methodologies used to discover these special heads, dividing them into two categories: Modeling-Free methods and Modeling-Required methods. Also, we outline relevant evaluation methods and benchmarks. Finally, we discuss the limitations of current research and propose several potential future directions., Comment: 29 pages, 11 figures, 4 tables, 5 equations

Reconstructing the high-fidelity surface from multi-view images, especially sparse images, is a critical and practical task that has attracted widespread attention in recent years. However, existing methods are impeded by the memory constraint or the requirement of ground-truth depths and cannot recover satisfactory geometric details. To this end, we propose SuRF, a new Surface-centric framework that incorporates a new Region sparsification based on a matching Field, achieving good trade-offs between performance, efficiency and scalability. To our knowledge, this is the first unsupervised method achieving end-to-end sparsification powered by the introduced matching field, which leverages the weight distribution to efficiently locate the boundary regions containing surface. Instead of predicting an SDF value for each voxel, we present a new region sparsification approach to sparse the volume by judging whether the voxel is inside the surface region. In this way, our model can exploit higher frequency features around the surface with less memory and computational consumption. Extensive experiments on multiple benchmarks containing complex large-scale scenes show that our reconstructions exhibit high-quality details and achieve new state-of-the-art performance, i.e., 46% improvements with 80% less memory consumption. Code is available at https://github.com/prstrive/SuRF., Comment: ECCV 2024 Accepted

Audio-driven talking face generation aims to synthesize video with lip movements synchronized to input audio. However, current generative techniques face challenges in preserving intricate regional textures (skin, teeth). To address the aforementioned challenges, we propose a novel framework called SegTalker to decouple lip movements and image textures by introducing segmentation as intermediate representation. Specifically, given the mask of image employed by a parsing network, we first leverage the speech to drive the mask and generate talking segmentation. Then we disentangle semantic regions of image into style codes using a mask-guided encoder. Ultimately, we inject the previously generated talking segmentation and style codes into a mask-guided StyleGAN to synthesize video frame. In this way, most of textures are fully preserved. Moreover, our approach can inherently achieve background separation and facilitate mask-guided facial local editing. In particular, by editing the mask and swapping the region textures from a given reference image (e.g. hair, lip, eyebrows), our approach enables facial editing seamlessly when generating talking face video. Experiments demonstrate that our proposed approach can effectively preserve texture details and generate temporally consistent video while remaining competitive in lip synchronization. Quantitative and qualitative results on the HDTF and MEAD datasets illustrate the superior performance of our method over existing methods., Comment: 10 pages, 7 figures, 3 tables

The communication-assisted sensing (CAS) systems are expected to endow the users with beyond-line-of-sight sensing capabilities without the aid of additional sensors. In this paper, we study the dual-functional signaling strategy, focusing on three primary aspects, namely, the information-theoretic framework, the optimal distribution of channel input, and the optimal waveform design for Gaussian signals. First, we establish the information-theoretic framework and develop a modified source-channel separation theorem (MSST) tailored for CAS systems. The proposed MSST elucidates the relationship between achievable distortion, coding rate, and communication channel capacity in cases where the distortion metric is separable for sensing and communication (S\&C) processes. Second, we present an optimal channel input design for dual-functional signaling, which aims to minimize total distortion under the constraints of the MSST and resource budget. We then conceive a two-step Blahut-Arimoto (BA)-based optimal search algorithm to numerically solve the functional optimization problem. Third, in light of the current signaling strategy, we further propose an optimal waveform design for Gaussian signaling in multi-input multi-output (MIMO) CAS systems. The associated covariance matrix optimization problem is addressed using a successive convex approximation (SCA)-based waveform design algorithm. Finally, we provide numerical simulation results to demonstrate the effectiveness of the proposed algorithms and to show the unique performance tradeoff between S\&C processes.

The human skin exhibits remarkable capability to perceive contact forces and environmental temperatures, providing intricate information essential for nuanced manipulation. Despite recent advancements in soft tactile sensors, a significant challenge remains in accurately decoupling signals - specifically, separating force from directional orientation and temperature - resulting in fail to meet the advanced application requirements of robots. This research proposes a multi-layered soft sensor unit (F3T) designed to achieve isolated measurements and mathematical decoupling of normal pressure, omnidirectional tangential forces, and temperature. We developed a circular coaxial magnetic film featuring a floating-mountain multi-layer capacitor, facilitating the physical decoupling of normal and tangential forces in all directions. Additionally, we incorporated an ion gel-based temperature sensing film atop the tactile sensor. This sensor is resilient to external pressure and deformation, enabling it to measure temperature and, crucially, eliminate capacitor errors induced by environmental temperature changes. This innovative design allows for the decoupled measurement of multiple signals, paving the way for advancements in higher-level robot motion control, autonomous decision-making, and task planning.

Cancer Screening has been widely recognized as an effective strategy for preventing the disease. Despite its effectiveness, determining when to start screening is complicated, because starting too early increases the number of screenings over lifetime and thus costs but starting too late may miss the cancer that could have been prevented. Therefore, to make an informed recommendation on the age to start screening, it is necessary to conduct cost-effectiveness analysis to assess the gain in life years relative to the cost of screenings. As more large-scale observational studies become accessible, there is growing interest in evaluating cost-effectiveness based on empirical evidence. In this paper, we propose a unified measure for evaluating cost-effectiveness and a causal analysis for the continuous intervention of screening initiation age, under the multi-state modeling with semi-competing risks. Extensive simulation results show that the proposed estimators perform well in realistic scenarios. We perform a cost-effectiveness analysis of the colorectal cancer screening, utilizing data from the large-scale Women's Health Initiative. Our analysis reveals that initiating screening at age 50 years yields the highest quality-adjusted life years with an acceptable incremental cost-effectiveness ratio compared to no screening, providing real-world evidence in support of screening recommendation for colorectal cancer., Comment: 37 pages, 2 figures, 8 tables

The proper flow time for the Wilson flow in calculating the topological charge, topological susceptibility, and topological charge density correlator (TCDC) using the gluonic definition is analyzed. The proper flow time for the topological charge and TCDC is determined by different methods. The proper flow time may vary depending on whether the calculation is for the topological charge, topological density, topological susceptibility, or TCDC. Specifically, the flow time identified using the matching procedure is optimal for TCDC and a good choice for the topological susceptibility. Additionally, the pseudoscalar glueball is extracted from the TCDC using the bosonic definition at the identified proper flow time for three ensembles, and the continuum mass value is obtained through continuous extrapolation., Comment: 21 pages, 6 figures.V2: minor modifications, corrected typos

Multiply-accumulation (MAC) is a crucial computing operation in signal processing, numerical simulations, and machine learning. This work presents a scalable, programmable, frequency-domain parallel computing leveraging gigahertz (GHz)-frequency acoustic-wave nonlinearities. By encoding data in the frequency domain, a single nonlinear acoustic-wave device can perform a billion arithmetic operations simultaneously. A single device with a footprint of 0.03 mm$^2$ on lithium niobate (LN) achieves 0.0144 tera floating-point operations per second (TFLOPS), leading to a computing area density of 0.48 TFLOPS/mm$^2$ and a core power efficiency of 0.14 TFLOPS/Watt. As applications, we demonstrate multiplications of two 16-by-16 matrices and convolutional imaging processing of 128-by-128-pixel photos. Our technology could find versatile applications in near-sensor signal processing and edge computing.

Autonomous agents powered by large language models (LLMs) have attracted significant research interest. However, the open-source community faces many challenges in developing specialized models for agent tasks, driven by the scarcity of high-quality agent datasets and the absence of standard protocols in this area. We introduce and publicly release xLAM, a series of large action models designed for AI agent tasks. The xLAM series includes five models with both dense and mixture-of-expert architectures, ranging from 1B to 8x22B parameters, trained using a scalable, flexible pipeline that unifies, augments, and synthesizes diverse datasets to enhance AI agents' generalizability and performance across varied environments. Our experimental results demonstrate that xLAM consistently delivers exceptional performance across multiple agent ability benchmarks, notably securing the 1st position on the Berkeley Function-Calling Leaderboard, outperforming GPT-4, Claude-3, and many other models in terms of tool use. By releasing the xLAM series, we aim to advance the performance of open-source LLMs for autonomous AI agents, potentially accelerating progress and democratizing access to high-performance models for agent tasks. Models are available at https://huggingface.co/collections/Salesforce/xlam-models-65f00e2a0a63bbcd1c2dade4, Comment: Technical report for the Salesforce xLAM model series

Multimodal Large Language Models (MLLMs) have significantly improved performance across various image-language applications. Recently, there has been a growing interest in adapting image pre-trained MLLMs for video-related tasks. However, most efforts concentrate on enhancing the vision encoder and projector components, while the core part, Large Language Models (LLMs), remains comparatively under-explored. In this paper, we propose two strategies to enhance the model's capability in video understanding tasks by improving inter-layer attention computation in LLMs. Specifically, the first approach focuses on the enhancement of Rotary Position Embedding (RoPE) with Temporal-Aware Dual RoPE, which introduces temporal position information to strengthen the MLLM's temporal modeling capabilities while preserving the relative position relationships of both visual and text tokens. The second approach involves enhancing the Attention Mask with the Frame-wise Block Causal Attention Mask, a simple yet effective method that broadens visual token interactions within and across video frames while maintaining the causal inference mechanism. Based on these proposed methods, we adapt LLaVA for video understanding tasks, naming it Temporal-Considered LLaVA (TC-LLaVA). Our TC-LLaVA achieves new state-of-the-art performance across various video understanding benchmarks with only supervised fine-tuning (SFT) on video-related datasets.

Let $n\geqslant3$ be an odd integer and $u$ an element in the finite field $\gf_{3^n}$. The Ness-Helleseth function is the binomial $f_u(x)=ux^{d_1}+x^{d_2}$ over $\gf_{3^n}$, where $d_1=\frac{3^n-1}{2}-1$ and $d_2=3^n-2$. In 2007, Ness and Helleseth showed that $f_u$ is an APN function when $\chi(u+1)=\chi(u-1)=\chi(u)$, is differentially $3$-uniform when $\chi(u+1)=\chi(u-1)\neq\chi(u)$, and has differential uniformity at most 4 if $ \chi(u+1)\neq\chi(u-1)$ and $u\notin\gf_3$. Here $\chi(\cdot)$ denotes the quadratic character on $\gf_{3^n}$. Recently, Xia et al. determined the differential uniformity of $f_u$ for all $u$ and computed the differential spectrum of $f_u$ for $u$ satisfying $\chi(u+1)=\chi(u-1)$ or $u\in\gf_3$. The remaining problem is the differential spectrum of $f_u$ with $\chi(u+1)\neq\chi(u-1)$ and $u\notin\gf_3$. In this paper, we fill in the gap. By studying differential equations arising from the Ness-Helleseth function $f_u$ more carefully, we express the differential spectrum of $f_u$ for such $u$ in terms of two quadratic character sums. This complements the previous work of Xia et al.

Void number counts (VNC) indicates the number of low-density regions in the large-scale structure (LSS) of the Universe, and we propose to use it as an effective cosmological probe. By generating the galaxy mock catalog based on Jiutian simulations and considering the spectroscopic survey strategy and instrumental design of the China Space Station Telescope (CSST), which can reach a magnitude limit $\sim$23 AB mag and spectral resolution $R\gtrsim200$ with a sky coverage 17,500 deg$^2$, we identify voids using the watershed algorithm without any assumption of void shape, and obtain the mock void catalog and data of the VNC in six redshift bins from $z=0.3$ to1.3. We use the Markov Chain Monte Carlo (MCMC) method to constrain the cosmological and VNC parameters. The void linear underdensity threshold $\delta_{\rm v}$ in the theoretical model is set to be a free parameter at a given redshift to fit the VNC data and explore its redshift evolution. We find that, the VNC can correctly derive the cosmological information, and the constraint strength on the cosmological parameters is comparable to that from the void size function (VSF) method, which can reach a few percentage levels in the CSST full spectroscopic survey. This is because that, since the VNC is not sensitive to void shape, the modified theoretical model can match the data better by integrating over void features, and more voids could be included in the VNC analysis by applying simpler selection criteria, which will improve the statistical significance. It indicates that the VNC can be an effective cosmological probe for exploring the LSS., Comment: 8 pages, 5 figures, 2 tables. Accepted for publication in MNRAS

Currently, prompting techniques can be mainly divided into two categories:1)shot method implicitly inspires the model to answer the question by mimicing the steps in the given example, e.g., the few-shot CoT. 2) Guideline method explicitly instructs the model to reason by following guidelines, which contains succinct and concise task-specific knowledge. Shot method is prone to difficulties in terms of selection of shots type, the number of shots, and the design of the reasoning steps, so a question arises: can we only use guideline instead of shot in the prompt? To this end, we propose the FGT framework to automatically learn task-specific guidelines from dataset consisting of Feedback, Guideline, and Tree-gather agents. First, the feedback agent is designed to evaluate the outcomes, both right and wrong, of each Q&A to gather insights guiding more effective optimization strategies. Next, the guideline agent is tasked with deriving guidelines from each piece of feedback and storing them in local memory. Lastly, the tree-gather agent aggregates all guidelines hierarchically through a tree structure, ultimately obtaining all unduplicated guidelines from a global perspective. In addition, we induce the model to generate intermediate processes to ensure the reasoning consistent with the guidelines. Experimental results demonstrate that our approach achieves superior performance across multiple tasks, thereby highlighting the effectiveness of using the guidelines in prompt.

Recent studies have shown that large language models' (LLMs) mathematical problem-solving capabilities can be enhanced by integrating external tools, such as code interpreters, and employing multi-turn Chain-of-Thought (CoT) reasoning. While current methods focus on synthetic data generation and Supervised Fine-Tuning (SFT), this paper studies the complementary direct preference learning approach to further improve model performance. However, existing direct preference learning algorithms are originally designed for the single-turn chat task, and do not fully address the complexities of multi-turn reasoning and external tool integration required for tool-integrated mathematical reasoning tasks. To fill in this gap, we introduce a multi-turn direct preference learning framework, tailored for this context, that leverages feedback from code interpreters and optimizes trajectory-level preferences. This framework includes multi-turn DPO and multi-turn KTO as specific implementations. The effectiveness of our framework is validated through training of various language models using an augmented prompt set from the GSM8K and MATH datasets. Our results demonstrate substantial improvements: a supervised fine-tuned Gemma-1.1-it-7B model's performance increased from 77.5% to 83.9% on GSM8K and from 46.1% to 51.2% on MATH. Similarly, a Gemma-2-it-9B model improved from 84.1% to 86.3% on GSM8K and from 51.0% to 54.5% on MATH., Comment: A multi-turn direct preference learning framework for tool-integrated reasoning tasks

HERMES (High Energy Rapid Modular Ensemble of Satellites) Pathfinder is a space-borne mission based on a constellation of six nano-satellites flying in a low-Earth orbit (LEO). The 3U CubeSats, to be launched in early 2025, host miniaturized instruments with a hybrid Silicon Drift Detector/GAGG:Ce scintillator photodetector system, sensitive to X-rays and gamma-rays in a large energy band. HERMES will operate in conjunction with Australian Space Industry Responsive Intelligent Thermal (SpIRIT) 6U CubeSat, launched in December 2023. HERMES will probe the temporal emission of bright high-energy transients such as Gamma-Ray Bursts (GRBs), ensuring a fast transient localization in a field of view of several steradians exploiting the triangulation technique. HERMES intrinsically modular transient monitoring experiment represents a keystone capability to complement the next generation of gravitational wave experiments. In this paper we outline the scientific case, development and programmatic status of the mission, Comment: 13 pages, 16 figures. Proceedings of SPIE "Astronomical Telescopes and Instrumentation" 2024. arXiv admin note: text overlap with arXiv:2210.13866

In this paper, we calculate the semileptonic decay $D\to K_0^\ast(1430)\ell\nu_{\ell}$ with $\ell=(e,\mu)$ induced by $c\to s\ell\nu_{\ell}$ transition. For the key component, $D\to K_0^\ast(1430)$ transition form factors (TFFs) $f_{\pm}(q^2)$ are calculated within the framework of QCD light-cone sum rule. Then, we consider two scenarios for $K_0^\ast(1430)$-meson twist-2 distribution amplitude. For the scenario 1 (S1), we take the truncated form based on Gegenbauer polynomial series. Meanwhile, we also consider the scenario 2 (S2) constructed by light-cone harmonic oscillator model, where the model parameters are fixed by the $K_0^\ast(1430)$-meson twist-2 distribution amplitude $10$th-order $\xi$-moments calculated by using the background field theory. For the TFFs at large recoil point, we have $f_+^{\rm (S1)}(0)=0.597^{+0.122}_{-0.121}$ and $f_-^{\rm (S1)}(0)=-0.136^{+0.023}_{-0.035}$, $f_+^{\rm (S2)}(0)= 0.663^{+0.135}_{-0.134}$ and $f_-^{\rm (S2)}(0)=-0.202^{+0.026}_{-0.046}$. After extrapolating TFFs to the whole physical $q^2$-region, we calculate the branching fractions of $D^0\to K_0^{\ast +}(1430)\ell^-\bar\nu_\ell$ and $D^+\to K_0^{\ast 0}(1430)\ell^+\nu_\ell$ which at $10^{-4}$-order level for the S1 and S2 cases. Meanwhile, we predict the CKM matrix $|V_{cs}|^{\rm (S1)}=0.997^{+0.258}_{-0.172}, |V_{cs}|^{\rm (S2)}=0.903^{+0.233}_{-0.155}$, and leptonic flavour universality $\mathcal{R}^{\rm (S1)}_{K_0^*}=0.768^{+0.560}_{-0.368}, \mathcal{R}_{K_0^*}^{\rm (S2)}=0.764^{+0.555}_{-0.365}$. Finally, we discuss the angular observables of forward-backward asymmetries, lepton polarization asymmetries and $q^2$-differential flat terms for this decay., Comment: 10 pages, 5 figures, comments welcome

Recent advancements in large language models (LLMs) have significantly enhanced their capacity to aggregate and process information across multiple modalities, enabling them to perform a wide range of tasks such as multimodal data querying, tool usage, web interactions, and handling long documents. These capabilities pave the way for transforming LLMs from mere chatbots into general-purpose agents capable of interacting with the real world. This paper explores the concept of using a language model as the core component of an operating system (OS), effectively acting as a CPU that processes data stored in a context window, which functions as RAM. A key challenge in realizing such an LM OS is managing the life-long context and ensuring statefulness across sessions, a feature limited by the current session-based interaction paradigm due to context window size limit. To address this, we introduce compressor-retriever, a model-agnostic architecture designed for life-long context management. Unlike other long-context solutions such as retrieval-augmented generation, our approach exclusively uses the base model's forward function to compress and retrieve context, ensuring end-to-end differentiability. Preliminary experiments demonstrate the effectiveness of this architecture in in-context learning tasks, marking a step towards the development of a fully stateful LLM OS. Project repo available at: https://github.com/gblackout/LM-OS

The evolving speech processing landscape is increasingly focused on complex scenarios like meetings or cocktail parties with multiple simultaneous speakers and far-field conditions. Existing methodologies for addressing these challenges fall into two categories: multi-channel and single-channel solutions. Single-channel approaches, notable for their generality and convenience, do not require specific information about microphone arrays. This paper presents a large-scale far-field overlapping speech dataset, crafted to advance research in speech separation, recognition, and speaker diarization. This dataset is a critical resource for decoding ``Who said What and When'' in multi-talker, reverberant environments, a daunting challenge in the field. Additionally, we introduce a pipeline system encompassing speech separation, recognition, and diarization as a foundational benchmark. Evaluations on the WHAMR! dataset validate the broad applicability of the proposed data., Comment: InterSpeech 2024

This paper investigates discrete and continuous speech representations in Large Language Model (LLM)-based Automatic Speech Recognition (ASR), organizing them by feature continuity and training approach into four categories: supervised and unsupervised for both discrete and continuous types. We further classify LLMs based on their input and autoregressive feedback into continuous and discrete-space models. Using specialized encoders and comparative analysis with a Joint-Training-From-Scratch Language Model (JTFS LM) and pre-trained LLaMA2-7b, we provide a detailed examination of their effectiveness. Our work marks the first extensive comparison of speech representations in LLM-based ASR and explores various modeling techniques. We present an open-sourced achievement of a state-of-the-art Word Error Rate (WER) of 1.69\% on LibriSpeech using a HuBERT encoder, offering valuable insights for advancing ASR and natural language processing (NLP) research., Comment: InterSpeech 2024

Axion-like particles (ALPs) and dark photons (DPs) are viable dark matter particle candidates. We have searched for possible ALP/DP signals in the PandaX-4T liquid xenon detector using 94.8 days of data. A binned likelihood fit is constructed to search for possible mono-energetic peaks induced by the absorption processes between ALPs/DPs and atomic electrons of xenon. A detailed temporal model of decays associated with xenon isotopes is introduced to constrain the number of background events. No signal excess over background expectations is observed, and we have established the most stringent exclusion limits for most ALP/DP masses ranging from 150 keV/$c^2$ to 1 MeV/$c^2$.

Virtual reality (VR) apps can harvest a wider range of user data than web/mobile apps running on personal computers or smartphones. Existing law and privacy regulations emphasize that VR developers should inform users of what data are collected/used/shared (CUS) through privacy policies. However, privacy policies in the VR ecosystem are still in their early stages, and many developers fail to write appropriate privacy policies that comply with regulations and meet user expectations. In this paper, we propose VPVet to automatically vet privacy policy compliance issues for VR apps. VPVet first analyzes the availability and completeness of a VR privacy policy and then refines its analysis based on three key criteria: granularity, minimization, and consistency of CUS statements. Our study establishes the first and currently largest VR privacy policy dataset named VRPP, consisting of privacy policies of 11,923 different VR apps from 10 mainstream platforms. Our vetting results reveal severe privacy issues within the VR ecosystem, including the limited availability and poor quality of privacy policies, along with their coarse granularity, lack of adaptation to VR traits and the inconsistency between CUS statements in privacy policies and their actual behaviors. We open-source VPVet system along with our findings at repository https://github.com/kalamoo/PPAudit, aiming to raise awareness within the VR community and pave the way for further research in this field., Comment: 18 pages, 13 figures (including subfigures), 13 tables. To appear on ACM CCS 2024

Targeted community detection aims to distinguish a particular type of community in the network. This is an important task with a lot of real-world applications, e.g., identifying fraud groups in transaction networks. Traditional community detection methods fail to capture the specific features of the targeted community and detect all types of communities indiscriminately. Semi-supervised community detection algorithms, emerged as a feasible alternative, are inherently constrained by their limited adaptability and substantial reliance on a large amount of labeled data, which demands extensive domain knowledge and manual effort. In this paper, we address the aforementioned weaknesses in targeted community detection by focusing on few-shot scenarios. We propose ProCom, a novel framework that extends the ``pre-train, prompt'' paradigm, offering a low-resource, high-efficiency, and transferable solution. Within the framework, we devise a dual-level context-aware pre-training method that fosters a deep understanding of latent communities in the network, establishing a rich knowledge foundation for downstream task. In the prompt learning stage, we reformulate the targeted community detection task into pre-training objectives, allowing the extraction of specific knowledge relevant to the targeted community to facilitate effective and efficient inference. By leveraging both the general community knowledge acquired during pre-training and the specific insights gained from the prompt communities, ProCom exhibits remarkable adaptability across different datasets. We conduct extensive experiments on five benchmarks to evaluate the ProCom framework, demonstrating its SOTA performance under few-shot scenarios, strong efficiency, and transferability across diverse datasets., Comment: Accepted by SIGKDD'2024

Data science and engineering workflows often span multiple stages, from warehousing to orchestration, using tools like BigQuery, dbt, and Airbyte. As vision language models (VLMs) advance in multimodal understanding and code generation, VLM-based agents could potentially automate these workflows by generating SQL queries, Python code, and GUI operations. This automation can improve the productivity of experts while democratizing access to large-scale data analysis. In this paper, we introduce Spider2-V, the first multimodal agent benchmark focusing on professional data science and engineering workflows, featuring 494 real-world tasks in authentic computer environments and incorporating 20 enterprise-level professional applications. These tasks, derived from real-world use cases, evaluate the ability of a multimodal agent to perform data-related tasks by writing code and managing the GUI in enterprise data software systems. To balance realistic simulation with evaluation simplicity, we devote significant effort to developing automatic configurations for task setup and carefully crafting evaluation metrics for each task. Furthermore, we supplement multimodal agents with comprehensive documents of these enterprise data software systems. Our empirical evaluation reveals that existing state-of-the-art LLM/VLM-based agents do not reliably automate full data workflows (14.0% success). Even with step-by-step guidance, these agents still underperform in tasks that require fine-grained, knowledge-intensive GUI actions (16.2%) and involve remote cloud-hosted workspaces (10.6%). We hope that Spider2-V paves the way for autonomous multimodal agents to transform the automation of data science and engineering workflow. Our code and data are available at https://spider2-v.github.io., Comment: 34 pages, 14 figures, 10 tables

The progenitors of gamma-ray bursts (GRBs) have long been an unresolved issue. GRB 230307A stands out as an exceptionally bright event, belonging to the long-duration GRBs but also exhibiting a late emission component reminiscent of a kilonova. Together with the similar events GRBs 060614 and 211211A, they make up a new sub-group of GRBs with intriguing progenitors. If such long-duration merger-type GRBs originated from the coalescence of a white dwarf (WD) with a neutron star (NS) or a black hole (BH), as proposed in the recent literature, then the larger tidal disruption radius of the WD, together with a non-negligible residual orbital eccentricity, would make repeated partial tidal disruptions inevitable. This may modulate the mass accretion and jet launching process at the NS or BH, resulting in a quasi-periodic modulation (QPM) in the light curve of the GRB, on the orbital period. The detection of potential QPMs during the early episode of prompt emission of these three GRBs supports this scenario, and the relatively slow QPM ($>$ 1 s) suggests that the lighter object can not be a NS. We propose that the progenitor system of GRBs 230307A, 060614, and 211211A consist of a WD of mass 1.3 $M_\odot$, 0.9 $M_\odot$ and 1.4 $M_\odot$, respectively, and a NS (or BH). After several cycles of modulations, the WD is completely destructed, and the accretion of the remaining debris dominates the extended emission episode., Comment: 13 pages, 1 appendix, accepted for publication in ApJL

Global localization using onboard perception sensors, such as cameras and LiDARs, is crucial in autonomous driving and robotics applications when GPS signals are unreliable. Most approaches achieve global localization by sequential place recognition (PR) and pose estimation (PE). Some methods train separate models for each task, while others employ a single model with dual heads, trained jointly with separate task-specific losses. However, the accuracy of localization heavily depends on the success of place recognition, which often fails in scenarios with significant changes in viewpoint or environmental appearance. Consequently, this renders the final pose estimation of localization ineffective. To address this, we introduce a new paradigm, PR-by-PE localization, which bypasses the need for separate place recognition by directly deriving it from pose estimation. We propose RING#, an end-to-end PR-by-PE localization network that operates in the bird's-eye-view (BEV) space, compatible with both vision and LiDAR sensors. RING# incorporates a novel design that learns two equivariant representations from BEV features, enabling globally convergent and computationally efficient pose estimation. Comprehensive experiments on the NCLT and Oxford datasets show that RING# outperforms state-of-the-art methods in both vision and LiDAR modalities, validating the effectiveness of the proposed approach. The code will be publicly released., Comment: 25 pages, 21 figures

Recognizing overlapping speech from multiple speakers in conversational scenarios is one of the most challenging problem for automatic speech recognition (ASR). Serialized output training (SOT) is a classic method to address multi-talker ASR, with the idea of concatenating transcriptions from multiple speakers according to the emission times of their speech for training. However, SOT-style transcriptions, derived from concatenating multiple related utterances in a conversation, depend significantly on modeling long contexts. Therefore, compared to traditional methods that primarily emphasize encoder performance in attention-based encoder-decoder (AED) architectures, a novel approach utilizing large language models (LLMs) that leverages the capabilities of pre-trained decoders may be better suited for such complex and challenging scenarios. In this paper, we propose an LLM-based SOT approach for multi-talker ASR, leveraging pre-trained speech encoder and LLM, fine-tuning them on multi-talker dataset using appropriate strategies. Experimental results demonstrate that our approach surpasses traditional AED-based methods on the simulated dataset LibriMix and achieves state-of-the-art performance on the evaluation set of the real-world dataset AMI, outperforming the AED model trained with 1000 times more supervised data in previous works., Comment: 8 pages, accepted by IEEE SLT 2024

With advancements in video generative AI models (e.g., SORA), creators are increasingly using these techniques to enhance video previsualization. However, they face challenges with incomplete and mismatched AI workflows. Existing methods mainly rely on text descriptions and struggle with camera placement, a key component of previsualization. To address these issues, we introduce CinePreGen, a visual previsualization system enhanced with engine-powered diffusion. It features a novel camera and storyboard interface that offers dynamic control, from global to local camera adjustments. This is combined with a user-friendly AI rendering workflow, which aims to achieve consistent results through multi-masked IP-Adapter and engine simulation guidelines. In our comprehensive evaluation study, we demonstrate that our system reduces development viscosity (i.e., the complexity and challenges in the development process), meets users' needs for extensive control and iteration in the design process, and outperforms other AI video production workflows in cinematic camera movement, as shown by our experiments and a within-subjects user study. With its intuitive camera controls and realistic rendering of camera motion, CinePreGen shows great potential for improving video production for both individual creators and industry professionals.

Explainable Artificial Intelligence (XAI) plays a crucial role in fostering transparency and trust in AI systems, where traditional XAI approaches typically offer one level of abstraction for explanations, often in the form of heatmaps highlighting single or multiple input features. However, we ask whether abstract reasoning or problem-solving strategies of a model may also be relevant, as these align more closely with how humans approach solutions to problems. We propose a framework, called Symbolic XAI, that attributes relevance to symbolic queries expressing logical relationships between input features, thereby capturing the abstract reasoning behind a model's predictions. The methodology is built upon a simple yet general multi-order decomposition of model predictions. This decomposition can be specified using higher-order propagation-based relevance methods, such as GNN-LRP, or perturbation-based explanation methods commonly used in XAI. The effectiveness of our framework is demonstrated in the domains of natural language processing (NLP), vision, and quantum chemistry (QC), where abstract symbolic domain knowledge is abundant and of significant interest to users. The Symbolic XAI framework provides an understanding of the model's decision-making process that is both flexible for customization by the user and human-readable through logical formulas.

This paper provides the background, methodology and validation for constructing a representation of the European high-voltage grid, including and above 200 kV, based on public data provided by OpenStreetMap. The model-independent grid dataset is published under the Open Data Commons Open Database (ODbL 1.0) licence and can be used for large-scale electricity as well as energy system modelling. The dataset and workflow are provided as part of PyPSA-Eur -- an open-source, sector-coupled optimisation model of the European energy system. By integrating with the codebase for initiatives such as PyPSA-Earth, the value of open and maintainable high-voltage grid data extends to the global context. By accessing the latest data through the the Overpass turbo API, the dataset can be easily reconstructed and updated within minutes. To assess the data quality, this paper further compares the dataset with official statistics and representative model runs using PyPSA-Eur based on different electricity grid representations., Comment: 20 pages, 15 figures, 8 tables. For associated prebuilt electricity network, see https://doi.org/10.5281/zenodo.13358976

The Muonium-to-Antimuonium Conversion Experiment (MACE) is proposed to search for charged lepton flavor violation and increase the sensitivity by three orders of magnitude compared to the PSI experiment in the 1990s. A clear signature of this conversion is the positron produced from antimuonium decay. This paper presents a near-$4\pi$-coverage calorimeter designed for MACE, which can provide an energy resolution of 9% at 511 keV, and a signal efficiency of 55.5% for annihilation $\gamma$-ray events at maximum. Detailed Monte-Carlo simulations using MACE offline software based on \textsc{Geant4} are performed for geometry optimization, coincidence system design, background estimation, and benchmark detector validation., Comment: 9 pages, 8 figures

This paper investigates a zero-sum stochastic linear-quadratic (SLQ, for short) Stackelberg differential game problem, where the coefficients of the state equation and the weighting matrices in the performance functional are regulated by a Markov chain. By utilizing the findings in \citet{Zhang.X.2021_ILQM}, we directly present the feedback representation to the rational reaction of the follower. For the leader's problem, we derive the optimality system through the variational method and study its unique solvability from the Hilbert space point of view. We construct the explicit optimal control for the leader based on the solution to coupled differential Riccati equations (CDREs, for short) and obtain the solvability of CDREs under the one-dimensional framework. Finally, we provide two concrete examples to illustrate the results developed in this paper.

Binary stars are responsible for many unusual astrophysical phenomena, including some important explosive cosmic events. The stability criteria for rapid mass transfer and common-envelope evolution are fundamental to binary star evolution. They determine the mass, mass ratio, and orbital distribution of systems such as X-ray binaries and merging gravitational-wave sources. We use our adiabatic mass-loss model to systematically survey metal-poor and solar-metallicity donor thresholds for dynamical timescale mass transfer. The critical mass ratios qad are systematically explored, and the impact of metallicity and nonconservative mass transfer are studied. For metal-poor radiative-envelope donors, qad are smaller than those for solar-metallicity stars at the same evolutionary stage. However, qad do the opposite for convective-envelope donors. Nonconservative mass transfer significantly decreases qad for massive donors. This is because it matters how conservative mass transfer is during the thermal timescale phase immediately preceding a delayed dynamical mass transfer. We apply our theoretical predictions to observed high-mass X-ray binaries that have overfilled their Roche lobes and find a good agreement with their mass ratios. Our results can be applied to study individual binary objects or large samples of binary objects with binary population synthesis codes., Comment: Submitted to ApJ. Comments are welcome

Several software mitigations have been proposed to defend against Spectre vulnerabilities. However, these countermeasures often suffer from high performance overhead, largely due to unnecessary protections. We propose LightSLH, designed to mitigate this overhead by hardening instructions only when they are under threat from Spectre vulnerabilities. LightSLH leverages program analysis techniques based on abstract interpretation to identify all instructions that could potentially lead to Spectre vulnerabilities and provides provable protection. To enhance analysis efficiency and precision, LightSLH employs novel taint and value domains. The taint domain enables bit-level taint tracking, while the value domain allows LightSLH to analyze complex program structures such as pointers and structures. Furthermore, LightSLH uses a two-stage abstract interpretation approach to circumvent potential analysis paralysis issues. We demonstrate the security guarantees of LightSLH and evaluate its performance on cryptographic algorithm implementations from OpenSSL. LightSLH significantly reduces the overhead associated with speculative-load-hardening techniques. Our results show that LightSLH introduces no protection and thus no overhead on 4 out of the 7 studied algorithms, which contrasts with existing countermeasures that introduce additional overhead due to unnecessary hardening. Additionally, LightSLH performs, for the first time, a rigorous analysis of the security guarantees of RSA against Spectre v1, highlighting that the memory access patterns generated by the scatter-gather algorithm depend on secrets, even for observers at the cache line granularity, necessitating protection for such accesses.

Time series forecasting requires balancing short-term and long-term dependencies for accurate predictions. Existing methods mainly focus on long-term dependency modeling, neglecting the complexities of short-term dynamics, which may hinder performance. Transformers are superior in modeling long-term dependencies but are criticized for their quadratic computational cost. Mamba provides a near-linear alternative but is reported less effective in time series longterm forecasting due to potential information loss. Current architectures fall short in offering both high efficiency and strong performance for long-term dependency modeling. To address these challenges, we introduce Mixture of Universals (MoU), a versatile model to capture both short-term and long-term dependencies for enhancing performance in time series forecasting. MoU is composed of two novel designs: Mixture of Feature Extractors (MoF), an adaptive method designed to improve time series patch representations for short-term dependency, and Mixture of Architectures (MoA), which hierarchically integrates Mamba, FeedForward, Convolution, and Self-Attention architectures in a specialized order to model long-term dependency from a hybrid perspective. The proposed approach achieves state-of-the-art performance while maintaining relatively low computational costs. Extensive experiments on seven real-world datasets demonstrate the superiority of MoU. Code is available at https://github.com/lunaaa95/mou/., Comment: Code at https://github.com/lunaaa95/mou/

The rapid obsolescence of information in Large Language Models (LLMs) has driven the development of various techniques to incorporate new facts. However, existing methods for knowledge editing still face difficulties with multi-hop questions that require accurate fact identification and sequential logical reasoning, particularly among numerous fact updates. To tackle these challenges, this paper introduces Graph Memory-based Editing for Large Language Models (GMeLLo), a straitforward and effective method that merges the explicit knowledge representation of Knowledge Graphs (KGs) with the linguistic flexibility of LLMs. Beyond merely leveraging LLMs for question answering, GMeLLo employs these models to convert free-form language into structured queries and fact triples, facilitating seamless interaction with KGs for rapid updates and precise multi-hop reasoning. Our results show that GMeLLo significantly surpasses current state-of-the-art knowledge editing methods in the multi-hop question answering benchmark, MQuAKE, especially in scenarios with extensive knowledge edits.

Backdoor attacks present a serious security threat to deep neuron networks (DNNs). Although numerous effective defense techniques have been proposed in recent years, they inevitably rely on the availability of either clean or poisoned data. In contrast, data-free defense techniques have evolved slowly and still lag significantly in performance. To address this issue, different from the traditional approach of pruning followed by fine-tuning, we propose a novel data-free defense method named Optimal Transport-based Backdoor Repairing (OTBR) in this work. This method, based on our findings on neuron weight changes (NWCs) of random unlearning, uses optimal transport (OT)-based model fusion to combine the advantages of both pruned and backdoored models. Specifically, we first demonstrate our findings that the NWCs of random unlearning are positively correlated with those of poison unlearning. Based on this observation, we propose a random-unlearning NWC pruning technique to eliminate the backdoor effect and obtain a backdoor-free pruned model. Then, motivated by the OT-based model fusion, we propose the pruned-to-backdoored OT-based fusion technique, which fuses pruned and backdoored models to combine the advantages of both, resulting in a model that demonstrates high clean accuracy and a low attack success rate. To our knowledge, this is the first work to apply OT and model fusion techniques to backdoor defense. Extensive experiments show that our method successfully defends against all seven backdoor attacks across three benchmark datasets, outperforming both state-of-the-art (SOTA) data-free and data-dependent methods. The code implementation and Appendix are provided in the Supplementary Material.

We calculate the amplitudes of the axion photoproduction off the nucleon, i.e., $\gamma N \to a N$, within the framework of chiral effective field theory. Several different types of contributions are simultaneously included in our calculation, namely the nucleon exchanges up to next-to-leading order, the $a\gamma\gamma$ vertex and the vector meson exchanges in the $t$-channel. We utilize the existing hadronic inputs as much as possible to fix the unknown couplings. A comprehensive study of the phenomenological discussions is then provided in this work. Different mechanisms in the $\gamma N \to a N$ processes manifest distinct behaviors in the total and differential cross sections, which could provide useful quantities to distinguish different axion models., Comment: 25 pages, 9 figures

Chiral superconductors are unconventional superconducting states that break time reversal symmetry spontaneously and typically feature Cooper pairing at non-zero angular momentum. Such states may host Majorana fermions and provide an important platform for topological physics research and fault-tolerant quantum computing. Despite of intensive search and prolonged studies of several candidate systems, chiral superconductivity has remained elusive so far. Here we report the discovery of unconventional superconductivity in rhombohedral tetra-layer graphene. We observed two superconducting states in the gate-induced flat conduction bands with Tc up to 300 mK and charge density ne as low as 2.4*1011 cm-2, appearing robustly in three different devices, where electrons reside close to a proximate WSe2 layer, far away from WSe2, and in the absence of WSe2 respectively. Spontaneous time-reversal-symmetry-breaking (TRSB) due to electron's orbital motion is found, and several observations indicate the chiral nature of these superconducting states, including 1. In the superconducting state, Rxx shows fluctuations at zero magnetic field and magnetic hysteresis versus an out-of-plane magnetic field B, which are absent from all other superconductors; 2. one superconducting state develops within a spin- and valley-polarized quarter-metal phase, and is robust against the neighboring spin-valley-polarized quarter-metal state under B; 3. the normal states show anomalous Hall signals at zero magnetic field and magnetic hysteresis. We also observed a critical B > 0.9 Tesla, higher than any graphene superconductivity reported so far and indicates a strong-coupling superconductivity close the BCS-BEC crossover. Our observations establish a pure carbon material for the study of topological superconductivity, and pave the way to explore Majorana modes and topological quantum computing.