Search

Foley is a term commonly used in filmmaking, referring to the addition of daily sound effects to silent films or videos to enhance the auditory experience. Video-to-Audio (V2A), as a particular type of automatic foley task, presents inherent challenges related to audio-visual synchronization. These challenges encompass maintaining the content consistency between the input video and the generated audio, as well as the alignment of temporal and loudness properties within the video. To address these issues, we construct a controllable video-to-audio synthesis model, termed Draw an Audio, which supports multiple input instructions through drawn masks and loudness signals. To ensure content consistency between the synthesized audio and target video, we introduce the Mask-Attention Module (MAM), which employs masked video instruction to enable the model to focus on regions of interest. Additionally, we implement the Time-Loudness Module (TLM), which uses an auxiliary loudness signal to ensure the synthesis of sound that aligns with the video in both loudness and temporal dimensions. Furthermore, we have extended a large-scale V2A dataset, named VGGSound-Caption, by annotating caption prompts. Extensive experiments on challenging benchmarks across two large-scale V2A datasets verify Draw an Audio achieves the state-of-the-art. Project page: https://yannqi.github.io/Draw-an-Audio/., Comment: 14 pages, 11 figures

Recent advances in large language models (LLMs) have shown significant potential to automate various software development tasks, including code completion, test generation, and bug fixing. However, the application of LLMs for automated bug fixing remains challenging due to the complexity and diversity of real-world software systems. In this paper, we introduce MarsCode Agent, a novel framework that leverages LLMs to automatically identify and repair bugs in software code. MarsCode Agent combines the power of LLMs with advanced code analysis techniques to accurately localize faults and generate patches. Our approach follows a systematic process of planning, bug reproduction, fault localization, candidate patch generation, and validation to ensure high-quality bug fixes. We evaluated MarsCode Agent on SWE-bench, a comprehensive benchmark of real-world software projects, and our results show that MarsCode Agent achieves a high success rate in bug fixing compared to most of the existing automated approaches., Comment: Yizhou Liu and Pengfei Gao contributed equally and the order is determined by rolling the dice. Chao Peng is the corresponding author

With the growing reliance on automated code completion tools in software development, the need for robust evaluation benchmarks has become critical. However, existing benchmarks focus more on code generation tasks in function and class level and provide rich text description to prompt the model. By contrast, such descriptive prompt is commonly unavailable in real development and code completion can occur in wider range of situations such as in the middle of a function or a code block. These limitations makes the evaluation poorly align with the practical scenarios of code completion tools. In this paper, we propose RepoMasterEval, a novel benchmark for evaluating code completion models constructed from real-world Python and TypeScript repositories. Each benchmark datum is generated by masking a code snippet (ground truth) from one source code file with existing test suites. To improve test accuracy of model generated code, we employ mutation testing to measure the effectiveness of the test cases and we manually crafted new test cases for those test suites with low mutation score. Our empirical evaluation on 6 state-of-the-art models shows that test argumentation is critical in improving the accuracy of the benchmark and RepoMasterEval is able to report difference in model performance in real-world scenarios. The deployment of RepoMasterEval in a collaborated company for one month also revealed that the benchmark is useful to give accurate feedback during model training and the score is in high correlation with the model's performance in practice. Based on our findings, we call for the software engineering community to build more LLM benchmarks tailored for code generation tools taking the practical and complex development environment into consideration.

Recently, an audio-visual segmentation (AVS) task has been introduced, aiming to group pixels with sounding objects within a given video. This task necessitates a first-ever audio-driven pixel-level understanding of the scene, posing significant challenges. In this paper, we propose an innovative audio-visual transformer framework, termed COMBO, an acronym for COoperation of Multi-order Bilateral relatiOns. For the first time, our framework explores three types of bilateral entanglements within AVS: pixel entanglement, modality entanglement, and temporal entanglement. Regarding pixel entanglement, we employ a Siam-Encoder Module (SEM) that leverages prior knowledge to generate more precise visual features from the foundational model. For modality entanglement, we design a Bilateral-Fusion Module (BFM), enabling COMBO to align corresponding visual and auditory signals bi-directionally. As for temporal entanglement, we introduce an innovative adaptive inter-frame consistency loss according to the inherent rules of temporal. Comprehensive experiments and ablation studies on AVSBench-object (84.7 mIoU on S4, 59.2 mIou on MS3) and AVSBench-semantic (42.1 mIoU on AVSS) datasets demonstrate that COMBO surpasses previous state-of-the-art methods. Code and more results will be publicly available at https://yannqi.github.io/AVS-COMBO/., Comment: CVPR 2024 Highlight. 13 pages, 10 figures

Fault injection attacks represent a type of active, physical attack against cryptographic circuits. Various countermeasures have been proposed to thwart such attacks, the design and implementation of which are, however, intricate, error-prone, and laborious. The current formal fault-resistance verification approaches are limited in efficiency and scalability. In this paper, we formalize the fault-resistance verification problem which is shown to be NP-complete. We then devise a novel approach for encoding the fault-resistance verification problem as the Boolean satisfiability (SAT) problem so that off-the-shelf SAT solvers can be utilized. The approach is implemented in an open-source tool FIRMER which is evaluated extensively on realistic cryptographic circuit benchmarks. The experimental results show that FIRMER is able to verify fault-resistance of almost all (46/48) benchmarks in 3 minutes (the other two are verified in 35 minutes). In contrast, the prior approach fails on 23 fault-resistance verification tasks even after 24 hours (per task).

In this paper, we present the Multi-Forgery Detection Challenge held concurrently with the IEEE Computer Society Workshop on Biometrics at CVPR 2022. Our Multi-Forgery Detection Challenge aims to detect automatic image manipulations including but not limited to image editing, image synthesis, image generation, image photoshop, etc. Our challenge has attracted 674 teams from all over the world, with about 2000 valid result submission counts. We invited the Top 10 teams to present their solutions to the challenge, from which three teams are awarded prizes in the grand finale. In this paper, we present the solutions from the Top 3 teams, in order to boost the research work in the field of image forgery detection., Comment: Workshop and challenge summary paper, containing technical reports from different teams

Many important classification problems in the real-world consist of a large number of closely related categories in a hierarchical structure or taxonomy. Hierarchical multi-label text classification (HMTC) with higher accuracy over large sets of closely related categories organized in a hierarchy or taxonomy has become a challenging problem. In this paper, we present a hierarchical and fine-tuning approach based on the Ordered Neural LSTM neural network, abbreviated as HFT-ONLSTM, for more accurate level-by-level HMTC. First, we present a novel approach to learning the joint embeddings based on parent category labels and textual data for accurately capturing the joint features of both category labels and texts. Second, a fine tuning technique is adopted for training parameters such that the text classification results in the upper level should contribute to the classification in the lower one. At last, the comprehensive analysis is made based on extensive experiments in comparison with the state-of-the-art hierarchical and flat multi-label text classification approaches over two benchmark datasets, and the experimental results show that our HFT-ONLSTM approach outperforms these approaches, in particular reducing computational costs while achieving superior performance., Comment: 31 pages, 8 tables, 4 figures

Room temperature two-dimensional (2D) ferromagnetism is highly desired in practical spintronics applications. Recently, 1T phase CrTe2 (1T-CrTe2) nanosheets with five and thicker layers have been successfully synthesized, which all exhibit the properties of ferromagnetic (FM) metals with Curie temperatures around 305 K. However, whether the ferromagnetism therein can be maintained when continuously reducing the nanosheet's thickness to monolayer limit remains unknown. Here, through first-principles calculations, we explore the evolution of magnetic properties of 1 to 6 layers CrTe2 nanosheets and several interesting points are found: First, unexpectedly, monolayer CrTe2 prefers a zigzag antiferromagnetic (AFM) state with its energy much lower than that of FM state. Second, in 2 to 4 layers CrTe2, both the intralayer and interlayer magnetic coupling are AFM. Last, when the number of layers is equal to or greater than five, the intralayer and interlayer magnetic coupling become FM. Theoretical analysis reveals that the in-plane lattice contraction of few layer CrTe2 compared to bulk is the main factor producing intralayer AFM-FM magnetic transition. At the same time, as long as the intralayer coupling gets FM, the interlayer coupling will concomitantly switch from AFM to FM. Such highly thickness dependent magnetism provides a new perspective to control the magnetic properties of 2D materials., Comment: 4 pages, 3 figures

As facial interaction systems are prevalently deployed, security and reliability of these systems become a critical issue, with substantial research efforts devoted. Among them, face anti-spoofing emerges as an important area, whose objective is to identify whether a presented face is live or spoof. Recently, a large-scale face anti-spoofing dataset, CelebA-Spoof which comprised of 625,537 pictures of 10,177 subjects has been released. It is the largest face anti-spoofing dataset in terms of the numbers of the data and the subjects. This paper reports methods and results in the CelebA-Spoof Challenge 2020 on Face AntiSpoofing which employs the CelebA-Spoof dataset. The model evaluation is conducted online on the hidden test set. A total of 134 participants registered for the competition, and 19 teams made valid submissions. We will analyze the top ranked solutions and present some discussion on future work directions., Comment: Technical report. Challenge website: https://competitions.codalab.org/competitions/26210

Side-channel attacks, which are capable of breaking secrecy via side-channel information, pose a growing threat to the implementation of cryptographic algorithms. Masking is an effective countermeasure against side-channel attacks by removing the statistical dependence between secrecy and power consumption via randomization. However, designing efficient and effective masked implementations turns out to be an error-prone task. Current techniques for verifying whether masked programs are secure are limited in their applicability and accuracy, especially when they are applied. To bridge this gap, in this article, we first propose a sound type system, equipped with an efficient type inference algorithm, for verifying masked arithmetic programs against higher-order attacks. We then give novel model-counting based and pattern-matching based methods which are able to precisely determine whether the potential leaky observable sets detected by the type system are genuine or simply spurious. We evaluate our approach on various implementations of arithmetic cryptographicprograms.The experiments confirm that our approach out performs the state-of-the-art base lines in terms of applicability, accuracy and efficiency.

JavaScript has become one of the most widely used languages for Web development. However, it is challenging to ensure the correctness and reliability of Web applications written in JavaScript, due to their dynamic and event-driven features. A variety of dynamic analysis techniques for JavaScript Web applications have been proposed, but they are limited in either coverage or scalability. In this paper, we propose a model-based automated approach to achieve high code coverage in a reasonable amount of time via testing with longer event sequences. We implement our approach as the tool LJS, and perform extensive experiments on 21 publicly available benchmarks (18,559 lines of code in total). On average, LJS achieves 86.4\% line coverage in 10 minutes, which is 5.4\% higher than that of JSDep, a breadth-first search based automated testing tool enriched with partial order reduction. In particular, on large applications, the coverage of LJS is 11-18\% higher than that of JSDep. Our empirical finding supports that longer test sequences can achieve higher code coverage in JavsScript testing.

Recently we have reported the direct observation of two-dimensional (2D) Ca-Cl crystals on reduced graphene oxide (rGO) membranes, in which the calcium ions are only about monovalent (i.e. ~+1) and metallic rather than insulating properties are displayed by those CaCl crystals. Here, we report the experimental observation and demonstration of the formation of graphene-Ca-Cl heterojunction owing to the metallicity of 2D Ca-Cl crystals, unexpected piezoelectric effect, room-temperature ferromagnetism, as well as the distinct hydrogen storage and release capability of the Ca-Cl crystals in rGO membranes. Theoretical studies show that the formation of those abnormal crystals is attributed to the strong cation-pi interactions of the Ca2+ with the aromatic rings in the graphitic surfaces. Since strong cation-pi interactions also exist between other metal ions (such as Mg2+, Fe2+, Co2+, Cu2+, Cd2+, Cr2+ and Pb2+) and graphitic surfaces, similar 2D crystals with abnormal valence state of the metal cations and corresponding abnormal properties as well as novel applications are highly expected. Those findings further show the realistically potential applications of such abnormal CaCl material with unusual electronic properties in designing novel transistors and magnetic devices, hydrogen storage, catalyzer, high-performance conducting electrodes and sensors, with a size down to atomic scale., Comment: 7 pages, 4 figures. arXiv admin note: text overlap with arXiv:1812.07195

Power side-channel attacks, which can deduce secret data via statistical analysis, have become a serious threat. Masking is an effective countermeasure for reducing the statistical dependence between secret data and side-channel information. However, designing masking algorithms is an error-prone process. In this paper, we propose a hybrid approach combing type inference and model-counting to verify masked arithmetic programs against side-channel attacks. The type inference allows an efficient, lightweight procedure to determine most observable variables whereas model-counting accounts for completeness. In case that the program is not perfectly masked, we also provide a method to quantify the security level of the program. We implement our methods in a tool QMVerif and evaluate it on cryptographic benchmarks. The experimental results show the effectiveness and efficiency of our approach., Comment: 19 pages

Recently, we report the direct observation, under ambient conditions, of Na2Cl and Na3Cl as two-dimensional (2D) Na-Cl crystals, together with regular NaCl, on reduced graphene oxide membranes and on the surfaces of natural graphite powders from salt solutions far below the saturated concentration. However, what are these abnormal stoichiometries for high valence ions, such as calcium ions and copper ions still remain unknown. Here, using cryo-electron microscopy, we report the direct observation of two-dimensional (2D) Ca-Cl crystals on reduced graphene oxide (rGO) membranes, in which the calcium ions are only monovalent (i.e. +1). Remarkably, metallic properties rather than insulating are displayed by those CaCl crystals. We note that such CaCl crystals are obtained by simply incubating rGO membranes in salt solutions below the saturated concentration, under ambient conditions. Theoretical studies show that the formation of those abnormal crystals is attributed to the strong cation-pi interactions of the Ca2+ ions with the aromatic rings in the graphitic surfaces. Since strong cation-pi interactions also exist between other metal ions (such as Mg2+, Fe2+, Co2+, Cu2+, Cd2+, Cr2+ and Pb2+) and graphitic surfaces, similar 2D crystals with abnormal valence state of the metal cations and corresponding abnormal properties are highly expected. The 2D crystals with monovalent calcium ions show unusual electronic properties, and can be applicated in catalyzer, hydrogen storage, high-performance conducting electrodes and sensors. These findings also produce functionalized graphene including compact "graphene-metallic CaCl-insulating CaCl2" junction that can serve as transistors down to the atomic scale, and other devices for magnetic, optical and mechanical applications., Comment: 10 pages, 3 figures