Search

With the growing reliance on automated code completion tools in software development, the need for robust evaluation benchmarks has become critical. However, existing benchmarks focus more on code generation tasks in function and class level and provide rich text description to prompt the model. By contrast, such descriptive prompt is commonly unavailable in real development and code completion can occur in wider range of situations such as in the middle of a function or a code block. These limitations makes the evaluation poorly align with the practical scenarios of code completion tools. In this paper, we propose RepoMasterEval, a novel benchmark for evaluating code completion models constructed from real-world Python and TypeScript repositories. Each benchmark datum is generated by masking a code snippet (ground truth) from one source code file with existing test suites. To improve test accuracy of model generated code, we employ mutation testing to measure the effectiveness of the test cases and we manually crafted new test cases for those test suites with low mutation score. Our empirical evaluation on 6 state-of-the-art models shows that test argumentation is critical in improving the accuracy of the benchmark and RepoMasterEval is able to report difference in model performance in real-world scenarios. The deployment of RepoMasterEval in a collaborated company for one month also revealed that the benchmark is useful to give accurate feedback during model training and the score is in high correlation with the model's performance in practice. Based on our findings, we call for the software engineering community to build more LLM benchmarks tailored for code generation tools taking the practical and complex development environment into consideration.

Recent years have seen the development of LLM-based code generation. Compared to generating code in a software project, incremental code edits are empirically observed to be more frequent. The emerging code editing approaches usually formulate the problem as generating an edit based on known relevant prior edits and context. However, practical code edits can be more complicated. First, an editing session can include multiple (ir)relevant edits to the code under edit. Second, the inference of the subsequent edits is non-trivial as the scope of its ripple effect can be the whole project. In this work, we propose CoEdPilot, an LLM-driven solution to recommend code edits by discriminating the relevant edits, exploring their interactive natures, and estimating its ripple effect in the project. Specifically, CoEdPilot orchestrates multiple neural transformers to identify what and how to edit in the project regarding both edit location and edit content. When a user accomplishes an edit with an optional editing description, a Subsequent Edit Analysis first reports the most relevant files in the project with what types of edits (e.g., keep, insert, and replace) can happen for each line of their code. Next, an Edit-content Generator generates concrete edit options for the lines of code, regarding its relevant prior changes reported by an Edit-dependency Analyzer. Lastly, both the Subsequent Edit Analysis and the Edit-content Generator capture relevant prior edits as feedback to readjust their recommendations. We train our models by collecting over 180K commits from 471 open-source projects in 5 programming languages. Our extensive experiments show that CoEdPilot can well predict the edits (i.e., predicting edit location with an accuracy of 70.8%-85.3%, and the edit content with an exact match rate of 41.8% and BLEU4 score of 60.7)..., Comment: 13 pages, 7 figures

Significant research has focused on improving the performance of large language model on code-related tasks due to their practical importance. Although performance is typically evaluated using public benchmark datasets, the existing datasets do not account for the concept of \emph{version}, which is crucial in professional software development. In this paper, we introduce VersiCode, the first comprehensive dataset designed to assess the ability of large language models to generate verifiable code for specific library versions. VersiCode encompasses 300 libraries across more than 2,000 versions spanning 9 years. We design two dedicated evaluation tasks: version-specific code completion (VSCC) and version-aware code editing (VACE). Comprehensive experiments are conducted to benchmark the performance of LLMs, revealing the challenging nature of these tasks and VersiCode, that even state-of-the-art LLMs struggle to generate version-correct code. This dataset, together with the proposed tasks, sheds light on LLMs' capabilities and limitations in handling version-specific code generation, and opens up an important new area of research for further investigation. The resources can be found at https://github.com/wutong8023/VersiCode.

In spectral CT reconstruction, the basis materials decomposition involves solving a large-scale nonlinear system of integral equations, which is highly ill-posed mathematically. This paper proposes a model that parameterizes the attenuation coefficients of the object using a neural field representation, thereby avoiding the complex calculations of pixel-driven projection coefficient matrices during the discretization process of line integrals. It introduces a lightweight discretization method for line integrals based on a ray-driven neural field, enhancing the accuracy of the integral approximation during the discretization process. The basis materials are represented as continuous vector-valued implicit functions to establish a neural field parameterization model for the basis materials. The auto-differentiation framework of deep learning is then used to solve the implicit continuous function of the neural base-material fields. This method is not limited by the spatial resolution of reconstructed images, and the network has compact and regular properties. Experimental validation shows that our method performs exceptionally well in addressing the spectral CT reconstruction. Additionally, it fulfils the requirements for the generation of high-resolution reconstruction images., Comment: 14 pages,16 figures

Wireless communication is rapidly evolving, and future wireless communications (6G and beyond) will be more heterogeneous, multi-layered, and complex, which poses challenges to traditional communications. Adaptive technologies in traditional communication systems respond to environmental changes by modifying system parameters and structures on their own and are not flexible and agile enough to satisfy requirements in future communications. To tackle these challenges, we propose a novel self-evolving communication framework, which consists of three layers: data layer, information layer, and knowledge layer. The first two layers allow communication systems to sense environments, fuse data, and generate a knowledge base for the knowledge layer. When dealing with a variety of application scenarios and environments, the generated knowledge is subsequently fed back to the first two layers for communication in practical application scenarios to obtain self-evolving ability and enhance the robustness of the system. In this paper, we first highlight the limitations of current adaptive communication systems and the need for intelligence, automation, and self-evolution in future wireless communications. We overview the development of self-evolving technologies and conceive the concept of self-evolving communications with its hypothetical architecture. To demonstrate the power of self-evolving modules, we compare the performances of a communication system with and without evolution. We then provide some potential techniques that enable self-evolving communications and challenges in implementing them.

Recent advancements in large language models (LLMs) have significantly enhanced their coding capabilities. However, existing benchmarks predominantly focused on simplified or isolated aspects of programming, such as single-file code generation or repository issue debugging, falling short of measuring the full spectrum of challenges raised by real-world programming activities. To this end, we propose DevBench, a comprehensive benchmark that evaluates LLMs across various stages of the software development lifecycle, including software design, environment setup, implementation, acceptance testing, and unit testing. DevBench features a wide range of programming languages and domains, high-quality data collection, and carefully designed and verified metrics for each task. Empirical studies show that current LLMs, including GPT-4-Turbo, fail to solve the challenges presented within DevBench. Analyses reveal that models struggle with understanding the complex structures in the repository, managing the compilation process, and grasping advanced programming concepts. Our findings offer actionable insights for the future development of LLMs toward real-world programming applications. Our benchmark is available at https://github.com/open-compass/DevBench, Comment: Our data and code are available at https://github.com/open-compass/DevBench

Topotactic reduction utilizing metal hydrides as reagents emerges as an effective approach to achieve exceptionally low oxidization states of metal ions and unconventional coordination networks. This method opens avenues to the development of entirely new functional materials, with one notable example being the infinite-layer nickelate superconductors. However, the reduction effect on the atomic reconstruction and electronic structures -- crucial for superconductivity -- remains largely unresolved. We design two sets of control Nd$_{0.8}$Sr$_{0.2}$NiO$_2$ thin films and implement secondary ion mass spectroscopy to highlight the absence of reduction-induced hydrogen intercalation. X-ray absorption spectroscopy shows a significant linear dichroism with dominant Ni 3d$_{x2{-}y2}$ orbitals on superconducting samples, indicating a Ni single-band nature of infinite-layer nickelates. Consistent with the superconducting $T_c$, the Ni 3d orbitals asymmetry manifests a dome-like reduction duration dependence. Our results unveil the critical role of reduction in modulating the Ni-3d orbital polarization and its impact on the superconducting properties., Comment: Main: 21 pages, 4 figure - Supplementary: 7 pages, 5 figure

Simulating the cold $^{87}Rb$ atom with a three-level quantum system interacting with two orthogonal standing-wave fields, the localization within half-wavelength domain in the x-y plane is achieved by monitoring the probe absorption. Within the half-wavelength domain, the single absorption peak increases from 0.2 to 1.0 via the spontaneously generated coherence (SGC), while the diameters of the single absorption peaks are diminished by the increasing incoherent pumping field. Our scheme provides the flexible parameters manipulating manner for the localization of cold $^{87}Rb$ atom., Comment: 10 pages, 5 figures

The atom localization of a V-type atomic system is discussed by the detunings associated with the probe and the two orthogonal standing-wave fields, and by the spontaneously generated coherence (SGC). Within the half-wavelength domain in the 2-dimensional(2-D) plane, the atom localization depicted by the probe dual absorption peaks is achieved when the detunings are tuned. However, the dual peaks change into single-peak when the SGC arises. The single-peak 2-D localization demonstrated the advantage for atom localization achieved by the flexible manipulating parameters in our scheme., Comment: 0 pages, 6 figures

We present a novel, deep-learning based method -- dubbed Galactic-Seismology Substructures and Streams Hunter, or GS$^{3}$ Hunter for short, to search for substructures and streams in stellar kinematics data. GS$^{3}$ Hunter relies on a combined application of Siamese Neural Networks to transform the phase space information and the K-means algorithm for the clustering. As a validation test, we apply GS$^{3}$ Hunter to a subset of the Feedback in Realistic Environments (FIRE) cosmological simulations. The stellar streams and substructures thus identified are in good agreement with corresponding results reported earlier by the FIRE team. In the same vein, we apply our method to a subset of local halo stars from the Gaia Early Data Release 3 and GALAH DR3 datasets, and recover several, previously known dynamical groups, such as Thamnos 1+2, Hot Thick Disk, ED-1, L-RL3, Helmi 1+2, and Gaia-Sausage-Enceladus, Sequoia, VRM, Cronus, Nereus. Finally, we apply our method without fine-tuning to a subset of K-giant stars located in the inner halo region, obtained from the LAMOST Data Release 5 (DR5) dataset. We recover three, previously known structures (Sagittarius, Hercules-Aquila Cloud, and the Virgo Overdensity), but we also discover a number of new substructures. We anticipate that GS$^{3}$ Hunter will become a useful tool for the community dedicated to the search of stellar streams and structures in the Milky Way (MW) and the Local group, thus helping advance our understanding of the stellar inner and outer halos, and of the assembly and tidal stripping history in and around the MW., Comment: 41 pages, 34 figures, 3 tables. Accepted for Publication in the Astrophysical Journal

Various large language models (LLMs) have been proposed in recent years, including closed- and open-source ones, continually setting new records on multiple benchmarks. However, the development of LLMs still faces several issues, such as high cost of training models from scratch, and continual pre-training leading to catastrophic forgetting, etc. Although many such issues are addressed along the line of research on LLMs, an important yet practical limitation is that many studies overly pursue enlarging model sizes without comprehensively analyzing and optimizing the use of pre-training data in their learning process, as well as appropriate organization and leveraging of such data in training LLMs under cost-effective settings. In this work, we propose Ziya2, a model with 13 billion parameters adopting LLaMA2 as the foundation model, and further pre-trained on 700 billion tokens, where we focus on pre-training techniques and use data-centric optimization to enhance the learning process of Ziya2 on different stages. We define three data attributes and firstly establish data-centric scaling laws to illustrate how different data impacts LLMs. Experiments show that Ziya2 significantly outperforms other models in multiple benchmarks especially with promising results compared to representative open-source ones. Ziya2 (Base) is released at https://huggingface.co/IDEA-CCNL/Ziya2-13B-Base and https://modelscope.cn/models/Fengshenbang/Ziya2-13B-Base/summary.

The application of asymmetric drift (AD) tomography across different populations provides valuable insights into the kinematics, dynamics, and rotation curves of the Galactic disk. By leveraging common stars identified in both the LAMOST and Gaia surveys, alongside Gaia DR3's circular velocity curve, we conducted a qualitative exploration of asymmetric drift distributions within the Galactic disk spanning distances from 8 to 16 kpc. In the R-Z plane, we observed that the asymmetric drift is minimal near the mid-plane of the Galactic disk and gradually increases with vertical distance, resulting in a distinctive ``horn" shape. Additionally, our analysis revealed that populations with higher [$\alpha$/Fe] ratios exhibit greater asymmetric drift compared to those with lower [$\alpha$/Fe] ratios. Specifically, we found the asymmetric drift around the solar location to be approximately 6 km s$^{-1}$, with a median value of 16 km s$^{-1}$ across the entire sample. Notably, the median asymmetric drift in the northern region of the Galactic disk (20 km s$^{-1}$) surpasses that in the southern region (13 km s$^{-1}$), with errors remaining within 2 km s$^{-1}$. Furthermore, our investigation into mono-age stellar populations unveiled that older stellar populations tend to exhibit larger asymmetric drift and velocity dispersion, aligning closely with predictions from previous numerical models. Finally, based on chemical compositions, we observed that the median asymmetric drift of the thick disk significantly exceeds that of the thin disk and found that star formation within the thick disk primarily occurred earlier than 8-10 billion years, whereas the thin disk's predominant star formation period spanned 6-8 billion years ago., Comment: 14 pages and 17 figures. The Open Journal of Astrophysics. AD analysis is the key target and the time scale picture of the milky way disk are also revealed here in another point of view, see the Fig. 9

Android Apps are frequently updated to keep up with changing user, hardware, and business demands. Ensuring the correctness of App updates through extensive testing is crucial to avoid potential bugs reaching the end user. Existing Android testing tools generate GUI events focussing on improving the test coverage of the entire App rather than prioritising updates and its impacted elements. Recent research has proposed change-focused testing but relies on random exploration to exercise the updates and impacted GUI elements that is ineffective and slow for large complex Apps with a huge input exploration space. We propose directed testing of App updates with Hawkeye that is able to prioritise executing GUI actions associated with code changes based on deep reinforcement learning from historical exploration data. Our empirical evaluation compares Hawkeye with state-of-the-art model-based and reinforcement learning-based testing tools FastBot2 and ARES using 10 popular open-source and 1 commercial App. We find that Hawkeye is able to generate GUI event sequences targeting changed functions more reliably than FastBot2 and ARES for the open source Apps and the large commercial App. Hawkeye achieves comparable performance on smaller open source Apps with a more tractable exploration space. The industrial deployment of Hawkeye in the development pipeline also shows that Hawkeye is ideal to perform smoke testing for merge requests of a complicated commercial App.

We investigate the existence of two nontrivial solutions for a poly-Laplacian system involving concave-convex nonlinearities and parameters with Dirichlet boundary condition on locally finite graphs. By using the mountain pass theorem and Ekeland's variational principle, we obtain that system has at least one nontrivial solution of positive energy and one nontrivial solution of negative energy, respectively. We also obtain an estimate about semi-trivial solutions. Moreover, by using a result in [4] which is based on the fibering method and Nehari manifold, we obtain the existence of ground state solution to the single equation corresponding to poly-Laplacian system. Especially, we present some ranges of parameters in all of results.

In recent years, the task of weakly supervised audio-visual violence detection has gained considerable attention. The goal of this task is to identify violent segments within multimodal data based on video-level labels. Despite advances in this field, traditional Euclidean neural networks, which have been used in prior research, encounter difficulties in capturing highly discriminative representations due to limitations of the feature space. To overcome this, we propose HyperVD, a novel framework that learns snippet embeddings in hyperbolic space to improve model discrimination. Our framework comprises a detour fusion module for multimodal fusion, effectively alleviating modality inconsistency between audio and visual signals. Additionally, we contribute two branches of fully hyperbolic graph convolutional networks that excavate feature similarities and temporal relationships among snippets in hyperbolic space. By learning snippet representations in this space, the framework effectively learns semantic discrepancies between violent and normal events. Extensive experiments on the XD-Violence benchmark demonstrate that our method outperforms state-of-the-art methods by a sizable margin., Comment: 11 pages, 12 figures, typos are fixed

While traditional distributionally robust optimization (DRO) aims to minimize the maximal risk over a set of distributions, Agarwal and Zhang (2022) recently proposed a variant that replaces risk with excess risk. Compared to DRO, the new formulation$\unicode{x2013}$minimax excess risk optimization (MERO) has the advantage of suppressing the effect of heterogeneous noise in different distributions. However, the choice of excess risk leads to a very challenging minimax optimization problem, and currently there exists only an inefficient algorithm for empirical MERO. In this paper, we develop efficient stochastic approximation approaches which directly target MERO. Specifically, we leverage techniques from stochastic convex optimization to estimate the minimal risk of every distribution, and solve MERO as a stochastic convex-concave optimization (SCCO) problem with biased gradients. The presence of bias makes existing theoretical guarantees of SCCO inapplicable, and fortunately, we demonstrate that the bias, caused by the estimation error of the minimal risk, is under-control. Thus, MERO can still be optimized with a nearly optimal convergence rate. Moreover, we investigate a practical scenario where the quantity of samples drawn from each distribution may differ, and propose a stochastic approach that delivers distribution-dependent convergence rates.

We propose a new paradigm for universal information extraction (IE) that is compatible with any schema format and applicable to a list of IE tasks, such as named entity recognition, relation extraction, event extraction and sentiment analysis. Our approach converts the text-based IE tasks as the token-pair problem, which uniformly disassembles all extraction targets into joint span detection, classification and association problems with a unified extractive framework, namely UniEX. UniEX can synchronously encode schema-based prompt and textual information, and collaboratively learn the generalized knowledge from pre-defined information using the auto-encoder language models. We develop a traffine attention mechanism to integrate heterogeneous factors including tasks, labels and inside tokens, and obtain the extraction target via a scoring matrix. Experiment results show that UniEX can outperform generative universal IE models in terms of performance and inference-speed on $14$ benchmarks IE datasets with the supervised setting. The state-of-the-art performance in low-resource scenarios also verifies the transferability and effectiveness of UniEX.

The integration of high-temperature superconducting YBa2Cu3O6+x (YBCO) into flexible electronic devices has the potential to revolutionize the technology industry. The effective preparation of high-quality flexible YBCO films therefore plays a key role in this development. We present a novel approach for transferring water-sensitive YBCO films onto flexible substrates without any buffer layer. Freestanding YBCO film on a polydimethylsiloxane substrate is extracted by etching the Sr3Al2O6 sacrificial layer from the LaAlO3 substrate. In addition to the obtained freestanding YBCO thin film having a Tc of 89.1 K, the freestanding YBCO thin films under inward and outward bending conditions have Tc of 89.6 K and 88.9 K, respectively. A comprehensive characterization involving multiple experimental techniques including high-resolution transmission electron microscopy, scanning electron microscopy, Raman and X-ray Absorption Spectroscopy is conducted to investigate the morphology, structural and electronic properties of the YBCO film before and after the extraction process where it shows the preservation of the structural and superconductive properties of the freestanding YBCO virtually in its pristine state. Further investigation reveals the formation of a YBCO passivated layer serves as a protective layer which effectively preserves the inner section of the freestanding YBCO during the etching process. This work plays a key role in actualizing the fabrication of flexible oxide thin films and opens up new possibilities for a diverse range of device applications involving thin-films and low-dimensional materials., Comment: 22 pages,4 figures,references added

Accurate distance determination to astrophysical objects is essential for the understanding of their intrinsic brightness and size. The distance to SN 1987A has been previously measured by the expanding photosphere method, and by using the angular size of the circumstellar rings with absolute sizes derived from light curves of narrow UV emission lines, with reported distances ranging from 46.77 kpc to 55 kpc. In this study, we independently determined the distance to SN 1987A using photometry and imaging polarimetry observations of AT 2019xis, a light echo of SN 1987A, by adopting a radiative transfer model of the light echo developed in Ding et al. (2021). We obtained distances to SN 1987A in the range from 49.09 $\pm$ 2.16 kpc to 59.39 $\pm$ 3.27 kpc, depending on the interstellar polarization and extinction corrections, which are consistent with the literature values. This study demonstrates the potential of using light echoes as a tool for distance determination to astrophysical objects in the Milky Way, up to kiloparsec level scales., Comment: Accepted for publication in ApJL

We generalize two embedding theorems and investigate the existence and multiplicity of nontrivial solutions for a $(p,q)$-Laplacian coupled system with perturbations and two parameters $\lambda_1$ and $\lambda_2$ on locally finite graph. By using the Ekeland's variational principle, we obtain that system has at least one nontrivial solution when the nonlinear term satisfies the sub-$(p,q)$ conditions. We also obtain a necessary condition for the existence of semi-trivial solutions to the system. Moreover, by using the mountain pass theorem and Ekeland's variational principle, we obtain that system has at least one solution of positive energy and one solution of negative energy when the nonlinear term satisfies the super-$(p,q)$ conditions which is weaker than the well-known Ambrosetti-Rabinowitz condition. Especially, in all of the results, we present the concrete ranges of the parameters $\lambda_1$ and $\lambda_2$.