Search

Large language models (LLMs) face significant challenges in handling long-context tasks because of their limited effective context window size during pretraining, which restricts their ability to generalize over extended sequences. Meanwhile, extending the context window in LLMs through post-pretraining is highly resource-intensive. To address this, we introduce LongRecipe, an efficient training strategy for extending the context window of LLMs, including impactful token analysis, position index transformation, and training optimization strategies. It simulates long-sequence inputs while maintaining training efficiency and significantly improves the model's understanding of long-range dependencies. Experiments on three types of LLMs show that LongRecipe can utilize long sequences while requiring only 30% of the target context window size, and reduces computational training resource over 85% compared to full sequence training. Furthermore, LongRecipe also preserves the original LLM's capabilities in general tasks. Ultimately, we can extend the effective context window of open-source LLMs from 8k to 128k, achieving performance close to GPT-4 with just one day of dedicated training using a single GPU with 80G memory. Our code is released at https://github.com/zhiyuanhubj/LongRecipe., Comment: Work in Progress

Multi-Label Text Classification (MLTC) is a practical yet challenging task that involves assigning multiple non-exclusive labels to each document. Previous studies primarily focus on capturing label correlations to assist label prediction by introducing special labeling schemes, designing specific model structures, or adding auxiliary tasks. Recently, the $k$ Nearest Neighbor ($k$NN) framework has shown promise by retrieving labeled samples as references to mine label co-occurrence information in the embedding space. However, two critical biases, namely embedding alignment bias and confidence estimation bias, are often overlooked, adversely affecting prediction performance. In this paper, we introduce a DEbiased Nearest Neighbors (DENN) framework for MLTC, specifically designed to mitigate these biases. To address embedding alignment bias, we propose a debiased contrastive learning strategy, enhancing neighbor consistency on label co-occurrence. For confidence estimation bias, we present a debiased confidence estimation strategy, improving the adaptive combination of predictions from $k$NN and inductive binary classifications. Extensive experiments conducted on four public benchmark datasets (i.e., AAPD, RCV1-V2, Amazon-531, and EUR-LEX57K) showcase the effectiveness of our proposed method. Besides, our method does not introduce any extra parameters.

Recent Pre-trained Language Models (PLMs) usually only provide users with the inference APIs, namely the emerging Model-as-a-Service (MaaS) setting. To adapt MaaS PLMs to downstream tasks without accessing their parameters and gradients, some existing methods focus on the output-side adaptation of PLMs, viewing the PLM as an encoder and then optimizing a task-specific decoder for decoding the output hidden states and class scores of the PLM. Despite the effectiveness of these methods, they only use a single prompt to query PLMs for decoding, leading to a heavy reliance on the quality of the adopted prompt. In this paper, we propose a simple yet effective Multi-Prompting Decoder (MPD) framework for MaaS adaptation. The core idea is to query PLMs with multiple different prompts for each sample, thereby obtaining multiple output hidden states and class scores for subsequent decoding. Such multi-prompting decoding paradigm can simultaneously mitigate reliance on the quality of a single prompt, alleviate the issue of data scarcity under the few-shot setting, and provide richer knowledge extracted from PLMs. Specifically, we propose two decoding strategies: multi-prompting decoding with optimal transport for hidden states and calibrated decoding for class scores. Extensive experiments demonstrate that our method achieves new state-of-the-art results on multiple natural language understanding datasets under the few-shot setting.

In the field of portrait video generation, the use of single images to generate portrait videos has become increasingly prevalent. A common approach involves leveraging generative models to enhance adapters for controlled generation. However, control signals (e.g., text, audio, reference image, pose, depth map, etc.) can vary in strength. Among these, weaker conditions often struggle to be effective due to interference from stronger conditions, posing a challenge in balancing these conditions. In our work on portrait video generation, we identified audio signals as particularly weak, often overshadowed by stronger signals such as facial pose and reference image. However, direct training with weak signals often leads to difficulties in convergence. To address this, we propose V-Express, a simple method that balances different control signals through the progressive training and the conditional dropout operation. Our method gradually enables effective control by weak conditions, thereby achieving generation capabilities that simultaneously take into account the facial pose, reference image, and audio. The experimental results demonstrate that our method can effectively generate portrait videos controlled by audio. Furthermore, a potential solution is provided for the simultaneous and effective use of conditions of varying strengths.

The success of the text-guided diffusion model has inspired the development and release of numerous powerful diffusion models within the open-source community. These models are typically fine-tuned on various expert datasets, showcasing diverse denoising capabilities. Leveraging multiple high-quality models to produce stronger generation ability is valuable, but has not been extensively studied. Existing methods primarily adopt parameter merging strategies to produce a new static model. However, they overlook the fact that the divergent denoising capabilities of the models may dynamically change across different states, such as when experiencing different prompts, initial noises, denoising steps, and spatial locations. In this paper, we propose a novel ensembling method, Adaptive Feature Aggregation (AFA), which dynamically adjusts the contributions of multiple models at the feature level according to various states (i.e., prompts, initial noises, denoising steps, and spatial locations), thereby keeping the advantages of multiple diffusion models, while suppressing their disadvantages. Specifically, we design a lightweight Spatial-Aware Block-Wise (SABW) feature aggregator that adaptive aggregates the block-wise intermediate features from multiple U-Net denoisers into a unified one. The core idea lies in dynamically producing an individual attention map for each model's features by comprehensively considering various states. It is worth noting that only SABW is trainable with about 50 million parameters, while other models are frozen. Both the quantitative and qualitative experiments demonstrate the effectiveness of our proposed Adaptive Feature Aggregation method. The code is available at https://github.com/tenvence/afa/.

The hydrodeoxygenation (HDO) of lignin-derived feedstocks into value-added chemicals with high efficiency and selectivity is desirable for the utilization of biomass resource. The complex oxygen-containing groups of lignin-derived substance result in the challenge of the low selectivity toward the required product. In this work, highly dispersed Ru nanoparticles anchored on Ni3Al1 layered double oxides (LDOs) catalyst derived from NiAl layered double hydroxides (LDHs) with flower-shaped morphology was constructed by a simple deposition-reduction method. The introduction of LDHs-derived support can significantly impact the catalytic activity for the HDO of lignin-derived vanillin (VL) into 2-methoxy-4-methylphenol (MMP). The Ru/Ni3Al1-400 catalyst obtained complete conversion of VL and 94.2% yield of MMP at 130 {\deg}C in methanol solvent, much better than the catalysts without LDHs-derived support. The methanol solvent is beneficial for the conversion of reaction intermediate of vanillin alcohol (VA). Detailed characterization reveals that the existence of the enhanced metal-support interaction over Ru/Ni3Al1-400 and the easily accessible acid sites facilitate the production of MMP.

Nanostructured catalysts have attracted the increased attention for biomass conversion into high-valued chemicals due to the rapid depletion of fossil resources and increasingly severe environmental issues. Supercritical carbon dioxide (scCO2) fluid is an attractive medium for synthesizing nanostructured materials due to its favorable properties. In this review, the properties of scCO2 and the roles of scCO2 in the fabrication of metal nanoparticles were assessed in detailed. A general overview of the synthesis of different types of metal nanoparticles (including metal oxide nanoparticles) using scCO2 and the relationship between the structure of the obtained metal nanoparticles and the preparation conditions such as reaction temperature and pressure, types of metal precursors, and deposition time are system summarized and compared in tables. Besides, compared to the meatal catalysts using the conventional methods, the catalysts obtained using scCO2 exhibited excellent catalytic performance on biomass conversion reactions, mainly focused on oxidation, hydrogenation reactions. Finally, opportunities and challenges of metal nanoparticle preparation using scCO2 for biomass valorization to chemicals and liquid fuels are highlighted. This review could be helpful for the rational design of more efficient metal catalysts for the selective synthesis of fine chemicals and fuels from biomass-derived chemicals., Comment: 66 pages, 21 figures

Developing a green and cost-effective catalytic system for the selective oxidation of biomass-derived alcohols is vital for the sustainable synthesis of fine chemicals. Herein, highly dispersed subnanometric amorphous CoOx clusters in anatase TiO2 nanosheets (Co-TiO2) fabricated by green solvent CO2 assisted approach could directly activate peroxymonosulfate (PMS) for the highly selective oxidation of various biomass-derived alcohols. Advanced characterizations (e.g., EXAFS, EPR, AC HAADF-STEM) reveal that a strong interaction of CoOx clusters and the anatase TiO2 support exist in Co-TiO2 and Co atom in Co-TiO2 is mainly consisted of Co2+ and Co3+. The Co-TiO2 catalyst offers superior catalytic performance in the conversion of six types of alcohols (e.g., benzyl alcohol (BAL), 5-hydroxymethylfurfural (HMF)) with high selectivity to produce corresponding aldehydes. Highly dispersed CoOx clusters and the interaction between CoOx clusters and TiO2 support contribute to the superior performance. Mechanism studies show that SO4 radicals play the dominant role in the selective oxidation of model reactant BAL and 1O2 participates in the non-radical pathway. DFT calculations are well matched with experiment and decipher that the strong interaction between CoOx clusters and TiO2 support promotes the formation of SO4 and SO5., Comment: 29 pages, 9 figures

Rational design and green synthesis of low-cost and robust catalysts efficient for the selective oxidation of various alcohols are full of challenges. Herein, we report a fast and solvent-free arc-melting (AM) method to controllably synthesize semimetal CoSi alloy (abbreviated as AM-CoSi) that is efficient for the base- and solvent-free oxidation of six types of aromatic alcohols. X-ray absorption fine structure (XAFS), electron paramagnetic resonance (EPR), and aberration corrected high angle annular dark field scanning transmission electron microscope (AC HAADF-STEM) confirmed the successful synthesis of AM-CoSi with rich Si vacancy (Siv). The as-prepared CoSi alloy catalysts exhibit an order of magnitude activity enhancement in the oxidation of model reactant benzyl alcohol (BAL) to benzyl benzoate (BBE) compared with its mono counterparts, whereas 70 % yield of BBE which is the highest yield to date. Experimental results and DFT calculations well verify that the CoSi alloy structure improves the BAL conversion and Si vacancy mainly contributes to the generation of BBE. After that, CoSi alloy maintains high stability and a potential pathway is rationally proposed. Besides, CoSi alloy also efficiently works for the selective oxidation of various alcohols with different groups. This work demonstrates for the first time that semimetal CoSi alloy is robust for the green oxidation of various alcohols and provides a vast opportunity for reasonable design and application of other semimetal alloy catalysts., Comment: 22 pages, 6 figures

Iron-based molybdenum disulfide (Fe-MoS$_2$) has emerged as a Fenton-like catalyst for the highly efficient degradation of antibiotics, but the structure-activity relationship remains elusive. Herein, garland-like MIL-101(Fe)/MoS$_2$ nanosheets (MMS) with dual metal active sites (Fe and Mo) and rich sulfur vacancies were fabricated to directly activate peroxymonosulfate (PMS) for fast degradation of different organic pollutants (phenols, dyes and drugs), even in real water bodies. The MMS exhibited extremely fast catalytic rate constant of 0.289 min$^{-1}$ in the degradation of sulfamethoxazole (SMX), which was about 36 and 29 times that of single MoS$_2$ (0.008 min$^{-1}$) and MIL-101(Fe) (0.01 min$^{-1}$). Moreover, MMS with good stability and reusability could reach 92% degradation of SMX after 5 cycles. Quenching experiments and electron spin resonance (ESR) tests revealed that hydroxyl radicals (.OH) and singlet oxygen ($^1$O$_2$) were the dominant reactive oxygen species (ROS) for SMX degradation. The integration of experimental works, characterization techniques and density functional theory (DFT) calculations unraveled that the formation of sulfur vacancies in MMS catalyst could expose more Mo sites, improve the charge density and boost the electron transfer, which was conducive to accelerating the Fe$^{3+}$/Fe$^{2+}$ cycle for enhancing the activation of PMS. Finally, the C-N, N-O, S-N, C-O and C-S bonds of SMX were easily attacked by ROS to generate the nontoxic intermediates in the MMS/PMS/SMX system. This study offers a new approach to designing high-performance Fe-MoS$_2$ catalysts for the removal of organic pollutants.

The coexistence of tetracycline (TC) and arsenite (As(III)) in livestock wastewater threatens public health, and the heterogeneous Fenton-like system is a practical approach for the simultaneous removal of TC and As(III). In this work, fine CoFe$_2$O$_4$ nanoparticles are facilely anchored on heretically porous carbon (CoFe$_2$O$_4$@PC) via a microwave-assisted calcination method and used for eliminating TC and As(III) via peroxymonosulfate (PMS) activation.

Spinel oxides are recognized as promising Fenton-like catalysts for the degradation of antibiotics. However, the catalytic performance is restrained by the poor electron transfer rate (ETR). Herein, hollow NiCo2O4@C nanocages are rationally designed and prepared to accelerate ETR in peroxymonosulfate (PMS) activation for tetracycline (TC) degradation.

Channel attention mechanisms endeavor to recalibrate channel weights to enhance representation abilities of networks. However, mainstream methods often rely solely on global average pooling as the feature squeezer, which significantly limits the overall potential of models. In this paper, we investigate the statistical moments of feature maps within a neural network. Our findings highlight the critical role of high-order moments in enhancing model capacity. Consequently, we introduce a flexible and comprehensive mechanism termed Extensive Moment Aggregation (EMA) to capture the global spatial context. Building upon this mechanism, we propose the Moment Channel Attention (MCA) framework, which efficiently incorporates multiple levels of moment-based information while minimizing additional computation costs through our Cross Moment Convolution (CMC) module. The CMC module via channel-wise convolution layer to capture multiple order moment information as well as cross channel features. The MCA block is designed to be lightweight and easily integrated into a variety of neural network architectures. Experimental results on classical image classification, object detection, and instance segmentation tasks demonstrate that our proposed method achieves state-of-the-art results, outperforming existing channel attention methods.

Despite Large Language Models (LLMs) like GPT-4 achieving impressive results in function-level code generation, they struggle with repository-scale code understanding (e.g., coming up with the right arguments for calling routines), requiring a deeper comprehension of complex file interactions. Also, recently, people have developed LLM agents that attempt to interact with repository code (e.g., compiling and evaluating its execution), prompting the need to evaluate their performance. These gaps have motivated our development of ML-Bench, a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. Addressing the need for LLMs to interpret long code contexts and translate instructions into precise, executable scripts, ML-Bench encompasses annotated 9,641 examples across 18 GitHub repositories, challenging LLMs to accommodate user-specified arguments and documentation intricacies effectively. To evaluate both LLMs and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment. Our findings indicate that while GPT-4o leads with a Pass@5 rate surpassing 50%, there remains significant scope for improvement, highlighted by issues such as hallucinated outputs and difficulties with bash script generation. Notably, in the more demanding ML-Agent-Bench, GPT-4o achieves a 76.47% success rate, reflecting the efficacy of iterative action and feedback in complex task resolution. Our code, dataset, and models are available at https://github.com/gersteinlab/ML-bench.

Reconfigurable Intelligent Surface (RIS) or metasurface is one of the important enabling technologies in mobile cellular networks that can effectively enhance the signal coverage performance in obstructed regions, and it is generally deployed on surfaces different from obstacles to redirect electromagnetic (EM) waves by reflection, or covered on objects' surfaces to manipulate EM waves by refraction. In this paper, Reconfigurable Intelligent Surface & Edge (RISE) is proposed to extend RIS' abilities of reflection and refraction over surfaces to diffraction around obstacles' edge for better adaptation to specific coverage scenarios. Based on that, this paper analyzes the performance of several different deployment locations and EM manipulation structure designs for different coverage scenarios. Then a novel EM manipulation structure deployed at the obstacles' edge is proposed to achieve static EM environment modification. Simulations validate the preference of the schemes for different scenarios and the new structure achieves better coverage performance than other typical structures in the static scheme.

Few-shot sequence labeling aims to identify novel classes based on only a few labeled samples. Existing methods solve the data scarcity problem mainly by designing token-level or span-level labeling models based on metric learning. However, these methods are only trained at a single granularity (i.e., either token level or span level) and have some weaknesses of the corresponding granularity. In this paper, we first unify token and span level supervisions and propose a Consistent Dual Adaptive Prototypical (CDAP) network for few-shot sequence labeling. CDAP contains the token-level and span-level networks, jointly trained at different granularities. To align the outputs of two networks, we further propose a consistent loss to enable them to learn from each other. During the inference phase, we propose a consistent greedy inference algorithm that first adjusts the predicted probability and then greedily selects non-overlapping spans with maximum probability. Extensive experiments show that our model achieves new state-of-the-art results on three benchmark datasets., Comment: Accepted by ACM Transactions on Information Systems

For deep ordinal classification, learning a well-structured feature space specific to ordinal classification is helpful to properly capture the ordinal nature among classes. Intuitively, when Euclidean distance metric is used, an ideal ordinal layout in feature space would be that the sample clusters are arranged in class order along a straight line in space. However, enforcing samples to conform to a specific layout in the feature space is a challenging problem. To address this problem, in this paper, we propose a novel Constrained Proxies Learning (CPL) method, which can learn a proxy for each ordinal class and then adjusts the global layout of classes by constraining these proxies. Specifically, we propose two kinds of strategies: hard layout constraint and soft layout constraint. The hard layout constraint is realized by directly controlling the generation of proxies to force them to be placed in a strict linear layout or semicircular layout (i.e., two instantiations of strict ordinal layout). The soft layout constraint is realized by constraining that the proxy layout should always produce unimodal proxy-to-proxies similarity distribution for each proxy (i.e., to be a relaxed ordinal layout). Experiments show that the proposed CPL method outperforms previous deep ordinal classification methods under the same setting of feature extractor., Comment: Accepted by AAAI 2023

Open intent classification is a practical yet challenging task in dialogue systems. Its objective is to accurately classify samples of known intents while at the same time detecting those of open (unknown) intents. Existing methods usually use outlier detection algorithms combined with K-class classifier to detect open intents, where K represents the class number of known intents. Different from them, in this paper, we consider another way without using outlier detection algorithms. Specifically, we directly train a (K+1)-class classifier for open intent classification, where the (K+1)-th class represents open intents. To address the challenge that training a (K+1)-class classifier with training samples of only K classes, we propose a deep model based on Soft Labeling and Manifold Mixup (SLMM). In our method, soft labeling is used to reshape the label distribution of the known intent samples, aiming at reducing model's overconfident on known intents. Manifold mixup is used to generate pseudo samples for open intents, aiming at well optimizing the decision boundary of open intents. Experiments on four benchmark datasets demonstrate that our method outperforms previous methods and achieves state-of-the-art performance. All the code and data of this work can be obtained at https://github.com/zifengcheng/SLMM., Comment: Accepted by IEEE/ACM Transactions on Audio Speech and Language Processing