Search

Broad textual understanding and in-context learning require language models that utilize full document contexts. Due to the implementation challenges associated with directly training long-context models, many methods have been proposed for extending models to handle long contexts. However, owing to differences in data and model classes, it has been challenging to compare these approaches, leading to uncertainty as to how to evaluate long-context performance and whether it differs from standard evaluation. We implement a controlled protocol for extension methods with a standardized evaluation, utilizing consistent base models and extension data. Our study yields several insights into long-context behavior. First, we reaffirm the critical role of perplexity as a general-purpose performance indicator even in longer-context tasks. Second, we find that current approximate attention methods systematically underperform across long-context tasks. Finally, we confirm that exact fine-tuning based methods are generally effective within the range of their extension, whereas extrapolation remains challenging. All codebases, models, and checkpoints will be made available open-source, promoting transparency and facilitating further research in this critical area of AI development.

The increasing sophistication of text-to-image generative models has led to complex challenges in defining and enforcing copyright infringement criteria and protection. Existing methods, such as watermarking and dataset deduplication, fail to provide comprehensive solutions due to the lack of standardized metrics and the inherent complexity of addressing copyright infringement in diffusion models. To deal with these challenges, we propose a Reinforcement Learning-based Copyright Protection(RLCP) method for Text-to-Image Diffusion Model, which minimizes the generation of copyright-infringing content while maintaining the quality of the model-generated dataset. Our approach begins with the introduction of a novel copyright metric grounded in copyright law and court precedents on infringement. We then utilize the Denoising Diffusion Policy Optimization (DDPO) framework to guide the model through a multi-step decision-making process, optimizing it using a reward function that incorporates our proposed copyright metric. Additionally, we employ KL divergence as a regularization term to mitigate some failure modes and stabilize RL fine-tuning. Experiments conducted on 3 mixed datasets of copyright and non-copyright images demonstrate that our approach significantly reduces copyright infringement risk while maintaining image quality., Comment: arXiv admin note: text overlap with arXiv:2403.12052 by other authors

Robustness is a measure of functional reliability of a system against perturbations. To achieve a good and robust performance, a system must filter out external perturbations by its internal priors. These priors are usually distilled in the structure and the states of the system. Biophysical neural network are known to be robust but the exact mechanisms are still elusive. In this paper, we probe how orientation-selective neurons organized on a 1-D ring network respond to perturbations in the hope of gaining some insights on the robustness of visual system in brain. We analyze the steady-state of the rate-based network and prove that the activation state of neurons, rather than their firing rates, determines how the model respond to perturbations. We then identify specific perturbation patterns that induce the largest responses for different configurations of activation states, and find them to be sinusoidal or sinusoidal-like while other patterns are largely attenuated. Similar results are observed in a spiking ring model. Finally, we remap the perturbations in orientation back into the 2-D image space using Gabor functions. The resulted optimal perturbation patterns mirror adversarial attacks in deep learning that exploit the priors of the system. Our results suggest that based on different state configurations, these priors could underlie some of the illusionary experiences as the cost of visual robustness.

In recent years, the increasing demand for dynamic 3D assets in design and gaming applications has given rise to powerful generative pipelines capable of synthesizing high-quality 4D objects. Previous methods generally rely on score distillation sampling (SDS) algorithm to infer the unseen views and motion of 4D objects, thus leading to unsatisfactory results with defects like over-saturation and Janus problem. Therefore, inspired by recent progress of video diffusion models, we propose to optimize a 4D representation by explicitly generating multi-view videos from one input image. However, it is far from trivial to handle practical challenges faced by such a pipeline, including dramatic temporal inconsistency, inter-frame geometry and texture diversity, and semantic defects brought by video generation results. To address these issues, we propose DG4D, a novel multi-stage framework that generates high-quality and consistent 4D assets without score distillation. Specifically, collaborative techniques and solutions are developed, including an attention injection strategy to synthesize temporal-consistent multi-view videos, a robust and efficient dynamic reconstruction method based on Gaussian Splatting, and a refinement stage with diffusion prior for semantic restoration. The qualitative results and user preference study demonstrate that our framework outperforms the baselines in generation quality by a considerable margin. Code will be released at \url{https://github.com/jasongzy/EG4D}.

Interest modeling in recommender system has been a constant topic for improving user experience, and typical interest modeling tasks (e.g. multi-interest, long-tail interest and long-term interest) have been investigated in many existing works. However, most of them only consider one interest in isolation, while neglecting their interrelationships. In this paper, we argue that these tasks suffer from a common "interest amnesia" problem, and a solution exists to mitigate it simultaneously. We figure that long-term cues can be the cornerstone since they reveal multi-interest and clarify long-tail interest. Inspired by the observation, we propose a novel and unified framework in the retrieval stage, "Trinity", to solve interest amnesia problem and improve multiple interest modeling tasks. We construct a real-time clustering system that enables us to project items into enumerable clusters, and calculate statistical interest histograms over these clusters. Based on these histograms, Trinity recognizes underdelivered themes and remains stable when facing emerging hot topics. Trinity is more appropriate for large-scale industry scenarios because of its modest computational overheads. Its derived retrievers have been deployed on the recommender system of Douyin, significantly improving user experience and retention. We believe that such practical experience can be well generalized to other scenarios.

Flocking, as paradigmatically exemplified by birds, is the coherent collective motion of active agents. As originally conceived, flocking emerges through alignment interactions between the agents. Here, we report that flocking can also emerge through interactions that turn agents away from each other. Combining simulations, kinetic theory, and experiments, we demonstrate this mechanism of flocking in self-propelled Janus colloids with stronger repulsion on the front than on the rear. The polar state is stable because particles achieve a compromise between turning away from left and right neighbors. Unlike for alignment interactions, the emergence of polar order from turn-away interactions requires particle repulsion. At high concentration, repulsion produces flocking Wigner crystals. Whereas repulsion often leads to motility-induced phase separation of active particles, here it combines with turn-away torques to produce flocking. Therefore, our findings bridge the classes of aligning and non-aligning active matter. Our results could help to reconcile the observations that cells can flock despite turning away from each other via contact inhibition of locomotion. Overall, our work shows that flocking is a very robust phenomenon that arises even when the orientational interactions would seem to prevent it.

Token-free language models learn directly from raw bytes and remove the inductive bias of subword tokenization. Operating on bytes, however, results in significantly longer sequences. In this setting, standard autoregressive Transformers scale poorly as the effective memory required grows with sequence length. The recent development of the Mamba state space model (SSM) offers an appealing alternative approach with a fixed-sized memory state and efficient decoding. We propose MambaByte, a token-free adaptation of the Mamba SSM trained autoregressively on byte sequences. In terms of modeling, we show MambaByte to be competitive with, and even to outperform, state-of-the-art subword Transformers on language modeling tasks while maintaining the benefits of token-free language models, such as robustness to noise. In terms of efficiency, we develop an adaptation of speculative decoding with tokenized drafting and byte-level verification. This results in a $2.6\times$ inference speedup to the standard MambaByte implementation, showing similar decoding efficiency as the subword Mamba. These findings establish the viability of SSMs in enabling token-free language modeling., Comment: Published at COLM 2024

Comparative reasoning plays a crucial role in text preference prediction; however, large language models (LLMs) often demonstrate inconsistencies in their reasoning. While approaches like Chain-of-Thought improve accuracy in many other settings, they struggle to consistently distinguish the similarities and differences of complex texts. We introduce SC, a prompting approach that predicts text preferences by generating structured intermediate comparisons. SC begins by proposing aspects of comparison, followed by generating textual comparisons under each aspect. We select consistent comparisons with a pairwise consistency comparator that ensures each aspect's comparisons clearly distinguish differences between texts, significantly reducing hallucination and improving consistency. Our comprehensive evaluations across various NLP tasks, including summarization, retrieval, and automatic rating, demonstrate that SC equips LLMs to achieve state-of-the-art performance in text preference prediction.

Large language models (LLMs) have shown remarkable capabilities in various natural language understanding tasks. With only a few demonstration examples, these LLMs can quickly adapt to target tasks without expensive gradient updates. Common strategies to boost such 'in-context' learning ability are to ensemble multiple model decoded results and require the model to generate an explanation along with the prediction. However, these models often treat different class predictions equally and neglect the potential discrepancy between the explanations and predictions. To fully unleash the power of explanations, we propose EASE, an Explanation-Aware Soft Ensemble framework to empower in-context learning with LLMs. We design two techniques, explanation-guided ensemble, and soft probability aggregation, to mitigate the effect of unreliable explanations and improve the consistency between explanations and final predictions. Experiments on seven natural language understanding tasks and four varying-size LLMs demonstrate the effectiveness of our proposed framework.

Assessing causal effects in the presence of unobserved confounding is a challenging problem. Existing studies leveraged proxy variables or multiple treatments to adjust for the confounding bias. In particular, the latter approach attributes the impact on a single outcome to multiple treatments, allowing estimating latent variables for confounding control. Nevertheless, these methods primarily focus on a single outcome, whereas in many real-world scenarios, there is greater interest in studying the effects on multiple outcomes. Besides, these outcomes are often coupled with multiple treatments. Examples include the intensive care unit (ICU), where health providers evaluate the effectiveness of therapies on multiple health indicators. To accommodate these scenarios, we consider a new setting dubbed as multiple treatments and multiple outcomes. We then show that parallel studies of multiple outcomes involved in this setting can assist each other in causal identification, in the sense that we can exploit other treatments and outcomes as proxies for each treatment effect under study. We proceed with a causal discovery method that can effectively identify such proxies for causal estimation. The utility of our method is demonstrated in synthetic data and sepsis disease., Comment: Preprint, under review

Lifelogs are descriptions of experiences that a person had during their life. Lifelogs are created by fusing data from the multitude of digital services, such as online photos, maps, shopping and content streaming services. Question answering over lifelogs can offer personal assistants a critical resource when they try to provide advice in context. However, obtaining answers to questions over lifelogs is beyond the current state of the art of question answering techniques for a variety of reasons, the most pronounced of which is that lifelogs combine free text with some degree of structure such as temporal and geographical information. We create and publicly release TimelineQA1, a benchmark for accelerating progress on querying lifelogs. TimelineQA generates lifelogs of imaginary people. The episodes in the lifelog range from major life episodes such as high school graduation to those that occur on a daily basis such as going for a run. We describe a set of experiments on TimelineQA with several state-of-the-art QA models. Our experiments reveal that for atomic queries, an extractive QA system significantly out-performs a state-of-the-art retrieval-augmented QA system. For multi-hop queries involving aggregates, we show that the best result is obtained with a state-of-the-art table QA technique, assuming the ground truth set of episodes for deriving the answer is available.

We have performed a spectral-timing analysis on the black hole X-ray binary MAXI J1535--571 during its 2017 outburst, with the aim of exploring the evolution of the inner accretion flow geometry. X-ray reverberation lags are observed in the hard-intermediate state (HIMS) and soft-intermediate state (SIMS) of the outburst. During the HIMS, the characteristic frequency of the reverberation lags $\nu_0$ (the frequency at which the soft lag turns to zero in the lag-frequency spectra) increases when the spectrum softens. This reflects a reduction of the spatial distance between the corona and accretion disc, when assuming the measured time lags are associated with the light travel time. We also find a strong correlation between $\nu_0$ and type-C Quasi Periodic Oscillation (QPO) centroid frequency $\nu_{QPO}$, which can be well explained by the Lense-Thirring (L-T) precession model under a truncated disk geometry. Despite the degeneracy in the spectral modellings, our results suggest that the accretion disc is largely truncated in the low hard state (LHS), and moves inward as the spectrum softens. Combine the spectral modelling results with the $\nu_0$ - $\nu_{QPO}$ evolution, we are inclined to believe that this source probably have a truncated disk geometry in the hard state.

We present time-frequency analysis, based on the Hilbert-Huang transform (HHT), of the evolution on the low-frequency quasi-periodic oscillations (LFQPOs) observed in the black hole X-ray binary MAXI J1820+070. Through the empirical mode decomposition (EMD) method, we decompose the light curve of the QPO component and measure its intrinsic phase lag between photons from different energy bands. We find that the QPO phase lag is negative (low energy photons lag behind high energy photons), meanwhile the absolute value of the lag increases with energy. By applying the Hilbert transform to the light curve of the QPO, we further extract the instantaneous frequency and amplitude of the QPO. Compared these results with those from the Fourier analysis, we find that the broadening of the QPO peak is mainly caused by the frequency modulation. Through further analysis, we find that these modulations could share a common physical origin with the broad-band noise, and can be well explained by the internal shock model of the jet.

Virtual reality interview simulator (VRIS) provides an effective and manageable approach for candidates prone to being very nervous during interviews, yet, the major anxiety-inducing elements remain unknown. During an interview, the anxiety levels, overall experience, and performance of interviewees might be affected by various circumstances. By analyzing electrodermal activity and questionnaire, we investigated the influence of five variables: (I) \textit{Realism}; (II) \textit{Question type}; (III) \textit{Interviewer attitude}; (IV) \textit{Timing}; and (V) \textit{Preparation}. As such, an orthogonal design $L_8(4^1 \times 2^4)$ with eight experiments ($O A_8$ matrix) was implemented, in which 19 college students took part in the experiments. Considering the anxiety, overall experience, and performance of the interviewees, results indicate that \textit{Question type} plays a major role; secondly, \textit{Realism}, \textit{Preparation}, and \textit{Interviewer attitude} all have some degree of influence; lastly, \textit{Timing} have little to no impact. Specifically, professional interview questions elicited a greater degree of anxiety than personal ones among the categories of interview questions. This work contributes to our understanding of anxiety-stimulating factors during job interviews in virtual reality and provides cues for designing future VRIS., Comment: 12 pages, 4 pictures, 9 tables

Recent studies reveal that various biases exist in different NLP tasks, and over-reliance on biases results in models' poor generalization ability and low adversarial robustness. To mitigate datasets biases, previous works propose lots of debiasing techniques to tackle specific biases, which perform well on respective adversarial sets but fail to mitigate other biases. In this paper, we propose a new debiasing method Sparse Mixture-of-Adapters (SMoA), which can mitigate multiple dataset biases effectively and efficiently. Experiments on Natural Language Inference and Paraphrase Identification tasks demonstrate that SMoA outperforms full-finetuning, adapter tuning baselines, and prior strong debiasing methods. Further analysis indicates the interpretability of SMoA that sub-adapter can capture specific pattern from the training data and specialize to handle specific bias.

Transformers have been essential to pretraining success in NLP. While other architectures have been used, downstream accuracy is either significantly worse, or requires attention layers to match standard benchmarks such as GLUE. This work explores pretraining without attention by using recent advances in sequence routing based on state-space models (SSMs). Our proposed model, Bidirectional Gated SSM (BiGS), combines SSM layers with a multiplicative gating architecture that has been effective in simplified sequence modeling architectures. The model learns static layers that do not consider pair-wise interactions. Even so, BiGS is able to match BERT pretraining accuracy on GLUE and can be extended to long-form pretraining of 4096 tokens without approximation. Analysis shows that while the models have similar average accuracy, the approach has different inductive biases than BERT in terms of interactions and syntactic representations. All models from this work are available at https://github.com/jxiw/BiGS.

Living active collectives have evolved with remarkable self-patterning ability to meet the physical and biological constraints for growth and survival. However, how complex multicellular patterns emerge from a single founder cell remains elusive. Here, by recourse to an agent-based model, we track the three-dimensional (3D) morphodynamics and cell orientational order of growing bacterial biofilms encased by agarose gels. Confined growth causes spatiotemporally heterogeneous stress buildup in the biofilm. High hydrostatic and low shear stresses at the core of the biofilm promote viscous-to-elastic transition and randomize cell packing, whereas the opposite stress state near the gel-cell interface drives nematic ordering with a time delay inherent to shear stress relaxation. Overall, stress anisotropy spatiotemporally coincides with nematic order in the confined biofilms, suggesting an anisotropic-stress-driven ordering mechanism. The strong reciprocity between stress anisotropy and cell ordering inspires innovative 3D mechano-lithography of living active collectives for a variety of environmental and biomedical applications.

Single-image super-resolution (SISR) typically focuses on restoring various degraded low-resolution (LR) images to a single high-resolution (HR) image. However, during SISR tasks, it is often challenging for models to simultaneously maintain high quality and rapid sampling while preserving diversity in details and texture features. This challenge can lead to issues such as model collapse, lack of rich details and texture features in the reconstructed HR images, and excessive time consumption for model sampling. To address these problems, this paper proposes a Latent Feature-oriented Diffusion Probability Model (LDDPM). First, we designed a conditional encoder capable of effectively encoding LR images, reducing the solution space for model image reconstruction and thereby improving the quality of the reconstructed images. We then employed a normalized flow and multimodal adversarial training, learning from complex multimodal distributions, to model the denoising distribution. Doing so boosts the generative modeling capabilities within a minimal number of sampling steps. Experimental comparisons of our proposed model with existing SISR methods on mainstream datasets demonstrate that our model reconstructs more realistic HR images and achieves better performance on multiple evaluation metrics, providing a fresh perspective for tackling SISR tasks.

Active nematics are the nonequilibrium analog of passive liquid crystals in which anisotropic units consume free energy to drive emergent behavior. Similar to liquid crystal (LC) molecules in displays, ordering and dynamics in active nematics are sensitive to boundary conditions; however, unlike passive liquid crystals, active nematics, such as those composed of living matter, have the potential to regulate their boundaries through self-generated stresses. Here, using bacterial biofilms confined by a hydrogel as a model system, we show how a three-dimensional, living nematic can actively shape itself and its boundary in order to regulate its internal architecture through growth-induced stresses. We show that biofilms exhibit a sharp transition in shape from domes to lenses upon changing environmental stiffness or cell-substrate friction, which is explained by a theoretical model considering the competition between confinement and interfacial forces. The growth mode defines the progression of the boundary, which in turn determines the trajectories and spatial distribution of cell lineages. We further demonstrate that the evolving boundary defines the orientational ordering of cells and the emergence of topological defects in the interior of the biofilm. Our findings reveal novel self-organization phenomena in confined active matter and provide strategies for guiding the development of programmed microbial consortia with emergent material properties.

Thanks to the development of deep learning, research on machine anomalous sound detection based on self-supervised learning has made remarkable achievements. However, there are differences in the acoustic characteristics of the test set and the training set under different operating conditions of the same machine (domain shifts). It is challenging for the existing detection methods to learn the domain shifts features stably with low computation overhead. To address these problems, we propose a domain shift-oriented machine anomalous sound detection model based on self-supervised learning (TranSelf-DyGCN) in this paper. Firstly, we design a time-frequency domain feature modeling network to capture global and local spatial and time-domain features, thus improving the stability of machine anomalous sound detection stability under domain shifts. Then, we adopt a Dynamic Graph Convolutional Network (DyGCN) to model the inter-dependence relationship between domain shifts features, enabling the model to perceive domain shifts features efficiently. Finally, we use a Domain Adaptive Network (DAN) to compensate for the performance decrease caused by domain shifts, making the model adapt to anomalous sound better in the self-supervised environment. The performance of the suggested model is validated on DCASE 2020 task 2 and DCASE 2022 task 2.

Immunoaffinity-based liquid biopsies of circulating tumor cells (CTCs) hold great promise for cancer management, but typically suffer from low throughput, relative complexity and post-processing limitations. Here we address these issues simultaneously by decoupling and independently optimizing the nano-, micro- and macro-scales of an enrichment device that is simple to fabricate and operate. Unlike other affinity-based devices, our scalable mesh approach enables optimum capture conditions at any flow rate, as demonstrated with constant capture efficiencies, above 75% between 50-200 uL/min. The device achieved 96% sensitivity and 100% specificity when used to detect CTCs in the blood of 79 cancer patients and 20 healthy controls. We demonstrate its post processing capacity with the identification of potential responders to immune checkpoint inhibition therapy and the detection of HER2 positive breast cancer. The results compare well with other assays, including clinical standards. This suggests that our approach, which overcomes major limitations associated with affinity-based liquid biopsies, could help improve cancer management., Comment: 19 pages, 5 figures (+ supplementary materials: 16 pages, 10 figures)

Recent research has revealed that deep neural networks often take dataset biases as a shortcut to make decisions rather than understand tasks, leading to failures in real-world applications. In this study, we focus on the spurious correlation between word features and labels that models learn from the biased data distribution of training data. In particular, we define the word highly co-occurring with a specific label as biased word, and the example containing biased word as biased example. Our analysis shows that biased examples are easier for models to learn, while at the time of prediction, biased words make a significantly higher contribution to the models' predictions, and models tend to assign predicted labels over-relying on the spurious correlation between words and labels. To mitigate models' over-reliance on the shortcut (i.e. spurious correlation), we propose a training strategy Less-Learn-Shortcut (LLS): our strategy quantifies the biased degree of the biased examples and down-weights them accordingly. Experimental results on Question Matching, Natural Language Inference and Sentiment Analysis tasks show that LLS is a task-agnostic strategy and can improve the model performance on adversarial data while maintaining good performance on in-domain data.

Cavitation has long been recognized as a crucial predictor, or precursor, to the ultimate failure of various materials, ranging from ductile metals to soft and biological materials. Traditionally, cavitation in solids is defined as an unstable expansion of a void or a defect within a material. The critical applied load needed to trigger this instability - the critical pressure - is a lengthscale independent material property and has been predicted by numerous theoretical studies for a breadth of constitutive models. While these studies usually assume that cavitation initiates from defects in the bulk of an otherwise homogeneous medium, an alternative and potentially more ubiquitous scenario can occur if the defects are found at interfaces between two distinct media within the body. Such interfaces are becoming increasingly common in modern materials with the use of multi-material composites and layer-by-layer additive manufacturing methods. However, a criterion to determine the threshold for interfacial failure, in analogy to the bulk cavitation limit, has yet to be reported. In this work we fill this gap. Our theoretical model captures a lengthscale independent limit for interfacial cavitation, and is shown to agree with our observations at two distinct lengthscales, via two different experimental systems. To further understand the competition between the two cavitation modes (bulk versus interface) we expand our investigation beyond the elastic response to understand the ensuing unstable propagation of delamination at the interface. A phase diagram summarizes these results, showing regimes in which interfacial failure becomes the dominant mechanism.