Search

Encoding constraints into neural networks is attractive. This paper studies how to introduce the popular positive linear satisfiability to neural networks. We propose the first differentiable satisfiability layer based on an extension of the classic Sinkhorn algorithm for jointly encoding multiple sets of marginal distributions. We further theoretically characterize the convergence property of the Sinkhorn algorithm for multiple marginals. In contrast to the sequential decision e.g.\ reinforcement learning-based solvers, we showcase our technique in solving constrained (specifically satisfiability) problems by one-shot neural networks, including i) a neural routing solver learned without supervision of optimal solutions; ii) a partial graph matching network handling graphs with unmatchable outliers on both sides; iii) a predictive network for financial portfolios with continuous constraints. To our knowledge, there exists no one-shot neural solver for these scenarios when they are formulated as satisfiability problems. Source code is available at https://github.com/Thinklab-SJTU/LinSATNet, Comment: This is a revised version of our ICML'23 publication that fixes a minor issue in Eq (11). In Proceedings of the 40th International Conference on Machine Learning (ICML'23)

This technical report presents the implementation details of 2nd winning for CVPR'24 UG2 WeatherProof Dataset Challenge. This challenge aims at semantic segmentation of images degraded by various degrees of weather from all around the world. We addressed this problem by introducing a pre-trained large-scale vision foundation model: InternImage, and trained it using images with different levels of noise. Besides, we did not use additional datasets in the training procedure and utilized dense-CRF as post-processing in the final testing procedure. As a result, we achieved 2nd place in the challenge with 45.1 mIOU and fewer submissions than the other winners.

Neural language models, particularly large-scale ones, have been consistently proven to be most effective in predicting brain neural activity across a range of studies. However, previous research overlooked the comparison of these models with psychologically plausible ones. Moreover, evaluations were reliant on limited, single-modality, and English cognitive datasets. To address these questions, we conducted an analysis comparing encoding performance of various neural language models and psychologically plausible models. Our study utilized extensive multi-modal cognitive datasets, examining bilingual word and discourse levels. Surprisingly, our findings revealed that psychologically plausible models outperformed neural language models across diverse contexts, encompassing different modalities such as fMRI and eye-tracking, and spanning languages from English to Chinese. Among psychologically plausible models, the one incorporating embodied information emerged as particularly exceptional. This model demonstrated superior performance at both word and discourse levels, exhibiting robust prediction of brain activation across numerous regions in both English and Chinese.

Despite differing from the human language processing mechanism in implementation and algorithms, current language models demonstrate remarkable human-like or surpassing language capabilities. Should computational language models be employed in studying the brain, and if so, when and how? To delve into this topic, this paper reviews efforts in using computational models for brain research, highlighting emerging trends. To ensure a fair comparison, the paper evaluates various computational models using consistent metrics on the same dataset. Our analysis reveals that no single model outperforms others on all datasets, underscoring the need for rich testing datasets and rigid experimental control to draw robust conclusions in studies involving computational models.

Pre-trained computational language models have recently made remarkable progress in harnessing the language abilities which were considered unique to humans. Their success has raised interest in whether these models represent and process language like humans. To answer this question, this paper proposes MulCogBench, a multi-modal cognitive benchmark dataset collected from native Chinese and English participants. It encompasses a variety of cognitive data, including subjective semantic ratings, eye-tracking, functional magnetic resonance imaging (fMRI), and magnetoencephalography (MEG). To assess the relationship between language models and cognitive data, we conducted a similarity-encoding analysis which decodes cognitive data based on its pattern similarity with textual embeddings. Results show that language models share significant similarities with human cognitive data and the similarity patterns are modulated by the data modality and stimuli complexity. Specifically, context-aware models outperform context-independent models as language stimulus complexity increases. The shallow layers of context-aware models are better aligned with the high-temporal-resolution MEG signals whereas the deeper layers show more similarity with the high-spatial-resolution fMRI. These results indicate that language models have a delicate relationship with brain language representations. Moreover, the results between Chinese and English are highly consistent, suggesting the generalizability of these findings across languages.

Transformer-based models, even though achieving super-human performance on several downstream tasks, are often regarded as a black box and used as a whole. It is still unclear what mechanisms they have learned, especially their core module: multi-head attention. Inspired by functional specialization in the human brain, which helps to efficiently handle multiple tasks, this work attempts to figure out whether the multi-head attention module will evolve similar function separation under multi-tasking training. If it is, can this mechanism further improve the model performance? To investigate these questions, we introduce an interpreting method to quantify the degree of functional specialization in multi-head attention. We further propose a simple multi-task training method to increase functional specialization and mitigate negative information transfer in multi-task learning. Experimental results on seven pre-trained transformer models have demonstrated that multi-head attention does evolve functional specialization phenomenon after multi-task training which is affected by the similarity of tasks. Moreover, the multi-task training strategy based on functional specialization boosts performance in both multi-task learning and transfer learning without adding any parameters., Comment: Accepted to EMNLP 2023 main conference. Our code is available at https://github.com/ZNLP/FunctionalSpecializationInMHA

Decoding visual stimuli from neural responses recorded by functional Magnetic Resonance Imaging (fMRI) presents an intriguing intersection between cognitive neuroscience and machine learning, promising advancements in understanding human visual perception and building non-invasive brain-machine interfaces. However, the task is challenging due to the noisy nature of fMRI signals and the intricate pattern of brain visual representations. To mitigate these challenges, we introduce a two-phase fMRI representation learning framework. The first phase pre-trains an fMRI feature learner with a proposed Double-contrastive Mask Auto-encoder to learn denoised representations. The second phase tunes the feature learner to attend to neural activation patterns most informative for visual reconstruction with guidance from an image auto-encoder. The optimized fMRI feature learner then conditions a latent diffusion model to reconstruct image stimuli from brain activities. Experimental results demonstrate our model's superiority in generating high-resolution and semantically accurate images, substantially exceeding previous state-of-the-art methods by 39.34% in the 50-way-top-1 semantic classification accuracy. Our research invites further exploration of the decoding task's potential and contributes to the development of non-invasive brain-machine interfaces., Comment: Accepted by NeurIPS2023

With the success of down streaming task using English pre-trained language model, the pre-trained Chinese language model is also necessary to get a better performance of Chinese NLP task. Unlike the English language, Chinese has its special characters such as glyph information. So in this article, we propose the Chinese pre-trained language model StyleBERT which incorporate the following embedding information to enhance the savvy of language model, such as word, pinyin, five stroke and chaizi. The experiments show that the model achieves well performances on a wide range of Chinese NLP tasks.

This paper presents Basil, the first transactional, leaderless Byzantine Fault Tolerant key-value store. Basil leverages ACID transactions to scalably implement the abstraction of a trusted shared log in the presence of Byzantine actors. Unlike traditional BFT approaches, Basil executes non-conflicting operations in parallel and commits transactions in a single round-trip during fault-free executions. Basil improves throughput over traditional BFT systems by four to five times, and is only four times slower than TAPIR, a non-Byzantine replicated system. Basil's novel recovery mechanism further minimizes the impact of failures: with 30% Byzantine clients, throughput drops by less than 25% in the worst-case., Comment: 24 pages. 7 Figures. To be published at SOSP'21

Wave steering by artificial materials (for example, phononic crystals and acoustic metamaterials) is a fascinating frontier in modern physics and engineering, but suffers from bulky sizes and intractable challenges in fabrication. Here, a sparse layer of identical tiny scatters, which we call meta-fences, is presented with a non-destructive way to omnidirectionally block flexural waves in plates. The underlying mechanism is that the restraining force and moment of the scatter are tuned simultaneously to counter-balance the incident wave. Both our experimental results and numerical analysis have demonstrated that broadband wave sources ranging from 3 to 7 kHz can be segregated from the protected area by the meta-fence. In addition, the meta-fence is further assembled into a waveguide routing with an arbitrary configuration. Compared with previous isolators and waveguides, our meta-fences exhibit absolute advantages in compact size, flexible configuration, and high structural strength. The current scenario sheds light on the design of lightweight-and-strong architectures for vibration control and energy harvesting with a high efficiency, and can be extended to microfluidics, acoustics, seismology and other fields., Comment: 17 pages, 5 figures

Time-series is ubiquitous across applications, such as transportation, finance and healthcare. Time-series is often influenced by external factors, especially in the form of asynchronous events, making forecasting difficult. However, existing models are mainly designated for either synchronous time-series or asynchronous event sequence, and can hardly provide a synthetic way to capture the relation between them. We propose Variational Synergetic Multi-Horizon Network (VSMHN), a novel deep conditional generative model. To learn complex correlations across heterogeneous sequences, a tailored encoder is devised to combine the advances in deep point processes models and variational recurrent neural networks. In addition, an aligned time coding and an auxiliary transition scheme are carefully devised for batched training on unaligned sequences. Our model can be trained effectively using stochastic variational inference and generates probabilistic predictions with Monte-Carlo simulation. Furthermore, our model produces accurate, sharp and more realistic probabilistic forecasts. We also show that modeling asynchronous event sequences is crucial for multi-horizon time-series forecasting., Comment: Accepted by AAAI 2021 conference

Zeckendorf's Theorem states that every positive integer can be uniquely represented as a sum of non-adjacent Fibonacci numbers, indexed from $1, 2, 3, 5,\ldots$. This has been generalized by many authors, in particular to constant coefficient fixed depth linear recurrences with positive (or in some cases non-negative) coefficients. In this work we extend this result to a recurrence with non-constant coefficients, $a_{n+1} = n a_{n} + a_{n-1}$. The decomposition law becomes every $m$ has a unique decomposition as $\sum s_i a_i$ with $s_i \le i$, where if $s_i = i$ then $s_{i-1} = 0$. Similar to Zeckendorf's original proof, we use the greedy algorithm. We show that almost all the gaps between summands, as $n$ approaches infinity, are of length zero, and give a heuristic that the distribution of the number of summands tends to a Gaussian. Furthermore, we build a game based upon this recurrence relation, generalizing a game on the Fibonacci numbers. Given a fixed integer $n$ and an initial decomposition of $n= na_1$, the players alternate by using moves related to the recurrence relation, and whoever moves last wins. We show that the game is finite and ends at the unique decomposition of $n$, and that either player can win in a two-player game. We find the strategy to attain the shortest game possible, and the length of this shortest game. Then we show that in this generalized game when there are more than three players, no player has the winning strategy. Lastly, we demonstrate how one player in the two-player game can force the game to progress to their advantage., Comment: 21 pages, 1 figure, from Zeckendorf Polymath REU and the Eureka Program

We present a submillimetre continuum survey ('SCUBA2 High rEdshift bRight quasaR surveY', hereafter SHERRY) of 54 high redshift quasars at $5.63.5\sigma$). The new SHERRY detections indicate far-infrared (FIR) luminosities of $\rm 3.5\times10^{12}$ to $\rm 1.4\times10^{13}$ $L_{\odot}$, implying extreme star formation rates of 90 to 1060 $M_{\odot}$ yr$^{-1}$ in the quasar host galaxies. Compared with $z =$ 2$-$5 samples, the FIR luminous quasars ($L_{\rm FIR} > 10^{13}\,L_{\odot}$) are more rare at $z \sim 6$. The optical/near-infrared (NIR) spectra of these objects show 11% (6/54) of the sources have weak Ly$\alpha$, emission line features, which may relate to different sub-phases of the central active galactic nuclei (AGNs). Our SCUBA2 survey confirms the trend reported in the literature that quasars with submillimeter detections tend to have weaker ultraviolet (UV) emission lines compared to quasars with nondetections. The connection between weak UV quasar line emission and bright dust continuum emission powered by massive star formation may suggest an early phase of AGN-galaxy evolution, in which the broad line region is starting to develop slowly or is shielded from the central ionization source, and has unusual properties such as weak line features or bright FIR emission., Comment: 28 pages, 10 figures, published in ApJ

As 2D materials with subwavelength structures, elastic metasurfaces show remarkable abilities to manipulate elastic waves at will through artificial boundary conditions. However, the application prospects of current metasurfaces may be restricted by their phase-only modulating boundaries. Herein, we present the next generation of elastic metasurfaces by additionally incorporating amplitude-shift modulation. A general theory for target wave fields steered by metasurfaces is proposed by modifying the Huygens-Fresnel principle. As examples, two amplitude-shift metasurfaces concerning flexural waves in thin plates are carried out: one is to transform a cylindrical wave into a Gaussian beam by elaborating both amplitude and phase shifts, and the other one is to focus the incidence by amplitude modulations only. These examples coincide well over theoretical calculations, numerical simulations and experimental tests. This work may underlie the design of metasurfaces with complete control over guided elastic waves, and may extend to more sophisticated applications, such as analog signal processing and holographic imaging., Comment: 32 pages, 12 figures

We use persistent homology along with the eigenfunctions of the Laplacian to study similarity amongst triangulated 2-manifolds. Our method relies on studying the lower-star filtration induced by the eigenfunctions of the Laplacian. This gives us a shape descriptor that inherits the rich information encoded in the eigenfunctions of the Laplacian. Moreover, the similarity between these descriptors can be easily computed using tools that are readily available in Topological Data Analysis. We provide experiments to illustrate the effectiveness of the proposed method.