Search

Why do biological and artificial neurons sometimes modularise, each encoding a single meaningful variable, and sometimes entangle their representation of many variables? In this work, we develop a theory of when biologically inspired representations -- those that are nonnegative and energy efficient -- modularise with respect to source variables (sources). We derive necessary and sufficient conditions on a sample of sources that determine whether the neurons in an optimal biologically-inspired linear autoencoder modularise. Our theory applies to any dataset, extending far beyond the case of statistical independence studied in previous work. Rather, we show that sources modularise if their support is "sufficiently spread". From this theory, we extract and validate predictions in a variety of empirical studies on how data distribution affects modularisation in nonlinear feedforward and recurrent neural networks trained on supervised and unsupervised tasks. Furthermore, we apply these ideas to neuroscience data. First, we explain why two studies that recorded prefrontal activity in working memory tasks conflict on whether memories are encoded in orthogonal subspaces: the support of the sources differed due to a critical discrepancy in experimental protocol. Second, we use similar arguments to understand why preparatory and potent subspaces in RNN models of motor cortex are only sometimes orthogonal. Third, we study spatial and reward information mixing in entorhinal recordings, and show our theory matches data better than previous work. And fourth, we suggest a suite of surprising settings in which neurons can be (or appear) mixed selective, without requiring complex nonlinear readouts as in traditional theories. In sum, our theory prescribes precise conditions on when neural activities modularise, providing tools for inducing and elucidating modular representations in brains and machines., Comment: 47 pages, 23 figures. WD and KH contributed equally; LH and JHL contributed equally

Compositionality, the notion that the meaning of an expression is constructed from the meaning of its parts and syntactic rules, permits the infinite productivity of human language. For the first time, artificial language models (LMs) are able to match human performance in a number of compositional generalization tasks. However, much remains to be understood about the representational mechanisms underlying these abilities. We take a high-level geometric approach to this problem by relating the degree of compositionality in a dataset to the intrinsic dimensionality of its representations under an LM, a measure of feature complexity. We find not only that the degree of dataset compositionality is reflected in representations' intrinsic dimensionality, but that the relationship between compositionality and geometric complexity arises due to learned linguistic features over training. Finally, our analyses reveal a striking contrast between linear and nonlinear dimensionality, showing that they respectively encode formal and semantic aspects of linguistic composition., Comment: Under review as a conference paper at ICLR 2025

The United States has experienced a significant increase in violent extremism, prompting the need for automated tools to detect and limit the spread of extremist ideology online. This study evaluates the performance of Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-Trained Transformers (GPT) in detecting and classifying online domestic extremist posts. We collected social media posts containing "far-right" and "far-left" ideological keywords and manually labeled them as extremist or non-extremist. Extremist posts were further classified into one or more of five contributing elements of extremism based on a working definitional framework. The BERT model's performance was evaluated based on training data size and knowledge transfer between categories. We also compared the performance of GPT 3.5 and GPT 4 models using different prompts: na\"ive, layperson-definition, role-playing, and professional-definition. Results showed that the best performing GPT models outperformed the best performing BERT models, with more detailed prompts generally yielding better results. However, overly complex prompts may impair performance. Different versions of GPT have unique sensitives to what they consider extremist. GPT 3.5 performed better at classifying far-left extremist posts, while GPT 4 performed better at classifying far-right extremist posts. Large language models, represented by GPT models, hold significant potential for online extremism classification tasks, surpassing traditional BERT models in a zero-shot setting. Future research should explore human-computer interactions in optimizing GPT models for extremist detection and classification tasks to develop more efficient (e.g., quicker, less effort) and effective (e.g., fewer errors or mistakes) methods for identifying extremist content.

While significant advancements have been made in music generation and differentiable sound synthesis within machine learning and computer audition, the simulation of instrument vibration guided by physical laws has been underexplored. To address this gap, we introduce a novel model for simulating the spatio-temporal motion of nonlinear strings, integrating modal synthesis and spectral modeling within a neural network framework. Our model leverages physical properties and fundamental frequencies as inputs, outputting string states across time and space that solve the partial differential equation characterizing the nonlinear string. Empirical evaluations demonstrate that the proposed architecture achieves superior accuracy in string motion simulation compared to existing baseline architectures. The code and demo are available online., Comment: 38th Conference on Neural Information Processing Systems (NeurIPS 2024)

To ensure safe urban driving for autonomous platforms, it is crucial not only to develop high-performance object detection techniques but also to establish a diverse and representative dataset that captures various urban environments and object characteristics. To address these two issues, we have constructed a multi-class 3D LiDAR dataset reflecting diverse urban environments and object characteristics, and developed a robust 3D semi-supervised object detection (SSOD) based on a multiple teachers framework. This SSOD framework categorizes similar classes and assigns specialized teachers to each category. Through collaborative supervision among these category-specialized teachers, the student network becomes increasingly proficient, leading to a highly effective object detector. We propose a simple yet effective augmentation technique, Pie-based Point Compensating Augmentation (PieAug), to enable the teacher network to generate high-quality pseudo-labels. Extensive experiments on the WOD, KITTI, and our datasets validate the effectiveness of our proposed method and the quality of our dataset. Experimental results demonstrate that our approach consistently outperforms existing state-of-the-art 3D semi-supervised object detection methods across all datasets. We plan to release our multi-class LiDAR dataset and the source code available on our Github repository in the near future., Comment: under review

Wide-field calcium imaging (WFCI) that records neural calcium dynamics allows for identification of functional brain networks (FBNs) in mice that express genetically encoded calcium indicators. Estimating FBNs from WFCI data is commonly achieved by use of seed-based correlation (SBC) analysis and independent component analysis (ICA). These two methods are conceptually distinct and each possesses limitations. Recent success of unsupervised representation learning in neuroimage analysis motivates the investigation of such methods to identify FBNs. In this work, a novel approach referred as LSTM-AER, is proposed in which a long short-term memory (LSTM) autoencoder (AE) is employed to learn spatial-temporal latent embeddings from WFCI data, followed by an ordinary least square regression (R) to estimate FBNs. The goal of this study is to elucidate and illustrate, qualitatively and quantitatively, the FBNs identified by use of the LSTM-AER method and compare them to those from traditional SBC and ICA. It was observed that spatial FBN maps produced from LSTM-AER resembled those derived by SBC and ICA while better accounting for intra-subject variation, data from a single hemisphere, shorter epoch lengths and tunable number of latent components. The results demonstrate the potential of unsupervised deep learning-based approaches to identifying and mapping FBNs.

In this paper, we investigate $L^p$ bounds of maximal Fourier multiplier operators with dilation of fractional dimensions. For the Fourier multipliers, we suggest a criterion related to dimensions of dilation sets which guarantees $L^p$ bounds of the maximal operators for each $p$. Our criterion covers Mikhlin-type multipliers, multipliers with limited decay, and multipliers with slow decay., Comment: 15 pages

Parallel tensor network contraction algorithms have emerged as the pivotal benchmarks for assessing the classical limits of computation, exemplified by Google's demonstration of quantum supremacy through random circuit sampling. However, the massive parallelization of the algorithm makes it vulnerable to computer node failures. In this work, we apply coded computing to a practical parallel tensor network contraction algorithm. To the best of our knowledge, this is the first attempt to code tensor network contractions. Inspired by matrix multiplication codes, we provide two coding schemes: 2-node code for practicality in quantum simulation and hyperedge code for generality. Our 2-node code successfully achieves significant gain for $f$-resilient number compared to naive replication, proportional to both the number of node failures and the dimension product of sliced indices. Our hyperedge code can cover tensor networks out of the scope of quantum, with degraded gain in the exchange of its generality., Comment: Accepted to ISIT2024

This position paper presents the AR Language Map, a speculative artifact designed to enhance understanding of linguistic justice among middle and high school students through augmented reality (AR) that allows students to map their linguistic experiences. Through a social justice-oriented academic outreach program aimed at linguistically, economically, and racially minoritized students, academic concepts on language, culture, race, and power are introduced to California middle school and high school students. The curriculum has activities for each lesson plan drawn from students' culturally relevant experiences. By enabling interactive exploration of linguistic justice, this tool aims to foster empathy, challenge linguistic racism, and valorize linguistic diversity. We discuss its conceptualization within the broader context of AR in social justice education. The AR Language Map not only deepens students' understanding of these critical issues but also enables them to become co-creators of their learning experiences., Comment: Presented at CHI 2024 (arXiv:2404.05889)

Survival analysis is essential for studying time-to-event outcomes and providing a dynamic understanding of the probability of an event occurring over time. Various survival analysis techniques, from traditional statistical models to state-of-the-art machine learning algorithms, support healthcare intervention and policy decisions. However, there remains ongoing discussion about their comparative performance. We conducted a comparative study of several survival analysis methods, including Cox proportional hazards (CoxPH), stepwise CoxPH, elastic net penalized Cox model, Random Survival Forests (RSF), Gradient Boosting machine (GBM) learning, AutoScore-Survival, DeepSurv, time-dependent Cox model based on neural network (CoxTime), and DeepHit survival neural network. We applied the concordance index (C-index) for model goodness-of-fit, and integral Brier scores (IBS) for calibration, and considered the model interpretability. As a case study, we performed a retrospective analysis of patients admitted through the emergency department of a tertiary hospital from 2017 to 2019, predicting 90-day all-cause mortality based on patient demographics, clinicopathological features, and historical data. The results of the C-index indicate that deep learning achieved comparable performance, with DeepSurv producing the best discrimination (DeepSurv: 0.893; CoxTime: 0.892; DeepHit: 0.891). The calibration of DeepSurv (IBS: 0.041) performed the best, followed by RSF (IBS: 0.042) and GBM (IBS: 0.0421), all using the full variables. Moreover, AutoScore-Survival, using a minimal variable subset, is easy to interpret, and can achieve good discrimination and calibration (C-index: 0.867; IBS: 0.044). While all models were satisfactory, DeepSurv exhibited the best discrimination and calibration. In addition, AutoScore-Survival offers a more parsimonious model and excellent interpretability.

Diverse studies in systems neuroscience begin with extended periods of curriculum training known as `shaping' procedures. These involve progressively studying component parts of more complex tasks, and can make the difference between learning a task quickly, slowly or not at all. Despite the importance of shaping to the acquisition of complex tasks, there is as yet no theory that can help guide the design of shaping procedures, or more fundamentally, provide insight into its key role in learning. Modern deep reinforcement learning systems might implicitly learn compositional primitives within their multilayer policy networks. Inspired by these models, we propose and analyse a model of deep policy gradient learning of simple compositional reinforcement learning tasks. Using the tools of statistical physics, we solve for exact learning dynamics and characterise different learning strategies including primitives pre-training, in which task primitives are studied individually before learning compositional tasks. We find a complex interplay between task complexity and the efficacy of shaping strategies. Overall, our theory provides an analytical understanding of the benefits of shaping in a class of compositional tasks and a quantitative account of how training protocols can disclose useful task primitives, ultimately yielding faster and more robust learning., Comment: Accepted to ICML 2024. 5 figures, 9 pages and Appendix

Gaussian and discrete non-Gaussian spatial datasets are prevalent across many fields such as public health, ecology, geosciences, and social sciences. Bayesian spatial generalized linear mixed models (SGLMMs) are a flexible class of models designed for these data, but SGLMMs do not scale well, even to moderately large datasets. State-of-the-art scalable SGLMMs (i.e., basis representations or sparse covariance/precision matrices) require posterior sampling via Markov chain Monte Carlo (MCMC), which can be prohibitive for large datasets. While variational Bayes (VB) have been extended to SGLMMs, their focus has primarily been on smaller spatial datasets. In this study, we propose two computationally efficient VB approaches for modeling moderate-sized and massive (millions of locations) Gaussian and discrete non-Gaussian spatial data. Our scalable VB method embeds semi-parametric approximations for the latent spatial random processes and parallel computing offered by modern high-performance computing systems. Our approaches deliver nearly identical inferential and predictive performance compared to 'gold standard' methods but achieve computational speedups of up to 1000x. We demonstrate our approaches through a comparative numerical study as well as applications to two real-world datasets. Our proposed VB methodology enables practitioners to model millions of non-Gaussian spatial observations using a standard laptop within a short timeframe., Comment: 34 Pages for the main paper, 72 pages for the supplemental information, 4 tables, 5 figures

Acousto-optics consists of launching acoustic waves in a medium (usually a crystal) in order to modulate its refractive index and create a tunable optical grating. In this article, we present the theoretical basis of a new scheme to generate acousto-optics in a gas, where the acoustic waves are initiated by the localized absorption (and thus gas heating) of spatially-modulated UV light, as was demonstrated in Y. Michine and H. Yoneda, Commun. Phys. 3, 24 (2020). We identify the chemical reactions initiated by the absorption of UV light via the photodissociation of ozone molecules present in the gas, and calculate the resulting temperature increase in the gas as a function of space and time. Solving the Euler fluid equations shows that the modulated, isochoric heating initiates a mixed acoustic/entropy wave in the gas, whose high-amplitude density (and thus refractive index) modulation can be used to manipulate a high-power laser. We calculate that diffraction efficiencies near 100% can be obtained using only a few millimeters of gas containing a few percent ozone fraction at room temperature, with UV fluences of less than 100 mJ/cm2, consistent with the experimental measurements. Our analysis suggests possible ways to optimize the diffraction efficiency by changing the buffer gas composition. Gases have optics damage thresholds two to three orders of magnitude beyond those of solids; these optical elements should therefore be able to manipulate kJ-class lasers., Comment: 11 pages, 7 figures

In this article, we focus on $L^{2}(\mathbb{R}^d)\times\cdots\times L^{2}(\mathbb{R}^d)\rightarrow L^{2/m}(\mathbb{R}^d)$ estimates for multilinear maximal averages over non-degenerate hypersurfaces. Our findings is new for $m$-linear averages with $m\geq3$, and represent a reproof of the recent result of T. Borges, B. Foster, and Y. Ou on the curvature conditions of the hypersurfaces required in establishing $L^{2}(\mathbb{R}^d)\times L^{2}(\mathbb{R}^d)\rightarrow L^{1}(\mathbb{R}^d)$ estimates of bilinear maximal functions., Comment: 14 pages. We changed Lemma 2.2 due to some errors, hence the condition in Theorem 1.2, $s>(m-2)d/2 +1$ is replaced by $s>(m-1)d/2 +1/2$

This paper presents an in-depth examination of the nonlinear deformation induced by dielectric actuation in pre-stressed ideal dielectric elastomers. A nonlinear ordinary differential equation that governs this deformation is formulated based on the hyperelastic model under dielectric stress. By means of numerical integration and neural network approximations, the relationship between voltage and stretch is established. Neural networks are utilized to approximate solutions for voltage-to-stretch and stretch-to-voltage transformations obtained via an explicit Runge-Kutta method. The efficacy of these approximations is illustrated by their use in compensating for nonlinearity through the waveshaping of the input signal. The comparative analysis demonstrates that the approximated solutions are more accurate than baseline methods, resulting in reduced harmonic distortions when dielectric elastomers are used as acoustic actuators. This study highlights the effectiveness of the proposed approach in mitigating nonlinearities and enhancing the performance of dielectric elastomers in acoustic actuation applications.

An important aspect of solar energetic particle (SEP) events is their source populations. Elemental abundance enhancements of impulsive SEP events, originating in presumed coronal reconnection episodes, can be fitted to steep power laws of A/Q, where A and Q are the atomic mass and ionic charge. Since thermal electron energies are enhanced and nonthermal electron distributions arise in the reconnection process, we might expect that ionic charge states Q would be increased through ionization interactions with those electron populations during the acceleration process. The temperature estimated from the SEPs corresponds to the charge state during the acceleration process, while the actual charge state measured in situ may be modified as the SEPs pass through the corona. We examine whether the temperature estimation from the A/Q would differ with various kappa values in a kappa function representing high-energy tail deviating from a Maxwellian velocity distribution. We find that the differences in the A/Q between a Maxwellian and an extreme kappa distribution are about 10-30%. We fit power-law enhancement of element abundances as a function of their A/Q with various kappa values. Then, we find that the derived source region temperature is not significantly affected by whether or not the electron velocity distribution deviates from a Maxwellian, i.e., thermal, distribution. Assuming that electrons are heated in the acceleration region, the agreement of the SEP charge state during acceleration with typical active region temperatures suggests that SEPs are accelerated and leave the acceleration region in a shorter time than the ionization time scale., Comment: 14 pages, 10 figures