Search

We present revised point-spread functions (PSFs) for the Atmospheric Imaging Assembly (AIA) onboard the Solar Dynamics Observatory (SDO). These PSFs provide a robust estimate of the light diffracted by the meshes holding the entrance and focal plane filters and the light that is diffusely scattered over medium- to long-distance by the micro-roughness of the mirrors. We first calibrate the diffracted light using flare images. Our modeling of the diffracted light provides reliable determinations of the mesh parameters and finds that about 24 to 33% of the collected light is diffracted, depending on the AIA channel. Then, we fit for the diffuse scattered light using partially lunar occulted images. We find that the diffuse scattered light can be modeled as a superposition of two power law functions that scatter light over the entire length of the detector. The amount of diffuse scattered light ranges from 10 to 35 %, depending on the AIA channel. In total, AIA diffracts and scatters about 40 to 60 % of the collected light over medium to long distances. When correcting for this, bright image regions increase in intensity by about 30 %, dark image regions decrease in intensity by up to 90 %, and the associated differential emission measure analysis of solar features are affected accordingly. Finally, we compare the image reconstructions using our new PSFs to those using the AIA team PSFs and the PSFs of Poduval et al. (2013). We find that our PSFs outperform the others; our PSFs correct well for the flare diffraction pattern and predict accurately the long-distance scattered light in lunar occultations.

A major challenge for transformers is generalizing to sequences longer than those observed during training. While previous works have empirically shown that transformers can either succeed or fail at length generalization depending on the task, theoretical understanding of this phenomenon remains limited. In this work, we introduce a rigorous theoretical framework to analyze length generalization in causal transformers with learnable absolute positional encodings. In particular, we characterize those functions that are identifiable in the limit from sufficiently long inputs with absolute positional encodings under an idealized inference scheme using a norm-based regularizer. This enables us to prove the possibility of length generalization for a rich family of problems. We experimentally validate the theory as a predictor of success and failure of length generalization across a range of algorithmic and formal language tasks. Our theory not only explains a broad set of empirical observations but also opens the way to provably predicting length generalization capabilities in transformers.

Solar physicists routinely utilize observations of Ar-like Fe IX and Cl-like Fe X emission to study a variety of solar structures. However, unidentified lines exist in the Fe IX and Fe X spectra, greatly impeding the spectroscopic diagnostic potential of these ions. Here, we present measurements using the Lawrence Livermore National Laboratory EBIT-I electron beam ion trap in the wavelength range 238-258 A. These studies enable us to unambiguously identify the charge state associated with each of the observed lines. This wavelength range is of particular interest because it contains the Fe IX density diagnostic line ratio 241.74 A/244.91 A, which is predicted to be one of the best density diagnostics of the solar corona, as well as the Fe X 257.26 A magnetic-field-induced transition. We compare our measurements to the Fe IX and Fe X lines tabulated in CHIANTI v10.0.1, which is used for modeling the solar spectrum. In addition, we have measured previously unidentified Fe X lines that will need to be added to CHIANTI and other spectroscopic databases.

Transformer architectures have been widely adopted in foundation models. Due to their high inference costs, there is renewed interest in exploring the potential of efficient recurrent architectures (RNNs). In this paper, we analyze the differences in the representational capabilities of Transformers and RNNs across several tasks of practical relevance, including index lookup, nearest neighbor, recognizing bounded Dyck languages, and string equality. For the tasks considered, our results show separations based on the size of the model required for different architectures. For example, we show that a one-layer Transformer of logarithmic width can perform index lookup, whereas an RNN requires a hidden state of linear size. Conversely, while constant-size RNNs can recognize bounded Dyck languages, we show that one-layer Transformers require a linear size for this task. Furthermore, we show that two-layer Transformers of logarithmic size can perform decision tasks such as string equality or disjointness, whereas both one-layer Transformers and recurrent models require linear size for these tasks. We also show that a log-size two-layer Transformer can implement the nearest neighbor algorithm in its forward pass; on the other hand recurrent models require linear size. Our constructions are based on the existence of $N$ nearly orthogonal vectors in $O(\log N)$ dimensional space and our lower bounds are based on reductions from communication complexity problems. We supplement our theoretical results with experiments that highlight the differences in the performance of these architectures on practical-size sequences., Comment: Preprint

Recently, recurrent models based on linear state space models (SSMs) have shown promising performance in language modeling (LM), competititve with transformers. However, there is little understanding of the in-principle abilities of such models, which could provide useful guidance to the search for better LM architectures. We present a comprehensive theoretical study of the capacity of such SSMs as it compares to that of transformers and traditional RNNs. We find that SSMs and transformers have overlapping but distinct strengths. In star-free state tracking, SSMs implement straightforward and exact solutions to problems that transformers struggle to represent exactly. They can also model bounded hierarchical structure with optimal memory even without simulating a stack. On the other hand, we identify a design choice in current SSMs that limits their expressive power. We discuss implications for SSM and LM research, and verify results empirically on a recent SSM, Mamba.

The inner workings of neural networks can be better understood if we can fully decipher the information encoded in neural activations. In this paper, we argue that this information is embodied by the subset of inputs that give rise to similar activations. We propose InversionView, which allows us to practically inspect this subset by sampling from a trained decoder model conditioned on activations. This helps uncover the information content of activation vectors, and facilitates understanding of the algorithms implemented by transformer models. We present four case studies where we investigate models ranging from small transformers to GPT-2. In these studies, we show that InversionView can reveal clear information contained in activations, including basic information about tokens appearing in the context, as well as more complex information, such as the count of certain tokens, their relative positions, and abstract knowledge about the subject. We also provide causally verified circuits to confirm the decoded information., Comment: NeurIPS 2024; ICML 2024 Mechanistic Interpretability Workshop oral

Human language is a unique form of communication in the natural world, distinguished by its structured nature. Most fundamentally, it is systematic, meaning that signals can be broken down into component parts that are individually meaningful -- roughly, words -- which are combined in a regular way to form sentences. Furthermore, the way in which these parts are combined maintains a kind of locality: words are usually concatenated together, and they form contiguous phrases, keeping related parts of sentences close to each other. We address the challenge of understanding how these basic properties of language arise from broader principles of efficient communication under information processing constraints. Here we show that natural-language-like systematicity arises in codes that are constrained by predictive information, a measure of the amount of information that must be extracted from the past of a sequence in order to predict its future. In simulations, we show that such codes approximately factorize their source distributions, and then express the resulting factors systematically and locally. Next, in a series of cross-linguistic corpus studies, we show that human languages are structured to have low predictive information at the levels of phonology, morphology, syntax, and semantics. Our result suggests that human language performs a sequential, discrete form of Independent Components Analysis on the statistical distribution over meanings that need to be expressed. It establishes a link between the statistical and algebraic structure of human language, and reinforces the idea that the structure of human language is shaped by communication under cognitive constraints.

Empirical studies have identified a range of learnability biases and limitations of transformers, such as a persistent difficulty in learning to compute simple formal languages such as PARITY, and a bias towards low-degree functions. However, theoretical understanding remains limited, with existing expressiveness theory either overpredicting or underpredicting realistic learning abilities. We prove that, under the transformer architecture, the loss landscape is constrained by the input-space sensitivity: Transformers whose output is sensitive to many parts of the input string inhabit isolated points in parameter space, leading to a low-sensitivity bias in generalization. We show theoretically and empirically that this theory unifies a broad array of empirical observations about the learning abilities and biases of transformers, such as their generalization bias towards low sensitivity and low degree, and difficulty in length generalization for PARITY. This shows that understanding transformers' inductive biases requires studying not just their in-principle expressivity, but also their loss landscape., Comment: ACL 2024

We report the first experimental detection of a reflected Alfv\'en wave from an Alfv\'en-speed gradient under conditions similar to those in coronal holes. The experiments were conducted in the Large Plasma Device at the University of California, Los Angeles. We present the experimentally measured dependence of the coefficient of reflection versus the wave inhomogeneity parameter, i.e., the ratio of the wave length of the incident wave to the length scale of the gradient. Two-fluid simulations using the Gkeyll code qualitatively agree with and support the experimental findings. Our experimental results support models of wave heating that rely on wave reflection at low heights from a smooth Alfv\'en-speed gradient to drive turbulence.

While natural languages differ widely in both canonical word order and word order flexibility, their word orders still follow shared cross-linguistic statistical patterns, often attributed to functional pressures. In the effort to identify these pressures, prior work has compared real and counterfactual word orders. Yet one functional pressure has been overlooked in such investigations: the uniform information density (UID) hypothesis, which holds that information should be spread evenly throughout an utterance. Here, we ask whether a pressure for UID may have influenced word order patterns cross-linguistically. To this end, we use computational models to test whether real orders lead to greater information uniformity than counterfactual orders. In our empirical study of 10 typologically diverse languages, we find that: (i) among SVO languages, real word orders consistently have greater uniformity than reverse word orders, and (ii) only linguistically implausible counterfactual orders consistently exceed the uniformity of real orders. These findings are compatible with a pressure for information uniformity in the development and usage of natural languages.

We have measured line widths in active region coronal loops in order to determine whether the non-thermal broadening is anisotropic with respect to the magnetic field direction. These non-thermal velocities are caused by unresolved fluid motions. Our analysis method combines spectroscopic data and a magnetic field extrapolation. We analyzed spectra from the Extreme Ultraviolet Imaging Spectrometer on Hinode. A differential emission measure analysis showed that many spectral lines that are commonly considered to be formed in the active region have a substantial contribution from the background quiet Sun. From these spectra we identified lines whose emission was dominated by the active region loops rather than background sources. Using these lines, we constructed maps of the non-thermal velocity. With data from the Helioseismic Magnetic Imager on the Solar Dynamics Observatory and the Coronal Modeling System nonlinear force-free magnetic field reconstruction code, we traced several of the magnetic field lines through the active region. Comparing the spectroscopic and magnetic data, we looked for correlations of non-thermal velocity with the viewing angle between the line of sight and the magnetic field. We found that non-thermal velocities show a weak anti-correlation with the viewing angle. That is, the tendency is for the non-thermal velocity to be slightly larger in the parallel direction. This parallel broadening may be due to acoustic waves or unresolved parallel flows., Comment: Submitted to the Astrophysical Journal

Scaling large language models (LLMs) leads to an emergent capacity to learn in-context from example demonstrations. Despite progress, theoretical understanding of this phenomenon remains limited. We argue that in-context learning relies on recombination of compositional operations found in natural language data. We derive an information-theoretic bound showing how in-context learning abilities arise from generic next-token prediction when the pretraining distribution has sufficient amounts of compositional structure, under linguistically motivated assumptions. A second bound provides a theoretical justification for the empirical success of prompting LLMs to output intermediate steps towards an answer. To validate theoretical predictions, we introduce a controlled setup for inducing in-context learning; unlike previous approaches, it accounts for the compositional nature of language. Trained transformers can perform in-context learning for a range of tasks, in a manner consistent with the theoretical results. Mirroring real-world LLMs in a miniature setup, in-context learning emerges when scaling parameters and data, and models perform better when prompted to output intermediate steps. Probing shows that in-context learning is supported by a representation of the input's compositional structure. Taken together, these results provide a step towards theoretical understanding of emergent behavior in large language models.

The point-spread function (PSF) of an imaging system describes the response of the system to a point source. Accurately determining the PSF enables one to correct for the combined effects of focussing and scattering within the imaging system, and thereby enhance the spatial resolution and dynamic contrast of the resulting images. We present a semi-empirical semi-blind methodology to derive a PSF from partially occulted images. We partition the two-dimensional PSF into multiple segments, set up a multi-linear system of equations, and directly fit the system of equations to determine the PSF weight in each segment. The algorithm is guaranteed to converge towards the correct instrumental PSF for a large class of occultations, does not require a predefined functional form of the PSF, and can be applied to a large variety of partially occulted images, such as within laboratory settings, regular calibrations within a production line or in the field, astronomical images of distant clusters of stars, or partial solar eclipse images. We show that the central weight of the PSF, which gives the percentage of photons that are not scattered by the instrument, is accurate to bettern than 1.2%. The mean absolute percentage error between the reconstructed and true PSF is usually between 0.5% and 5% for the entire PSF, between 0.5% and 5% for the PSF core, and between 0.5% and 3% for the PSF tail.

Languages vary considerably in syntactic structure. About 40% of the world's languages have subject-verb-object order, and about 40% have subject-object-verb order. Extensive work has sought to explain this word order variation across languages. However, the existing approaches are not able to explain coherently the frequency distribution and evolution of word order in individual languages. We propose that variation in word order reflects different ways of balancing competing pressures of dependency locality and information locality, whereby languages favor placing elements together when they are syntactically related or contextually informative about each other. Using data from 80 languages in 17 language families and phylogenetic modeling, we demonstrate that languages evolve to balance these pressures, such that word order change is accompanied by change in the frequency distribution of the syntactic structures which speakers communicate to maintain overall efficiency. Variability in word order thus reflects different ways in which languages resolve these evolutionary pressures. We identify relevant characteristics that result from this joint optimization, particularly the frequency with which subjects and objects are expressed together for the same verb. Our findings suggest that syntactic structure and usage across languages co-adapt to support efficient communication under limited cognitive resources., Comment: Preprint of peer-reviewed paper published in PNAS. Final copyedited version is available at: https://www.pnas.org/doi/10.1073/pnas.2122604119

We find evidence for the first observation of the parametric decay instability (PDI) in the lower solar atmosphere. Specifically, we find that the power spectrum of density fluctuations near the solar transition region resembles the power spectrum of the velocity fluctuations, but with the frequency axis scaled up by about a factor of two. These results are from an analysis of the Si IV lines observed by the Interface Region Imaging Spectrometer (IRIS) in the transition region of a polar coronal hole. We also find that the density fluctuations have radial velocity of about 75 km/s and that the velocity fluctuations are much faster with an estimated speed of 250 km/s, as is expected for sound waves and Alfv\'en waves, respectively, in the transition region. Theoretical calculations show that this frequency relationship is consistent with those expected from PDI for the plasma conditions of the observed region. These measurements suggest an interaction between sound waves and Alfv\'en waves in the transition region that is evidence for the parametric decay instability., Comment: Submitted to the Astrophysical Journal