Search

We present a novel OCR-free document understanding framework based on pretrained Multimodal Large Language Models (MLLMs). Our approach employs multi-scale visual features to effectively handle various font sizes within document images. To address the increasing costs of considering the multi-scale visual inputs for MLLMs, we propose the Hierarchical Visual Feature Aggregation (HVFA) module, designed to reduce the number of input tokens to LLMs. Leveraging a feature pyramid with cross-attentive pooling, our approach effectively manages the trade-off between information loss and efficiency without being affected by varying document image sizes. Furthermore, we introduce a novel instruction tuning task, which facilitates the model's text-reading capability by learning to predict the relative positions of input text, eventually minimizing the risk of truncated text caused by the limited capacity of LLMs. Comprehensive experiments validate the effectiveness of our approach, demonstrating superior performance in various document understanding tasks., Comment: NeurIPS 2024

The single variant spin cycloid and associated antiferromagnetic order in multiferroic BiFeO3 can provide a direct and predictable magnetoelectric coupling to ferroelectric order for deterministic switching, and also a key to fundamental understanding of spin transport and magnon-based applications in the system. (111) oriented BiFeO3 supplies an easy magnetic plane for the spin cycloid, but despite previous efforts, achieving deterministic switching of the single spin cycloid over multiple cycles remains challenging due to the presence of multiple spin cycloid domains in the (111) plane. Here we show that anisotropic in-plane strain engineering can stabilize a single antiferromagnetic domain and provide robust, deterministic switching. We grow BiFeO3 on orthorhombic NdGaO3 (011)o [(111)pc] substrates, breaking the spin cycloid degeneracy and by imposing a uniaxial strain in the (111) plane. This stabilization is confirmed through direct imaging with scanning NV microscopy and non-resonant X-ray diffraction. Remarkably, we achieved deterministic and non-volatile 180{\deg} switching of ferroelectric and associated antiferromagnetic domains over 1,000 cycles, significantly outperforming existing approaches. Our findings underscore that anisotropic strain engineering opens up exciting possibilities for (111)pc monodomain BiFeO3 in potential magnetoelectric and emerging magnonic applications., Comment: 17 pages, 4 figures

In-context learning (ICL) describes a language model's ability to generate outputs based on a set of input demonstrations and a subsequent query. To understand this remarkable capability, researchers have studied simplified, stylized models. These studies have consistently observed long loss plateaus, during which models exhibit minimal improvement, followed by a sudden, rapid surge of learning. In this work, we reveal that training on multiple diverse ICL tasks simultaneously shortens the loss plateaus, making each task easier to learn. This finding is surprising as it contradicts the natural intuition that the combined complexity of multiple ICL tasks would lengthen the learning process, not shorten it. Our result suggests that the recent success in large-scale training of language models may be attributed not only to the richness of the data at scale but also to the easier optimization (training) induced by the diversity of natural language training data.

Environmental hazards place certain individuals at disproportionately higher risks. As these hazards increasingly endanger human health, precise identification of the most vulnerable population subgroups is critical for public health. Moderated multiple regression (MMR) offers a straightforward method for investigating this by adding interaction terms between the exposure to a hazard and other population characteristics to a linear regression model. However, when the vulnerabilities are hidden within a cross-section of many characteristics, MMR is often limited in its capabilities to find any meaningful discoveries. Here, we introduce a hybrid method, named regression-guided neural networks (ReGNN), which utilizes artificial neural networks (ANNs) to non-linearly combine predictors, generating a latent representation that interacts with a focal predictor (i.e. variable measuring exposure to an environmental hazard). We showcase the use of ReGNN for investigating the population heterogeneity in the health effects of exposure to air pollution (PM2.5) on cognitive functioning scores. We demonstrate that population heterogeneity that would otherwise be hidden using traditional MMR can be found using ReGNN by comparing its results to the fit results of the traditional MMR models. In essence, ReGNN is a novel tool that enhances traditional regression models by effectively summarizing and quantifying an individual's susceptibility to health risks.

This paper describes the 2nd edition of the ICML Topological Deep Learning Challenge that was hosted within the ICML 2024 ELLIS Workshop on Geometry-grounded Representation Learning and Generative Modeling (GRaM). The challenge focused on the problem of representing data in different discrete topological domains in order to bridge the gap between Topological Deep Learning (TDL) and other types of structured datasets (e.g. point clouds, graphs). Specifically, participants were asked to design and implement topological liftings, i.e. mappings between different data structures and topological domains --like hypergraphs, or simplicial/cell/combinatorial complexes. The challenge received 52 submissions satisfying all the requirements. This paper introduces the main scope of the challenge, and summarizes the main results and findings., Comment: Proceedings of the Geometry-grounded Representation Learning and Generative Modeling Workshop (GRaM) at ICML 2024

Unconventional spin-orbit torques arising from electric-field-generated spin currents in anisotropic materials have promising potential for spintronic applications, including for perpendicular magnetic switching in high-density memory applications. Here we determine all the independent elements of the spin torque conductivity tensor allowed by bulk crystal symmetries for the tetragonal conductor IrO2, via measurements of conventional (in plane) antidamping torques for IrO2 thin films in the high-symmetry (001) and (100) orientations. We then test that rotational transformations of this same tensor can predict both the conventional and unconventional anti-damping torques for IrO2 thin films in the lower-symmetry (101), (110), and (111) orientations, finding good agreement. The results confirm that spin-orbit torques from all these orientations are consistent with the bulk symmetries of IrO2, and show how simple measurements of conventional torques from high-symmetry orientations of anisotropic thin films can provide an accurate prediction of the unconventional torques from lower-symmetry orientations., Comment: Main text: 14 pages, 3 figures. Supporting Information: 19 pages, 10 figures, and 1 table

We propose a novel and comprehensive particle physics framework that addresses multiple cosmological tensions observed in recent measurements of the Hubble parameter, $S_8$, and Lyman-$\alpha$ forest data. Our model, termed `{\bf SIDR$+\boldsymbol{z_t}$}' (Self Interacting Dark Radiation with transition redshift), is based on an inelastic dark matter (IDM) scenario coupled with dark radiation, governed by a $U(1)_D$ gauge symmetry. This framework naturally incorporates cold dark matter (DM), strongly interacting dark radiation (SIDR), and the interactions between these components. The fluid-like behavior of the dark radiation component which originates from the self-quartic coupling of the $U(1)_D$ breaking scalar can suppress the free-streaming effects. Simultaneously, the interacting DM-DR system can attenuate the matter power spectrum at small scales. The inelastic nature of DM provides a distinct temperature dependence for the DM-DR interaction rate determined by the mass-splitting between the inelastic dark fermions which is crucial for resolving the Ly-$\alpha$ discrepancies. We present a cosmologically consistent analysis of the model by solving the relevant Boltzmann equations to obtain the energy density and number density evolution of different species of the model. The DR undergoes two ``steps" of increased energy density when the heavier dark species freeze out and become non-relativistic, transferring their entropy to the dark radiation and enhancing $\Delta N_{\rm eff}$. The analysis showcases the model's potential to uphold the Big Bang Nucleosynthesis (BBN) prediction of $\Delta N_{\rm eff}$ but dominantly producing additional contributions prior to recombination, while simultaneously achieving correct relic density of DM though an hybrid of freeze-in and non-thermal production., Comment: 27 pages, 7 captioned figures, Updated to match published version

We study a mechanism to make dark matter stable based on the Pauli blocking in the fermion background. In the background where fermions occupy the states, the decay of dark matter to those final states is not allowed, as a result, DM becomes stable. We derive the evolution equations of the distribution function in the quantum field theory and compare it with the Boltzmann equation. We apply this mechanism to a realistic model of neutrino and dark matter., Comment: 19 pages, 9 figures

Securing a sufficient amount of paired data is important to train an image-text retrieval (ITR) model, but collecting paired data is very expensive. To address this issue, in this paper, we propose an active learning algorithm for ITR that can collect paired data cost-efficiently. Previous studies assume that image-text pairs are given and their category labels are asked to the annotator. However, in the recent ITR studies, the importance of category label is decreased since a retrieval model can be trained with only image-text pairs. For this reason, we set up an active learning scenario where unpaired images (or texts) are given and the annotator provides corresponding texts (or images) to make paired data. The key idea of the proposed AL algorithm is to select unpaired images (or texts) that can be hard negative samples for existing texts (or images). To this end, we introduce a novel scoring function to choose hard negative samples. We validate the effectiveness of the proposed method on Flickr30K and MS-COCO datasets.

Current state-of-the-art diffusion models employ U-Net architectures containing convolutional and (qkv) self-attention layers. The U-Net processes images while being conditioned on the time embedding input for each sampling step and the class or caption embedding input corresponding to the desired conditional generation. Such conditioning involves scale-and-shift operations to the convolutional layers but does not directly affect the attention layers. While these standard architectural choices are certainly effective, not conditioning the attention layers feels arbitrary and potentially suboptimal. In this work, we show that simply adding LoRA conditioning to the attention layers without changing or tuning the other parts of the U-Net architecture improves the image generation quality. For example, a drop-in addition of LoRA conditioning to EDM diffusion model yields FID scores of 1.91/1.75 for unconditional and class-conditional CIFAR-10 generation, improving upon the baseline of 1.97/1.79.

In this article, we investigate the stochastic gravitational waves (GWs) spectrum, resulting from the emission of gravitons through bremsstrahlung, in the decay of particles produced by Hawking radiation. Although particle decays inevitably entail the emission of graviton due to bremsstrahlung, the associated decay width is notably suppressed due to the Planck scale suppression in the coupling of matter fields to gravitons. Consequently, the relic abundance of such GWs constituted of these gravitons undergoes a corresponding reduction. However, we demonstrate that super-heavy particles, reaching masses as high as Planck scale, can emerge naturally in the Hawking radiation of evaporating primordial black holes (PBHs) and can compensate for this suppression. In addition, we also discuss the stochastic gravitational waves constituted out of the gravitons directly radiated from such evaporating PBHs. When the super-heavy particle decays promptly after its production, then the corresponding GW spectrum remains subdominant to the one arising from direct PBH evaporation. However, if this particle is long-lived and decays after PBH evaporation, then the resulting GWs produced in these two processes have two distinct spectra with their peaks at extremely high frequencies, providing avenues for proposed ultra-high frequency gravitational wave detectors. We also show that such gravitational waves contribute significantly to substantial dark radiation, which can be probed with the enhanced sensitivity of future experiments., Comment: 19 pages, 5 figures; matches the version published in JCAP

While existing motion style transfer methods are effective between two motions with identical content, their performance significantly diminishes when transferring style between motions with different contents. This challenge lies in the lack of clear separation between content and style of a motion. To tackle this challenge, we propose a novel motion style transformer that effectively disentangles style from content and generates a plausible motion with transferred style from a source motion. Our distinctive approach to achieving the goal of disentanglement is twofold: (1) a new architecture for motion style transformer with `part-attentive style modulator across body parts' and `Siamese encoders that encode style and content features separately'; (2) style disentanglement loss. Our method outperforms existing methods and demonstrates exceptionally high quality, particularly in motion pairs with different contents, without the need for heuristic post-processing. Codes are available at https://github.com/Boeun-Kim/MoST., Comment: Accepted by CVPR 2024

We study the expected sensitivity to the mixing between sterile and tau neutrinos directly from the tau neutrino disappearance in the high-energy fixed target experiment. Here, the beam energy is large enough to produce tau neutrinos at the target with large luminosity. During their propagation to the detector, tau neutrinos may oscillate into sterile neutrinos. By examining the energy spectrum of the observed tau neutrino events, we can probe the mixing between sterile and tau neutrinos directly. In this paper, we consider Scattering and Neutrino Detector (SND) at SHiP experiment as a showcase, which uses 400 GeV protons from SPS at CERN, and expect to observe 7,300 tau and anti-tau neutrinos from the $2\times 10^{20}$ POT for 5 years operation. Assuming the uncertainty of 10\%, we find the sensitivity $|U_{\tau 4}|^2 \sim 0.08$\, (90\% CL) for $\Delta m_{41}^2 \sim 500\ \mathrm{eV}^2$ with 10\% background to the signal. We also consider a far SND at the end of the SHiP Hidden Sector Decay Spectrometer (HSDS), in which case the sensitivity would be enhanced to $|U_{\tau 4}|^2 \sim 0.02$. Away from this mass, the sensitivity becomes lower than $|U_{\tau 4}|^2 \sim 0.15$ for $\Delta m_{41}^2 \lesssim 100\ \mathrm{eV}^2$ or $\Delta m_{41}^2\gtrsim 10^4 \mathrm{eV}^2$., Comment: 14 pages, 8 figures

Ferroelectric domain walls, recognized as distinct from the bulk in terms of symmetry, structure, and electronic properties, host exotic phenomena including conductive walls, ferroelectric vortices, novel topologies, and negative capacitance. Contrary to conventional understanding, our study reveals that the structure of domain walls in HfO2 closely resembles its bulk. First, our first-principles simulations unveil that the robust ferroelectricity is supported by bosonic pairing of all the anionic phonons in bulk HfO2. Strikingly, the paired phonons strongly bond with each other and successfully reach the center of the domain wall without losing their integrity and produce bulk-like domain walls. We then confirmed preservation of the bulk phonon displacements and consequently full revival of the bulk structure at domain walls via aberration-corrected STEM. The newly found duality between the bulk and the domain wall sheds light on previously enigmatic properties such as zero-energy domain walls, perfect Ising-type polar ordering, and exceptionally robust ferroelectricity at the sub-nm scales. The phonon-pairing discovered here is robust against physical boundaries such as domain walls and enables zero momentum and zero-energy cost local ferroelectric switching. This phenomenon demonstrated in Si-compatible ferroelectrics provides a novel technological platform where data storage on domain walls is as feasible as that within the domains, thereby expanding the potential for high-density data storage and advanced ferroelectric applications., Comment: 24 pages, 4 figures

Transformers have revolutionized performance in Natural Language Processing and Vision, paving the way for their integration with Graph Neural Networks (GNNs). One key challenge in enhancing graph transformers is strengthening the discriminative power of distinguishing isomorphisms of graphs, which plays a crucial role in boosting their predictive performances. To address this challenge, we introduce 'Topology-Informed Graph Transformer (TIGT)', a novel transformer enhancing both discriminative power in detecting graph isomorphisms and the overall performance of Graph Transformers. TIGT consists of four components: A topological positional embedding layer using non-isomorphic universal covers based on cyclic subgraphs of graphs to ensure unique graph representation: A dual-path message-passing layer to explicitly encode topological characteristics throughout the encoder layers: A global attention mechanism: And a graph information layer to recalibrate channel-wise graph features for better feature representation. TIGT outperforms previous Graph Transformers in classifying synthetic dataset aimed at distinguishing isomorphism classes of graphs. Additionally, mathematical analysis and empirical evaluations highlight our model's competitive edge over state-of-the-art Graph Transformers across various benchmark datasets.

Graph Neural Networks (GNNs) and Transformer have been increasingly adopted to learn the complex vector representations of spatio-temporal graphs, capturing intricate spatio-temporal dependencies crucial for applications such as traffic datasets. Although many existing methods utilize multi-head attention mechanisms and message-passing neural networks (MPNNs) to capture both spatial and temporal relations, these approaches encode temporal and spatial relations independently, and reflect the graph's topological characteristics in a limited manner. In this work, we introduce the Cycle to Mixer (Cy2Mixer), a novel spatio-temporal GNN based on topological non-trivial invariants of spatio-temporal graphs with gated multi-layer perceptrons (gMLP). The Cy2Mixer is composed of three blocks based on MLPs: A message-passing block for encapsulating spatial information, a cycle message-passing block for enriching topological information through cyclic subgraphs, and a temporal block for capturing temporal properties. We bolster the effectiveness of Cy2Mixer with mathematical evidence emphasizing that our cycle message-passing block is capable of offering differentiated information to the deep learning model compared to the message-passing block. Furthermore, empirical evaluations substantiate the efficacy of the Cy2Mixer, demonstrating state-of-the-art performances across various traffic benchmark datasets.

In this paper, we raise a new issue on Unidentified Foreground Object (UFO) detection in 3D point clouds, which is a crucial technology in autonomous driving in the wild. UFO detection is challenging in that existing 3D object detectors encounter extremely hard challenges in both 3D localization and Out-of-Distribution (OOD) detection. To tackle these challenges, we suggest a new UFO detection framework including three tasks: evaluation protocol, methodology, and benchmark. The evaluation includes a new approach to measure the performance on our goal, i.e. both localization and OOD detection of UFOs. The methodology includes practical techniques to enhance the performance of our goal. The benchmark is composed of the KITTI Misc benchmark and our additional synthetic benchmark for modeling a more diverse range of UFOs. The proposed framework consistently enhances performance by a large margin across all four baseline detectors: SECOND, PointPillars, PV-RCNN, and PartA2, giving insight for future work on UFO detection in the wild., Comment: Under review

In the weakly supervised temporal video grounding study, previous methods use predetermined single Gaussian proposals which lack the ability to express diverse events described by the sentence query. To enhance the expression ability of a proposal, we propose a Gaussian mixture proposal (GMP) that can depict arbitrary shapes by learning importance, centroid, and range of every Gaussian in the mixture. In learning GMP, each Gaussian is not trained in a feature space but is implemented over a temporal location. Thus the conventional feature-based learning for Gaussian mixture model is not valid for our case. In our special setting, to learn moderately coupled Gaussian mixture capturing diverse events, we newly propose a pull-push learning scheme using pulling and pushing losses, each of which plays an opposite role to the other. The effects of components in our scheme are verified in-depth with extensive ablation studies and the overall scheme achieves state-of-the-art performance. Our code is available at https://github.com/sunoh-kim/pps., Comment: Accepted in AAAI 2024

Prior work has argued for a more justice-centered approach to postsecondary computing education by emphasizing ethics, identity, and political vision. In this experience report, we examine how postsecondary students of diverse gender and racial identities experience a justice-centered Data Structures and Algorithms designed for undergraduate non-computer science majors. Through a quantitative and qualitative analysis of two quarters of student survey data collected at the start and end of each quarter, we report on student attitudes and expectations. Across the class, we found a significant increase in the following attitudes: computing confidence and sense of belonging. While women, non-binary, and other students not identifying as men (WNB+) also increased in these areas, they still reported significantly lower confidence and sense of belonging than men at the end of the quarter. Black, Latinx, Middle Eastern and North African, Native American, and Pacific Islander (BLMNPI) students had no significant differences compared to white and Asian students. We also analyzed end-of-quarter student self-reflections on their fulfillment of expectations prior to taking the course. While the majority of students reported a positive overall sentiment about the course and many students specifically appreciated the justice-centered approach, some desired more practice with program implementation and interview preparation. We discuss implications for practice and articulate a political vision for holding both appreciation for computing ethics and a desire for professional preparation together through iterative design., Comment: Experience Reports and Tools paper in the Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE 2024); 7 pages