Search

The digitization of historical documents is crucial for preserving the cultural heritage of the society. An important step in this process is converting scanned images to text using Optical Character Recognition (OCR), which can enable further search, information extraction, etc. Unfortunately, this is a hard problem as standard OCR tools are not tailored to deal with historical orthography as well as with challenging layouts. Thus, it is standard to apply an additional text correction step on the OCR output when dealing with such documents. In this work, we focus on Bulgarian, and we create the first benchmark dataset for evaluating the OCR text correction for historical Bulgarian documents written in the first standardized Bulgarian orthography: the Drinov orthography from the 19th century. We further develop a method for automatically generating synthetic data in this orthography, as well as in the subsequent Ivanchev orthography, by leveraging vast amounts of contemporary literature Bulgarian texts. We then use state-of-the-art LLMs and encoder-decoder framework which we augment with diagonal attention loss and copy and coverage mechanisms to improve the post-OCR text correction. The proposed method reduces the errors introduced during recognition and improves the quality of the documents by 25\%, which is an increase of 16\% compared to the state-of-the-art on the ICDAR 2019 Bulgarian dataset. We release our data and code at \url{https://github.com/angelbeshirov/post-ocr-text-correction}.}, Comment: Accepted for publication in the International Journal on Digital Libraries

We prove the irrationality of the classical Dirichlet L-value $L(2,\chi_{-3})$. The argument applies a new kind of arithmetic holonomy bound to a well-known construction of Zagier. In fact our work also establishes the $\mathbf{Q}$-linear independence of $1$, $\zeta(2)$, and $L(2,\chi_{-3})$. We also give a number of other applications of our method to other problems in irrationality., Comment: 218 pages, comments welcome, minor expository changes, this is the submitted version of the paper

In (J. Stat. Phys. 132, 275-290, 2008) Borodin and P\'ech\'e introduced a generalization of the extended Airy kernel based on two infinite sets of parameters. For an arbitrary choice of parameters we construct determinantal point processes on $\mathbb{R}^2$ for these generalized kernels. In addition, for a subset of the parameter space we show that the point processes can be lifted to line ensembles on $\mathbb{R}$, which satisfy the Brownian Gibbs property. Our ensembles generalize the wanderer line ensembles introduced by Corwin and Hammond in (Invent. Math. 195, 441-508, 2014)., Comment: 76 pages, 8 figures

In this paper we propose a new generalizations of amicable numbers. We also give examples and prove properties of these new concepts.

In this paper we establish three Barban-Davenport-Halberstam type theorems. Namely for exponential sums over primes, for Piatetski-Shapiro primes and for exponential sums over Piatetski-Shapiro primes.

Event sequences (ESs) arise in many practical domains including finance, retail, social networks, and healthcare. In the context of machine learning, event sequences can be seen as a special type of tabular data with annotated timestamps. Despite the importance of ESs modeling and analysis, little effort was made in adapting large language models (LLMs) to the ESs domain. In this paper, we highlight the common difficulties of ESs processing and propose a novel solution capable of solving multiple downstream tasks with little or no finetuning. In particular, we solve the problem of working with long sequences and improve time and numeric features processing. The resulting method, called ESQA, effectively utilizes the power of LLMs and, according to extensive experiments, achieves state-of-the-art results in the ESs domain., Comment: 25 pages, 3 figures

GRBs produced by the collapse of massive stars are usually found near the most prominent star-forming regions of star-forming galaxies. GRB 171205A happened in the outskirts of a spiral galaxy, a peculiar location in an atypical GRB host. In this paper we present a highly-resolved study of the molecular gas of this host, with CO(1-0) observations from ALMA. We compare with GMRT atomic HI observations, and with data at other wavelengths to provide a broad-band view of the galaxy. The ALMA observations have a spatial resolution of 0.2" and a spectral resolution of 10 km/s, observed when the afterglow had a flux density of ~53 mJy. This allowed a molecular study both in emission and absorption. The HI observations allowed to study the host galaxy and its extended environment. The CO emission shows an undisturbed spiral structure with a central bar, and no significant emission at the location of the GRB. Our CO spectrum does not reveal any CO absorption, with a column density limit of < 10^15 cm^-2. This argues against the progenitor forming in a massive molecular cloud. The molecular gas traces the galaxy arms with higher concentration in the regions dominated by dust. The HI gas does not follow the stellar light or the molecular gas and is concentrated in two blobs, with no emission towards the centre of the galaxy, and is slightly displaced towards the southwest of the galaxy, where the GRB exploded. Within the extended neighbourhood of the host galaxy, we identify another prominent HI source at the same redshift, at a projected distance of 188 kpc. Our observations show that the progenitor of this GRB is not associated to a massive molecular cloud, but more likely related to low-metallicity atomic gas. The distortion in the HI gas field is indicator of an odd environment that could have triggered star formation and could be linked to a past interaction with the companion galaxy., Comment: 13 pages, 10 figures, 8 tables, A&A submitted after 1st referee review

A key challenge of quantum programming is uncomputation: the reversible deallocation of qubits. And while there has been much recent progress on automating uncomputation, state-of-the-art methods are insufficient for handling today's expressive quantum programming languages. A core reason is that they operate on primitive quantum circuits, while quantum programs express computations beyond circuits, for instance, they can capture families of circuits defined recursively in terms of uncomputation and adjoints. In this paper, we introduce the first modular automatic approach to synthesize correct and efficient uncomputation for expressive quantum programs. Our method is based on two core technical contributions: (i) an intermediate representation (IR) that can capture expressive quantum programs and comes with support for uncomputation, and (ii) modular algorithms over that IR for synthesizing uncomputation and adjoints. We have built a complete end-to-end implementation of our method, including an implementation of the IR and the synthesis algorithms, as well as a translation from an expressive fragment of the Silq programming language to our IR and circuit generation from the IR. Our experimental evaluation demonstrates that we can handle programs beyond the capabilities of existing uncomputation approaches, while being competitive on the benchmarks they can handle. More broadly, we show that it is possible to benefit from the greater expressivity and safety offered by high-level quantum languages without sacrificing efficiency., Comment: 25 pages, 9 figures

Federated learning works by aggregating locally computed gradients from multiple clients, thus enabling collaborative training without sharing private client data. However, prior work has shown that the data can actually be recovered by the server using so-called gradient inversion attacks. While these attacks perform well when applied on images, they are limited in the text domain and only permit approximate reconstruction of small batches and short input sequences. In this work, we propose DAGER, the first algorithm to recover whole batches of input text exactly. DAGER leverages the low-rank structure of self-attention layer gradients and the discrete nature of token embeddings to efficiently check if a given token sequence is part of the client data. We use this check to exactly recover full batches in the honest-but-curious setting without any prior on the data for both encoder- and decoder-based architectures using exhaustive heuristic search and a greedy approach, respectively. We provide an efficient GPU implementation of DAGER and show experimentally that it recovers full batches of size up to 128 on large language models (LLMs), beating prior attacks in speed (20x at same batch size), scalability (10x larger batches), and reconstruction quality (ROUGE-1/2 > 0.99).

This paper reveals a novel linear characteristic exclusive to transformer decoders, including models such as GPT, LLaMA, OPT, BLOOM and others. We analyze embedding transformations between sequential layers, uncovering a near-perfect linear relationship (Procrustes similarity score of 0.99). However, linearity decreases when the residual component is removed due to a consistently low output norm of the transformer layer. Our experiments show that removing or linearly approximating some of the most linear blocks of transformers does not affect significantly the loss or model performance. Moreover, in our pretraining experiments on smaller models we introduce a cosine-similarity-based regularization, aimed at reducing layer linearity. This regularization improves performance metrics on benchmarks like Tiny Stories and SuperGLUE and as well successfully decreases the linearity of the models. This study challenges the existing understanding of transformer architectures, suggesting that their operation may be more linear than previously assumed., Comment: 9 pages, 9 figures

The pursuit to operate photocathodes at high accelerating gradients to increase brightness of electron beams is gaining interests within the accelerator community. Cesium telluride (Cs2Te) is a widely used photocathode material and it is presumed to offer resilience to higher gradients because of its wider band gap compared to other semiconductors. Despite its advantages, crucial material properties of Cs2Te remain largely unknown both in theory and experiments. In this study, we employ first-principles calculations to provide detailed structural, elastic, electronic and transport properties of Cs2Te. It is found that Cs2Te has an intrinsic mobility of 20 cm2/Vs for electrons and 2.0 cm2/Vs for holes at room temperature. The low mobility is primarily limited by the strong polar optical phonon scattering. Cs2Te also exhibits ultralow lattice thermal conductivity of 0.2 W/(m*K) at room temperature. Based on the energy gain/loss balance under external field and electron-phonon scattering, we predict that Cs2Te has a dielectric breakdown field in the range from ~60 MV/m to ~132 MV/m at room temperature dependent on the doping level of Cs2Te. Our results are crucial to advance the understanding of applicability of Cs2Te photocathodes for high-gradient operation.

In this paper we establish lower bounds on several expressions dependent on functions $\varphi(n)$, $\psi(n)$ and $\sigma(n)$.

This paper presents a brief survey of the most important and the most remarkable inequalities involving the basic arithmetic functions.

We have developed an advanced workflow for reservoir characterization, effectively addressing the challenges of reservoir history matching through a novel approach. This method integrates a Physics Informed Neural Operator (PINO) as a forward model within a sophisticated Cluster Classify Regress (CCR) framework. The process is enhanced by an adaptive Regularized Ensemble Kalman Inversion (aREKI), optimized for rapid uncertainty quantification in reservoir history matching. This innovative workflow parameterizes unknown permeability and porosity fields, capturing non-Gaussian posterior measures with techniques such as a variational convolution autoencoder and the CCR. Serving as exotic priors and a supervised model, the CCR synergizes with the PINO surrogate to accurately simulate the nonlinear dynamics of Peaceman well equations. The CCR approach allows for flexibility in applying distinct machine learning algorithms across its stages. Updates to the PINO reservoir surrogate are driven by a loss function derived from supervised data, initial conditions, and residuals of governing black oil PDEs. Our integrated model, termed PINO-Res-Sim, outputs crucial parameters including pressures, saturations, and production rates for oil, water, and gas. Validated against traditional simulators through controlled experiments on synthetic reservoirs and the Norne field, the methodology showed remarkable accuracy. Additionally, the PINO-Res-Sim in the aREKI workflow efficiently recovered unknown fields with a computational speedup of 100 to 6000 times faster than conventional methods. The learning phase for PINO-Res-Sim, conducted on an NVIDIA H100, was impressively efficient, compatible with ensemble-based methods for complex computational tasks., Comment: 55 pages, 46 figures

Last year, multimodal architectures served up a revolution in AI-based approaches and solutions, extending the capabilities of large language models (LLM). We propose an \textit{OmniFusion} model based on a pretrained LLM and adapters for visual modality. We evaluated and compared several architecture design principles for better text and visual data coupling: MLP and transformer adapters, various CLIP ViT-based encoders (SigLIP, InternVIT, etc.), and their fusing approach, image encoding method (whole image or tiles encoding) and two 7B LLMs (the proprietary one and open-source Mistral). Experiments on 8 visual-language benchmarks show the top score for the best OmniFusion setup in terms of different VQA tasks in comparison with open-source LLaVA-like solutions: VizWiz, Pope, MM-Vet, ScienceQA, MMBench, TextVQA, VQAv2, MMMU. We also propose a variety of situations, where OmniFusion provides highly-detailed answers in different domains: housekeeping, sightseeing, culture, medicine, handwritten and scanned equations recognition, etc. Mistral-based OmniFusion model is an open-source solution with weights, training and inference scripts available at https://github.com/AIRI-Institute/OmniFusion., Comment: 17 pages, 4 figures, 9 tables, 2 appendices

Consider a search from the root of an ordered tree with $n$ edges to some target node at a fixed distance $\ell$ from that root. We compare the average time complexity of the breadth-first search (BFS) and depth-first search (DFS) algorithms, when the target node is selected uniformly at random among all nodes at level $\ell$ in the ordered trees with $n$ edges. Intuition suggests that BFS should have better average performance when $\ell$ is small, while DFS must have an advantage when $\ell$ is large. But where exactly is the threshold, as a function of $n$, and is it unique? We obtain explicit formulas for the expected number of steps of both BFS and DFS, by using results on the occupation measure of Brownian excursions, as well as a combinatorial proof of an identity related to lattice paths. This allows us to show that there exists a unique constant $\lambda\approx 0.789004$, such that in expectation BFS is asymptotically faster than DFS if and only if $\ell\leq \lambda\sqrt{n}$. Furthermore, we find the asymptotic average time complexity of BFS in the given setting for any class of Galton$\unicode{x2013}$Watson trees, including binary trees and ordered trees. Finally, we introduce the truncated DFS algorithm, which performs better than both BFS and DFS when $\ell$ is known in advance, and we find a formula evaluating the average time complexity of this algorithm., Comment: 32 pages

Pre-trained Language Models (PLMs) are known to contain various kinds of knowledge. One method to infer relational knowledge is through the use of cloze-style prompts, where a model is tasked to predict missing subjects or objects. Typically, designing these prompts is a tedious task because small differences in syntax or semantics can have a substantial impact on knowledge retrieval performance. Simultaneously, evaluating the impact of either prompt syntax or information is challenging due to their interdependence. We designed CONPARE-LAMA - a dedicated probe, consisting of 34 million distinct prompts that facilitate comparison across minimal paraphrases. These paraphrases follow a unified meta-template enabling the controlled variation of syntax and semantics across arbitrary relations. CONPARE-LAMA enables insights into the independent impact of either syntactical form or semantic information of paraphrases on the knowledge retrieval performance of PLMs. Extensive knowledge retrieval experiments using our probe reveal that prompts following clausal syntax have several desirable properties in comparison to appositive syntax: i) they are more useful when querying PLMs with a combination of supplementary information, ii) knowledge is more consistently recalled across different combinations of supplementary information, and iii) they decrease response uncertainty when retrieving known facts. In addition, range information can boost knowledge retrieval performance more than domain information, even though domain information is more reliably helpful across syntactic forms., Comment: Accepted for NAACL 2024

We introduce a two-parameter family of probability distributions, indexed by $\beta/2 = \theta > 0$ and $K \in \mathbb{Z}_{\geq 0}$, that are called $\beta$-Krawtchouk corners processes. These measures are related to Jack symmetric functions, and can be thought of as integrable discretizations of $\beta$-corners processes from random matrix theory, or alternatively as non-determinantal measures on lozenge tilings of infinite domains. We show that as $K$ tends to infinity the height function of these models concentrates around an explicit limit shape, and prove that its fluctuations are asymptotically described by a pull-back of the Gaussian free field, which agrees with the one for Wigner matrices. The main tools we use to establish our results are certain multi-level loop equations introduced in our earlier work arXiv:2108.07710., Comment: 88 pages, 5 figures

We introduce EXAMS-V, a new challenging multi-discipline multimodal multilingual exam benchmark for evaluating vision language models. It consists of 20,932 multiple-choice questions across 20 school disciplines covering natural science, social science, and other miscellaneous studies, e.g., religion, fine arts, business, etc. EXAMS-V includes a variety of multimodal features such as text, images, tables, figures, diagrams, maps, scientific symbols, and equations. The questions come in 11 languages from 7 language families. Unlike existing benchmarks, EXAMS-V is uniquely curated by gathering school exam questions from various countries, with a variety of education systems. This distinctive approach calls for intricate reasoning across diverse languages and relies on region-specific knowledge. Solving the problems in the dataset requires advanced perception and joint reasoning over the text and the visual content of the image. Our evaluation results demonstrate that this is a challenging dataset, which is difficult even for advanced vision-text models such as GPT-4V and Gemini; this underscores the inherent complexity of the dataset and its significance as a future benchmark.

Let $F$ be a totally real field and $p$ a rational prime unramified in $F$. For each subset $P$ of primes of $F$ above $p$, there is the notion of partially classical Hilbert modular forms, where $P=\varnothing$ recovers the overconvergent forms and $P$ being the full set of primes above $p$ recovers classical forms. Given $P$, we $p$-adically interpolate the classical modular sheaves to construct families of partially classical Hilbert modular forms with weights varying in appropriate weight spaces. We then construct the corresponding eigenvariety, which recovers the eigenvariety constructed by Andreatta--Iovita--Pilloni when $P=\varnothing$. By contrast, it is unclear whether each partially classical form can be put into a partially classical family.

Federated learning is a framework for collaborative machine learning where clients only share gradient updates and not their private data with a server. However, it was recently shown that gradient inversion attacks can reconstruct this data from the shared gradients. In the important honest-but-curious setting, existing attacks enable exact reconstruction only for a batch size of $b=1$, with larger batches permitting only approximate reconstruction. In this work, we propose SPEAR, the first algorithm reconstructing whole batches with $b >1$ exactly. SPEAR combines insights into the explicit low-rank structure of gradients with a sampling-based algorithm. Crucially, we leverage ReLU-induced gradient sparsity to precisely filter out large numbers of incorrect samples, making a final reconstruction step tractable. We provide an efficient GPU implementation for fully connected networks and show that it recovers high-dimensional ImageNet inputs in batches of up to $b \lesssim 25$ exactly while scaling to large networks. Finally, we show theoretically that much larger batches can be reconstructed with high probability given exponential time.

In this paper, we establish a Bombieri-Vinogradov type result for prime numbers of the form $p=x^2+y^2+1$. The proof is based on the enveloping sieve.

Let $[\, \cdot\,]$ be the floor function. In this paper, we show that when $1

Over the past few years, one of the most notable advancements in AI research has been in foundation models (FMs), headlined by the rise of language models (LMs). As the models' size increases, LMs demonstrate enhancements in measurable aspects and the development of new qualitative features. However, despite researchers' attention and the rapid growth in LM application, the capabilities, limitations, and associated risks still need to be better understood. To address these issues, we introduce an open Multimodal Evaluation of Russian-language Architectures (MERA), a new instruction benchmark for evaluating foundation models oriented towards the Russian language. The benchmark encompasses 21 evaluation tasks for generative models in 11 skill domains and is designed as a black-box test to ensure the exclusion of data leakage. The paper introduces a methodology to evaluate FMs and LMs in zero- and few-shot fixed instruction settings that can be extended to other modalities. We propose an evaluation methodology, an open-source code base for the MERA assessment, and a leaderboard with a submission system. We evaluate open LMs as baselines and find that they are still far behind the human level. We publicly release MERA to guide forthcoming research, anticipate groundbreaking model features, standardize the evaluation procedure, and address potential societal drawbacks., Comment: The paper version comparable with the release code v.1.1.0 of the benchmark MERA. ACL-2024 main track camera ready version

We present Kandinsky 3.0, a large-scale text-to-image generation model based on latent diffusion, continuing the series of text-to-image Kandinsky models and reflecting our progress to achieve higher quality and realism of image generation. In this report we describe the architecture of the model, the data collection procedure, the training technique, and the production system for user interaction. We focus on the key components that, as we have identified as a result of a large number of experiments, had the most significant impact on improving the quality of our model compared to the others. We also describe extensions and applications of our model, including super resolution, inpainting, image editing, image-to-video generation, and a distilled version of Kandinsky 3.0 - Kandinsky 3.1, which does inference in 4 steps of the reverse process and 20 times faster without visual quality decrease. By side-by-side human preferences comparison, Kandinsky becomes better in text understanding and works better on specific domains. The code is available at https://github.com/ai-forever/Kandinsky-3, Comment: Project page: https://ai-forever.github.io/Kandinsky-3

Multimedia generation approaches occupy a prominent place in artificial intelligence research. Text-to-image models achieved high-quality results over the last few years. However, video synthesis methods recently started to develop. This paper presents a new two-stage latent diffusion text-to-video generation architecture based on the text-to-image diffusion model. The first stage concerns keyframes synthesis to figure the storyline of a video, while the second one is devoted to interpolation frames generation to make movements of the scene and objects smooth. We compare several temporal conditioning approaches for keyframes generation. The results show the advantage of using separate temporal blocks over temporal layers in terms of metrics reflecting video generation quality aspects and human preference. The design of our interpolation model significantly reduces computational costs compared to other masked frame interpolation approaches. Furthermore, we evaluate different configurations of MoVQ-based video decoding scheme to improve consistency and achieve higher PSNR, SSIM, MSE, and LPIPS scores. Finally, we compare our pipeline with existing solutions and achieve top-2 scores overall and top-1 among open-source solutions: CLIPSIM = 0.2976 and FVD = 433.054. Project page: https://ai-forever.github.io/kandinsky-video/, Comment: Project page: https://ai-forever.github.io/kandinsky-video/

We present a comprehensive study of the geometry of Hilbert $p$-adic eigenvarieties at parallel weight one intersection points of their cuspidal and Eisenstein loci. The Galois theoretic approach presents genuine difficulties due to the lack of good deformation theory for pseudo-characters irregular at $p$ and reflects the rich local geometry at such points. We believe that our geometric results lead to deeper insight into the arithmetic of Hilbert automorphic forms and we produce several applications in Iwasawa theory to support that., Comment: 42 pages

In this study, we present an investigation into the anisotropy dynamics and intrinsic dimension of embeddings in transformer architectures, focusing on the dichotomy between encoders and decoders. Our findings reveal that the anisotropy profile in transformer decoders exhibits a distinct bell-shaped curve, with the highest anisotropy concentrations in the middle layers. This pattern diverges from the more uniformly distributed anisotropy observed in encoders. In addition, we found that the intrinsic dimension of embeddings increases in the initial phases of training, indicating an expansion into higher-dimensional space. Which is then followed by a compression phase towards the end of training with dimensionality decrease, suggesting a refinement into more compact representations. Our results provide fresh insights to the understanding of encoders and decoders embedding properties., Comment: Accepted to EACL-2024

Accurate and scalable methods for computational quantum chemistry can accelerate research and development in many fields, ranging from drug discovery to advanced material design. Solving the electronic Schrodinger equation is the core problem of computational chemistry. However, the combinatorial complexity of this problem makes it intractable to find exact solutions, except for very small systems. The idea of quantum computing originated from this computational challenge in simulating quantum-mechanics. We propose an end-to-end quantum chemistry pipeline based on the variational quantum eigensolver (VQE) algorithm and integrated with both HPC-based simulators and a trapped-ion quantum computer. Our platform orchestrates hundreds of simulation jobs on compute resources to efficiently complete a set of ab initio chemistry experiments with a wide range of parameterization. Per- and poly-fluoroalkyl substances (PFAS) are a large family of human-made chemicals that pose a major environmental and health issue globally. Our simulations includes breaking a Carbon-Fluorine bond in trifluoroacetic acid (TFA), a common PFAS chemical. This is a common pathway towards destruction and removal of PFAS. Molecules are modeled on both a quantum simulator and a trapped-ion quantum computer, specifically IonQ Aria. Using basic error mitigation techniques, the 11-qubit TFA model (56 entangling gates) on IonQ Aria yields near-quantitative results with milli-Hartree accuracy. Our novel results show the current state and future projections for quantum computing in solving the electronic structure problem, push the boundaries for the VQE algorithm and quantum computers, and facilitates development of quantum chemistry workflows.