Search

Large Language Models (LLMs) are currently under exploration for various tasks, including Automatic Speech Recognition (ASR), Machine Translation (MT), and even End-to-End Speech Translation (ST). In this paper, we present KIT's offline submission in the constrained + LLM track by incorporating recently proposed techniques that can be added to any cascaded speech translation. Specifically, we integrate Mistral-7B\footnote{mistralai/Mistral-7B-Instruct-v0.1} into our system to enhance it in two ways. Firstly, we refine the ASR outputs by utilizing the N-best lists generated by our system and fine-tuning the LLM to predict the transcript accurately. Secondly, we refine the MT outputs at the document level by fine-tuning the LLM, leveraging both ASR and MT predictions to improve translation quality. We find that integrating the LLM into the ASR and MT systems results in an absolute improvement of $0.3\%$ in Word Error Rate and $0.65\%$ in COMET for tst2019 test set. In challenging test sets with overlapping speakers and background noise, we find that integrating LLM is not beneficial due to poor ASR performance. Here, we use ASR with chunked long-form decoding to improve context usage that may be unavailable when transcribing with Voice Activity Detection segmentation alone.

With the rapid development of Large Language Models (LLMs), it is crucial to have benchmarks which can evaluate the ability of LLMs on different domains. One common use of LLMs is performing tasks on scientific topics, such as writing algorithms, querying databases or giving mathematical proofs. Inspired by the way university students are evaluated on such tasks, in this paper, we propose SciEx - a benchmark consisting of university computer science exam questions, to evaluate LLMs ability on solving scientific tasks. SciEx is (1) multilingual, containing both English and German exams, and (2) multi-modal, containing questions that involve images, and (3) contains various types of freeform questions with different difficulty levels, due to the nature of university exams. We evaluate the performance of various state-of-the-art LLMs on our new benchmark. Since SciEx questions are freeform, it is not straightforward to evaluate LLM performance. Therefore, we provide human expert grading of the LLM outputs on SciEx. We show that the free-form exams in SciEx remain challenging for the current LLMs, where the best LLM only achieves 59.4\% exam grade on average. We also provide detailed comparisons between LLM performance and student performance on SciEx. To enable future evaluation of new LLMs, we propose using LLM-as-a-judge to grade the LLM answers on SciEx. Our experiments show that, although they do not perform perfectly on solving the exams, LLMs are decent as graders, achieving 0.948 Pearson correlation with expert grading.

Many management decisions involve accumulated random realizations for which the expected value and variance are assumed to be known. We revisit the tail behavior of such quantities when individual realizations are independent, and we develop new sharper bounds on the tail probability and expected linear loss. The underlying distribution is semi-parametric in the sense that it remains unrestricted other than the assumed mean and variance. Our bounds complement well-established results in the literature, including those based on aggregation, which often fail to take full account of independence and use less elegant proofs. New insights include a proof that in the non-identical case, the distributions attaining the bounds have the equal range property, and that the impact of each random variable on the expected value of the sum can be isolated using an extension of the Korkine identity. We show that the new bounds %not only complement the extant results but also open up abundant practical applications, including improved pricing of product bundles, more precise option pricing, more efficient insurance design, and better inventory management. For example, we establish a new solution to the optimal bundling problem, yielding a 17\% uplift in per-bundle profits, and a new solution to the inventory problem, yielding a 5.6\% cost reduction for a model with 20 retailers.

The challenge of low-latency speech translation has recently draw significant interest in the research community as shown by several publications and shared tasks. Therefore, it is essential to evaluate these different approaches in realistic scenarios. However, currently only specific aspects of the systems are evaluated and often it is not possible to compare different approaches. In this work, we propose the first framework to perform and evaluate the various aspects of low-latency speech translation under realistic conditions. The evaluation is carried out in an end-to-end fashion. This includes the segmentation of the audio as well as the run-time of the different components. Secondly, we compare different approaches to low-latency speech translation using this framework. We evaluate models with the option to revise the output as well as methods with fixed output. Furthermore, we directly compare state-of-the-art cascaded as well as end-to-end systems. Finally, the framework allows to automatically evaluate the translation quality as well as latency and also provides a web interface to show the low-latency model outputs to the user., Comment: Demo paper at EMNLP 2023

Let $\mathbb{A}$ be the ring of adeles of a number field $k$ and $\pi$ be an irreducible cuspidal automorphic representation of $\mathrm{GL}_n(\mathbb{A})$. In the previous work of the first author with Zhilin Luo, they introduced $\pi$-Schwartz space $\mathcal{S}_\pi(\mathbb{A}^\times)$ and $\pi$-Fourier transform $\mathcal{F}_{\pi,\psi}$ with a non-trivial additive character $\psi$ of $k\backslash\mathbb{A}$, proved the associated Poisson summation formula over $\mathbb{A}^\times$, based on the Godement-Jacquet theory for the standard $L$-functions $L(s,\pi)$, and provided interesting applications. In this paper, in addition to the further development of the local theory, we found two global applications. First, we find a Poisson summation formula proof of the Voronoi summation formula for $\mathrm{GL}_n$ over a number field, which was first proved by A. Ichino and N. Templier. Then we introduce the notion of the Godement-Jacquet kernels $H_{\pi,s}$ and their dual kernels $K_{\pi,s}$ for any irreducible cuspidal automorphic representation $\pi$ of $\mathrm{GL}_n(\mathbb{A})$ and show that $H_{\pi,s}$ and $K_{\pi,1-s}$ are related by the nonlinear $\pi_\infty$-Fourier transform if and only if $s\in\mathbb{C}$ is a zero of $L_f(s,\pi_f)=0$, the finite part of the standard automorphic $L$-function $L(s,\pi)$, which are the $(\mathrm{GL}_n,\pi)$-versions of a Clozel's Theorem, where the Tate kernel with $n=1$ and $\pi$ the trivial character are considered.

We prove the uniqueness of the Ginzburg-Rallis models over $p$-adic local fields of characteristic zero, which completes the local uniqueness problem for the Ginzburg-Rallis models starting from the work of C.-F. Nien in \cite{MR2709083} that proves the non-split case, and the work of D. Jiang, B. Sun and C. Zhu in \cite{MR2763736} that proves the general case over Archimedean local fields. Our proof extends the strategy of \cite{MR2763736} to the $p$-adic case with the help of the refined structure of the wavefront sets of $\mathfrak {z}$-finite distributions as developed by A. Aizenbud, D. Gourevitch and E. Sayag in \cite{MR3406530}.

When building state-of-the-art speech translation models, the need for large computational resources is a significant obstacle due to the large training data size and complex models. The availability of pre-trained models is a promising opportunity to build strong speech translation systems efficiently. In a first step, we investigate efficient strategies to build cascaded and end-to-end speech translation systems based on pre-trained models. Using this strategy, we can train and apply the models on a single GPU. While the end-to-end models show superior translation performance to cascaded ones, the application of this technology has a limitation on the need for additional end-to-end training data. In a second step, we proposed an additional similarity loss to encourage the model to generate similar hidden representations for speech and transcript. Using this technique, we can increase the data efficiency and improve the translation quality by 6 BLEU points in scenarios with limited end-to-end training data.