Search

Large Language Models (LLMs) are increasingly serving as evaluators in Natural Language Generation (NLG) tasks. However, the capabilities of LLMs in scoring NLG quality remain inadequately explored. Current studies depend on human assessments and simple metrics that fail to capture the discernment of LLMs across diverse NLG tasks. To address this gap, we propose the Discernment of Hierarchical Perturbation (DHP) benchmarking framework, which provides quantitative discernment scores for LLMs utilizing hierarchically perturbed text data and statistical tests to measure the NLG evaluation capabilities of LLMs systematically. We have re-established six evaluation datasets for this benchmark, covering four NLG tasks: Summarization, Story Completion, Question Answering, and Translation. Our comprehensive benchmarking of five major LLM series provides critical insight into their strengths and limitations as NLG evaluators.

While large language models (LLMs) have demonstrated impressive capabilities across various natural language processing tasks by acquiring rich factual knowledge from their broad training data, their ability to synthesize and logically reason with this knowledge in complex ways remains underexplored. In this work, we present a systematic evaluation of state-of-the-art LLMs' complex logical reasoning abilities through a novel benchmark of automatically generated complex reasoning questions over general domain and biomedical knowledge graphs. Our extensive experiments, employing diverse in-context learning techniques, reveal that LLMs excel at reasoning over general world knowledge but face significant challenges with specialized domain-specific knowledge. We find that prompting with explicit Chain-of-Thought demonstrations can substantially improve LLM performance on complex logical reasoning tasks with diverse logical operations. Interestingly, our controlled evaluations uncover an asymmetry where LLMs display proficiency at set union operations, but struggle considerably with set intersections - a key building block of logical reasoning. To foster further work, we will publicly release our evaluation benchmark and code., Comment: 9 pages

We have built and demonstrated a digital four-arm bridge for the comparison of resistance with capacitance. The digital four-arm bridge mimics the classical quad bridge in the digital domain with three balances: the source balance, the detector balance, as well as the main balance. Due to correlation, the required precision of the source voltages is only of the order of the square root of the ultimate bridge precision. For the comparison of a $100\,$k$\Omega$ resistor with a $1\,$nF capacitor near 1592 Hz, the combined standard uncertainty $(k = 1)$ is $5\times 10^{-9}$., Comment: 8 pages 5 figures

As a promising field in open-world learning, \textit{Novel Class Discovery} (NCD) is usually a task to cluster unseen novel classes in an unlabeled set based on the prior knowledge of labeled data within the same domain. However, the performance of existing NCD methods could be severely compromised when novel classes are sampled from a different distribution with the labeled ones. In this paper, we explore and establish the solvability of NCD in cross domain setting with the necessary condition that style information must be removed. Based on the theoretical analysis, we introduce an exclusive style removal module for extracting style information that is distinctive from the baseline features, thereby facilitating inference. Moreover, this module is easy to integrate with other NCD methods, acting as a plug-in to improve performance on novel classes with different distributions compared to the seen labeled set. Additionally, recognizing the non-negligible influence of different backbones and pre-training strategies on the performance of the NCD methods, we build a fair benchmark for future NCD research. Extensive experiments on three common datasets demonstrate the effectiveness of our proposed module.

Automatic speech recognition (ASR) techniques have become powerful tools, enhancing efficiency in law enforcement scenarios. To ensure fairness for demographic groups in different acoustic environments, ASR engines must be tested across a variety of speakers in realistic settings. However, describing the fairness discrepancies between models with confidence remains a challenge. Meanwhile, most public ASR datasets are insufficient to perform a satisfying fairness evaluation. To address the limitations, we built FairLENS - a systematic fairness evaluation framework. We propose a novel and adaptable evaluation method to examine the fairness disparity between different models. We also collected a fairness evaluation dataset covering multiple scenarios and demographic dimensions. Leveraging this framework, we conducted fairness assessments on 1 open-source and 11 commercially available state-of-the-art ASR models. Our results reveal that certain models exhibit more biases than others, serving as a fairness guideline for users to make informed choices when selecting ASR models for a given real-world scenario. We further explored model biases towards specific demographic groups and observed that shifts in the acoustic domain can lead to the emergence of new biases.

This paper introduces an innovative parameter extraction method for BSIM-CMG compact models, seamlessly integrating curve feature extraction and machine learning techniques. This method offers a promising solution for bridging the division between TCAD and compact model, significantly contributing to the Design Technology Co-Optimization (DTCO) process. The key innovation lies in the development of an automated IV and CV curve feature extractor, which not only streamlines the analysis of device IV and CV curves but also enhances the consistency and efficiency of data processing. Validation on 5-nm nanosheet devices underscores the extractor's remarkable precision, with impressively low fitting errors of 0.42% for CV curves and 1.28% for IV curves. Furthermore, its adaptability to parameter variations, including those in Equivalent Oxide Thickness and Gate Length, solidifies its potential to revolutionize the TCAD-to-compact model transition. This universal BSIM-CMG model parameter extractor promises to improve the DTCO process, offering efficient process optimization and accurate simulations for semiconductor device performance prediction., Comment: 6 pages

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.

Emitters based on photoconductive materials excited by ultrafast lasers are well established and popular devices for THz generation. However, so far, these emitters, both photoconductive antennas and large area emitters, were mostly explored using driving lasers with moderate average powers (either fiber lasers with up to hundreds of milliwatts or Ti:Sapphire systems up to few watts). In this paper, we explore the use of high power, MHz repetition rate Ytterbium (Yb) based oscillator for THz emission using a microstructured large area photoconductive emitter, consist of semi insulating GaAs with a 10 by 10 mm2 active area. As a driving source, we use a frequency doubled home built high average power ultrafast Yb oscillator, delivering 22 W of average power, 115 fs pulses with 91 MHz repetition rate at a central wavelength of 516 nm. When applying 9 W of average power (after an optical chopper with a duty cycle of 50 percent) on the structure without optimized heatsinking, we obtain 65 uW THz average power, 4 THz bandwidth; furthermore, we safely apply up to 18 W of power on the structure without observing damage. We investigate the impact of excitation power, bias voltage, optical fluence, and their interplay on the emitter performance and explore in detail the sources of thermal load originating from electrical and optical power. Optical power is found to have a more critical impact on LAE saturation than electrical power, thus optimized heatsinking will allow us to improve the conversion efficiency in the near future towards much higher emitter power. This work paves the way towards achieving hundreds of MHz or even GHz repetition rates, high power THz sources based on photoconductive emitters, that are of great interest for example for future THz imaging applications.

Recently, sharpness-aware minimization (SAM) has attracted much attention because of its surprising effectiveness in improving generalization performance. However, compared to stochastic gradient descent (SGD), it is more prone to getting stuck at the saddle points, which as a result may lead to performance degradation. To address this issue, we propose a simple renormalization strategy, dubbed Stable SAM (SSAM), so that the gradient norm of the descent step maintains the same as that of the ascent step. Our strategy is easy to implement and flexible enough to integrate with SAM and its variants, almost at no computational cost. With elementary tools from convex optimization and learning theory, we also conduct a theoretical analysis of sharpness-aware training, revealing that compared to SGD, the effectiveness of SAM is only assured in a limited regime of learning rate. In contrast, we show how SSAM extends this regime of learning rate and then it can consistently perform better than SAM with the minor modification. Finally, we demonstrate the improved performance of SSAM on several representative data sets and tasks., Comment: 33 pages

Abductive reasoning is the process of making educated guesses to provide explanations for observations. Although many applications require the use of knowledge for explanations, the utilization of abductive reasoning in conjunction with structured knowledge, such as a knowledge graph, remains largely unexplored. To fill this gap, this paper introduces the task of complex logical hypothesis generation, as an initial step towards abductive logical reasoning with KG. In this task, we aim to generate a complex logical hypothesis so that it can explain a set of observations. We find that the supervised trained generative model can generate logical hypotheses that are structurally closer to the reference hypothesis. However, when generalized to unseen observations, this training objective does not guarantee better hypothesis generation. To address this, we introduce the Reinforcement Learning from Knowledge Graph (RLF-KG) method, which minimizes differences between observations and conclusions drawn from generated hypotheses according to the KG. Experiments show that, with RLF-KG's assistance, the generated hypotheses provide better explanations, and achieve state-of-the-art results on three widely used KGs., Comment: Accepted by ACL 2024

We report on a GHz fundamental repetition rate Kerr-lens mode-locked Ho:CALGO laser emitting at 2.1 um. The laser employs a ring-cavity to increase the fundamental repetition rate to 1.179 GHz and can be made to oscillate in both directions stably with nearly identical performance: for counterclockwise oscillation, it generates 93-fs pulses at 1.68 W of average power, whereas 92 fs and 1.69 W were measured for clockwise operation. Our current results represent the highest average power from a 2-um GHz oscillator and the first sub-100-fs pulse duration from a Ho-based oscillator., Comment: 4 pages, 6 figures

Question generation is a widely used data augmentation approach with extensive applications, and extracting qualified candidate answers from context passages is a critical step for most question generation systems. However, existing methods for candidate answer extraction are reliant on linguistic rules or annotated data that face the partial annotation issue and challenges in generalization. To overcome these limitations, we propose a novel unsupervised candidate answer extraction approach that leverages the inherent structure of context passages through a Differentiable Masker-Reconstructor (DMR) Model with the enforcement of self-consistency for picking up salient information tokens. We curated two datasets with exhaustively-annotated answers and benchmark a comprehensive set of supervised and unsupervised candidate answer extraction methods. We demonstrate the effectiveness of the DMR model by showing its performance is superior among unsupervised methods and comparable to supervised methods., Comment: EMNLP 2023 - Findings

Parkinson's disease (PD), a neurodegenerative disorder, often manifests as speech and voice dysfunction. While utilizing voice data for PD detection has great potential in clinical applications, the widely used deep learning models currently have fairness issues regarding different ages of onset. These deep models perform well for the elderly group (age $>$ 55) but are less accurate for the young group (age $\leq$ 55). Through our investigation, the discrepancy between the elderly and the young arises due to 1) an imbalanced dataset and 2) the milder symptoms often seen in early-onset patients. However, traditional debiasing methods are impractical as they typically impair the prediction accuracy for the majority group while minimizing the discrepancy. To address this issue, we present a new debiasing method using GradCAM-based feature masking combined with ensemble models, ensuring that neither fairness nor accuracy is compromised. Specifically, the GradCAM-based feature masking selectively obscures age-related features in the input voice data while preserving essential information for PD detection. The ensemble models further improve the prediction accuracy for the minority (young group). Our approach effectively improves detection accuracy for early-onset patients without sacrificing performance for the elderly group. Additionally, we propose a two-step detection strategy for the young group, offering a practical risk assessment for potential early-onset PD patients.

Ultrafast laser driven, single-cycle THz pulsed sources hold immense potential for scientific and industrial applications; however, their limited average power hinders their widespread application. In particular, applications where high repetition rates in the multi-MHz region and beyond are required are more severely affected, due to the lower pulse energies available for frequency conversion. In this respect, resonant enhancement both in passive and active resonators is a well-known technique for boosting the efficiency of nonlinear frequency conversion; however, this route has remained poorly explored for the generation of broadband THz pulses due to the inadequacy of typically employed nonlinear crystals. Here, we demonstrate that thin lithium niobate crystals used intracavity of multimode diode-pumped mode-locked thin-disk lasers are a promising platform to circumvent these difficulties. Using a 50-{\mu}m thin lithium niobate plate intracavity of a compact high-power mode-locked thin-disk laser, we generate milliwatt-level broadband THz pulses with a spectrum extending up to 3 THz at 44.8 MHz repetition rate, driven by 264 W of intracavity average power. This approach opens the door to efficient high-power single-cycle THz generation using affordable nonlinear crystals at very high repetition rates, scalable to kilowatt-level driving power with low cost and complexity.

Earth can act as a transducer to convert ultralight bosonic dark matter (axions and hidden photons) into an oscillating magnetic field with a characteristic pattern across its surface. Here we describe the first results of a dedicated experiment, the Search for Non-Interacting Particles Experimental Hunt (SNIPE Hunt), that aims to detect such dark-matter-induced magnetic-field patterns by performing correlated measurements with a network of magnetometers in relatively quiet magnetic environments (in the wilderness far from human-generated magnetic noise). Our experiment constrains parameter space describing hidden-photon and axion dark matter with Compton frequencies in the 0.5-5.0 Hz range. Limits on the kinetic-mixing parameter for hidden-photon dark matter represent the best experimental bounds to date in this frequency range., Comment: 22 pages, 12 figures. Accepted for publication in PRD. Minor changes in text and updated / improved limits

We report on an in-band pumped soft-aperture Kerr-lens mode-locked Ho:CALGO bulk laser at 2.1 um, generating 2 W of average power with 112 fs pulses at 91-MHz repetition rate. To the best of our knowledge, this is the highest average power from a 100-fs class mode-locked laser based on a Tm3+ or Ho3+ doped bulk material. We show that the laser has excellent noise properties with an integrated relative intensity noise of 0.02% and a timing jitter of 950 fs (RMS phase noise 0.543 mrad) in the integration interval from 10 Hz to 10 MHz. The demonstrated combination of high average power, short pulses, and low-noise make this an outstanding laser source for spectroscopy and many other applications at 2.1 um., Comment: 4 pages, 6 figures

Various depth estimation models are now widely used on many mobile and IoT devices for image segmentation, bokeh effect rendering, object tracking and many other mobile tasks. Thus, it is very crucial to have efficient and accurate depth estimation models that can run fast on low-power mobile chipsets. In this Mobile AI challenge, the target was to develop deep learning-based single image depth estimation solutions that can show a real-time performance on IoT platforms and smartphones. For this, the participants used a large-scale RGB-to-depth dataset that was collected with the ZED stereo camera capable to generated depth maps for objects located at up to 50 meters. The runtime of all models was evaluated on the Raspberry Pi 4 platform, where the developed solutions were able to generate VGA resolution depth maps at up to 27 FPS while achieving high fidelity results. All models developed in the challenge are also compatible with any Android or Linux-based mobile devices, their detailed description is provided in this paper., Comment: arXiv admin note: substantial text overlap with arXiv:2105.08630, arXiv:2211.03885; text overlap with arXiv:2105.08819, arXiv:2105.08826, arXiv:2105.08629, arXiv:2105.07809, arXiv:2105.07825

Picking the first arrival times of prestack gathers is called First Arrival Time (FAT) picking, which is an indispensable step in seismic data processing, and is mainly solved manually in the past. With the current increasing density of seismic data collection, the efficiency of manual picking has been unable to meet the actual needs. Therefore, automatic picking methods have been greatly developed in recent decades, especially those based on deep learning. However, few of the current supervised deep learning-based method can avoid the dependence on labeled samples. Besides, since the gather data is a set of signals which are greatly different from the natural images, it is difficult for the current method to solve the FAT picking problem in case of a low Signal to Noise Ratio (SNR). In this paper, for hard rock seismic gather data, we propose a Multi-Stage Segmentation Pickup Network (MSSPN), which solves the generalization problem across worksites and the picking problem in the case of low SNR. In MSSPN, there are four sub-models to simulate the manually picking processing, which is assumed to four stages from coarse to fine. Experiments on seven field datasets with different qualities show that our MSSPN outperforms benchmarks by a large margin.Particularly, our method can achieve more than 90\% accurate picking across worksites in the case of medium and high SNRs, and even fine-tuned model can achieve 88\% accurate picking of the dataset with low SNR.

We report on an in-band pumped SESAM mode-locked Ho:CALGO bulk laser with a record-high average power of 8.7 W and an optical-to-optical efficiency of 38.2% at a central wavelength of 2.1 um. At this power level, the bulk laser generates pulses with a duration of 369 fs at 84.4-MHz repetition rate, corresponding to a pulse energy of 103 nJ and a peak power of 246 kW. To the best of our knowledge, this is the highest average power and pulse energy directly generated from a mode-locked bulk laser in the 2-3 um wavelength region. Our current results indicate that Ho:CALGO is a competitive candidate for average power scaling of 2 um femtosecond lasers., Comment: 9 pages, 5 figures

Physics-informed neural networks (PINNs) are revolutionizing science and engineering practice by bringing together the power of deep learning to bear on scientific computation. In forward modeling problems, PINNs are meshless partial differential equation (PDE) solvers that can handle irregular, high-dimensional physical domains. Naturally, the neural architecture hyperparameters have a large impact on the efficiency and accuracy of the PINN solver. However, this remains an open and challenging problem because of the large search space and the difficulty of identifying a proper search objective for PDEs. Here, we propose Auto-PINN, the first systematic, automated hyperparameter optimization approach for PINNs, which employs Neural Architecture Search (NAS) techniques to PINN design. Auto-PINN avoids manually or exhaustively searching the hyperparameter space associated with PINNs. A comprehensive set of pre-experiments using standard PDE benchmarks allows us to probe the structure-performance relationship in PINNs. We find that the different hyperparameters can be decoupled, and that the training loss function of PINNs is a good search objective. Comparison experiments with baseline methods demonstrate that Auto-PINN produces neural architectures with superior stability and accuracy over alternative baselines.

Ultrafast laser systems operating with high-average power in the wavelength range from 1.9 um to 3 um are of interest for a wide range of applications for example in spectroscopy, material processing and as drivers for secondary sources in the XUV spectral region. In this area, laser systems based on holmium-doped gain materials directly emitting at 2.1 um have made significant progress over the past years, however so far only very few results were demonstrated in power-scalable high-power laser geometries. In particular, the thin-disk geometry is promising for directly modelocked oscillators with high average power levels that are comparable to amplifier systems at MHz repetition rate. In this paper, we demonstrate Semiconductor Saturable Absorber Mirror (SESAM) modelocked Ho:YAG thin-disk lasers (TDLs) emitting at 2.1 um wavelength with record-holding performance levels. In our highest average power configuration, we reach 50 W of average power, with 1.13 ps pulses, 2.11 uJ of pulse energy and ~1.9 MW of peak power. To the best of our knowledge, this represents the highest average power, as well as the highest output pulse energy so far demonstrated from a modelocked laser in the 2 um wavelength region. This record performance level was enabled by the recent development of high-power GaSb-based SESAMs with low loss, adapted for high intracavity power and pulse energy. We also explore the limitations in terms of reaching shorter pulse durations at high power with this gain material in the disk geometry and using SESAM modelocking, and present first steps in this direction, with the demonstration of 30 W of output power, with 692 fs pulses in another laser configuration., Comment: 12 pages, 10 figures