Search

Core-collapse supernovae (CCSNe) are potential multimessenger events detectable by current and future gravitational wave (GW) detectors. The GW signals emitted during these events are expected to provide insights into the explosion mechanism and the internal structures of neutron stars. In recent years, several studies have empirically derived the relationship between the frequencies of the GW signals originating from the oscillations of protoneutron stars (PNSs) and the physical parameters of these stars. This study applies the Hilbert-Huang transform (HHT) [Proc. R. Soc. A 454, 903 (1998)] to extract the frequencies of these modes to infer the physical properties of the PNSs. The results exhibit comparable accuracy to a short-time Fourier transform-based estimation, highlighting the potential of this approach as a complementary method for extracting physical information from GW signals of CCSNe., Comment: 12 pages, 12 figures

Decadal changes in a nearby supernova remnant (SNR) were analyzed using a multiepoch maximum likelihood estimation (MLE) approach. To achieve greater accuracy in capturing the dynamics of SNRs, kinematic features and point-spread function effects were integrated into the MLE framework. Using Cassiopeia A as a representative example, data obtained by the Chandra X-ray Observatory in 2000, 2009, and 2019 were utilized. The proposed multiepoch MLE was qualitatively and quantitatively demonstrated to provide accurate estimates of various motions, including shock waves and faint features, across all regions. To investigate asymmetric structures, such as singular components that deviate from the direction of expansion, the MLE method was extended to combine multiple computational domains and classify kinematic properties using the $k$-means algorithm. This approach allowed for the mapping of different physical states onto the image, and one classified component was suggested to interact with circumstellar material by comparison with infrared observations from the James Webb Space Telescope. Thus, this technique will help quantify the dynamics of SNRs and discover their unique evolution., Comment: Accepted for publication in ApJ. The paper is 20 pages long with 13 figures. Animations for Fig. 8 and Fig. 9, which provide a visual illustration of the results, are available in ApJ

Text generation commonly relies on greedy and beam decoding that limit the search space and degrade output quality. Minimum Bayes Risk (MBR) decoding can mitigate this problem by utilizing automatic evaluation metrics and model-generated pseudo-references. Previous studies have conducted empirical analyses to reveal the improvement by MBR decoding, and reported various observations. However, despite these observations, the theoretical relationship between them remains uncertain. To address this, we present a novel theoretical interpretation of MBR decoding from the perspective of bias-diversity decomposition. We decompose errors in the estimated quality of generated hypotheses in MBR decoding into two key factors: bias, which reflects the closeness between utility functions and human evaluations, and diversity, which represents the variation in the estimated quality of utility functions. Our theoretical analysis reveals the difficulty in simultaneously improving both bias and diversity, and highlights the effectiveness of increasing diversity to enhance MBR decoding performance. This analysis verifies the alignment between our theoretical insights and the empirical results reported in previous work. Furthermore, to support our theoretical findings, we propose a new metric, pseudo-bias, which approximates the bias term using gold references. We also introduce a new MBR approach, Metric-augmented MBR (MAMBR), which increases diversity by adjusting the behavior of utility functions without altering the pseudo-references. Experimental results across multiple NLP tasks show that the decomposed terms in the bias-diversity decomposition correlate well with performance, and that MAMBR improves text generation quality by modifying utility function behavior. Our code will be available at https://github.com/naist-nlp/mbr-bias-diversity.

A large part of human communication relies on nonverbal cues such as facial expressions, eye contact, and body language. Unlike language or sign language, such nonverbal communication lacks formal rules, requiring complex reasoning based on commonsense understanding. Enabling current Video Large Language Models (VideoLLMs) to accurately interpret body language is a crucial challenge, as human unconscious actions can easily cause the model to misinterpret their intent. To address this, we propose a dataset, BQA, a body language question answering dataset, to validate whether the model can correctly interpret emotions from short clips of body language comprising 26 emotion labels of videos of body language. We evaluated various VideoLLMs on BQA and revealed that understanding body language is challenging, and our analyses of the wrong answers by VideoLLMs show that certain VideoLLMs made significantly biased answers depending on the age group and ethnicity of the individuals in the video. The dataset is available.

As the performance of Large-scale Vision Language Models (LVLMs) improves, they are increasingly capable of responding in multiple languages, and there is an expectation that the demand for explanations generated by LVLMs will grow. However, pre-training of Vision Encoder and the integrated training of LLMs with Vision Encoder are mainly conducted using English training data, leaving it uncertain whether LVLMs can completely handle their potential when generating explanations in languages other than English. In addition, multilingual QA benchmarks that create datasets using machine translation have cultural differences and biases, remaining issues for use as evaluation tasks. To address these challenges, this study created an extended dataset in multiple languages without relying on machine translation. This dataset that takes into account nuances and country-specific phrases was then used to evaluate the generation explanation abilities of LVLMs. Furthermore, this study examined whether Instruction-Tuning in resource-rich English improves performance in other languages. Our findings indicate that LVLMs perform worse in languages other than English compared to English. In addition, it was observed that LVLMs struggle to effectively manage the knowledge learned from English data.

The natural language understanding (NLU) performance of large language models (LLMs) has been evaluated across various tasks and datasets. The existing evaluation methods, however, do not take into account the variance in scores due to differences in prompts, which leads to unfair evaluation and comparison of NLU performance. Moreover, evaluation designed for specific prompts is inappropriate for instruction tuning, which aims to perform well with any prompt. It is therefore necessary to find a way to measure NLU performance in a fair manner, considering score variance between different instruction templates. In this study, we provide English and Japanese cross-lingual datasets for evaluating the NLU performance of LLMs, which include multiple instruction templates for fair evaluation of each task, along with regular expressions to constrain the output format. Furthermore, we propose the Sharpe score as an evaluation metric that takes into account the variance in scores between templates. Comprehensive analysis of English and Japanese LLMs reveals that the high variance among templates has a significant impact on the fair evaluation of LLMs., Comment: 19 pages, 7 figures

The grammatical knowledge of language models (LMs) is often measured using a benchmark of linguistic minimal pairs, where LMs are presented with a pair of acceptable and unacceptable sentences and required to judge which is acceptable. The existing dominant approach, however, naively calculates and compares the probabilities of paired sentences using LMs. Additionally, large language models (LLMs) have yet to be thoroughly examined in this field. We thus investigate how to make the most of LLMs' grammatical knowledge to comprehensively evaluate it. Through extensive experiments of nine judgment methods in English and Chinese, we demonstrate that a probability readout method, in-template LP, and a prompting-based method, Yes/No probability computing, achieve particularly high performance, surpassing the conventional approach. Our analysis reveals their different strengths, e.g., Yes/No probability computing is robust against token-length bias, suggesting that they harness different aspects of LLMs' grammatical knowledge. Consequently, we recommend using diverse judgment methods to evaluate LLMs comprehensively.

With the increase in the more fluent ad texts automatically created by natural language generation technology, it is in the high demand to verify the quality of these creatives in a real-world setting. We propose AdTEC, the first public benchmark to evaluate ad texts in multiple aspects from the perspective of practical advertising operations. Our contributions are: (i) Defining five tasks for evaluating the quality of ad texts and building a dataset based on the actual operational experience of advertising agencies, which is typically kept in-house. (ii) Validating the performance of existing pre-trained language models (PLMs) and human evaluators on the dataset. (iii) Analyzing the characteristics and providing challenges of the benchmark. The results show that while PLMs have already reached the practical usage level in several tasks, human still outperforms in certain domains, implying that there is significant room for improvement in such area.

Minimum Bayes risk (MBR) decoding is a decision rule of text generation tasks that outperforms conventional maximum a posterior (MAP) decoding using beam search by selecting high-quality outputs based on a utility function rather than those with high-probability. Typically, it finds the most suitable hypothesis from the set of hypotheses under the sampled pseudo-references. mbrs is a library of MBR decoding, which can flexibly combine various metrics, alternative expectation estimations, and algorithmic variants. It is designed with a focus on speed measurement and calling count of code blocks, transparency, reproducibility, and extensibility, which are essential for researchers and developers. We published our mbrs as an MIT-licensed open-source project, and the code is available on GitHub. GitHub: https://github.com/naist-nlp/mbrs, Comment: Accepted at EMNLP2024 System Demonstration track

It is very challenging to curate a dataset for language-specific knowledge and common sense in order to evaluate natural language understanding capabilities of language models. Due to the limitation in the availability of annotators, most current multilingual datasets are created through translation, which cannot evaluate such language-specific aspects. Therefore, we propose Multilingual CommonsenseQA (mCSQA) based on the construction process of CSQA but leveraging language models for a more efficient construction, e.g., by asking LM to generate questions/answers, refine answers and verify QAs followed by reduced human efforts for verification. Constructed dataset is a benchmark for cross-lingual language-transfer capabilities of multilingual LMs, and experimental results showed high language-transfer capabilities for questions that LMs could easily solve, but lower transfer capabilities for questions requiring deep knowledge or commonsense. This highlights the necessity of language-specific datasets for evaluation and training. Finally, our method demonstrated that multilingual LMs could create QA including language-specific knowledge, significantly reducing the dataset creation cost compared to manual creation. The datasets are available at https://huggingface.co/datasets/yusuke1997/mCSQA., Comment: Accepted at Findings of ACL 2024

In Simultaneous Machine Translation (SiMT) systems, training with a simultaneous interpretation (SI) corpus is an effective method for achieving high-quality yet low-latency systems. However, it is very challenging to curate such a corpus due to limitations in the abilities of annotators, and hence, existing SI corpora are limited. Therefore, we propose a method to convert existing speech translation corpora into interpretation-style data, maintaining the original word order and preserving the entire source content using Large Language Models (LLM-SI-Corpus). We demonstrate that fine-tuning SiMT models in text-to-text and speech-to-text settings with the LLM-SI-Corpus reduces latencies while maintaining the same level of quality as the models trained with offline datasets. The LLM-SI-Corpus is available at \url{https://github.com/yusuke1997/LLM-SI-Corpus}., Comment: 23 pages, 9 figures

Large-scale vision-language models (LVLMs) output text from images and instructions, demonstrating advanced capabilities in text generation and comprehension. However, it has not been clarified to what extent LVLMs understand the knowledge necessary for explaining images, the complex relationships between various pieces of knowledge, and how they integrate these understandings into their explanations. To address this issue, we propose a new task: the artwork explanation generation task, along with its evaluation dataset and metric for quantitatively assessing the understanding and utilization of knowledge about artworks. This task is apt for image description based on the premise that LVLMs are expected to have pre-existing knowledge of artworks, which are often subjects of wide recognition and documented information. It consists of two parts: generating explanations from both images and titles of artworks, and generating explanations using only images, thus evaluating the LVLMs' language-based and vision-based knowledge. Alongside, we release a training dataset for LVLMs to learn explanations that incorporate knowledge about artworks. Our findings indicate that LVLMs not only struggle with integrating language and visual information but also exhibit a more pronounced limitation in acquiring knowledge from images alone. The datasets (ExpArt=Explain Artworks) are available at https://huggingface.co/datasets/naist-nlp/ExpArt.

Large-scale vision language models (LVLMs) are language models that are capable of processing images and text inputs by a single model. This paper explores the use of LVLMs to generate review texts for images. The ability of LVLMs to review images is not fully understood, highlighting the need for a methodical evaluation of their review abilities. Unlike image captions, review texts can be written from various perspectives such as image composition and exposure. This diversity of review perspectives makes it difficult to uniquely determine a single correct review for an image. To address this challenge, we introduce an evaluation method based on rank correlation analysis, in which review texts are ranked by humans and LVLMs, then, measures the correlation between these rankings. We further validate this approach by creating a benchmark dataset aimed at assessing the image review ability of recent LVLMs. Our experiments with the dataset reveal that LVLMs, particularly those with proven superiority in other evaluative contexts, excel at distinguishing between high-quality and substandard image reviews., Comment: 9pages, under reviewing

Minimum Bayes risk (MBR) decoding achieved state-of-the-art translation performance by using COMET, a neural metric that has a high correlation with human evaluation. However, MBR decoding requires quadratic time since it computes the expected score between a translation hypothesis and all reference translations. We propose centroid-based MBR (CBMBR) decoding to improve the speed of MBR decoding. Our method clusters the reference translations in the feature space, and then calculates the score using the centroids of each cluster. The experimental results show that our CBMBR not only improved the decoding speed of the expected score calculation 5.7 times, but also outperformed vanilla MBR decoding in translation quality by up to 0.5 COMET in the WMT'22 En$\leftrightarrow$Ja, En$\leftrightarrow$De, En$\leftrightarrow$Zh, and WMT'23 En$\leftrightarrow$Ja translation tasks., Comment: Accepted at Findings of ACL 2024

Recent advancements in gravitational wave astronomy have seen the application of convolutional neural networks (CNNs) in signal detection from compact binary coalescences. This study presents a comparative analysis of two CNN architectures: one-dimensional (1D) and two-dimensional (2D) along with an ensemble model combining both. We trained these models to detect gravitational wave signals from binary black hole (BBH) mergers, neutron star-black hole (NSBH) mergers, and binary neutron star (BNS) mergers within real detector noise. Our investigation entailed a comprehensive evaluation of the detection performance of each model type across different signal classes. To understand the models' decision-making processes, we employed feature map visualization and attribution analysis. The findings revealed that while the 1D model showed superior performance in detecting BBH signals, the 2D model excelled in identifying NSBH and BNS signals. Notably, the ensemble model outperformed both individual models across all signal types, demonstrating enhanced detection capabilities. Additionally, input feature visualization indicated distinct areas of focus in the data for the 1D and 2D models, emphasizing the effectiveness of their combination., Comment: 12 pages, 9 figures

Knowledge graphs (KGs) consist of links that describe relationships between entities. Due to the difficulty of manually enumerating all relationships between entities, automatically completing them is essential for KGs. Knowledge Graph Completion (KGC) is a task that infers unseen relationships between entities in a KG. Traditional embedding-based KGC methods, such as RESCAL, TransE, DistMult, ComplEx, RotatE, HAKE, HousE, etc., infer missing links using only the knowledge from training data. In contrast, the recent Pre-trained Language Model (PLM)-based KGC utilizes knowledge obtained during pre-training. Therefore, PLM-based KGC can estimate missing links between entities by reusing memorized knowledge from pre-training without inference. This approach is problematic because building KGC models aims to infer unseen links between entities. However, conventional evaluations in KGC do not consider inference and memorization abilities separately. Thus, a PLM-based KGC method, which achieves high performance in current KGC evaluations, may be ineffective in practical applications. To address this issue, we analyze whether PLM-based KGC methods make inferences or merely access memorized knowledge. For this purpose, we propose a method for constructing synthetic datasets specified in this analysis and conclude that PLMs acquire the inference abilities required for KGC through pre-training, even though the performance improvements mostly come from textual information of entities and relations., Comment: Accepted at NAACL 2024 main oral, 15 pages, 10 figures

In this study, we employ a convolutional neural network to classify gravitational waves originating from core-collapse supernovae. Training is conducted using spectrograms derived from three-dimensional numerical simulations of waveforms, which are injected onto real noise data from the third observing run of both Advanced LIGO and Advanced Virgo. To gain insights into the decision-making process of the model, we apply class activation mapping techniques to visualize the regions in the input image that are significant for the model's prediction. The class activation maps reveal that the model's predictions predominantly rely on specific features within the input spectrograms, namely, the $g$-mode and low-frequency modes. The visualization of convolutional neural network models provides interpretability to enhance their reliability and offers guidance for improving detection efficiency., Comment: 13 pages, 10 figures

This paper presents a state-of-the-art model for transcribing speech in any language into the International Phonetic Alphabet (IPA). Transcription of spoken languages into IPA is an essential yet time-consuming process in language documentation, and even partially automating this process has the potential to drastically speed up the documentation of endangered languages. Like the previous best speech-to-IPA model (Wav2Vec2Phoneme), our model is based on wav2vec 2.0 and is fine-tuned to predict IPA from audio input. We use training data from seven languages from CommonVoice 11.0, transcribed into IPA semi-automatically. Although this training dataset is much smaller than Wav2Vec2Phoneme's, its higher quality lets our model achieve comparable or better results. Furthermore, we show that the quality of our universal speech-to-IPA models is close to that of human annotators., Comment: 5 pages, 7 tables

Richardson-Lucy (RL) deconvolution is one of the classical methods widely used in X-ray astronomy and other areas. Amid recent progress in image processing, RL deconvolution still leaves much room for improvement under a realistic situations. One direction is to include the positional dependence of a point-spread function (PSF), so-called RL deconvolution with a spatially variant PSF (RL$_{\rm{sv}}$). Another is the method of estimating a reliable number of iterations and their associated uncertainties. We developed a practical method that incorporates the RL$_{\rm{sv}}$ algorithm and the estimation of uncertainties. As a typical example of bright and high-resolution images, the Chandra X-ray image of the supernova remnant Cassiopeia~A was used in this paper. RL$_{\rm{sv}}$ deconvolution enables us to uncover the smeared features in the forward/backward shocks and jet-like structures. We constructed a method to predict the appropriate number of iterations by using statistical fluctuation of the observed images. Furthermore, the uncertainties were estimated by error propagation from the last iteration, which was phenomenologically tested with the observed data. Thus, our method is a practically efficient framework to evaluate the time evolution of the remnants and their fine structures embedded in high-resolution X-ray images., Comment: 9 pages, 8 figures

We performed $42$ simulations of the first star formation with initial supersonic gas flows relative to the dark matter at the cosmic recombination era. Increasing the initial streaming velocities led to delayed halo formation and increased halo mass, enhancing the mass of the gravitationally shrinking gas cloud. For more massive gas clouds, the rate of temperature drop during contraction, in other words, the structure asymmetry, becomes more significant. When the maximum and minimum gas temperature ratios before and after contraction exceed about ten, the asymmetric structure of the gas cloud prevails, inducing fragmentation into multiple dense gas clouds. We continued our simulations until $10^5$ years after the first dense core formation to examine the final fate of the massive star-forming gas cloud. Among the $42$ models studied, we find the simultaneous formation of up to four dense gas clouds, with a total mass of about $2254\,M_\odot$. While the gas mass in the host halo increases with increasing the initial streaming velocity, the mass of the dense cores does not change significantly. The star formation efficiency decreases by more than one order of magnitude from $\epsilon_{\rm III} \sim 10^{-2}$ to $10^{-4}$ when the initial streaming velocity, normalised by the root mean square value, increases from 0 to 3., Comment: 15 pages, 16 figures, 1 table, Accepted for publication in MNRAS

Recently, many convolutional neural networks (CNNs) for classification by time domain data of multisignals have been developed. Although some signals are important for correct classification, others are not. When data that do not include important signals for classification are taken as the CNN input layer, the calculation, memory, and data collection costs increase. Therefore, identifying and eliminating nonimportant signals from the input layer are important. In this study, we proposed features gradient-based signals selection algorithm (FG-SSA), which can be used for finding and removing nonimportant signals for classification by utilizing features gradient obtained by the calculation process of grad-CAM. When we define N as the number of signals, the computational complexity of the proposed algorithm is linear time O(N), that is, it has a low calculation cost. We verified the effectiveness of the algorithm using the OPPORTUNITY Activity Recognition dataset, which is an open dataset comprising acceleration signals of human activities. In addition, we checked the average 6.55 signals from a total of 15 acceleration signals (five triaxial sensors) that were removed by FG-SSA while maintaining high generalization scores of classification. Therefore, the proposed algorithm FG-SSA has an effect on finding and removing signals that are not important for CNN-based classification., Comment: 10 pages, 7 figures

Transient noise appearing in the data from gravitational-wave detectors frequently causes problems, such as instability of the detectors and overlapping or mimicking gravitational-wave signals. Because transient noise is considered to be associated with the environment and instrument, its classification would help to understand its origin and improve the detector's performance. In a previous study, an architecture for classifying transient noise using a time-frequency 2D image (spectrogram) is proposed, which uses unsupervised deep learning combined with variational autoencoder and invariant information clustering. The proposed unsupervised-learning architecture is applied to the Gravity Spy dataset, which consists of Advanced Laser Interferometer Gravitational-Wave Observatory (Advanced LIGO) transient noises with their associated metadata to discuss the potential for online or offline data analysis. In this study, focused on the Gravity Spy dataset, the training process of unsupervised-learning architecture of the previous study is examined and reported., Comment: 17 pages, 10 figures, Matches version published in Annalen der Physik

In the data obtained by laser interferometric gravitational wave detectors, transient noise with non-stationary and non-Gaussian features occurs at a high rate. This often results in problems such as detector instability and the hiding and/or imitation of gravitational-wave signals. This transient noise has various characteristics in the time--frequency representation, which is considered to be associated with environmental and instrumental origins. Classification of transient noise can offer clues for exploring its origin and improving the performance of the detector. One approach for accomplishing this is supervised learning. However, in general, supervised learning requires annotation of the training data, and there are issues with ensuring objectivity in the classification and its corresponding new classes. By contrast, unsupervised learning can reduce the annotation work for the training data and ensure objectivity in the classification and its corresponding new classes. In this study, we propose an unsupervised learning architecture for the classification of transient noise that combines a variational autoencoder and invariant information clustering. To evaluate the effectiveness of the proposed architecture, we used the dataset (time--frequency two-dimensional spectrogram images and labels) of the Laser Interferometer Gravitational-wave Observatory (LIGO) first observation run prepared by the Gravity Spy project. The classes provided by our proposed unsupervised learning architecture were consistent with the labels annotated by the Gravity Spy project, which manifests the potential for the existence of unrevealed classes., Comment: 18 pages, 9 figures. Matches version published in Scientific Reports