Search

We report our efforts in identifying a set of previous human evaluations in NLP that would be suitable for a coordinated study examining what makes human evaluations in NLP more/less reproducible. We present our results and findings, which include that just 13\% of papers had (i) sufficiently low barriers to reproduction, and (ii) enough obtainable information, to be considered for reproduction, and that all but one of the experiments we selected for reproduction was discovered to have flaws that made the meaningfulness of conducting a reproduction questionable. As a result, we had to change our coordinated study design from a reproduce approach to a standardise-then-reproduce-twice approach. Our overall (negative) finding that the great majority of human evaluations in NLP is not repeatable and/or not reproducible and/or too flawed to justify reproduction, paints a dire picture, but presents an opportunity for a rethink about how to design and report human evaluations in NLP., Comment: 5 pages plus appendix, 4 tables, 1 figure. To appear at "Workshop on Insights from Negative Results in NLP" (co-located with EACL2023). Updated author list and acknowledgements

Steinerberger defined a notion of boundary for a graph and established a corresponding isoperimetric inquality. Hence, "large" graphs have more boundary vertices. In this paper, we first characterize graphs with three boundary vertices in terms of two infinite families of graphs. We then completely characterize graphs with four boundary vertices in terms of eight families of graphs, five of which are infinite. This parallels earlier work by Hasegawa and Saito as well as M\"uller, P\'or, and Sereni on another notion of boundary defined by Chartrand, Erwin, Johns, and Zhang., Comment: 16 pages, 9 figures

This study discusses the effect of semi-supervised learning in combination with pretrained language models for data-to-text generation. It is not known whether semi-supervised learning is still helpful when a large-scale language model is also supplemented. This study aims to answer this question by comparing a data-to-text system only supplemented with a language model, to two data-to-text systems that are additionally enriched by a data augmentation or a pseudo-labeling semi-supervised learning approach. Results show that semi-supervised learning results in higher scores on diversity metrics. In terms of output quality, extending the training set of a data-to-text system with a language model using the pseudo-labeling approach did increase text quality scores, but the data augmentation approach yielded similar scores to the system without training set extension. These results indicate that semi-supervised learning approaches can bolster output quality and diversity, even when a language model is also present., Comment: 22 pages (excluding bibliography and appendix)

This paper develops a unified framework for image-to-image translation based on conditional diffusion models and evaluates this framework on four challenging image-to-image translation tasks, namely colorization, inpainting, uncropping, and JPEG restoration. Our simple implementation of image-to-image diffusion models outperforms strong GAN and regression baselines on all tasks, without task-specific hyper-parameter tuning, architecture customization, or any auxiliary loss or sophisticated new techniques needed. We uncover the impact of an L2 vs. L1 loss in the denoising diffusion objective on sample diversity, and demonstrate the importance of self-attention in the neural architecture through empirical studies. Importantly, we advocate a unified evaluation protocol based on ImageNet, with human evaluation and sample quality scores (FID, Inception Score, Classification Accuracy of a pre-trained ResNet-50, and Perceptual Distance against original images). We expect this standardized evaluation protocol to play a role in advancing image-to-image translation research. Finally, we show that a generalist, multi-task diffusion model performs as well or better than task-specific specialist counterparts. Check out https://diffusion-palette.github.io for an overview of the results.

We present SpeechStew, a speech recognition model that is trained on a combination of various publicly available speech recognition datasets: AMI, Broadcast News, Common Voice, LibriSpeech, Switchboard/Fisher, Tedlium, and Wall Street Journal. SpeechStew simply mixes all of these datasets together, without any special re-weighting or re-balancing of the datasets. SpeechStew achieves SoTA or near SoTA results across a variety of tasks, without the use of an external language model. Our results include 9.0\% WER on AMI-IHM, 4.7\% WER on Switchboard, 8.3\% WER on CallHome, and 1.3\% on WSJ, which significantly outperforms prior work with strong external language models. We also demonstrate that SpeechStew learns powerful transfer learning representations. We fine-tune SpeechStew on a noisy low resource speech dataset, CHiME-6. We achieve 38.9\% WER without a language model, which compares to 38.6\% WER to a strong HMM baseline with a language model., Comment: submitted to INTERSPEECH

Preregistration refers to the practice of specifying what you are going to do, and what you expect to find in your study, before carrying out the study. This practice is increasingly common in medicine and psychology, but is rarely discussed in NLP. This paper discusses preregistration in more detail, explores how NLP researchers could preregister their work, and presents several preregistration questions for different kinds of studies. Finally, we argue in favour of registered reports, which could provide firmer grounds for slow science in NLP research. The goal of this paper is to elicit a discussion in the NLP community, which we hope to synthesise into a general NLP preregistration form in future research., Comment: Accepted at NAACL2021; pre-final draft, comments welcome

Many ultra-diffuse galaxies (UDGs) have been discovered in the Coma cluster, and there is evidence that some, notably Dragonfly 44, have Milky Way-like dynamical masses despite dwarf-like stellar masses. We used X-ray, UV, and optical data to investigate the star formation and nuclear activity in the Coma UDGs, and we obtained deep UV and X-ray data (Swift and XMM-Newton) for Dragonfly 44 to search for low-level star formation, hot circumgalactic gas, and the integrated emission from X-ray binaries. Among the Coma UDGs, we find UV luminosities consistent with quiescence but NUV$-r$ colors indicating star formation in the past Gyr. This indicates that the UDGs were recently quenched. The $r$-band luminosity declines with projected distance from the Coma core. The Dragonfly 44 UV luminosity is also consistent with quiescence, with SFR$<6\times 10^{-4} M_{\odot}$ yr$^{-1}$, and no X-rays are detected down to a sensitivity of $10^{38}$ erg s$^{-1}$. This rules out a hot corona with a $M > 10^8 M_{\odot}$ within the virial radius, which would normally be expected for a dynamically massive galaxy. The absence of bright, low mass X-ray binaries is consistent with the expectation from the galaxy total stellar mass, but it is unlikely if most low-mass X-ray binaries form in globular clusters, as Dragonfly 44 has a very large population. Based on the UV and X-ray analysis, the Coma UDGs are consistent with quenched dwarf galaxies, although we cannot rule out a dynamically massive population., Comment: 12 pages, 5 tables, 7 figures. Accepted for publication in MNRAS

Traditionally, most data-to-text applications have been designed using a modular pipeline architecture, in which non-linguistic input data is converted into natural language through several intermediate transformations. In contrast, recent neural models for data-to-text generation have been proposed as end-to-end approaches, where the non-linguistic input is rendered in natural language with much less explicit intermediate representations in-between. This study introduces a systematic comparison between neural pipeline and end-to-end data-to-text approaches for the generation of text from RDF triples. Both architectures were implemented making use of state-of-the art deep learning methods as the encoder-decoder Gated-Recurrent Units (GRU) and Transformer. Automatic and human evaluations together with a qualitative analysis suggest that having explicit intermediate steps in the generation process results in better texts than the ones generated by end-to-end approaches. Moreover, the pipeline models generalize better to unseen inputs. Data and code are publicly available., Comment: Preprint version of the EMNLP 2019 article

The absorption of electromagnetic radiation by a planar structure, consisting of a three-layer dielectric-metal-dielectric coating on a metal back-reflector is analysed. The conditions for total absorption are derived. Our analysis shows that, in contrast with bi-layer structures, the calculated layer thicknesses are feasible to fabricate for any metal. The proposed absorber design is of potential use in infrared and longer wavelength detectors and for radiant energy harvesting devices.