Search

This paper presents a shared task that we organized at the Foundations of Language Technology (FoLT) course in 2023/2024 at the Technical University of Darmstadt, which focuses on evaluating the output of Large Language Models (LLMs) in generating harmful answers to health-related clinical questions. We describe the task design considerations and report the feedback we received from the students. We expect the task and the findings reported in this paper to be relevant for instructors teaching natural language processing (NLP) and designing course assignments., Comment: accepted at the sixth Workshop on Teaching NLP (co-located with ACL 2024)

Applying differential privacy (DP) by means of the DP-SGD algorithm to protect individual data points during training is becoming increasingly popular in NLP. However, the choice of granularity at which DP is applied is often neglected. For example, neural machine translation (NMT) typically operates on the sentence-level granularity. From the perspective of DP, this setup assumes that each sentence belongs to a single person and any two sentences in the training dataset are independent. This assumption is however violated in many real-world NMT datasets, e.g., those including dialogues. For proper application of DP we thus must shift from sentences to entire documents. In this paper, we investigate NMT at both the sentence and document levels, analyzing the privacy/utility trade-off for both scenarios, and evaluating the risks of not using the appropriate privacy granularity in terms of leaking personally identifiable information (PII). Our findings indicate that the document-level NMT system is more resistant to membership inference attacks, emphasizing the significance of using the appropriate granularity when working with DP., Comment: Accepted at EMNLP Findings 2024

Neural machine translation (NMT) is a widely popular text generation task, yet there is a considerable research gap in the development of privacy-preserving NMT models, despite significant data privacy concerns for NMT systems. Differentially private stochastic gradient descent (DP-SGD) is a popular method for training machine learning models with concrete privacy guarantees; however, the implementation specifics of training a model with DP-SGD are not always clarified in existing models, with differing software libraries used and code bases not always being public, leading to reproducibility issues. To tackle this, we introduce DP-NMT, an open-source framework for carrying out research on privacy-preserving NMT with DP-SGD, bringing together numerous models, datasets, and evaluation metrics in one systematic software package. Our goal is to provide a platform for researchers to advance the development of privacy-preserving NMT systems, keeping the specific details of the DP-SGD algorithm transparent and intuitive to implement. We run a set of experiments on datasets from both general and privacy-related domains to demonstrate our framework in use. We make our framework publicly available and welcome feedback from the community., Comment: Accepted at EACL 2024

We present a "one-step" approach to modify ZnO electron transport layers (ETLs) used in organic solar cells. This approach involves adding benzoic acid (BZA) derivatives directly to the ZnO precursor solution, which are then present at the surface of the resulting ZnO film. We demonstrate this approach for three different BZA derivatives, namely benzoic acid, chlorobenzoic acid, and 4-hydrazinobenzoic acid. For all molecules, improved device performance and stability is demonstrated in solar cells using an active layer blend of PTQ10 (donor) and ITIC-Br (non-fullerene acceptor) compared to such cells prepared using untreated ZnO. Furthermore, similar or improved device performance and stability is demonstrated compared to conventional PEIE treatment of ZnO. The presence of the BZA derivatives at the surface after processing is established using X-ray photoelectron spectroscopy and near-edge X-ray absorption fine-structure spectroscopy. From atomic force microscopy analysis and X-ray diffraction studies, the addition of BZA derivatives appears to restrict ZnO grain growth; however, this does not negatively impact device performance. ZnO layers treated with BZA derivatives also exhibit higher water contact angle and lower work function compared to untreated ZnO. This approach enables simplification of device manufacture while still allowing optimization of the surface properties of metal oxide ETLs. Keywords: electron transport layers, zinc oxide, organic solar cells, surface modification, Comment: Manuscript: 25 pages, 8 figures, 5 tables. Supplementary Material: 36 pages, 22 figures, 13 tables. Submitted to Solar Energy Materials and Solar Cells

The evaluation of recent embedding-based evaluation metrics for text generation is primarily based on measuring their correlation with human evaluations on standard benchmarks. However, these benchmarks are mostly from similar domains to those used for pretraining word embeddings. This raises concerns about the (lack of) generalization of embedding-based metrics to new and noisy domains that contain a different vocabulary than the pretraining data. In this paper, we examine the robustness of BERTScore, one of the most popular embedding-based metrics for text generation. We show that (a) an embedding-based metric that has the highest correlation with human evaluations on a standard benchmark can have the lowest correlation if the amount of input noise or unknown tokens increases, (b) taking embeddings from the first layer of pretrained models improves the robustness of all metrics, and (c) the highest robustness is achieved when using character-level embeddings, instead of token-based embeddings, from the first layer of the pretrained model., Comment: COLING 2022 camera-ready version