Search

Large language models (LLMs) often inherit biases from vast amounts of training corpora. Traditional debiasing methods, while effective to some extent, do not completely eliminate memorized biases and toxicity in LLMs. In this paper, we study an unlearning-based approach to debiasing in LLMs by performing gradient ascent on hate speech against minority groups, i.e., minimizing the likelihood of biased or toxic content. Specifically, we propose a mask language modeling unlearning technique, which unlearns the harmful part of the text. This method enables LLMs to selectively forget and disassociate from biased and harmful content. Experimental results demonstrate the effectiveness of our approach in diminishing bias while maintaining the language modeling abilities. Surprisingly, the results also unveil an unexpected potential for cross-domain transfer unlearning: debiasing in one bias form (e.g. gender) may contribute to mitigating others (e.g. race and religion).

The creativity of scientists often manifests as localized hot streaks of significant success. Understanding the underlying mechanisms of these influential phases can enhance the effectiveness of support systems and funding allocation, fostering groundbreaking discoveries worthy of accolades. Historically, analyses have suggested that hot streaks occur randomly over time. However, our research, through meticulous examination, reveals that these phases are not flatly distributed but are more frequent at the early and late stages of scientists' careers. Notably, both early and late hot streaks are marked by dense tie collaborations, with the former typically involving close partnerships with particular authors and the latter being characterized by involvement in large-scale projects compared with single-top or ordinary papers. This pattern indicates that mid-career researchers lack both intimate relations and resources to keep big projects, leading to``mid-career pitfal'' of consecutive success. This insight holds profound implications for the development of policies and initiatives aimed at bolstering innovative research and discovery., Comment: 23 pages, 20 figures

Instruction tuning has been attracting much attention to achieve generalization ability across a wide variety of tasks. Although various types of instructions have been manually created for instruction tuning, it is still unclear what kind of instruction is optimal to obtain cross-task generalization ability. This work presents instruction optimization, which optimizes training instructions with respect to generalization ability. Rather than manually tuning instructions, we introduce learnable instructions and optimize them with gradient descent by leveraging bilevel optimization. Experimental results show that the learned instruction enhances the diversity of instructions and improves the generalization ability compared to using only manually created instructions., Comment: 14pages, 6 figures, accepted for Findings of ACL2023

Automatic literature review generation is one of the most challenging tasks in natural language processing. Although large language models have tackled literature review generation, the absence of large-scale datasets has been a stumbling block to the progress. We release SciReviewGen, consisting of over 10,000 literature reviews and 690,000 papers cited in the reviews. Based on the dataset, we evaluate recent transformer-based summarization models on the literature review generation task, including Fusion-in-Decoder extended for literature review generation. Human evaluation results show that some machine-generated summaries are comparable to human-written reviews, while revealing the challenges of automatic literature review generation such as hallucinations and a lack of detailed information. Our dataset and code are available at https://github.com/tetsu9923/SciReviewGen., Comment: ACL findings 2023 (to be appeared). arXiv admin note: text overlap with arXiv:1810.04020 by other authors

In the Cumulative Advantage(CA) model, which is one of the most fundamental approaches to understand the mechanism of citation dynamics, papers receive citations depending on how much they have been already cited. On the other hand, a substantial effect not included in CA is that some surprising discoveries suddenly acquire citations after a long time from publishing. This phenomenon is known as Sleeping Beauty(SB). Since disrupting discoveries need long-time discussion by the research community to accept, SBs can capture innovative findings and reveal the nature of disruptive scientific knowledge production. To research SBs citation burst mechanism, bibliometricians consider the existence of the Prince(PR) for each SBs, which can be the trigger of SBs awakeness. For example, the discovery of Green Fluorescent Protein(GFP), which got Nobel prize in chemistry, had been overlooked for 30 years until Chalfie and Tsien, who also received the prize, developed a method to use GFP as a marker protein in genetic engineering. However, how does Chalfies and Tsiens research relight the hidden knowledge in the research community? If we can clarify such a mechanism rediscovering from nearly nothing, it can be helpful in science support and policy decision-making. This study proposes a Storyteller that focuses on the connection between SB and PR before SB gets citation burst by co-citation. PR is found to be the paper awakening SB in retrospect, but it is not easy to detect it as the trigger of SBs awakeness at the time of PR submission. We named the papers which co-cites SB and PR before the citation burst of SB as Storyteller(ST) and analyze (1) how ST contributes to broadening the novelty of SB&PR connections and (2) how much ST leads the citation burst after awakening., Comment: preprint, submitted to ASIS&T SIG-MET Workshop, extended abstract

This paper presents a novel unsupervised abstractive summarization method for opinionated texts. While the basic variational autoencoder-based models assume a unimodal Gaussian prior for the latent code of sentences, we alternate it with a recursive Gaussian mixture, where each mixture component corresponds to the latent code of a topic sentence and is mixed by a tree-structured topic distribution. By decoding each Gaussian component, we generate sentences with tree-structured topic guidance, where the root sentence conveys generic content, and the leaf sentences describe specific topics. Experimental results demonstrate that the generated topic sentences are appropriate as a summary of opinionated texts, which are more informative and cover more input contents than those generated by the recent unsupervised summarization model (Bra\v{z}inskas et al., 2020). Furthermore, we demonstrate that the variance of latent Gaussians represents the granularity of sentences, analogous to Gaussian word embedding (Vilnis and McCallum, 2015)., Comment: accepted to TACL, pre-MIT Press publication version

This paper focuses on the end-to-end abstractive summarization of a single product review without supervision. We assume that a review can be described as a discourse tree, in which the summary is the root, and the child sentences explain their parent in detail. By recursively estimating a parent from its children, our model learns the latent discourse tree without an external parser and generates a concise summary. We also introduce an architecture that ranks the importance of each sentence on the tree to support summary generation focusing on the main review point. The experimental results demonstrate that our model is competitive with or outperforms other unsupervised approaches. In particular, for relatively long reviews, it achieves a competitive or better performance than supervised models. The induced tree shows that the child sentences provide additional information about their parent, and the generated summary abstracts the entire review., Comment: 13 pages, ACL 2019 (long paper)