Search

[Display omitted] • Iterative prompting is a simple yet effective strategy that enables us to interact with ChatGPT or GPT-4 more effectively and efficiently through multi-turn interaction, which iteratively improves the quality of text summary. • Extensive evaluations of our approach using 100 NDA review documents for food effect summarization show the versatility of employing multi-turn interaction to refine the quality of the generated summaries. • The superior performance of GPT-4 over ChatGPT is indicative of the potential of GPT-4 in automating evaluation while acknowledging certain limitations. Food effect summarization from New Drug Application (NDA) is an essential component of product-specific guidance (PSG) development and assessment, which provides the basis of recommendations for fasting and fed bioequivalence studies to guide the pharmaceutical industry for developing generic drug products. However, manual summarization of food effect from extensive drug application review documents is time-consuming. Therefore, there is a need to develop automated methods to generate food effect summary. Recent advances in natural language processing (NLP), particularly large language models (LLMs) such as ChatGPT and GPT-4, have demonstrated great potential in improving the effectiveness of automated text summarization, but its ability with regard to the accuracy in summarizing food effect for PSG assessment remains unclear. In this study, we introduce a simple yet effective approach, iterative prompting, which allows one to interact with ChatGPT or GPT-4 more effectively and efficiently through multi-turn interaction. Specifically, we propose a three-turn iterative prompting approach to food effect summarization in which the keyword-focused and length-controlled prompts are respectively provided in consecutive turns to refine the quality of the generated summary. We conduct a series of extensive evaluations, ranging from automated metrics to FDA professionals and even evaluation by GPT-4, on 100 NDA review documents selected over the past five years. We observe that the summary quality is progressively improved throughout the iterative prompting process. Moreover, we find that GPT-4 performs better than ChatGPT, as evaluated by FDA professionals (43% vs. 12%) and GPT-4 (64% vs. 35%). Importantly, all the FDA professionals unanimously rated that 85% of the summaries generated by GPT-4 are factually consistent with the golden reference summary, a finding further supported by GPT-4 rating of 72% consistency. Taken together, these results strongly suggest a great potential for GPT-4 to draft food effect summaries that could be reviewed by FDA professionals, thereby improving the efficiency of the PSG assessment cycle and promoting generic drug product development. [ABSTRACT FROM AUTHOR]

The problem of factual incorrectness in machine-generated abstractive summarization has received widespread attention in the past few years. Although large-scale neural models show excellent capability in generating fluent and coherent summaries, they still struggle with factual inconsistency, in which the named entity incorrectness is the most frequent and notable one, especially for the character-based languages, such as Chinese. Since abstractive summaries are mostly generated character by character in Chinese, the problem of hallucinated entities is more severe than that of other word-based languages. In this paper, we propose CC-Gens, a novel approach for Correctness Checking based on a Generative negative sampling strategy. Considering that the problem is due to the uncertain nature of the language generation process, we leverage fine-tuned generative language models, i.e., UniLMv2 and mT5, to generate summaries with incorrect entities, thereby constructing a synthetic binary classification dataset for the factuality discriminative model. We propose three strategies: Entity Sampling, Sequence Sampling, and Cloze Sampling, to generate summaries with incorrect entities. With such strategies, the negative samples are much more similar in nature to the output of the neural summarization model. We then train a BERT-based discriminator to identify factually incorrect machine-generated summaries with entity errors based on these generative negative samples. We further propose a novel PU learning algorithm to improve the performance of our approach by iteratively training the discriminator to select high confident negative samples from the unlabeled model generated summaries to replace the former artificially constructed ones. By generating multiple candidate summaries and selecting the one with the highest factual correctness score among them, our approach can significantly reduce the probability that an output summary contains factual errors. According to a comprehensive evaluation, the CC-Gens we proposed outperforms previous works in identifying faithless summaries as well as providing faithful ones. [Display omitted] • Named entity error ratio in Chinese abstractive summaries is as high as 58.1%. • We propose generative negative sampling strategies for training the discriminator. • We apply PU learning to improve performance by selecting high-confident samples. [ABSTRACT FROM AUTHOR]

Stemming is an important pre-processing step in the text analysis domains such as text mining, text summarization and information retrieval (IR). In this study, we build a Sanskrit text collection and explore different indexing, stemming and searching strategies in Sanskrit. We also propose two stemmers: a 'light' and an 'aggressive' and evaluate their effectiveness in the text analysis task. The performance of the stemmers is evaluated in two ways: a direct and an indirect IR-based evaluation. In direct evaluation, we found that the stemmers are effective. In indirect evaluation, we apply different retrieval models such as BM25, TF–IDF, Divergence from Randomness (DFR) based and language models. The proposed stemmers are compared with GRAS stemmer, language-independent indexing (trunc-n) and no stemming approach. Among different stemming methods, aggressive stemmers provide the best performance. Hiemstra language model outperforms other retrieval models we experimented with. In statistical analysis, we found that the proposed stemming approaches produce significantly better results than the no-stemming approach. • We build a Sanskrit text collection for the text analysis domain. • We proposed an inflectional and derivational stemmer in Sanskrit. • The performance of different stemmer is evaluated in the text analysis domain. [ABSTRACT FROM AUTHOR]

Abstractive summarization is flexible and allows the model to generate new words and phrases. However, the familiar words are more likely to be selected as abstract candidate words in the process of abstractive summarization, causing the generated abstract to diverge from the reference. In our consideration, this is caused by representation degeneration of the pre-trained word embedding. Therefore, this paper proposes a general abstractive summarization framework with dynamic word embedding representation correction (RepSum). The representation correction algorithm identifies the dimension most relevant to word frequency and eliminates the word frequency features. Then the distribution of word embeddings will be more even. As a result, the words will be selected as candidate words without frequency bias to improve the quality of the abstract. The experimental results illustrate that RepSum performs better than the benchmark model in summary quality, demonstrating our method's effectiveness. [Display omitted] • An embedding representation correction algorithm based on word frequency feature elimination is proposed. • The algorithm identifies the dimension most relevant to word frequency and eliminates the word frequency features. • Then the distribution of word embeddings will be more even and the words will be selected as candidate words without frequency bias. [ABSTRACT FROM AUTHOR]

Summarization is the process of compressing a text to obtain its important informative parts. In recent years, various methods have been presented to extract important parts of textual documents to present them in a summarized form. The first challenge of these methods is to detect the concepts that well convey the main topic of the text and extract sentences that better describe these essential concepts. The second challenge is the correct interpretation of the essential concepts to generate new paraphrased sentences such that they are not exactly the same as the sentences in the main text. The first challenge has been addressed by many researchers. However, the second one is still in progress. In this study, we focus on the abstractive summarization of biomedical documents. In this regard, for the first challenge, a new method is presented based on the graph generation and frequent itemset mining for generating extractive summaries by considering the concepts within the biomedical documents. Then, to address the second challenge, a transfer learning-based method is used to generate abstractive summarizations from extractive summaries. The efficiency of the proposed solution has been evaluated by conducting several experiments over BioMed Central and NLM's PubMed datasets. The obtained results show that the proposed approach admits a better interpretation of the main concepts and sentences of biomedical documents for the abstractive summarization by obtaining the overall ROUGE of 59.60%, which, on average, is 17% better than state-of-the-art summarization techniques. The source code, datasets, and results are available in GitHub1. [ABSTRACT FROM AUTHOR]