Search

Automatic Text Summarization (ATS) involves estimating the salience of information and creating coherent summaries that include all relevant and important information from the original text. Extensive research has been carried out on ATS since 1958, gradually evolving from simple to advanced techniques, including machine learning-based, neural network-based, and deep learning-based techniques. Progress has been made in both extractive and abstractive methods throughout this development. Despite numerous surveys on ATS, there remains a notable absence of a comprehensive literature review encompassing the latest advancements in deep learning techniques for text summarization. Therefore, this paper provides a Systematic Literature Review (SLR) of deep learning-based text summarization in both types (extractive and abstractive) between 2014 and 2023. To the best of our knowledge, this is the first SLR that offers a comprehensive overview of extractive and abstractive text summarization techniques based on Deep Learning models. According to the defined inclusion and exclusion criteria, 73 deep learning-based text summarization studies are selected for further investigation. The structure of the review is organized as follows. Firstly, it identifies and examines the deep learning models employed in both extractive and abstractive text summarization. Then, the input text's representation methods are identified and discussed clearly. Next, training strategies used in supervised extractive summarization techniques are identified. Furthermore, mechanisms that improve the abstractive summarization process are identified. Additionally, the most commonly used datasets and their advantages and disadvantages are discussed. The most commonly used evaluation metrics are also identified. Finally, the challenges and possible solutions to guide future research in the field are discussed. • Analyze the latest deep learning-based text summarization techniques. • Introduce a new taxonomy of deep learning models for text summarization. • Identify methods for representing input text clearly. • Identify training strategies for supervised extractive summarization techniques. • Identify the mechanisms used in abstractive summarization techniques. [ABSTRACT FROM AUTHOR]

The ever-increasing volume of research literature poses challenges for researchers in keeping up with related works in their fields. Automating the generation of related work section holds promise for saving time and effort. However, current models often fall short of producing coherent and logically correct related work with multiple sentences, a phenomenon we refer to as rhetorical structure chaos. Rhetorical structure describes how adjacent spans of units are connected to each other, and logically correct rhetorical structure is essential for a well-structured related work. Hence, to tackle the rhetorical structure chaos issue, this paper explicitly incorporates rhetorical structure information into related work generation. Firstly, we conduct the first rhetorical structure analysis for related work section, which provides insights into understanding the organization and arrangement of contents within related work. Then, based on two preliminary studies on rhetorical structure, we present a novel related work generation model called RSGen, which incorporates rhetorical structure at both the encoding and decoding stages. The encoding stage is facilitated with a rhetorical structure-based graph encoder, while the decoding process is guided by a rhetorical plan — ordered sequence of rhetorical functions of the related work. We conduct extensive experiments on three related work generation datasets to evaluate the performance of our model. The results show that our approach achieves state-of-the-art performance on ROUGE metrics. An ablation study and more analyses further highlight the remarkable efficacy of introducing rhetorical structure into both the encoding and decoding stages. • The first rhetorical structure analysis for related work section. • A model enhanced with rhetorical structure at both encoding and decoding stages. • Superior performance that exceeds all baselines and achieves state-of-the-art. [ABSTRACT FROM AUTHOR]

Text summarization research is significant and challenging in the domain of natural language processing. Abstractive text summarization mainly uses the encoder-decoder framework, wherein the encoder component does not have a sufficient semantic comprehension of the input text, and there are exposure biases and semantic inconsistencies between the reference and generated summaries during the training process. We propose an improved encoder-decoder model that incorporates a hierarchical attention mechanism and multiobjective reinforcement learning. The encoder introduces a multihead self-attention mechanism to allow for the acquisition of more comprehensive semantic information from multiple angles and dimensions, while the decoder introduces a pointer-generator network to solve the out-of-vocabulary problem. Multiobjective reinforcement learning methods are constructed throughout the training process to optimize the model in terms of addressing exposure bias, maintaining semantic consistency, and enhancing readability. The results of the comparative experiments demonstrate that the proposed model significantly improved in terms of the ROUGE evaluation metric, and the generated summaries were semantically similar to the reference summaries. [ABSTRACT FROM AUTHOR]

Text summarization is a widely-researched problem among scholars in the field of natural language processing. Multiple techniques have been proposed to help tackle this problem, yet some of these methodologies may still exhibit limitations such as the requirements for large training datasets, which might not always be possible, but more importantly, the lack of interpretability or transparency of the model. In this paper, we propose using meta-learning algorithm to train a deep learning model for extractive text summarization and then using various explanatory techniques such as SHAP (Shapley, 1953), linear regression (Lederer, 2022), decision trees (Fürnkranz, 2010), and input modification to gain insights into the model's decision making process. To evaluate the effectiveness of our approach, we will compare it to other popular natural language processing models like BERT (Miller, 2019) or XLNET (Yang et al., 2020) using the ROUGE metrics (Lin, 2004). Overall, our proposed approach provides a promising solution to the limitations of existing methods and a framework for improving the explainability of deep learning models in other natural language processing tasks. • Bi-LSTM and Meta-learning produce better results compared to other popular models. • Meta-learning can be utilized to improve performance in data-scarce scenarios. • Use SHAP, regression, decision tree, and text mutation to enhance interpretability. [ABSTRACT FROM AUTHOR]

With the advent of transformer-based architectures, the contextual representation of text data has leveraged the query and the document to be represented in low-dimensional dense vector space. These vectors are learned embeddings of fixed sizes, resulting in deeper text understanding. In this study, we designed a pipeline for effectively retrieving documents from a large search space by combining the deeper text understanding capabilities of the transformer-based BERT model and a phrase embedding-based query expansion model. To learn the contextual representations, we fine-tuned a deep semantic matching model by separately encoding the document and the query. The encoder model is based on the Sentence BERT (SBERT) architecture, which separately generates dense vector representations of documents and queries. The study has also addressed the maximum token length limitation of transformer-based models through the summarization of lengthy documents. In addition, to improve the clarity and completeness of short queries and reduce the semantic gap, a phrase embedding-based query expansion model is employed. The documents and their dense vectors are indexed using the Elasticsearch engine, and matched them with query vectors for retrieving query-specific documents. Finally, the BERT-based cross-encoder model is used to re-rank the relevant records for each query. It performs full self-attention over the inputs, and yields richer text interactions to produce the final results. To assess performance, experiments are conducted on two well-known datasets, TREC-CDS-2014 and OHSUMED. A comparative analysis is carried out, which clearly demonstrates that the proposed framework produced competitive retrieval results. • Transformer-based contextual text representation approach is proposed for IR. • Bi-Encoder architecture is applied to learn the representations of documents & query. • BERT-based summarization is applied to address the limitation on sequence length. • Phrase Embedding-based QE technique is employed to expand short and unclear queries. • Re-ranking of the retrieval results is performed using the cross encoder model. [ABSTRACT FROM AUTHOR]

The Transformer-based summarization models rely solely on the attention mechanism for document encoding, making it difficult to accurately capture long-range dependencies in long documents due to the presence of attention redundancy. To address this issue, we propose an extractive summarization framework guided by a topic model (TopicSum) that utilizes a heterogeneous graph neural network to leverage the topic information as document-level features during the sentence selection process, thereby capturing the long-range dependencies among sentences. The sentence-level features in this topic model align with the basic unit of the extractive summarization task. Additionally, a memory mechanism is employed to dynamically store and update the memory module, reducing the potential of repetitive information guiding sentence selection. We evaluated the model on three large document datasets, namely Pubmed, arXiv, and GovReport, and achieved significantly higher Rouge scores than previous works, including extractive and abstractive models. Furthermore, our experiments demonstrate that recent highly regarded large language models such as ChatGPT are insufficient to handle the long document summarization task directly. The proposed approach in this paper exhibits sufficient competitiveness in terms of both generation quality and deployment conditions. [ABSTRACT FROM AUTHOR]

Development of effective automatic summarization approaches for legal documents suffer from several challenges like extremely long document-summary pairs, lack of large scale training datasets with tractable document-summary token lengths, etc. In this work, we deal with the problem of legal document summarization by building a modified abstractive dataset from the original dataset. This ensures that the length of each document-summary pair is manageable and can be processed by the state of the art summarization approaches (such as BART). Secondly, we deal with the data scarcity problem by creating more number of training samples, from each of the original document-summary pair. This is done by creating multiple extractive summaries from each sample in the original dataset, following which ground-truth summary sentences are assigned to each of the extractive summary to generate new training samples. This results in a larger training dataset that can be utilized for fine-tuning summarization models. Our proposed approach has been evaluated on two different legal datasets- BillSum and Forum of Information Retrieval Evaluation (FIRE). With respect to the ROUGE metrics, the proposed approach is able to outperform pre-trained BART model fine-tuned on original dataset by (3 − 8) % for FIRE test sets, and by (1 − 3) % for the BillSum test sets. Considering the BERTScore metrics, the proposed approach obtains (1 − 2) % improvements on the FIRE test sets, while for the BillSum test sets (3 − 8) % improvements are observed. Such improvements suggest that the proposed dataset building approach can help achieve improved abstractive summarization of lengthy legal documents. • Lengthy nature and data scarcity are the two main challenges with legal documents. • Lengthy legal document summarization is handled by the proposed approach. • The data scarcity problem is handled by creating a feasible abstractive dataset. • A novel Extract-Then-Assign (ETA) approach is proposed. • ETA approach can greatly improve abstractive summarization of legal documents. [ABSTRACT FROM AUTHOR]

Automatically generating a survey draft is a challenge to text summarization research because it needs to select important sentences from important references in a large set of candidate papers for composing sections that are in line with section titles and different sections discuss the most relevant reference papers of different number, which are beyond the capability of previous text summarization approaches as they assume that all candidate papers should be included into one summary. This paper proposes an approach to generating survey draft according to a pattern consisting of sections with titles given by the user who requests the survey. The problem of generating each section can be divided into the following sub-problems: (1) rank the input scientific documents (in short documents) according to the title of a section, (2) determine the number of documents that are most relevant to the title, and (3) rank and select sentences from the selected documents according to the title. A position closeness distance of key word is proposed to rank a set of documents by measuring how closely two key words within section title are distributed within each document, which is used to rank the documents. The rationale is that the positions of the neighboring key words of a section title should be closer in more relevant documents than other words. As different sections have different number of selected documents, a method is proposed to determine the number of documents to be included into the current section based on the slope shape of the sorted rank curve of documents according to the section title. Based on the duality property of the closeness, ranks of sentences within a document can be directly obtained when the document is ranked according to the title of section, and both the importance and coherence of selected sentences can be reflected without extra calculation for ranking sentences. Experiments and manual evaluation show that the proposed methods achieve significant improvements compared with other approaches. The proposed approach is significant in applications as different surveys can be generated according to different patterns given by different users. [ABSTRACT FROM AUTHOR]

In Natural Language Processing (NLP) domain, the majority of automatic text summarization approaches depend on a prior knowledge of the language and/or the domain of the text being summarized. Such approaches requires language dependent part-of-speech taggers, parsers, databases, pre-structured lexicons, etc. In this research, we propose a novel automatic text summarization model, Text Documents - Language Agnostic Summarization Model (TxLASM), which is able to perform extractive text summarization task in language/domain agnostic manner. TxLASM depends on specific characteristics of the major elements of the text being summarized rather than its domain, context, or language and thus rules out the need for language dependent pre-processing tools, taggers, parsers, lexicons or databases. Within TxLASM , we present a novel technique for encoding the shapes of major text elements (paragraphs, sentences, n-grams and words); moreover, we present language independent preprocessing algorithms to normalize words and perform relative stemming or lemmatization. Those algorithms and its Shape-Coding technique enable the TxLASM to extract intrinsic features of text elements and score them statistically, and subsequently extract a representative summary that is independent of the text language, domain and context. TxLASM was applied on an English and Portuguese benchmark datasets, and the results were compared to twelve state-of-the-art approaches presented in recent literature. In addition, the model was applied on French and Spanish news datasets, and the results were compared to those obtained by standard commercial summarization tools. TxLASM has outperformed all the SOTA approaches as well as the commercial tools in all four languages while maintaining its language and domain agnostic nature. [ABSTRACT FROM AUTHOR]

With the development of pre-trained language models and large-scale datasets, automatic text summarization has attracted much attention from the community of natural language processing, but the progress of automatic summarization evaluation has stagnated. Although there have been efforts to improve automatic summarization evaluation, ROUGE has remained one of the most popular metrics for nearly 20 years due to its competitive evaluation performance. However, ROUGE is not perfect, there are studies have shown that it is suffering from inaccurate evaluation of abstractive summarization and limited diversity of generated summaries, both caused by lexical bias. To avoid the bias of lexical similarity, more and more meaningful embedding-based metrics have been proposed to evaluate summaries by measuring semantic similarity. Due to the challenge of accurately measuring semantic similarity, none of them can fully replace ROUGE as the default automatic evaluation toolkit for text summarization. To address the aforementioned problems, we propose a compromise evaluation framework (ROUGE-SEM) for improving ROUGE with semantic information, which compensates for the lack of semantic awareness through a semantic similarity module. According to the differences in semantic similarity and lexical similarity, summaries are classified into four categories for the first time, including good-summary, pearl-summary, glass-summary, and bad-summary. In particular, the back-translation technique is adopted to rewrite pearl-summary and glass-summary that are inaccurately evaluated by ROUGE to alleviate lexical bias. Through this pipeline framework, summaries are first classified by candidate summary classifier, then rewritten by categorized summary rewriter, and finally scored by rewritten summary scorer, which are efficiently evaluated in a manner consistent with human behavior. When measured using Pearson, Spearman, and Kendall rank coefficients, our proposal achieves comparable or higher correlations with human judgments than several state-of-the-art automatic summarization evaluation metrics in dimensions of coherence, consistency, fluency, and relevance. This also suggests that improving ROUGE with semantics is a promising direction for automatic summarization evaluation. [ABSTRACT FROM AUTHOR]

• The select-then-extract framework can better alleviate class imbalance. • Jointing the Extractive Summarization task contributes to the ECPE task. • Multi-granularity cooperative graph can better capture inter-clause relations. • The global keyword helps highlight causal cues implied in distant clauses. • Integrated inference predictor improves the efficiency of reasoning. The class imbalance problem arises in the current end-to-end models for the emotion-cause pair extraction (ECPE) task, and a large number of negative clause pairs interfere model to depict the characteristics of emotion and cause. Most existing methods adopt a window-constrained strategy to limit the relative distance between candidate clauses to a smaller range to filter negative pairs. However, those methods ignore the semantic association between distant clauses, and using an artificial window size is imperfect due to the inherent ambiguity and delicacy of emotion. Inspired by the centrality principle of discourse, in this paper we propose a select-then-extract framework for the ECPE task, where the core clauses are first selected from the original document as candidate emotion and cause clauses, and then emotions, causes and emotion-cause pairs are jointly extracted from the candidate clauses. Specifically, our model includes two main components: core clause selector and emotion-cause pairs extractor. For the Core Clause Selector , we introduce the extractive document summarization (EDS) task and present a multi-granularity semantic awareness cooperative graph model (MGCOG) to extract core clauses from documents. Compared to the previous methods, the core clause selector is more effective for alleviating the category imbalance because the global causal cues and context can be captured by learning global keywords and inter-clause relationships, which determines more efficient candidate clauses. For Emotion-Cause Pairs Extractor , we put forward a multi-task learning model to jointly extract emotions, causes and emotion-cause pairs from the selected core clauses. Here, a multi-head attention is used to further model the relationship between candidate clauses, and a co-predictor is designed for assigning scores to all possible emotion-cause pairs. We further investigate the pipeline and the joint model under the select-then-extract framework, and show that the experimental results on benchmark datasets are consistently superior to the comparative baseline models. Extensive ablation experiments also verify the effectiveness of each component. [ABSTRACT FROM AUTHOR]

Recently, large language models have shown great success on various abstractive summarization datasets. These datasets consist of numerous data that are enough to train a large number of parameters. However, for a new domain, there is a lack of labeled data to train those parameters and the model is easily overfitted to a small amount of data. In addition, because annotating document-summary pairs is too expensive and transfer learning using high-resource datasets causes a domain shifting problem, a low-resource abstractive summarization task is becoming necessary. Herein, we propose an efficient framework for low-resource abstractive summarization using a pointer-generator network and a meta-learning technique to address the above problems. Meta-learning using existing high-resource datasets enables our model to rapidly adapt to a new domain using limited data to solve the domain shifting problem. In addition, we explore the copy mechanism using a pointer-generator network that can copy words from a source document when generating a summary. The experimental results on 11 different datasets show that the proposed model outperforms the previous state-of-the-art models in low-resource abstractive summarization on most of the datasets. [ABSTRACT FROM AUTHOR]

The primary challenge faced in extractive text summarization is related to the scoring of sentences, with the critical factor for scoring being the manner in which the sentence representation is conducted. This study aims to investigate this hypothesis and to perform a detailed analysis of the impact of sentence representation techniques that have been used both semantically and syntactically. The study initially evaluated the empirical impact of individual syntactic and semantic features on the accuracy of summarization. To examine syntactic usage, a comprehensive list of 40 syntactic features was developed, while semantic representation was accomplished using sentence embeddings. Subsequently, an improved feature set was proposed that jointly utilizes syntactic and semantic features. To assess the impact of this feature set on the resulting summaries, the proposed sentence representation was tested on three distinct summarization corpora consisting of lengthy scientific documents across diverse domains. The assessment of summary evaluation and classification performance evaluation metrics was conducted to evaluate the quality of the resulting summaries. The findings of the experiments indicated that the summaries generated by the proposed feature set performed better than not only those obtained using individual features but even summaries produced by state-of-the-art methods. • In literature, text summarization has been handled by syntactic or semantic approach. • A new enhanced sentence representation was proposed to score sentences. • The sentence scoring model was a random forest fed by this enhanced representation. • The proposed model remarkably outperformed the baseline and state-of-the-art models. [ABSTRACT FROM AUTHOR]

Sentiment classification aims to predict the sentiment label for a given text. Recently, several research efforts have been devoted to incorporate matching clues between text words and class labels into the learning process of text representation. However, these methods heavily rely on the availability of label content. Moreover, they simply capture the label-specific signals to measure each word's contribution by either implicitly employing a learnable label representation or explicitly leveraging the interaction between text words and labels via the interaction mechanism. To deal with these issues, in this paper, we propose a novel framework called Label-Guided Dual-view Sentiment Classifier (LGDSC). We first introduce a new strategy for generating an effective label description and then design a novel Dual-Channel Label-guided Attention Network (DLAN) to learn a text representation via two different channels. DLAN will be further leveraged to learn label-guided text representations from two different views. Extensive experimental results on four real-world datasets demonstrate that LGDSC consistently outperforms the state-of-the-art baseline methods. • Propose an inverse label entropy based strategy for generating effective label descriptions. • Design a dual-channel label-guided attention network to learn text representation via two different channels. • Extensive experiments conducted on four widely used datasets demonstrate the effectiveness of the proposed approach. [ABSTRACT FROM AUTHOR]

In this paper, we propose DeepSumm, a novel method based on topic modeling and word embeddings for the extractive summarization of single documents. Recent summarization methods based on sequence networks fail to capture the long range semantics of the document which are encapsulated in the topic vectors of the document. In DeepSumm, our aim is to utilize the latent information in the document estimated via topic vectors and sequence networks to improve the quality and accuracy of the summarized text. Each sentence is encoded through two different recurrent neural networks based on probabilistic topic distributions and word embeddings, and then a sequence to sequence network is applied to each sentence encoding. The outputs of the encoder and the decoder in the sequence to sequence networks are combined after weighting using an attention mechanism and converted into a score through a multi-layer perceptron network. We refer to the score obtained through the topic model as Sentence Topic Score (STS) and to the score generated through word embeddings as Sentence Content Score (SCS). In addition, we propose Sentence Novelty Score (SNS) and Sentence Position Score (SPS) and perform a weighted fusion of the four scores for each sentence in the document to compute a Final Sentence Score (FSS). The proposed DeepSumm framework was evaluated on the standard DUC 2002 benchmark and CNN/DailyMail datasets. Experimentally, it was demonstrated that our method captures both the global and the local semantic information of the document and essentially outperforms existing state-of-the-art approaches for extractive text summarization with ROUGE-1, ROUGE-2, and ROUGE-L scores of 53.2, 28.7 and 49.2 on DUC 2002 and 43.3, 19.0 and 38.9 on CNN/DailyMail dataset. • DeepSumm for extractive summarization based on seq2seq networks. • Sentence encoded with probabilistic topic distributions and word embeddings RNNs. • Proposed Sentence Novelty Score, based on sentence content and topic embeddings. • Seq2seq attention networks extract content and topic Scores. • Summary generated using four sentence scores: Content, Topic, Novelty and Position. [ABSTRACT FROM AUTHOR]

• Automatic and accurate evaluation of summary is a key problem of text summarization research. • A framework for evaluating summary from fidelity, conciseness, and coherence. • An evaluation method based on the semantic link network representation of text. • The semantic link network plays an important role in representing and analyzing texts. Automatic evaluation of summary is one of the basic problems of automatic text summarization research. The commonly used approaches evaluate the informativeness, conciseness and coherence of summary separately based on different representations with weak semantics such as n-gram and vector of words. By transforming text into semantic link network, this paper proposes a general framework for automatically evaluating summary from the following aspects: (1) fidelity , the extent of a summary that conveys the core of its source text; (2) conciseness , the extent of non-redundancy within summary and, (3) coherence , the extent of relatedness between representations within summary, which are based on a uniform semantic representation of text. Experiments on the evaluation of the summarization of scientific papers and news show that the proposed framework achieves comparable or higher correlations with human judgments than the popular evaluation models. This research also verifies the role of the semantic link network in representing and analysing texts. [ABSTRACT FROM AUTHOR]

In this paper, we propose Ranksum, an approach for extractive text summarization of single documents based on the rank fusion of four multi-dimensional sentence features extracted for each sentence: topic information, semantic content, significant keywords, and position. The Ranksum obtains the sentence saliency rankings corresponding to each feature in an unsupervised way followed by the weighted fusion of the four scores to rank the sentences according to their significance. The scores are generated in completely unsupervised way, and a labeled document set is required to learn the fusion weights. Since we found that the fusion weights can generalize to other datasets, we consider the Ranksum as an unsupervised approach. To determine topic rank, we employ probabilistic topic models whereas semantic information is captured using sentence embeddings. To derive rankings using sentence embeddings, we utilize Siamese networks to produce abstractive sentence representation and then we formulate a novel strategy to arrange them in their order of importance. A graph-based strategy is applied to find the significant keywords and related sentence rankings in the document. We also formulate a sentence novelty measure based on bigrams, trigrams, and sentence embeddings to eliminate redundant sentences from the summary. The ranks of all the sentences – computed for each feature – are finally fused to get the final score for each sentence in the document. We evaluate our approach on publicly available summarization datasets — CNN/DailyMail and DUC 2002. Experimental results show that our approach outperforms other existing state-of-the-art summarization methods. [Display omitted] • A unified summarization framework with multi-dimensional sentence features. • Novel method to rank sentences using probabilistic topic vectors. • Sentences ranked with a new technique exploiting embeddings. • Novelty parameter utilizing bigrams, trigrams and sentence embeddings. • Rank-based fusion strategy for extractive summarization. [ABSTRACT FROM AUTHOR]

Automatic text summarization is a topic of great interest in many fields of knowledge. Particularly, query-oriented extractive multi-document text summarization methods have increased their importance recently, since they can automatically generate a summary according to a query given by the user. One way to address this problem is by multi-objective optimization approaches. In this paper, a memetic algorithm, specifically a Multi-Objective Shuffled Frog-Leaping Algorithm (MOSFLA) has been developed, implemented, and applied to solve the query-oriented extractive multi-document text summarization problem. Experiments have been conducted with datasets from Text Analysis Conference (TAC), and the obtained results have been evaluated with Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metrics. The results have shown that the proposed approach has achieved important improvements with respect to the works of scientific literature. Specifically, 25.41%, 7.13%, and 30.22% of percentage improvements in ROUGE-1, ROUGE-2, and ROUGE-SU4 scores have been respectively reached. In addition, MOSFLA has been applied to medicine texts from the Topically Diverse Query Focus Summarization (TD-QFS) dataset as a case study. • The query-oriented text summarization problem is addressed. • The criteria of query relevance and redundancy reduction are optimized simultaneously. • A multi-objective memetic algorithm (MOSFLA) is designed and developed. • The experiments use the Text Analysis Conference datasets and the ROUGE metrics. • MOSFLA outperforms the results of other 12 approaches from the scientific literature. [ABSTRACT FROM AUTHOR]