1. ASHuR: Evaluation of the Relation Summary-Content Without Human Reference Using ROUGE
- Author
-
Samantha Jiménez, Sergio Inzunza, Alan Ramírez-Noriega, Reyes Juárez-Ramírez, and Yobani Martínez-Ramírez
- Subjects
other areas of Computing and Informatics ,Relation (database) ,Sentence length ,business.industry ,Computer science ,General Engineering ,Construct (python library) ,computer.software_genre ,Automatic summarization ,Pearson product-moment correlation coefficient ,Test (assessment) ,symbols.namesake ,Text summarization, summary evaluation, ROUGE, sentences extraction ,Content (measure theory) ,symbols ,Artificial intelligence ,business ,68-U15, 68-T50 ,computer ,Natural language processing ,Reliability (statistics) - Abstract
In written documents, the summary is a brief description of important aspects of a text. The degree of similarity between the summary and the content of a document provides reliability about the summary. Some efforts have been done in order to automate the evaluation of a summary. ROUGE metrics can automatically evaluate a summary, but it needs a model summary built by humans. The goal of this study is to find a quantitative relation between an article content and its summary using ROUGE tests without a model summary built by humans. This work proposes a method for automatic text summarization to evaluate a summary (ASHuR) based on extraction of sentences. ASHuR extracts the best sentences of an article based on the frequency of concepts, cue-words, title words, and sentence length. Extracted sentences constitute the essence of the article; these sentences construct the model summary. We performed two experiments to assess the reliability of ASHuR. The first experiment compared ASHuR against similar approaches based on sentences extraction; the experiment placed ASHuR in the first place in each applied test. The second experiment compared ASHuR against human-made summaries, which yielded a Pearson correlation value of 0.86. Assessments made to ASHuR show reliability to evaluate summaries written by users in collaborative sites (e.g. Wikipedia) or to review texts generated by students in online learning systems (e.g. Moodle).
- Published
- 2018
- Full Text
- View/download PDF