Search

Textual understanding is the process of automatically extracting accurate high-quality information from text. The amount of textual data available from different sources such as news, blogs and social media is growing exponentially. These data encode significant latent information which if extracted accurately can be valuable in a variety of applications such as medical report analyses, news understanding and societal studies. Natural language processing techniques are often employed to develop customized algorithms to extract such latent information from text. Journalistic 5Ws refer to the basic information in news articles that describes an event and include where, when, who, what and why. Extracting them accurately may facilitate better understanding of many social processes including social unrest, human rights violations, propaganda spread, and population migration. Furthermore, the 5Ws information can be combined with socio-economic and demographic data to analyze state and trajectory of these processes. In this thesis, a data driven pipeline has been developed to extract the 5Ws from text using syntactic and semantic cues in the text. First, a classifier is developed to identify articles specifically related to social unrest. The classifier has been trained with a dataset of over 80K news articles. We then use NLP algorithms to generate a set of candidates for the 5Ws. Then, a series of algorithms to extract the 5Ws are developed. These algorithms based on heuristics leverage specific words and parts-of-speech customized for individual Ws to compute their scores. The heuristics are based on the syntactic structure of the document as well as syntactic and semantic representations of individual words and sentences. These scores are then combined and ranked to obtain the best answers to Journalistic 5Ws. The classification accuracy of the algorithms is validated using a manually annotated dataset of news articles.

Dynamic word embeddings refer to dividing data into time slices and obtaining the word vector representations of each time slice. By applying dynamic word embeddings, we can analyze how word trends evolve over time. Recently, a number of efforts have been applying dynamic word embeddings to understand how the meanings of words change over years or decades. However, little or no efforts have focused on the monthly changes, which are important for news analysis, such as how a word trend changes monthly. This thesis introduces how to apply dynamic word embeddings for news analysis and presents meaningful word trends evaluations. The application can be extended for news analysis of any time intervals, given enough data within each time slice.

The analysis of news content has been a central focus for media scholars, political scientists, sociologists and historians. Traditionally, it has been performed using relatively small news archives collected over limited periods. However, in the last 20 years, following the creation of the World-Wide Web, a dramatic change has occurred in the reporting and dissemination of news. One result of this change is that an ever growing archive of news is readily available as electronic text. This, in turn, is making it possible to analyze news on a large scale using methods developed in the fields of Web Intelligence, Data Mining and Machine Learning. The issues that news content analysis tries to address include: Identification of salient topics; summarization of stories; extraction of opinions; and characterization of news reports in terms of content, sentiment, bias, etc. These are also the motivating issues for this research.The research in this dissertation describes a framework called IONA: Intelligent Online News Analysis. This is meant to be a tool to accomplish four goals: 1) Extracting and visualizing important stories from real-time news streams; 2) Characterizing and comparing the cognitive/epistemic organization of all news in different media sources over the same time period; 3) Comparing the structure of specific stories from different media sources to characterize similarities, differences, and possible biases; and 4) Doing comparative analysis of how specific stories and the news streams from different media sources evolve over time in order to characterize the dynamics of news from each source.The IONA approach represents an innovative combination of methods from natural language processing, semantic analysis and complex networks. The identification of topics uses a novel algorithm that integrates Latent Dirichlet Allocation (\LDA) with tagging using Ngrams. The resulting topics are used to extract coherent sets of news reports from large corpora of very short news stories, each of which is then used to construct a summary in the form of a weighted semantic network. In addition to elucidating the global semantic structure of the story, this network also enables the identification of its core components and meaningful sub-units (motifs), which can then be used in the characterization of style, bias, sentiment, etc. The complexity of stories is characterized by both a graphical analysis of the extracted semantic networks and by a graph-based statistical analysis of the identified story sets. This analysis builds on deep theories of semantic cognition and creativity.IONA differs from other methods in current use in several ways: 1) It uses very large corpora with a large number of very short news stories, which presents unique challenges and opportunities for analysis; 2) It does not rely on any prior tagging of news stories, though such tagging could be used to improve performance further; 3) It relies on unsupervised learning rather than supervised or semi-supervised learning, which makes it more broadly applicable independent of specific domains; 4) It goes beyond the simple bag-of-words approach to include some structural and semantic information in the analysis; and 5) It represents a single integrated platform that ranges from data collection to highly abstract functions such as structural and temporal representation of extracted information.News plays a critical role in shaping opinions and thus driving events that affect billions of people. Deep analysis of news in terms of content, influence and bias are, therefore, not only important for media analysts but also for policy-makers, political and geopolitical thinkers, intelligence operations, corporate entities and, of course, the general public. IONA is expected to make a significant contribution in this regard.