Search

With the progress of the Web, significant amounts of information become available online. The information ranges from different data types such as news articles, tweets, cultural heritage objects to audio-visual archives and across various distribution channels such as traditional or social media. This democratization of information poses several challenges for search engines, information retrieval systems, and natural language processing systems, as they need to (1) extract meaningful information from any data modality (i.e., text, image, video) and (2) synthesize streams of data and information from various channels to provide concise pieces of information that answer the needs and requests of end-users. Events play an important role in understanding and contextualizing information, as well as influencing human interpretation. Nevertheless, by definition, events are complex entities, essential for querying, perceiving and consuming the meaning of the information surrounding us. Therefore, we need to understand what an event is, how to describe an event, and to what extent a document is meaningful or relevant for a given event or topic. Typically, events create context by introducing related properties or entities such as participants involved, locations where the event takes place or the period when the event takes place. The event space is typically represented in different data streams and channels. An event is likely mentioned both in news articles and tweets, as well as in textual and audio-visual media. Hence, besides relevance, we need to extend the event understanding with salience and novelty features to minimize redundancy and subjective semantics such as sentiments and sentiment intensities to account for the multitude of perspectives. The information extraction community acknowledges the importance of events. However, the accuracy of identifying events is still not optimal, as events (1) are vague, (2) carry multiple perspectives, and (3) have different granularity. The mainstream procedure for event annotation is through experts. However, even experts disagree to large extents. Crowdsourcing has emerged as a reliable, time and cost-efficient approach for gathering semantic annotations. However, a major crowdsourcing bottleneck is that most practices are not systematic and sustainable, while state-of-the-art methods exist only for a specific domain or input. Typically, solutions for assessing the quality of crowdsourced data are based on the hypothesis that there is only one right answer, which contradicts with the three angles of events, i.e., vagueness, multiple perspectives and granularities, and with the latent ambiguity in natural language. The recently proposed CrowdTruth methodology, however, addresses these issues. More precisely, experiments performed in the context of the CrowdTruth methodology showed that disagreement between workers and diversity between annotations are signals for identifying low-quality workers and better understanding data ambiguity. In this thesis, we investigate how diversity in crowdsourcing can improve the machine understanding of events and their characteristics. More precisely, we explore how events are perceived and represented across data modalities (e.g., text, image, video) and sources (e.g., news articles, tweets, video broadcasts). We systematically integrate machines and humans, i.e., with a focus on experimental methodologies and replicability, and sustainable way, i.e., with a focus on reusability of data, code and results. The research novelty is two-fold: (1) a context-sensitive approach to study and understand events, i.e., we do not study events in isolation, but we also study their properties (participating actors, locations, relevant information, salient information); (2) a diversity-driven methodology for gathering event and event-related ground truth, generalizable across domains and data modalities. In this thesis, we perform the research in the context of the CrowdTruth methodology and metrics.

Typically crowdsourcing-based approaches to gather annotated data use inter-annotator agreement as a measure of quality. However, in many domains, there is ambiguity in the data, as well as a multitude of perspectives of the information examples. In this paper, we present ongoing work into the CrowdTruth metrics, that capture and interpret inter-annotator disagreement in crowdsourcing. The CrowdTruth metrics model the inter-dependency between the three main components of a crowdsourcing system – worker, input data, and annotation. The goal of the metrics is to capture the degree of ambiguity in each of these three components. The metrics are available online at https://github.com/CrowdTruth/CrowdTruth-core.

Bias is inevitable and inherent in any form of communication. News often appear biased to citizens with different political orientations, and understood differently by news media scholars and the broader public. In this paper we advocate the need for accurate methods for bias identification in video news item, to enable rich analytics capabilities in order to assist humanities media scholars and social political scientists. We propose to analyze biases that are typical in video news (including framing, gender and racial biases) by means of a human-in-the-loop approach that combines text and image analysis with human computation techniques.

In the Web 2.0 environment, institutes and organizations are starting to open up their previously isolated and heterogeneous collections in order to provide visitors with maximal access. Semantic Web technologies act as instrumental in integrating these rich collections of metadata by defining ontologies which accommodate different representation schemata and inconsistent naming conventions over the various vocabularies. Facing the large amount of metadata with complex semantic structures, it is becoming more and more important to support visitors with a proper selection and presentation of information. In this context, the Dutch Science Foundation (NWO) funded the Cultural Heritage Information Personalization (CHIP) project in early 2005, as part of the Continuous Access to Cultural Heritage (CATCH) program in the Netherlands. It is a collaborative project between the Rijksmuseum Amsterdam, the Eindhoven University of Technology and the Telematica Instituut. The problem statement that guides the research of this thesis is as follows: Can we support visitors with personalized access to semantically-enriched collections? To study this question, we chose cultural heritage (museums) as an application domain, and the semantically rich background knowledge about the museum collection provides a basis to our research. On top of it, we deployed user modeling and recommendation technologies in order to provide personalized services for museum visitors. Our main contributions are: (i) we developed an interactive rating dialog of artworks and art concepts for a quick instantiation of the CHIP user model, which is built as a specialization of FOAF and mapped to an existing event model ontology SEM; (ii) we proposed a hybrid recommendation algorithm, combining both explicit and implicit relations from the semantic structure of the collection. On the presentation level, we developed three tools for end-users: Art Recommender, Tour Wizard and Mobile Tour Guide. Following a user-centered design cycle, we performed a series of evaluations with museum visitors to test the effectiveness of recommendations using the rating dialog, different ways to build an optimal user model and the prediction accuracy of the hybrid algorithm. Chapter 1 introduces the research questions, our approaches and the outline of this thesis. Chapter 2 gives an overview of our work at the first stage. It includes (i) the semantic enrichment of the Rijksmuseum collection, which is mapped to three Getty vocabularies (ULAN, AAT, TGN) and the Iconclass thesaurus; (ii) the minimal user model ontology defined as a specialization of FOAF, which only stores user ratings at that time, (iii) the first implementation of the content-based recommendation algorithm in our first tool, the CHIP Art Recommender. Chapter 3 presents two other tools: Tour Wizard and Mobile Tour Guide. Based on the user's ratings, the Web-based Tour Wizard recommends museum tours consisting of recommended artworks that are currently available for museum exhibitions. The Mobile Tour Guide converts recommended tours to mobile devices (e.g. PDA) that can be used in the physical museum space. To connect users' various interactions with these tools, we made a conversion of the online user model stored in RDF into XML format which the mobile guide can parse, and in this way we keep the online and on-site user models dynamically synchronized. Chapter 4 presents the second generation of the Mobile Tour Guide with a real time routing system on different mobile devices (e.g. iPod). Compared with the first generation, it can adapt museum tours based on the user's ratings artworks and concepts, her/his current location in the physical museum and the coordinates of the artworks and rooms in the museum. In addition, we mapped the CHIP user model to an existing event model ontology SEM. Besides ratings, it can store additional user activities, such as following a tour and viewing artworks. Chapter 5 identifies a number of semantic relations within one vocabulary (e.g. a concept has a broader/narrower concept) and across multiple vocabularies (e.g. an artist is associated to an art style). We applied all these relations as well as the basic artwork features in content-based recommendations and compared all of them in terms of usefulness. This investigation also enables us to look at the combined use of artwork features and semantic relations in sequence and derive user navigation patterns. Chapter 6 defines the task of personalized recommendations and decomposes the task into a number of inference steps for ontology-based recommender systems, from a perspective of knowledge engineering. We proposed a hybrid approach combining both explicit and implicit recommendations. The explicit relations include artworks features and semantic relations with preliminary weights which are derived from the evaluation in Chapter 5. The implicit relations are built between art concepts based on instance-based ontology matching. Chapter 7 gives an example of reusing user interaction data generated by one application into another one for providing cross-application recommendations. In this example, user tagging about cultural events, gathered by iCITY, is used to enrich the user model for generating content-based recommendations in the CHIP Art Recommender. To realize full tagging interoperability, we investigated the problems that arise in mapping user tags to domain ontologies, and proposed additional mechanisms, such as the use of SKOS matching operators to deal with the possible mis-alignment of tags and domain-specific ontologies. We summarized to what extent the problem statement and each of the research questions are answered in Chapter 8. We also discussed a number of limitations in our research and looked ahead at what may follow as future work.

In a variety of domains, developers nowadays are struggling with the dilemma of how they can provide a more personal service to their users. A more personal service can for example be facilitated by offering user-adapted search, generating recommendations, personalized content navigation, personalized user interfaces, etc. However, providing such functionality on top of a particular data set, requires to have good knowledge of the relevant domain items (which can for example represent books, songs, TV programs, art pieces, etc.) as well as to have good knowledge of the relevant users (in terms of the user’s behavior, interests, preferences, etc. with respect to those domain items). In this dissertation and more specifically in Chapter 3 and Chapter 4, we describe the re-quirements and a domain-independent approach respectively, to provide context-sensitive and user-adapted access to heterogeneous data sources. This approach consists of three main parts, including: 1) Data Integration, 2) User Modeling and 3) User-Adapted Data Access. Chapter 5 focusses on the integration of information from various heterogeneous data sources. To provide user-adapted access, a good description of the relevant domain items is key. The more descriptive information we have about every item, the more raw material is there to for example compare different items, compare items with user profiles, deduce new information, etc. Unfortunately, in the real world, items often come poorly described. On the other hand however, with the immense growth of available information on the Web, many different data sources (like IMDb, Wikipedia, social networks, etc.) exist and offer free access to their data. By using Semantic Web techniques we describe how we can enrich the descriptive metadata of those domain items by on the one hand integrating and matching information from different external sources providing instance metadata, and on the other hand taking relevant ontological background information into account. Chapter 6 concentrates on the second part in our approach: the creation of an extensive model of the end-user. Such a user model is the user’s digital representation and encompasses all valuable user data we can obtain. Information can be provided explicitly by the user himself (e.g. the user states that he is 45 years old, male, capable of speaking three languages, fond of tennis, etc.) but also implicitly. Implicit feedback includes all the information the user gives away without realizing it, by means of his behavioral patterns (e.g. the user watches the news every day at 8, he always adds books from the same author to his favorites, etc.). However, user feedback (both explicit and implicit) can be hard to interpret since it depends on a wide variety of parameters. Numerous influences like mood, location, time, environment, health, etc., make that people can behave very differently at any given time. Therefore, every statement in the user model is contextualized. In other words, the constrained setting in which a specific user statement was valid, which we call the statement’s context, is saved and is used later to predict the user’s interests accurately in any given situation. Further, since our approach depends on the quality and richness of the user model and new users usually start with an empty profile, we suffer from the so-called cold start problem. To deal with this situation where new users have an empty profile, we provide a number of strategies based on user statistics and stereotypes to alleviate this problem. The third and last part in our approach encompasses the strategies to adapt any user request and provide a personalized set of results, based on both the integrated data structure describing the domain and the user model. Chapter 7 describes a processing pipeline, consisting of three steps, which takes a user request as input and delivers a personal response. The first step involves cleaning and conceptualizing the user’s request with respect to the current domain. Secondly, the updated query is sent to the database to retrieve matching results. However, trying to find not only exactly matching results but also highly related results, the database automatically broadens the result space of the query in a controlled fashion. It does so by reasoning over well-chosen semantic relations including for example transitivity and synonymity. When matching results are retrieved, the last step filters them by following a set of rules. These rules are predefined and can include restrictions based on both ontological as well as user model information. This pipeline is employed both to provide user-adapted search as well as for the generation of personal recommendations. However, systems dealing with potentially large amounts of data and on top of that provide complex functionality like reasoning, user-adapted search, integration of data, recommendations, etc., require extra care in terms of their database setup. Moreover, efficiency in terms of querying speed is vital for any system’s long-term success. Therefore, in Chapter 8, we introduce a number of optimizations to improve the efficiency of the database in terms of size and querying speed. To illustrate our approach, we apply it in the television domain which we introduce in Chapter 2. Together with Stoneroos we developed a cross-platform application called iFanzy, which tries to bring personalized access to television programs to the user via a set-top box interface, a Web site and an iPhone application. All three of these platforms are synchronized and behave as one ubiquitous application supporting the user in putting together the best possible television experience, by finding exactly those TV programs fitting the user best. In Chapter 9, we give an overview of these three platforms in terms of functionality and user interface. Furthermore, we perform an evaluation on the interface of the iFanzy Web portal including experiments like a Cognitive Walkthrough, the Thinking Aloud method and a Heuristic Evaluation. Through the commercial availability of iFanzy we were able to further evaluate our approach, focussing on the recommendation quality and user satisfaction. In Chapter 10, we elucidate our evaluation which features a group of 60 people using the iFanzy Web interface for about two full weeks. From the data of this evaluation we can investigate the influence of both explicit and implicit user feedback on the predictive power of the system and the accuracy of generated recommendations. To further improve the quality of the recommendation strategy, Chapter 10 concludes with an approach to improve the serendipity of the recommender system, which leads to more surprising or serendipitous discoveries. Reusing the data from the previous evaluation, we describe a number of measurements to quantify the degree of serendipity in the recommendations of a recommender system.

Earth Observations (EO) collect various characteristics of the objective environment using sensors which often have different measuring, spatial and temporal coverage. Making individual observational data interoperable becomes equally important when viewed in the context of its expensive and time-consuming EO operations. Interoperability will improve reusability of existing observations in both the broader context, and with other observations. As a demonstration of the potential offered by semantic web technology, we have used the National Oceanography Centre Southampton’s Ferrybox project (where suites of environmental sensors installed on commercial ships collect near real time data) to set up an ontology based reference model of a Collaborative Ocean, where relevant oceanographic resources, such as sensors and observations, can be semantically annotated by their stakeholders to produce RDF format metadata to facilitate data/resource interoperability in a distributed environment. We have also demonstrated an infrastructure where common semantic management activities are supported, including ontology management, semantic annotation, storage, and reuse (navigating, inference and query). Once the method and infrastructure are adopted by other related oceanographic projects to describe their resources and move their metadata onto the semantic web, it would be possible to see better interoperability within the Collaborative Ocean initiative to facilitate multiuse of ocean data, as well as making more EO data available on the semantic web.

Ontology evolution is increasingly gaining momentum in the area of Semantic Web research. Current approaches target the evolution in terms of either content, or change management, without covering both aspects in the same framework. Moreover, they are slowed down as they heavily rely on user input. We tackle the aforementioned issues by proposing Evolva, a comprehensive ontology evolution framework, which handles a complete ontology evolution cycle, and makes use of background knowledge for decreasing user input.

Recent years have seen the utilisation of Semantic Web Service descriptions for automating a wide range of service-related activities, with a primary focus on service discovery, composition, execution and mediation. An important area which so far has received less attention is service validation, whereby advertised services are proven to conform to required behavioural specifications. This paper proposes a method for validation of service-oriented systems through automated functional testing. The method leverages ontology-based and rule-based descriptions of service inputs, outputs, preconditions and effects (IOPE) for constructing a stateful EFSM specification. The specification is subsequently utilised for functional testing and validation using the proven Stream X-machine (SXM) testing methodology. Complete functional test sets are generated automatically at an abstract level and are then applied to concrete Web services, using test drivers created from the Web service descriptions. The testing method comes with completeness guarantees and provides a strong method for validating the behaviour of Web services.