Search

Nowadays, different new learning methodologies have been proposed to achieve effective learning in University education. One of the most promising methodologies for teaching computer science is multimedia-based education. In order to empower the performance within the online learning platforms, such as Moodle or OLE, this paper proposes to integrate the multimedia-based education to learning systems, and conducts an experiment with the operating system course. We show that exploiting multimedia, such as educational video and smart text, can significantly improve the student's learning performance in terms of exam grade and knowledge transfer. Further, the paper presents a real-world case study depicting how to enhance the performance of learning platform with multimedia semantics.

Video annotation tools are often compared in the literature, however, most reviews mix unstructured, semi-structured, and the very few structured annotation software. This paper is a comprehensive review of video annotations tools generating structured data output for video clips, regions of interest, frames, and media fragments, with a focus on Linked Data support.The tools are compared in terms of supported input and output data formats, expressivity,annotation specificity, spatial and temporal fragmentation, the concept mapping sources used for Linked Open Data (LOD) interlinking, provenance data support, and standards alignment.Practicality and usability aspects of the user interface of these tools are highlighted. Moreover,this review distinguishes extensively researched yet discontinued semantic video annotation software from promising state-of-the-art tools that show new directions in this increasingly important field. Refereed/Peer-reviewed

This paper reviews various efforts to define and capture the semantics of multimedia data. These efforts are particularly relevant to the problem of storing, managing and querying the semantic content of such data. Since there is not yet an accepted solution to the problem of how to represent, organize and manage multimedia data and the related semantics by means of a formal framework, this paper aims at providing some major research trends in this area. The focus is on ontologies, which allow the exchange of semantics of multimedia content between distributed information systems. This paper aims at reporting on recent trends in the development of multimedia ontologies.

The complex nature of semantic video and audio content has made multimedia difficult to use within the traditional architecture of a knowledge-based intelligent tutoring system (ITS). This paper investigates how multimedia syntax and semantics may be integrated into the confines of the knowledge-based ITS architecture so as to develop a knowledge-based intelligent multimedia tutoring system (IMTS). The paper contributes a semantic content-based model that can be used for this purpose. Multimedia frames (m-frames) are used as the framework by which the syntactic and semantic information is represented within the model. The paper discusses how IMTSs may be developed using the model, and then demonstrates the use of our approach in the development of ARISTOTLE, an IMTS for teaching young children basic zoology.

The proliferation of multimedia narratives has contributed to what is known as the "crisis of choice", which demands a much more active participation on the part of the user to consume multimedia content. To address this issue, a strategy is to offer users efficient search mechanisms, sometimes based on ontologies. However, one may argue that such mechanisms are often based on abstractions that do not adequately capture the essential aspects of multimedia narratives. This paper proposes a conceptual model to specify multimedia narratives that overcomes this limitation. The model is based on the notion of event and is therefore called Nested Event Model (NEMo). The paper also includes a complete example to illustrate the use of the model.

The automatic inference of video semantics is an important but highly challenging problem whose solution can greatly contribute towards the annotation, retrieval, personalisation and reusability of video on the web. From a semantic annotation and retrieval perspective, this thesis investigates the influence of multiple video contexts for inferring video semantics, specifically aiming to improve video tagging and content description. The objective of the thesis is two- fold 1) formalising the representation of a video and its content via an ontological model and 2) inferring concepts to augment the model. First, a lightweight conceptual model of a video is proposed to describe a video object at four different structural abstractions (video, shot, frame, image region) and four different meta information categories (media, content feature, content semantics and context), and second, we investigate an ensemble of methods to infer video semantics from multiple contextual sources in order to augment the above model. The study showed that contextual sources positively contribute in understanding the ¿aboutness¿ of a video and one can discover many descriptive concepts not originally described by the creator. Experimenting with different contextual sources showed that contexts contributed semantic enrichment is not restricted to document level video annotations, but can go further and be used to localise the detected entities inside the video timeline for a fine grained time-stamped annotation. In all studies we found that a combination of cues results in robust concept detection compared to the cues in isolation. We evaluated our approaches using both quantitative analysis and user based qualitative feedback. The principal benefits of the context based approach over a content based approach is that it is computationally inexpensive, maximises the wisdom of crowds and easily adaptable across domains. Finally we built an integrated 'Annotate, Search and Browse' prototype building over the proposed framework that supports complex structured queries, ontology based concept querying, temporal segment querying as well as the normal keyword search.

Effective sharing of diverse social media is often inhibited by limitations in their search and discovery mechanisms, which are particularly restrictive for media that do not lend themselves to automatic processing or indexing. Here, we present the structure and mechanism of an adaptive search engine which is designed to overcome such limitations. The basic framework of the adaptive search engine is to capture human judgment in the course of normal usage from user queries in order to develop semantic indexes which link search terms to media objects semantics. This approach is particularly effective for the retrieval of multimedia objects, such as images, sounds, and videos, where a direct analysis of the object features does not allow them to be linked to search terms, for example, nontextual/icon-based search, deep semantic search, or when search terms are unknown at the time the media repository is built. An adaptive search architecture is presented to enable the index to evolve with respect to user feedback, while a randomized query-processing technique guarantees avoiding local minima and allows the meaningful indexing of new media objects and new terms. The present adaptive search engine allows for the efficient community creation and updating of social media indexes, which is able to instill and propagate deep knowledge into social media concerning the advanced search and usage of media resources. Experiments with various relevance distribution settings have shown efficient convergence of such indexes, which enable intelligent search and sharing of social media resources that are otherwise hard to discover.

Mobile Augmented Reality (MAR) enables overlays of semantically-enriched multimedia on video streams of smart phone cameras. These new ways to interact with digital objects via position and 3D movement can be helpful for on-site professional communities. However, current MAR applications lack real-time collaborative features. Moreover, blending multimedia semantics in collaborative MAR is still challenging. In this paper, we present a mobile real-time semantic multimedia-based collaborative system with augmented reality features based on open standards like XMPP and MPEG-7. The prototype system is evaluated in the digital documentation of historical sites for cultural heritage management. The evaluation results indicate increased productivity and awareness within on-site professional communities.

In this paper, we present one approach for extending the learning set of a classification algorithm with additional metadata. It is used as a base for giving appropriate names to found regularities. The analysis of correspondence between connections established in the attribute space and existing links between concepts can be used as a test for creation of an adequate model of the observed world. Meta-PGN classifier is suggested as a possible tool for establishing these connections. Applying this approach in the field of content-based image retrieval of art paintings provides a tool for extracting specific feature combinations, which represent different sides of artists' styles, periods and movements.

We present a method to automatically detect and identify events from social media sharing web sites. Our approach is based on the observation that many photos and videos are taken and shared when events occur. We select 9 venues across the globe that demonstrate a significant activity according to the EventMedia dataset and we thoroughly evaluate our approach against an official ground truth obtained directly by scraping the event venues' web sites. The results show our ability to not only detect events with high accuracy but also mine and identify events that have not been published in popular event directories such as Last.fm, Eventful or Upcoming. In addition to the textual identification of events, we show how we can build visual summaries of past events providing viewers with a more compelling feeling of the event's atmosphere.

This chapter presents the ISO/IEC MPEG-7 Multimedia Content description Interface Standard from the point of view of managing semantics in the context of multimedia applications. We describe the organisation and structure of the MPEG-7 Multimedia Description schemes which are metadata structures for describing and annotating multimedia content at several levels of granularity and abstraction. As we look at MPEG-7 semantic descriptions, we realise they provide a rich framework for static descriptions of content semantics. As content semantics evolves with interaction, the human user will have to compensate for the absence of detailed semantics that cannot be specified in advance. We explore the practical aspects of using these descriptions in the context of different applications and present some pros and cons from the point of view of managing multimedia semantics.

This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in Proceedings of the 19th ACM international conference on Multimedia, http://dx.doi.org/10.1145/2072298.2072401, This short overview describes the contents of the tutorial Semantic computing in multimedia, which was offered to the participants of ACM Multimedia 2011. Given the impossibility of summarizing properly the contents of the tutorial in just two pages, the purpose of this overview is mainly to introduce the reader to the relevant bibliography.

Even though current news websites use large amount of multimedia materials including image, video and audio, the multimedia materials are used as supplementary to the traditional text-based framework. As users always prefer multimedia, the traditional text-based news exploration interface receives more and more criticisms from both journalists and general audiences. To resolve this problem, we propose a novel framework for multimedia news exploration and analysis. The proposed framework adopts our semantic entity-relationship model to model the multimedia semantics. The proposed semantic entity-relationship model has three nice properties. First, it is able to model multimedia semantics with visual, audio and text properties in a uniform framework. Second, it can be extracted via existing semantic analysis and machine learning algorithms. Third, it is easy to implement sophisticated information mining and visualization algorithms based on the model. Based on this model, we implemented a novel multimedia news exploration and analysis system by integrating visual analytics and information mining techniques. Our system not only provides higher efficiency on news exploration and retrieval but also reveals extra interesting information that is not available on traditional news exploration systems.

With photo albums we aim to capture personal events such as weddings, vacations, and parties of family and friends. By arranging photo prints, captions and paper souvenirs such as tickets over the pages of a photo book we tell a story to capture and share our memories. The photo memories captured in such a photo book tell us a lot about the content and the relevance of the photos for the user. The way in which we select photos and arrange them in the photo book reveal a lot about the events, persons and places on the photos: captions describe content, closeness and arrangement of photos express relations between photos and their content. These semantics of our personal stories, however, are canned in the photo book but would form a very valuable asset for understanding and annotating our personal media collection. In this paper, we present the results of the analysis of a large repository of digitally mastered photo books to learn about the usage of digital photos. From our analysis of the books' content we derive additional semantics for photos, groups of photos and entire photo albums. With this semantic knowledge not only additional metadata can be provided for the organization of the personal photo collection. Also, we can characterize photo books and support users to create even more handsome photo books from large media collections.

The article presents a theoretical approach to semantics that embraces the complex and challenging problems associated with authoring multimedia albums.

Multimedia content exhibits multiple semantics that is influenced by different factors like time, contextual use, and personal background. With the semantics ecosystem, we find an elegant and high-level description of the different factors that influence the semantics of multimedia content. On the other hand, semantics derivation rules are a concrete means to extract and to derive semantics of multimedia content while authoring it. These rules are directly applicable in concrete applications and domains. Thus, there is a gap between the high-level ecosystem and the concrete semantics derivation rules. In this position paper, we propose the use of an ontology-based description of events to combine the high-level description of the semantics ecosystem with the concrete method of semantics derivation for page-based multimedia presentations.

The Second International Workshop on the Many Faces of Multimedia Semantics (MS'08) is the second such workshop held in conjunction with the ACM International Multimedia Conference. This workshop provides a forum for practitioners, artists, and scientists to exchange ideas on how to develop adaptive media retrieval techniques which satisfy individual user needs.

Content analysis technologies give more and more emphasis on multimedia semantics. However most watermarking systems are frame-oriented and do not focus on the protection of semantic regions. As a result, they fail to protect semantic content especially in case of the copy-paste attack. In this framework, a novel unsupervised semantic region watermark encoding scheme is proposed. The proposed scheme is applied to human objects, localized by a face and body detection method that is based on an adaptive two-dimensional Gaussian model of skin color distribution. Next, an invariant method is designed, based on Hu moments, for properly encoding the watermark information into each semantic region. Finally, experiments are carried out, illustrating the advantages of the proposed scheme, such as robustness to RST and copy-paste attacks, and low overhead transmission.

The digital media and games industry is one of the biggest IT based indus-tries worldwide. Recent observations therein showed that current production workflows may be potentially improved as multimedia objects are mostly created from scratch due to insu±cient reusability capacities of existing tools. In this paper we provide reasons for that, provide a potential solution based on semantic technologies, show the potential of ontologies, and provide scenarios for the application of semantic technologies in the digital media and games industry.

This book gives an overview on fundamental issues within the field of multimedia metadata focusing on contextualized, ubiquitous, accessible and interoperable services on a higher semantic level. The book provides a selection of basic articles being a base for multimedia metadata research. Furthermore, it brings together experts from research and industry to present a view on the current state-of-the-art in recent research in Multimedia Semantics and the role of Metadata.

Multimedia does not exhibit a unique semantics but multiple semantics that are influenced by many factors. Current approaches and systems lack from considering this problemin its entirety. What is needed is a holistic approach that describes and embraces the complex and challenging problem of multimedia semantics. Consequently, we are developing with the Semantics Ecosystem (SES) an approach that aimsat providing a "framework" for solving this problem. The SES defines five types of semantics and their relationships to each other and to the real world. In the paper, we present the SES and show its application to the domain of authoring multimedia albums. We show the benefits of having such a theoretical framework to handle the semantics of multimedia albums. The SES allows us to better understand, describe, and communicate the many different factors that are influencing multimedia semantics.

A wide variety of systems and architectures have been developed to manage multimedia metadata for a range of purposes. Each of these systems employs different underlying semantic and syntactic models to record, analyze and exchange data. Collaboration between members of the MUSCLE network, an EU project, aims to examine the difficulties in extracting and integrating multi-dimensional metadata from different sources and modes and to achieve a better understanding of the semantic models used within our different projects. This article describes the similarities and differences between four independent projects and discusses three levels of integration (syntactic, feature-level semantics and domain semantics). An example scenario is presented to illustrate potential applications of the integrated systems.

Learning semantics from low-level features of multimedia learning resources enables high-level access to multimedia content. Considerable amount of researches have been focused on multi-modal analysis to detect multimedia semantics. However, two fundamental issues have not been adequately addressed. First, given a set of raw features extracted from multimedia sources, what are the best independent modalities? Second, once a set of modalities has been identified, how are they optimally fused to map to the high-level semantics? In this paper, we apply statistical and machine learning techniques to answer the two questions. ISOMAP combining with support vector clustering are used to discover independent modalities from raw features. Then Maximum Entropy method is applied to optimally fuse the individual modalities. Experiments show that the proposed method can learn multimedia semantics more efficiently than traditional methods.

In the study of the language, the semantics is the discipline that takes care of the meaning of the expressions of the language. Next to it, the syntax studies the rules for the formation of the correct expressions, and the pragmatics studies the use of the expressions of the language for the communication. For a long time, it was thought that the meaning of linguistic expressions was given by abstract entities, such as ideas or concepts.

Multimedia plays an important role in the web-based learning environment. In this paper, we validate management and retrieval of large multimedia collections through high-level semantics. A novel algorithm is proposed for automatic annotation of image based on support vector machine and statistical learning. In addition, we construct cross-media indexing for multi-modal data upon the annotation result to support cross-media search. Experiments show that our algorithm can interpret multimedia semantics accurately and cross-media indexing can support cross-media search effectively.

In this paper, we present the tools standardized by MPEG-7 for describing the semantics of multimedia. In particular, we focus on the abstraction model, entities, attributes and relations of MPEG-7 semantic descriptions. MPEG-7 tools can describe the semantics of specific instances of multimedia such as one image or one video segment but can also generalize these descriptions either to multiple instances of multimedia or to a set of semantic descriptions. The key components of MPEG-7 semantic descriptions are semantic entities such as objects and events, attributes of these entities such as labels and properties, and, finally, relations of these entities such as an object being the patient of an event. The descriptive power and usability of these tools has been demonstrated in numerous experiments and applications, these make them key candidates to enable intelligent applications that deal with multimedia at human levels.

Multimedia semantics is the research area that deals with interpreting the meaning of a multimedia content and annotate the data in such a way that it will be useful for others to search, reuse, and repurpose the content (assuming the appropriate copyrights are available to do them!). The major hurdle in achieving this objective is well known: the binary nature of multimedia content does not lend itself to precise interpretations easily. Several machine learning and other artificial intelligence techniques have been developed for analyzing the binary data. However, these approaches have two major deficiencies in general

Although important advances in multimedia mining have been achieved, many old problems remain and are ever major research areas. These research areas include multimedia semantics and user interacti...

The aim of this chapter is to sketch a semantic taxonomy of representational forms (or “sign types” in the terminology of semiotics) relevant for the compositional analysis, design, and indexing of multimodal multimedia, specifically, in the domain of instructional design of learning objects, and to contribute to current attempts to bridge the “semantic gap” between the technical and spatio-temporal description of media objects and the contextual perception and interpretation by users of their content.q