Semantic Multimedia

In this paper, we describe a knowledge management framework that addresses the needs of multimedia analysis projects and provides a basis for information retrieval systems. The framework uses Semantic Web technologies to provide a shared knowledge environment, and active Knowledge Machines, wrapping multimedia processing tools, to exploit and/or export knowledge to this environment. This framework is able to handle a wide range of use cases, from an enhanced workspace for researchers to end-user information access. As an illustration of how the proposed framework can be used, we present a case study of music analysis.

During the last decades, digital media have revolutionised media reproduction. The availability of cheap consumer electronic devices that allow the consumption and management of digital multimedia content (eg MP3 players, digital cameras, DV camcorders, smart phones) has caused a media availability explosion. The amount of digital media that has been generated and stored, and which continues to be at an exponential rate, has already become unmanageable without fine-grained computerised support.

In parallel with the tremendously increasing number of video contents on the Web, many technical specifications and standards have been introduced to store technical details and describe the content of, and add subtitles to, online videos. Some of these specifications are based on unstructured data with limited machine-processability, data reuse, and interoperability, while others are XML-based, representing semi-structured data. While low-level video features can be derived automatically, high-level features are mainly related to a particular knowledge domain and heavily rely on human experience, judgment, and background. One of the approaches to solve this problem is to map standard, often semi-structured, vocabularies, such as that of MPEG-7, to machine-interpretable ontology. Another approach is to introduce new multimedia ontologies. While video contents can be annotated efficiently with terms defined by structured LOD datasets, such as DBpedia, ontology standardization would be desired in the video production and distribution domains. This paper compares the state-of-the-art video annotations in terms of descriptor level and machine-readability, highlights the limitations of the different approaches, and makes suggestions towards standard video annotations.

Multimedia reasoning, which is suitable for, among others, multimedia content analysis and high-level video scene interpretation, relies on the formal and comprehensive conceptualization of the represented knowledge domain. However, most multimedia ontologies are not exhaustive in terms of role definitions, and do not incorporate complex role inclusions and role interdependencies. In fact, most multimedia ontologies do not have a role box at all, and implement only a basic subset of the available logical constructors. Consequently, their application in multimedia reasoning is limited. To address the above issues, VidOnt, the very first multimedia ontology with SROIQ(D) expressivity and a DL-safe ruleset has been introduced for next-generation multimedia reasoning. In contrast to the common practice, the formal grounding has been set in one of the most expressive description logics, and the ontology validated with industry-leading reasoners, namely HermiT and FaCT++. This paper also presents best practices for developing multimedia ontologies, based on my ontology engineering approach.

Recent developments of the Internet and the World Wide Web (WWW) have resulted in the proliferation of multi-media resources and learning with existing multimedia web resources on the Web is becoming more prevalent and important. While recent standardization efforts in eLearning such as LOM, SCORM, and IMS Learning Design work towards learning content description, packaging, and delivery; existing eLearning solutions still lack the ability to adequately use multi-media resources to provide a learner with personalised learning resources. Effective use of multimedia for web-based learning provides quality interactive learning experience, but current techniques does not adequately provide a semantic approach for organising multimedia resources. With evolving trend in learning through the use of web technology, eLearning systems are expected to provide personalised learning resources for effective learning. We have accordingly proposed a way of organising existing multimedia resources based on contexts towards providing a truly personalised eLearning experience. This paper presents a framework for achieving contextbased multimedia semantic annotation and organisation towards personalised learning.

In this paper we describe the main aspects of a system to perform non-trivial segmentations of 3D surface meshes and to annotate the detected parts through concepts expressed by an ontology. Each part is connected to an instance in a knowledge base to ease the retrieval process in a semantics-based context. Through an intuitive interface, users create such instances by simply selecting proper classes in the ontology; attributes and relations with other instances can be computed automatically based on a customizable analysis of the underlying topology and geometry of the parts.

In this paper we describe the main aspects of a system to perform non-trivial segmentations of 3D surface meshes and to annotate the detected parts through concepts expressed by an ontology. Each part is connected to an instance in a knowledge base to ease the retrieval process in a semantics-based context. Through an intuitive interface, users create such instances by simply selecting proper classes in the ontology; attributes and relations with other instances can be computed automatically based on a customizable analysis of the underlying topology and geometry of the parts.

In this paper we present a framework for simultaneous image segmentation and region labeling leading to automatic image annotation. The proposed framework operates at semantic level using possible semantic labels to make decisions on handling image regions instead of visual features used traditionally. In order to stress its independence of a specific image segmentation approach we applied our idea on two region growing algorithms, i.e. watershed and recursive shortest spanning tree. Additionally we exploit the notion of visual context by employing fuzzy algebra and ontological taxonomic knowledge representation, incorporating in this way global information and improving region interpretation. In this process, semantic region growing labeling results are being re-adjusted appropriately, utilizing contextual knowledge in the form of domain-specific semantic concepts and relations. The performance of the overall methodology is demonstrated on a real-life still image dataset from the popular domains of beach holidays and motorsports.

Audiovisual archives hold enormous amounts of valuable content. However, many of these archives are difficult to access, as their holdings are documented using a range of different metadata formats. Being able to exchange metadata is the key to ensuring access to these collections and to establish interoperability among audiovisual collections, between audiovisual collections and other cultural heritage institutions as well as portals such as Europeana. In this work we attempt to model mappings between metadata formats based on a high-level intermediate concept representation in order to avoid hand-crafted one-to-one mappings between metadata formats. In addition, we define mapping templates on data type level, from which the code for mapping instructions between a pair of formats can be derived. The high-level intermediate concept representation is based on the existing meon ontology and the resulting mapping instructions are expressed using XSLT. As a proof of concept, mappings between two metadata formats are formalized and integrated in a web based prototype application.

Medical images coming from different sources can often provide different information. So, combining two or more co-registered multimodal medical images into a single image (image fusion) is an important support to the medical diagnosis. Most of the used image fusion techniques are based on the Multiresolution Analysis (MRA), which is able to decompose an image into several components at different scales. This paper presents a novel Wavelet-based method to fuse medical images according to the MRA approach, that aims to put the right "semantic" content in the fused image by applying two different quality indexes: variance and modulus maxima. Experimental tests show very encouraging results in terms of both quantitative and qualitative evaluations.

This paper presents an approach to semi-automate photo annotation. Instead of using content-recognition techniques this approach leverages context information available at the scene of the photo such as time and location in combination with existing photo annotations to provide suggestions to the user. An algorithm exploits a number of technologies including Global Positioning System (GPS), Semantic Web, Web services and Online Social Networks, considering all information and making a best-effort attempt to suggest both people and places depicted in the photo. The user then selects which of the suggestions are correct to annotate the photo. This process accelerates the photo annotation process dramatically which in turn aids photo search for a wide range of query tools that currently trawl the millions of photos on the Web.