Search

The present paper proposes an implementation of a hybrid hardware–software system for the visual servoing of prosthetic arms. We focus on the most critical vision analysis part of the system. The prosthetic system comprises a glass-worn eye tracker and a video camera, and the task is to recognize the object to grasp. The lightweight architecture for gaze-driven object recognition has to be implemented as a wearable device with low power consumption (less than 5.6 W). The algorithmic chain comprises gaze fixations estimation and filtering, generation of candidates, and recognition, with two backbone convolutional neural networks (CNN). The time-consuming parts of the system, such as SIFT (Scale Invariant Feature Transform) detector and the backbone CNN feature extractor, are implemented in FPGA, and a new reduction layer is introduced in the object-recognition CNN to reduce the computational burden. The proposed implementation is compatible with the real-time control of the prosthetic arm.

This paper proposes a fusion method of modalities extracted from video through a three-stream network with spatio-temporal and temporal convolutions for fine-grained action classification in sport. It is applied to TTStroke-21 dataset which consists of untrimmed videos of table tennis games. The goal is to detect and classify table tennis strokes in the videos, the first step of a bigger scheme aiming at giving feedback to the players for improving their performance. The three modalities are raw RGB data, the computed optical flow and the estimated pose of the player. The network consists of three branches with attention blocks. Features are fused at the latest stage of the network using bilinear layers. Compared to previous approaches, the use of three modalities allows faster convergence and better performances on both tasks: classification of strokes with known temporal boundaries and joint segmentation and classification. The pose is also further investigated in order to offer richer feedback to the athletes., Comment: MMSports '21, October 20, 2021, Virtual Event,, Oct 2021, Chengdu, China

International audience; The paper addresses the problem of recognition of actions in video with low inter-class variability such as Table Tennis strokes. Two stream, "twin" convolutional neural networks are used with 3D convolutions both on RGB data and optical flow. Actions are recognized by classification of temporal windows. We introduce 3D attention modules and examine their impact on classification efficiency. In the context of the study of sportsmen performances, a corpus of the particular actions of table tennis strokes is considered. The use of attention blocks in the network speeds up the training step and improves the classification scores up to 5% with our twin model. We visualize the impact on the obtained features and notice correlation between attention and player movements and position. Score comparison of state-of-the-art action classification method and proposed approach with attentional blocks is performed on the corpus. Proposed model with attention blocks outperforms previous model without them and our baseline.

Sports video analysis is a prevalent research topic due to the variety of application areas, ranging from multimedia intelligent devices with user-tailored digests up to analysis of athletes' performance. The Sports Video task is part of the MediaEval 2021 benchmark. This task tackles fine-grained action detection and classification from videos. The focus is on recordings of table tennis games. Running since 2019, the task has offered a classification challenge from untrimmed video recorded in natural conditions with known temporal boundaries for each stroke. This year, the dataset is extended and offers, in addition, a detection challenge from untrimmed videos without annotations. This work aims at creating tools for sports coaches and players in order to analyze sports performance. Movement analysis and player profiling may be built upon such technology to enrich the training experience of athletes and improve their performance., Comment: MediaEval 2021, Dec 2021, Online, Germany

In this chapter we are interested in how from high-resolution images and videos passing them through a Deep convolutional neural network we get reduced dimension which finally allows a classification decision. We are interested in two operations: convolution and pooling and trace analogy with these operations in a classical Image Processing framework.

After studies of fundamental operations of convolution and sub-sampling in previous chapter, we introduce here convolutional neural networks and consider those designed for particular data: images. First of all we will expose some general principles, then go into detail layer-by-layer and finally briefly overview most popular convolutional neural networks architectures.

Laparoscopic skill training and evaluation as well as identifying technical errors in surgical procedures have become important aspects in Surgical Quality Assessment (SQA). Typically performed in a manual, time-consuming and effortful post-surgical process, evaluating technical skills for a large part involves assessing proper instrument handling as the main cause for these type of errors. Therefore, when attempting to improve upon this situation using computer vision approaches, the automatic identification of instruments in laparoscopy videos is the very first step toward a semi-automatic assessment procedure. Within this work we summarize existing methodologies for instrument recognition, while proposing a state-of-the-art instance segmentation approach. As a first experiment in the domain of gynecology, our approach is able to segment instruments well but a much higher precision will be required, since this early step is critical before attempting any kind of skill recognition.

The automatic description of digital multimedia content was mainly developed for classification tasks, retrieval systems and massive ordering of data. Preservation of cultural heritage is a field of high importance of application of these methods. We address classification problem in cultural heritage such as classification of architectural styles in digital photographs of Mexican cultural heritage. In general, the selection of relevant content in the scene for training classification models makes the models more efficient in terms of accuracy and training time. Here we use a saliency-driven approach to predict visual attention in images and use it to train a Deep Convolutional Neural Network. Also, we present an analysis of the behavior of the models trained under the state-of-the-art image cropping and the saliency maps. To train invariant models to rotations, data augmentation of training set is required, which posses problems of filling normalization of crops, we study were different padding techniques and we find an optimal solution. The results are compared with the state-of-the-art in terms of accuracy and training time. Furthermore, we are studying saliency cropping in training and generalization for another classical task such as weak labeling of massive collections of images containing objects of interest. Here the experiments are conducted on a large subset of ImageNet database. This work is an extension of preliminary research in terms of image padding methods and generalization on large scale generic database.

Studies of visual attention of patients with Dementia such as Parkinson’s Disease Dementia and Alzheimer Disease is a promising way for non-invasive diagnostics. Past research showed, that people suffering from dementia are not reactive with regard to degradations on still images. Attempts are being made to study their visual attention relatively to the video content. Here the delays in their reactions on novelty and “unusual” novelty of the visual scene are expected. Nevertheless, large-scale screening of population is possible only if sufficiently robust automatic prediction models can be built. In the medical protocols the detection of Dementia behavior in visual content observation is always performed in comparison with healthy, “normal control” subjects. Hence, it is a research question per see as to develop an automatic prediction models for specific visual content to use in psycho-visual experience involving Patients with Dementia (PwD). The difficulty of such a prediction resides in a very small amount of training data. In this paper the reaction of healthy normal control subjects on degraded areas in videos was studied. Furthermore, in order to build an automatic prediction model for salient areas in intentionally degraded videos for PwD studies, a deep learning architecture was designed. Optimal transfer learning strategy for training the model in case of very small amount of training data was deployed. The comparison with gaze fixation maps and classical visual attention prediction models was performed. Results are interesting regarding the reaction of normal control subjects against degraded areas in videos.

International audience; Detecting and classifying human actions in videos is oneof the current challenges in visual content analysis and mining. This paper presents a method for performing a fine-grained classification of sport actions using a Siamese Spatio-Temporal Convolutional Neural Network (SSTCNN) model.This model takes RGB images and Optical Flow field as in-put data. Our first contribution is the comparison of different Optical flow methods and a study of their influence on the classification score. We also present different normalization methods for the optical flow that drastically impact results,boosting performances from 44% to 74% of accuracy. Our second contribution is the detection and classification of actions in videos performed using a sliding temporal window.It leads to a satisfying score of 81.3% over the whole dataset TTStroke-21.

International audience; Human action recognition in videos is one of the key problems in visual data interpretation. Despite intensive research, the recognition of actions with low inter-class variability remains a challenge. To answer this problem, my thesis focus on fine-grained classification challenge using a Siamese Spatio-Temporal Convolutional Neural Network and apply it to a new dataset we have introduced TTStroke-21. Our model take as input data RGB images and Optical Flow and is able to reach an accuracy of 91.4% against 43.1% for our baseline on temporal segmented videos. Detection and classification in videos using a sliding temporal window leads to a score of 81.3% over the whole dataset.

Prediction of visual saliency in images and video is needed for video understanding, search and retrieval, coding, watermarking and other applications. The majority of prediction models are founded only on “bottom-up” features. Nevertheless, the “top-down” component of human visual attention becomes prevalent as human observers explore the visual scene. Visual saliency which is always a mix of bottom-up and top-down cues can be predicted on the basis of seen data. In this paper, a model of prediction of visual saliency in video on the basis of Deep convolutional neural networks (CNNs) is proposed. A Deep CNN architecture is designed. Various input channels for a CNN architecture are studied: using the known sensitivity of human visual system to residual motion, pixel colour values are completed with residual motion map. The latter is a normalized energy of residual motion in video frames with regard to the estimated global affine motion model. The experiments show that the choice of the input features for the Deep CNN depends on visual task: for highly dynamic content, the proposed model with residual motion is more efficient and gives decent results with relatively shallow Deep architecture.

We tackle the problem of predicting a grasping action in ego-centric video for the assistance to upper limb amputees. Our work is based on paradigms of neuroscience that state that human gaze expresses intention and anticipates actions. In our scenario, human gaze fixations are recorded by a glass-worn eye-tracker and then used to predict the grasping actions. We have studied two aspects of the problem: which object from a given taxonomy will be grasped, and when is the moment to trigger the grasping action. To recognize objects, we using gaze to guide Convolutional Neural Networks (CNN) to focus on an object-to-grasp area. However, the acquired sequence of fixations is noisy due to saccades toward distractors and visual fatigue, and gaze is not always reliably directed toward the object-of-interest. To deal with this challenge, we use video-level annotations indicating the object to be grasped and a weak loss in Deep CNNs. To detect a moment when a person will take an object we take advantage of the predictive power of Long-Short Term Memory networks to analyze gaze and visual dynamics. Results show that our method achieves better performance than other approaches on a real-life dataset. (C) 2018 Elsevier Ltd. All rights reserved. This work was partially supported by French National Center of Scientific research with grant Suvipp PEPS CNRS-Idex 215-2016, by French National Center of Scientific research with Interdisciplinary project CNRS RoBioVis 2017–2019, the Scientific Council of Labri, University of Bordeaux, and the Spanish Ministry of Economy and Competitiveness under the National Grants TEC2014-53390-P and TEC2014-61729-EXP. Publicado

Incorporating Human Visual System (HVS) models into building of classifiers has become an intensively researched field in visual content mining. In the variety of models of HVS we are interested in so-called visual saliency maps. Contrarily to scan-paths they model instantaneous attention assigning the degree of interestingness/saliency for humans to each pixel in the image plane. In various tasks of visual content understanding, these maps proved to be efficient stressing contribution of the areas of interest in image plane to classifiers models. In previous works saliency layers have been introduced in Deep CNNs, showing that they allow reducing training time getting similar accuracy and loss values in optimal models. In case of large image collections efficient building of saliency maps is based on predictive models of visual attention. They are generally bottom-up and are not adapted to specific visual tasks. Unless they are built for specific content, such as "urban images"-targeted saliency maps we also compare in this paper. In present research we propose a "bootstrap" strategy of building visual saliency maps for particular tasks of visual data mining. A small collection of images relevant to the visual understanding problem is annotated with gaze fixations. Then the propagation to a large training dataset is ensured and compared with the classical GBVS model and a recent method of saliency for urban image content. The classification results within Deep CNN framework are promising compared to the purely automatic visual saliency prediction.

International audience; Human action recognition in video is one of the key problems in visual data interpretation. Despite intensive research, the recognition of actions with low inter-class variability remains a challenge. This paper presents a new Siamese Spatio-Temporal Convolutional neural network (SSTC) for this purpose. When applied to table tennis, it is possible to detect and recognize 20 table tennis strokes. The model has been trained on a specific dataset, TTStroke-21, recorded in natural condition (markerless) at the Faculty of Sports of the University of Bordeaux. Our model takes as inputs a RGB image sequence and its computed Optical Flow. After 3 spatio-temporal convolutions, data are fused in a fully connected layer of a proposed siamese network architecture. Our method reaches an accuracy of 91.4% against 43.1% for our baseline.

Recognition of objects of a given category in visual content is one of the key problems in computer vision and multimedia. It is strongly needed in wearable video shooting for a wide range of important applications in society. Supervised learning approaches are proved to be the most efficient in this task. They require available ground truth for training models. It is specifically true for Deep Convolution Networks, but is also hold for other popular models such as SVM on visual signatures. Annotation of ground truth when drawing bounding boxes (BB) is a very tedious task requiring important human resource. The research in prediction of visual attention in images and videos has attained maturity, specifically in what concerns bottom-up visual attention modeling. Hence, instead of annotating the ground truth manually with BB we propose to use automatically predicted salient areas as object locators for annotation. Such a prediction of saliency is not perfect, nevertheless. Hence active contours models on saliency maps are used in order to isolate the most prominent areas covering the objects. The approach is tested in the framework of a well-studied supervised learning model by SVM with psycho-visual weighted Bag-of-Words. An egocentric GTEA dataset was used in the experiment. The difference in mAP (mean average precision) is less than 10 percent while the mean annotation time is 36% lower.

The problem of object recognition in natural scenes has been recently successfully addressed with Deep Convolutional Neuronal Networks giving a significant break-through in recognition scores. The computational efficiency of Deep CNNs as a function of their depth, allows for their use in real-time applications. One of the key issues here is to reduce the number of windows selected from images to be submitted to a Deep CNN. This is usually solved by preliminary segmentation and selection of specific windows, having outstanding “objectiveness” or other value of indicators of possible location of objects. In this paper we propose a Deep CNN approach and the general framework for recognition of objects in a real-time scenario and in an egocentric perspective. Here the window of interest is built on the basis of visual attention map computed over gaze fixations measured by a glass-worn eye-tracker. The application of this set-up is an interactive user-friendly environment for upper-limb amputees. Vision has to help the subject to control his worn neuro-prosthesis in case of a small amount of remaining muscles when the EMG control becomes inefficient. The recognition results on a specifically recorded corpus of 151 videos with simple geometrical objects show the mean Average Precision (mAP) of 64,6% and the computational time at the generalization lower than a time of a visual fixation on the object of interest.

The present chapter is an introduction to the book. The subject we propose has seen an exploded interest since last decade from research community in computer vision and multimedia indexing. From the field of video quality assessment where models of Human Visual System (HVS) were generally used to predict where humans will foveate and how will they perceive the degradation, these methods moved to classical Image and Video Indexing and retrieval tasks, recognition of objects, events, actions in images and video. In this book we try to give the most complete overview of the methods for visual information indexing and retrieval using prediction of visual attention or saliency. But also consider new approaches specifically designed for these tasks.

Deep Neural Networks have become winners in indexing of visual information. They have allowed achievement of better performances in the fundamental tasks of visual information indexing and retrieval such as image classification and object recognition. In fine-grain indexing tasks, namely object recognition in visual scenes, the CNNs classifiers have to evaluate multiple “object proposals”, that is windows in the image plane of different size and location. Hence the problem of recognition is coupled with the problem of localization. In this chapter a model of prediction of Areas-if-Interest in video on the basis of Deep CNNs is proposed. A Deep CNN architecture is designed to classify windows in salient and non-salient. Then dense saliency maps are built upon classification score results. Using the known sensitivity of human visual system (HVS) to residual motion, the usual primary features such as pixel colour values are completed with residual motion features. The experiments show that the choice of the input features for the Deep CNN depends on visual task: for the interest in dynamic content, the proposed model with residual motion is more efficient.

International audience; The main purpose of transfer learning is to resolve the problem of different data distribution, generally, when the training samples of source domain are different from the training samples of the target domain. Prediction of salient areas in natural video suffers from the lack of large video benchmarks with human gaze fixations. Different databases only provide dozens up to one or two hundred of videos. The only public large database is HOLLYWOOD with 1707 videos available with gaze recordings. The main idea of this paper is to transfer the knowledge learned with the deep network on a large dataset to train the network on a small dataset to predict salient areas. The results show an improvement on two small publicly available video datasets.

International audience; Prediction of visual saliency in images and video is a highly researched topic. Target applications include Quality assessment of multimedia services in mobile context, video compression techniques, recognition of objects in video streams, etc. In the framework of mobile and egocentric perspectives, visual saliency models cannot be founded only on bottom-up features, as suggested by feature integration theory. The central bias hypothesis, is not respected neither. In this case, the top-down component of human visual attention becomes prevalent. Visual saliency can be predicted on the basis of seen data. Deep Convolutional Neural Networks (CNN) have proven to be a powerful tool for prediction of salient areas in stills. In our work we also focus on sensitivity of human visual system to residual motion in a video. A Deep CNN architecture is designed, where we incorporate input primary maps as color values of pixels and magnitude of local residual motion. Complementary contrast maps allow for a slight increase of accuracy compared to the use of color and residual motion only. The experiments show that the choice of the input features for the Deep CNN depends on visual task:for th eintersts in dynamic content, the 4K model with residual motion is more efficient, and for object recognition in egocentric video the pure spatial input is more appropriate. © (2016) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.

International audience; As a part of the automatic study of visual attention of affected populations with neurodegenerative diseases and to predict whether new gaze records a complaint of these diseases, we should design an automatic model that predicts salient areas in video. Past research showed, that people suffering form dementia are not reactive with regard to degradations on still images. In this paper we study the reaction of healthy normal control subjects on degraded area in videos. Furthermore, in the goal to build an automatic prediction model for salient areas in intentionally degraded videos, we design a deep learning architecture and measure its performances when predicting salient regions on completely unseen data. The obtained results are interesting regarding the reaction of normal control subjects against a degraded area in video.

International audience; In this paper, we propose a novel method for moving foreground object extraction in sequences taken by a wearable camera, with strong motion. We use camera motion compensated frame differencing, enhanced with a novel kernel-based estimation of the probability density function of background pixels. The probability density functions are used for filtering false foreground pixels on the motion compensated difference frame. The estimation is based on a limited number of measurements; therefore, we introduce a special, spatio-temporal sample point selection and an adaptive thresholding method to deal with this challenge. Foreground objects are built with the DBSCAN algorithm from detected foreground pixels.

International audience; The IRIM group is a consortium of French teams supported by the GDR ISIS and working on Multimedia Indexing and Retrieval. This paper describes its participation to the TRECVID 2015 semantic indexing (SIN). Our approach uses a six-stages processing pipelines for computing scores for the likelihood of a video shot to contain a target concept. These scores are then used for producing a ranked list of images or shots that are the most likely to contain the target concept. The pipeline is composed of the following steps: descriptor extraction, descriptor optimization, classification, fusion of descriptor variants, higher-level fusion, and re-ranking. We evaluated a number of different descriptors and tried different fusion strategies. The best IRIM run has a Mean Inferred Average Precision of 0.2947, which ranked it 4th out of 15 participants.

International audience; In the problem of “human sensing”, videos recorded with wearable cameras give an “egocentric” view of the world, capturing details of human activities. In this paper we continue research on visual saliency for such kind of content with the goal of “active” objects recognition in egocentric videos. In particular, a geometrical cue is considered in case when the central-bias hypothesis does not hold. The proposed visual saliency models are trained based on eye fixations of observers and incorporated into spatio-temporal saliency models. The proposed models have been compared to state of the art visual saliency models using a metric based on target object recognition performances. The results are promising:they highlight the necessity of a non-centered geometric saliency cue.

We propose a new top down probabilistic saliency model for egocentric video content. It aims to predict top-down visual attention maps focused on manipulated objects, that are then used for psycho-visual weighting of features in the problem of manipulated object recognition. The model is probabilistically defined using both global and local appearance features extracted from automatically segmented arm areas and objects. A psycho-visual experiment has been conducted in a guided framework that compares our proposal and other popular state-of-the-art models with respect to human gaze fixations. The obtained results show that our approach outperforms several popular bottom-up saliency approaches in a well-known egocentric dataset. Furthermore, an additional task-driven assessment for object recognition in egocentric video reveals that the proposed method improves the performance of several state-of-the-art techniques for object detection. HighlightsWe propose a new top down probabilistic saliency model for egocentric video content.Top-down saliency maps focused on manipulated objects based on the arms positions.Psycho-visual experiment conducted demonstrating our proposal increased performances over SOA saliency models.SOA performances in manipulated object recognition for this dataset.

International audience; Finding content in large video archives has so far required textual annotation to enable search by keywords. Our aim is to support retrieval from such archives using queries based on the example video clips that contain meaningful human actions. We propose a solution for the scalable search of actions in large-scale archives by leveraging the complementarity between the description at the frame level and the aggregation in time of descriptors. To permit fast search, we introduce a two-level cascade. The inexpensive first level employs aggregation to filter out a large part of the video. At the second level, aided by feature selection, a more discriminative comparison by frame alignment ranks the remaining video sequences. We improve upon the state of the art on popular data sets, and we introduce and show the results on a novel video archive data set that is significantly larger than previous ones.

In this paper we tackle the problem of Instrumental Activities of Daily Living (IADLs) recognition from wearable videos in a Home Clinical scenario. The aim of this research is to provide an accessible and yet detailed video-based navigation interface of patients with dementia/Alzheimer disease to doctors and caregivers. A joint work between a memory clinic and computer vision scientists enabled studying real-case life scenarios of a dyad couple consisting of a caregiver and patient with Alzheimer. As a result of this collaboration, a new @Home, real-life video dataset was recorded, from which a truly relevant taxonomy of activities was extracted. Following a state of the art Activity Recognition framework we further studied and assessed these IADLs in term of recognition performances with different calibration approaches.

In this chapter we study the problem of recognizing Instrumental Activities of Daily Living (IADL) in egocentric camera view. The target application of this research is the indexing of videos of patients with Alzehimer disease, thus providing medical staff with fast access and easy navigation through the video contents and helping them while assessing patients’ abilities to perform IADL. Driven by the consideration that an activity in egocentric videos can be defined as a sequence of interacted objects inside different rooms, we present a novel representation based on the output of object and room detectors over temporal segments. In addition, our object detection approach is extended by automatic detection of visually salient regions since distinguishing active objects from context has been proven to dramatically improve performances in egocentric ADL recognition. We have assessed our proposal on a publicly available egocentric dataset and show extensive experimental results that demonstrate our approach outperforms current state of the art for unconstrained scenarios in which training and testing environments may be notably different.

International audience; In this paper we present a Content-Based Image Retrieval (CBIR) system which extracts color features using Dominant Color Correlogram Descriptor (DCCD) and shape features using Pyramid Histogram of Oriented Gradients (PHOG). The DCCD is a descriptor which extracts global and local color features, whereas the PHOG descriptor extracts spatial information of shape in the image. In order to evaluate the image retrieval effectiveness of the proposed scheme, we used some metrics commonly used in the image retrieval task such as, the Average Retrieval Precision (ARP), the Average Retrieval Rate (ARR) and the Average Normalized Modified Retrieval Rank (ANMRR) and the Average Recall (R)-Average Precision (P) curve. The performance of the proposed algorithm is compared with some other methods which combine more than one visual feature (color, texture, shape). The results show a better performance of the proposed method compared with other methods previously reported in the literature.

International audience; The IRIM group is a consortium of French teams supported by the GDR ISIS and working on Multimedia Indexing and Retrieval. This paper describes its participation to the TRECVID 2014 semantic indexing (SIN) and instance search (INS) tasks. For the semantic indexing task, our approach uses a six-stages processing pipelines for computing scores for the likelihood of a video shot to contain a target concept. These scores are then used for producing a ranked list of images or shots that are the most likely to contain the target concept. The pipeline is composed of the following steps: descriptor extraction, descriptor optimization, classification, fusion of descriptor variants, higher-level fusion, and re-ranking. We evaluated a number of different descriptors and tried different fusion strategies. The best IRIM run has a Mean Inferred Average Precision of 0.2796, which ranked us 5th out of 15 participants.For INS 2014 task IRIM participation, the classical BoW approach was followed, trained only with east-enders dataset. Shot signatures were computed on one key frame, or several key frames (at 1fps) and average pooling. A dissimilarity, computing a distance only for words present in query, was tested. A saliency map, build from object ROI to incorporate background context, was tried. Late fusion of two individual BoWresults, with different detectors/descriptors (Hessian-Affine/SIFT and Harris-Laplace/Opponent SIFT), was used. The four submitted runs were the following:- Run F_D_IRIM_1 was the late fusion of BOW with SIFT, dissimilarity L2p, on several key frames per shot, with context for queries, and BOW with Opponent SIFT, dissimilarity L1p, on one key frame per shot.- Run F_D_IRIM_2 was similar to F_D_IRIM_1 but context for queries used also for second BoW.- Run F_D_IRIM_3 was similar to F_D_IRIM_1 but no context for queries used.- Run F_D_IRIM_4 was similar to F_D_IRIM_2 but using delta1 dissimilarity [46] (from INS 2013 best run).We found that extracting several key frames per shot coupled with average pooling improved results. We confirmed than including context in queries was also beneficial. Surprisingly, our dissimilarity performed better than delta1.

Submission to Reconnaissance de Formes et Intelligence Artificielle (RFIA) 2014, France (2014) - Paper ID 16 / Paper ID114; [Résumé en français] Cet article présente une nouvelle approche appelée bag-of-bags of words (BBoW) pour la recherche d'images par le contenu (CBIR). Il s'agit d'une extension du modèle classique dit sac-de-mots (bag of words - BoW). Dans notre approche, une image est représentée par un graphe placé sur une grille régulière de pixels de l'image. Les poids sur les arêtes dépendent de caractéristiques locales de couleur. Le graphe est découpé en un nombre fixe de régions qui constituent une partition irrégulière de l'image par la méthode Normalized Cuts. Enfin, chaque partition est représentée par sa propre signature suivant le même schéma que le BoW. Une image est donc décrite par un ensemble de signatures qui sont ensuite combinées pour la recherche d'images similaires dans une base de données. Contrairement aux méthodes existantes telle que Spatial Pyramid Matching (SPM), le modèle BBoW proposé ne repose pas sur l'hypothèse que des parties similaires d'une scène apparaissent toujours au même endroit dans des images d'une même catégorie. L'extension de cette méthode à une approche multi-échelle, appelée Irregular Pyramid Matching (IPM), est également décrite. Les résultats démontrent la qualité de notre approche lorsque les partitions obtenues sont stables au sein d'une même catégorie d'images. Une analyse statistique est menée pour définir concrètement la notion de partition stable. Mots Clef : Recherche d'images par le contenu, coupe de graphes, Normalized Cuts, Sac de mots. [Abstract in English] This paper presents a novel approach, named bag-of-bags of words (BBoW), to address the problem of Content-Based Image Retrieval (CBIR) from image databases. The proposed bag-of-bags of words model extends the classical bag-of-words (BoW) model. An image is represented as a connected full graph of local features on a regular grid. Then irregular partitions (subgraphs) of images are further built via Normalized Cuts. Each subgraph in the partition is then represented by its own signature. Compared to existing methods for image retrieval, such as Spatial Pyramid Matching (SPM), the BBoW model does not assume that similar parts of a scene always appear at the same location in images of the same category. The extension of the proposed model to pyramid gives rise to a method we name irregular pyramid matching (IPM). The experiments demonstrate the strength of our method for image retrieval when the partitions are stable across an image category. The statistical analysis of subgraphs is discussed in the paper. Keywords : Content-based image retrieval, graph partitioning, Normalized Cuts, Bag of words.

International audience; In this paper we study the problem of recognizing Instrumental Activities of Daily Living (IADL) in egocentric camera view. The target application of this research is the indexing of videos of patients with Alzehimer disease, thus providing medical staff with fast access and easy navigation through the video contents and helping them while assessing patients' abilities to perform IADL. Driven by the consideration that an activity in egocentric videos can be defined as a sequence of interacted objects inside different rooms, we present a novel representation based on the output of object and room detectors over temporal segments. In addition, our object detection approach is extended by automatic detection of visually salient regions since distinguishing active objects from context has been proven to dramatically improve performances in egocentric ADL recognition. We have assessed our proposal on a publicly available egocentric dataset and show extensive experimental results that demonstrate our approach outperforms current state of the art for unconstrained scenarios in which training and testing environments may be notably different.

18 pages; International audience; Grayscale methods have long been the focus of algorithms for recovering optical flow. Yet optical flow recovery from colour images can be implemented using direct methods, i.e. without using computationally costly iterations or search strategies. In this chapter, a brief introduction to optical flow is presented along with the optical flow constraint equation and proposed extensions to colour images. Methods for solving these extended equations are given for dense optical flows and the results of applying these methods on two synthetic image sequences are presented. The growing need for estimation of large magnitude optical flows and the filtering of singularities require more sophisticated approaches such as sparse optical flow. These sparse methods are described with a sample application in the analysis of High definition video in the compressed domain.

International audience; This paper presents a novel tracking method with a multi-resolution technique and a dual consistency check for model update to track a non-rigid target in an uncalibrated static multi-camera environment. It is based on particle filter methods using color appearance model. Compared to our previous work, the performance of tracking system is improved by proposing: i) a dual consistency check by Kolmo-grov-Smirnov test to evaluate the consistency of target estimate and ii) an interaction of cameras step by weighted least-squares method to compute the adaptive camera transformation matrix which is used to relocate the estimate in one camera by those in other cameras when tracking failure happens. After being tested in our multi-camera environment of one person tracking, a low failure rate in addition to a better tracking precision is achieved compared to mono-camera tracking.

International audience; Real time magnetic resonance (MR) imaging is a promising tool for image-guided interventions. For applications such as thermotherapy on moving organs, a fine image-based compensation of motion is required in real time to allow quantitative analysis, retro-control of the interventional device, or determination of the therapy endpoint. Since interventional procedures are usually restricted to a part of the organ/tissue under study, reduced FOV imaging represents a promising way to improve spatial and / or temporal resolution. However, it introduces new challenges for the target motion estimation since structures near the target may appear transiently due to the respiratory motion and the limited spatial coverage. In this paper, a new image based motion estimation method is proposed combining a global motion estimation with a novel optical flow approach extending the initial Horn & Schunck (H&S) method by an additional regularization term. This term integrates the displacement of physiological landmarks, which are obtained in a preparation step by pattern matching into the variational formulation of the optical flow problem. A smooth regulation of the constraint point influences is achieved using a spatial weighting function. The method was compared to the same registration pipeline employing the H&S approach. A first evaluation was performed on synthetic dataset where the accuracy of the motion estimated with the H&S method was improved by a factor of 2 using the proposed approach. An in vivo study was then realized on both the heart and the kidney of twelve volunteers. Compared to the H&S approach, a significant improvement (p

International audience; The paper contributes to No-Reference video quality assessment of broadcasted HD video over IP networks and DVB. In this work we have enhanced our bottom-up spatio-temporal saliency map model by considering semantics of the visual scene. Thus we propose a new saliency map model based on face detection that we called semantic saliency map. A new fusion method has been proposed to merge the bottom-up saliency maps with the semantic saliency map. We show that our NR metric WMBER weighted by the spatio-temporal-semantic saliency map provides higher results then the WMBER weighted by the bottom-up spatio-temporal saliency map. Tests are performed on two H.264/AVC video databases for video quality assessment over lossy networks.

International audience; In this paper we are interested in the saliency of visual content from wearable cameras. The subjective saliency in wearable video is studied first due to the psycho-visual experience on this content. Then the method for objective saliency map computation with a specific contribution based on geometrical saliency is proposed. Fusion of spatial, temporal and geometric cues in an objective saliency map is realized by the multiplicative operator. Resulting objective saliency maps are evaluated against the subjective maps with promising results, highlighting interesting performance of proposed geometric saliency model.

In this work we introduce the Bag-Of-Regions model, inspired from the Bag-Of-Visual-Words. Instead of clustering local image patches represented by SIFT or related descriptors, low level descriptors are extracted and clustered from image regions, as given by a segmentation algorithm. The Bag-Of-Region model allows to define visual dictionaries that capture extra information with respect to Bag-Of-Visual-Words. Combined description schemes and ad-hoc incremental clustering for visual dictionnaries are proposed. The results on public datasets are promising.

Recent trends in visual indexing make appear a large family of methods which use a local image representation via descriptors associated to the interest points, see chapter 2. Such approaches mostly ”forget” any structure in the image considering unordered sets of descriptors or their histograms as image model. Hence, more advanced approaches try to overcome this drawback by adding spatial arrangements to the interest points. In this chapter we will present two trends in incorporation of spatial context into visual description, such as considering spatial context in the process of matching of signatures on one hand and design of structural descriptors which are then used in a global Bag-of-Visual-Words (BoVW) approach on the other hand. As images and video are mainly available in a compressed form, we shortly review global descriptors extracted from compressed stream and hence less sensible to compression artifacts. Furthermore, on the basis of scalable, multiresolution/ multi-scale visual content representation in modern compression standards, we study how this multi-resolution context can be efficiently incorporated into a BoVW approach.

International audience; The paper addresses the problem of object search in video content. Both Query-By-Example paradigm and context search are explored. In QBE paradigm the object of interest is searched by matching of object signatures built from SURF descriptors with on-the-fly computed signatures in frames. The "context" search is understood as a query on the whole frame with features extracted after a region-based segmentation. Both kinds of features are transcribed in Bag-Of-Words framework. The combination of Bag-of-Visual-Words and Bag-of-Region-Words gives promising results in TRECVID'2011 Instance Search Task.

12 pages - TRECVID workshop notebook papers/slides available at http://www-nlpir.nist.gov/projects/tvpubs/tv.pubs.org.html; International audience; The IRIM group is a consortium of French teams work- ing on Multimedia Indexing and Retrieval. This paper describes its participation to the TRECVID 2011 se- mantic indexing and instance search tasks. For the semantic indexing task, our approach uses a six-stages processing pipelines for computing scores for the likeli- hood of a video shot to contain a target concept. These scores are then used for producing a ranked list of im- ages or shots that are the most likely to contain the tar- get concept. The pipeline is composed of the following steps: descriptor extraction, descriptor optimization, classification, fusion of descriptor variants, higher-level fusion, and re-ranking. We evaluated a number of dif- ferent descriptors and tried different fusion strategies. The best IRIM run has a Mean Inferred Average Pre- cision of 0.1387, which ranked us 5th out of 19 partic- ipants. For the instance search task, we we used both object based query and frame based query. We formu- lated the query in standard way as comparison of visual signatures either of object with parts of DB frames or as a comparison of visual signatures of query and DB frames. To produce visual signatures we also used two apporaches: the first one is the baseline Bag-Of-Visual- Words (BOVW) model based on SURF interest point descriptor; the second approach is a Bag-Of-Regions (BOR) model that extends the traditional notion of BOVW vocabulary not only to keypoint-based descrip- tors but to region based descriptors.

International audience; Human vision system is very complex and has been studied for many years specifically for purposes of efficient encoding of visual, e.g. video content from digital TV. There have been physiological and psychological evidences which indicate that viewers do not pay equal attention to all exposed visual information, but only focus on certain areas known as focus of attention (FOA) or saliency regions. In this work, we propose a novel based objective quality assessment metric, for assessing the perceptual quality of decoded video sequences affected by transmission errors and packed loses. The proposed method weights the Mean Square Error (MSE), Weighted-MSE (WMSE), according to the calculated saliency map at each pixel. Our method was validated trough subjective quality experiments.

International audience; Many efficient super-resolution methods have been presented in the past for improving resolution of images degraded by global blurs. Unfortunately, in video, more complex situations can be observed where local blurs appear in each frame which are mainly caused by object motions. To address this problem, we propose in this article a local super-resolution method which allows the restoration of such local blurs. Moreover, the motion of objects in video sequences may be very complex and particularly in very low-resolution sequences it is difficult to estimate their motion exactly enough to superimpose them for super-resolution. To this end, we present a generic method: An interpolation method is proposed to improve the resolution of moving objects and we derive from this a super-resolution method for the scene background.

International audience; This paper presents a method for indexing activities of daily living in videos obtained from wearable cameras. In the context of dementia diagnosis by doctors, the videos are recorded at patients' houses and later visualized by the medical practitioners. The videos may last up to two hours, therefore a tool for an efficient navigation in terms of activities of interest is crucial for the doctors. The specific recording mode provides video data which are really difficult, being a single sequence shot where strong motion and sharp lighting changes often appear. Our work introduces an automatic motion based segmentation of the video and a video structuring approach in terms of activities by a hierarchical two-level Hidden Markov Model. We define our description space over motion and visual characteristics of video and audio channels. Experiments on real data obtained from the recording at home of several patients show the difficulty of the task and the promising results of our approach.

This paper presents a method for indexing human ac- tivities in videos captured from a wearable camera being worn by patients, for studies of progression of the dementia diseases. Our method aims to produce indexes to facilitate the navigation throughout the individual video recordings, which could help doctors search for early signs of the dis- ease in the activities of daily living. The recorded videos have strong motion and sharp lighting changes, inducing noise for the analysis. The proposed approach is based on a two steps analysis. First, we propose a new approach to segment this type of video, based on apparent motion. Each segment is characterized by two original motion de- scriptors, as well as color, and audio descriptors. Second, a Hidden-Markov Model formulation is used to merge the multimodal audio and video features, and classify the test segments. Experiments show the good properties of the ap- proach on real data., Comment: 2011 9th International Workshop on Content-Based Multimedia Indexing (CBMI), Madrid : Spain (2011)

International audience; Our research focuses on analysing human activities according to a known behaviorist scenario, in case of noisy and high dimensional collected data. The data come from the monitoring of patients with dementia diseases by wearable cameras. We define a structural model of video recordings based on a Hidden Markov Model. New spatio-temporal features, color features and localization features are proposed as observations. First results in recognition of activities are promising.

HD video content represents a tremendous quantity of information that cannot be easily handled by all types of devices. Hence the scalability issues in its processing have become a focus of interest in HD video coding technologies. In this chapter, we focus on the natural scalability of hierarchical transforms to tackle video indexing and retrieval. In the first part of the chapter, we give an overview of the transforms used and then present the methods which aim at exploring the transform coefficients to extract meaningful features from video and embed metadata in the scalable code-stream. Statistical global object-based descriptor incorporating low frequency and high-frequency features is proposed. In the second part of the chapter, we introduce a video retrieval technique based on a multiscale description of the video content. Both spatial and temporal scalable descriptors are proposed on the basis of multi-scale patches. A statistical dissimilarity between videos is derived using Kullback-Leibler divergences to compare patch descriptors.

International audience; With exponentially growing quantity of video content in various formats, including the popularisation of HD (High Definition) video and cinematographic content, the problem of efficient indexing and retrieval in video databases becomes crucial. Despite efficient methods have been designed for the frame-based queries on video with local features, object-based indexing and retrieval attract attention of research community by the seducing possibility to formulate meaningful queries on semantic objects. In the case of HD video, the principle of scalability addressed by actual compression standards is of great importance. It allows for indexing and retrieval on the lower resolution available in the compressed bit-stream. The wavelet decomposition used in the JPEG2000 standard provides this property. In this paper, we propose a scalable indexing of video content by objects. First, a method for scalable moving object extraction is designed. Using the wavelet data, it relies on the combination of robust global motion estimation with morphological colour segmentation at a low spatial resolution. It is then refined using the scalable order of data. Second, a descriptor is built only on the objects extracted. This descriptor is based on multi-scale histograms of wavelet coefficients of objects. Comparison with SIFT features extracted on segmented object masks gives promising results.

International audience; The IRIM group is a consortium of French teams working on Multimedia Indexing and Retrieval. This paper describes our participation to the TRECVID 2010 semantic indexing and instance search tasks. For the semantic indexing task, we evaluated a number of different descriptors and tried different fusion strategies, in particular hierarchical fusion. The best IRIM run has a Mean Inferred Average Precision of 0.0442, which is above the task median performance. We found that fusion of the classification scores from different classifier types improves the performance and that even with a quite low individual performance, audio descriptors can help. For the instance search task, we used only one of the example images in our queries. The rank is nearly in the middle of the list of participants. The experiment showed that HSV features outperform the concatenation of HSV and Edge histograms or the Wavelet features.