Search

International audience; There has been an increasing interest in spiking neural networks in recent years. SNNs are seen as hypothetical solutions for the bottlenecks of ANNs in pattern recognition, such as energy efficiency. But current methods such as ANN-to-SNN conversion and back-propagation do not take full advantage of these networks, and unsupervised methods have not yet reached a success comparable to advanced artificial neural networks. It is important to study the behavior of SNNs trained with unsupervised learning methods such as spiketiming dependent plasticity (STDP) on video classification tasks, including mechanisms to model motion information using spikes, as this information is critical for video understanding. This paper presents multiple methods of transposing temporal information into a static format, and then transforming the visual information into spikes using latency coding. These methods are paired with two types of temporal fusion known as early and late fusion, and are used to help the spiking neural network in capturing the spatio-temporal features from videos. In this paper, we rely on the network architecture of a convolutional spiking neural network trained with STDP, and we test the performance of this network when challenged with action recognition tasks. Understanding how a spiking neural network responds to different methods of movement extraction and representation can help reduce the performance gap between SNNs and ANNs. In this paper we show the effect of the similarity in the shape and speed of certain actions on action recognition with spiking neural networks, we also highlight the effectiveness of some methods compared to others.

In recent years, spiking neural networks (SNNs) emerge as an alternative to deep neural networks (DNNs). SNNs present a higher computational efficiency – using low-power neuromorphic hardware – and require less labeled data for training – using local and unsupervised learning rules such as spike timing-dependent plasticity (STDP). SNNs have proven their effectiveness in image classification on simple datasets such as MNIST. However, to process natural images, a pre-processing step is required. Difference-of-Gaussians (DoG) filtering is typically used together with on-center / off-center coding, but it results in a loss of information that decreases the classification performance. In this paper, we propose to use whitening as a pre-processing step before learning features with STDP. Experiments on CIFAR-10 show that whitening allows STDP to learn visual features that are visually closer to the ones learned with standard neural networks, with a significantly increased classification performance as compared to DoG filtering. We also propose an approximation of whitening as convolution kernels that is computationally cheaper to learn and more suited to be implemented on neuromorphic hardware. Experiments on CIFAR-10 show that it performs similarly to regular whitening. Cross-dataset experiments on CIFAR-10 and STL-10 also show that it is stable across datasets, making it possible to learn a single whitening transformation to process different datasets.

Spiking neural networks (SNNs) equipped with latency coding and spike-timing dependent plasticity rules offer an alternative to solve the data and energy bottlenecks of standard computer vision approaches: they can learn visual features without supervision and can be implemented by ultra-low power hardware architectures. However, their performance in image classification has never been evaluated on recent image datasets. In this paper, we compare SNNs to auto-encoders on three visual recognition datasets, and extend the use of SNNs to color images. The analysis of the results helps us identify some bottlenecks of SNNs: the limits of on-center/off-center coding, especially for color images, and the ineffectiveness of current inhibition mechanisms. These issues should be addressed to build effective SNNs for image recognition.

International audience; Spiking neural networks (SNNs) are good candidates to produce ultra-energy-efficient hardware. However, the performance of these models is currently behind traditional methods. Introducing multi-layered SNNs is a promising way to reduce this gap. We propose in this paper a new threshold adaptation system which uses a timestamp objective at which neurons should fire. We show that our method leads to state-of-the-art classification rates on the MNIST dataset (98.60%) and the Faces/Motorbikes dataset (99.46%) with an unsupervised SNN followed by a linear SVM. We also investigate the sparsity level of the network by testing different inhibition policies and STDP rules.

Face alignment remains difficult under uncontrolled conditions due to the many variations that may considerably impact facial appearance. Recently, video-based approaches have been proposed, which take advantage of temporal coherence to improve robustness. These new approaches suffer from limited temporal connectivity. We show that early, direct pixel connectivity enables the detection of local motion patterns and the learning of a hierarchy of motion features. We integrate local motion to the two predominant models in the literature, coordinate regression networks and heatmap regression networks, and combine it with late connectivity based on recurrent neural networks. The experimental results on two datasets, 300VW and SNaP-2DFe, show that local motion improves video-based face alignment and is complementary to late temporal information. Despite the simplicity of the proposed architectures, our best model provides competitive performance with more complex models from the literature.

Image recognition tasks require multi-layer networks to achieve good performance on complex data. However, building multi-layer spiking neural networks (SNN) still remains unreachable. One cause is that the learning mechanism of these models decreases the spiking activity throughout the layers. We propose three mechanisms to solve this issue without impacting the performance of the network: target frequency threshold adaptation, which forces neurons to reach a desired frequency, binary coding, which improves the performance of the network at high levels of activity, and mirrored STDP, which improves the convergence of the training. Experiments on single layer networks show that these mechanisms preserve both the recognition rate and the level of spiking activity.

International audience; Face alignment is an essential task for many applications. Its objective is to locate feature points on the face, in order to identify its geometric structure. Under unconstrained conditions, the different variations that may occur in the visual context, together with the instability of face detection, make it a difficult problem to solve. While many methods have been proposed, their performances under these constraints are still not satisfactory. In this article, we claim that face alignment should be studied using image sequences rather than still images, as it has been done so far. We show the importance of taking into consideration the temporal information under unconstrained conditions.

Many large digital collections are organized by subject; these useful information organization structures are large and complex, thus difficult to browse. Current online tools and visualization prototypes show small localized subsets and do not provide the ability to explore the predominant patterns of the overall subject structure. This study builds on existing work concerning automatic subject hierarchy modification techniques that aim to facilitate browsing for documents by capitalizing on the highly uneven distribution of real-world collections. Specifically, previous work used an estimation of the number of accessible documents offered by each subject term, while the current study uses the exact number of accessible documents. The impact is demonstrated on a large collection organized using Medical Subject Headings (MeSH). Results show that, although computationally more demanding, pruning the MeSH hierarchy based on the exact access produces a different subject hierarchy under some conditions. The visual impact is demonstrated using traditional outline views. This study has implications for the development of information organization theory and human-information interaction techniques for subject hierarchies.

Image retrieval systems are generally based on the notion of image similarity: they compute similarity scores between the images of the database and a query (image or text), and organize the images according to these scores. However, this notion is ill-defined, and the collections used to train and evaluate image retrieval systems are based on similarity judgments that rely on simplistic, non-realistic, assumptions. This paper addresses the issue of the definition of image similarity, and more precisely the two following questions: do humans assess image similarity in the same way? Is it possible to define reference similarity judgments that would correspond to the perception of most users? An experiment is proposed, in which human subjects are assigned two tasks that fall in principle to the system: rating the similarity of images and ranking images according to a reference image. The data provided by the subjects is analyzed quantitatively to the light of the two aforementioned questions. Results show that the subjects do not have collective strategies of similarity assessment, but that a satisfying consensus can be found individually on the data samples used in the experiments. Based on this, methods to define reference similarity scores and rankings are proposed, that can be used on a larger scale to produce realistic ground truths for the evaluation of image retrieval systems. This study is a first step towards a general, realistic, definition of the notion of image similarity in the context of image retrieval.

International audience; This paper introduces a novel person track dataset dedicated to person re-identification. The dataset is built from a set of real life TV shows broadcasted from BFMTV and LCP TV french channels, provided during REPERE challenge. It contains a total 4,604 persontracks (short video sequences featuring an individual with no background) from 266 persons. The dataset has been built from the REPERE dataset by following several automated processing and manual selection/filtering steps. It is meant to serve as a benchmark in person re-identification from images/videos. The dataset also provides re-identifications results using space-time histograms as a baseline, together with an evaluation tool in order to ease the comparison to other re-identification methods.

International audience; — Bags of visual words are the most studied image description technique in the last years. This representation of images raised new possibilities as well as new research issues. In particular, it is important to automatically determine which visual words are the most relevant to describe the images, and which ones should be ignored. This issue is a classical problem of textual information retrieval, usually addressed by the use of weighting schemes. In this paper, the most common weighting schemes from text retrieval are applied to the case of visual word-based retrieval.New weighting schemes are also proposed, and several Minkowski-like distances are tested. The experiments are performed on four different datasets that correspond to two different retrieval tasks; it allows us to bring to light some properties of visual words and weighting schemes. This studyresults in several findings. It first shows that the optimal setting for distances and weighting schemes depends on the nature of the visual content of the images considered. Especially, raw frequency can be the most effective weight when dealing with complex datasets; it questions the habit to systematically use the tf . idf weighting scheme. It also shows that weighting schemes and Minkowski distances have similar effect and should be used together in a consistent way. Based on these findings, general guidelines for the choice of distances and weighting schemes are proposed.

International audience; In this paper, we propose an original content-based image retrieval method using bag-of-words dedicated to building matching on mobile devices. In the literature, the repetitiveness of visual words in natural scenes, and especially in building images, has been demonstrated. Assuming images are composed of a set of elementary blocks, we represent them using only a few well-chosen features. In the context of image search on mobile devices, this allows to considerably reduce the size of the data to be sent to the server. This method has been experimented using SIFT descriptors on three well-known databases. Experimental results show that this method can outperform the standard bag-of-words approach while reducing the number of features used to represent images. Moreover, this general framework can be used in conjunction with any kind of descriptors and indexing methods.

This paper describes a multi-modal person recognition system for video broadcast developed for participating in the DefiRepere challenge. The main track of this challenge targets the identification of all persons occurring in a video either in the audio modality (speakers) or the image modality (faces). This system is developed by the PERCOL team involving 4 research labs in France and was ranked first at the 2014 Defi-Repere challenge. The main scientific issue addressed by this challenge is the combination of audio and video information extraction processes for improving the extraction performance in both modalities. In this paper, we present the strategy followed by the PERCOL team for speaker identification based on enriching the speaker diarization with features related to the ”understanding” of the video scenes: text overlay transcription and analysis, automatic situation identification (TV set, report), the amount of people visible, TV set disposition and even the camera when available. Experiments on the REPERE corpus show interesting results on the speaker identification system enriched by the scene understanding features and the usefulness of the speaker to identify faces.

Many large digital collections are currently organized by subject; although useful, these information organization structures are large and complex and thus difficult to browse. Current online tools and visualization prototypes show small, localized subsets and do not provide the ability to explore the predominant patterns of the overall subject structure. This study describes subject tree modifications that facilitate browsing for documents by capitalizing on the highly uneven distribution of real-world collections. The approach is demonstrated on two large collections organized by the Library of Congress Subject Headings (LCSH) and Medical Subject Headings (MeSH). Results show that the LCSH subject tree can be reduced to 49% of its initial complexity while maintaining access to 83% of the collection, and the MeSH tree can be reduced to 45% of its initial complexity while maintaining access to 97% of the collection. A simple solution to negate the loss of access is discussed. The visual impact is demonstrated by using traditional outline views and a slider control allowing searchers to change the subject structure dynamically according to their needs. This study has implications for the development of information organization theory and human–information interaction techniques for subject trees.

Image indexing and retrieval systems mostly rely on the computation of similarity measures between images. This notion is ill-defined, generally based on simplistic assumptions that do not fit the actual context of use of image retrieval systems. This paper addresses two fundamental issues related to image similarity: checking whether the degree of similarity between two images is perceived consistently by different users and establishing the elements of the images on which users base their similarity judgment. A study is set up, in which human subjects have been asked to assess the degree of the pairwise similarity of images and describe the features on which they base their judgments. The quantitative analysis of the similarity scores reported by the subjects shows that users reach a certain consensus on similarity assessment. From the qualitative analysis of the transcripts of the records of the experiments, a list of the features used by the subjects to assess image similarity is built. From this, a new model of image description emerges. As compared to existing models, it is more realistic, free of preconceptions and more suited to the task of similarity computation. These results are discussed from the perspectives of psychology and computer science.

The Library of Congress Subject Headings (LCSH) is a subject structure used to index large library collections throughout the world. Browsing a collection through LCSH is difficult using current online tools in part because users cannot explore the structure using their existing experience navigating file hierarchies on their hard drives. This is due to inconsistencies in the LCSH structure, which does not adhere to the specific rules defining tree structures. This article proposes a method to adapt the LCSH structure to reflect a real-world collection from the domain of science and engineering. This structure is transformed into a valid tree structure using an automatic process. The analysis of the resulting LCSH tree shows a large and complex structure. The analysis of the distribution of information within the LCSH tree reveals a power law distribution where the vast majority of subjects contain few information items and a few subjects contain the vast majority of the collection. © 2012 Wiley Periodicals, Inc.

In recent years, attention has been raised on the use of image modality in medical image retrieval. Several methods have been developed to automatically identify the modality of images and integrate this information to image retrieval systems. Results show that using the modality can significantly improve the performance of these systems. However, doing so also requires to identify the modality expressed in the queries. This task is usually performed by elementary pattern matching techniques that can be applied only to a small proportion of queries. This paper addresses the issue of predicting the modality expressed in the queries in a general way. First, a taxonomy of queries and the specificities of the problem are described. Then, a Bayesian classifier is proposed to automatically predict the modality expressed in the queries, as well as two models to integrate these prediction to an image retrieval system. Experiments performed on data from the ImageCLEFMed 2009 and 2010 challenges show that our approach can outperform current systems in precision, although the performance can differ significantly from one query to another.

International audience; Image classification methods based on text utilize terms extracted from image annotations (image caption, image-related article, etc.) to achieve classification. For images involving different anatomical structures (chest, spine, etc.), however, the precision of pure textual classification often suffers from highly complex text contents (e.g. text terms extracted out of two MR abdomen images may be quite different from each other: terms from one image may concerns gastroenteritis while the other contains terms involving hysteromyoma). This paper tackles the anatomy image classification problem using a hybrid approach. First, a mutual information (MI) based filter is applied to select a set of terms with top MI scores for each anatomical class and help reduce the noise existing in the raw text. Second, local features extracted from the images are transformed as visual descriptors. Last, a hybrid scheme on the results from the textual and visual methods is applied to achieved further improvement of the classification results. Experiments show that this hybrid scheme improves the results over the sole textual or visual method on different anatomical class settings.

International audience; Medical databases have been a popular application field for image retrieval techniques during the last decade. More recently, much attention has been paid to the prediction of medical image modality (X-rays, MRI...) and the integration of the predicted modality into image retrieval systems. This paper addresses these two issues. On the one hand, we believe it is possible to design specific visual descriptors to determine image modality much more efficiently than the traditional image descriptors currently used for this task. We propose very light image descriptors that better describe the modality properties and show promising results. On the other hand, we present a comparison of different existing or new modality integration methods. This comprehensive study provide insights on the behavior of these models with respect to the initial classification and retrieval systems. These results can be extended to other applications with a similar framework. All the experiments presented in this work are performed using datasets provided during the 2009 and 2010 ImageCLEF medical tracks.

International audience; Current text retrieval techniques allow to index and retrieve text documents very efficiently and with a good accuracy. Image retrieval, on the contrary, is still very coarse and does not yield satisfying results. Therefore, computer vision researchers try to benefit from text retrieval contributions to enhance their retrieval systems. In particular, Sivic and Zisserman, with their video-google framework [1], propose a description of images similar to standard text descriptors: images are described by elementary image parts, called visual words. Thus, they perform image retrieval using the standard Vector Space Model (VSM) of Information Retrieval (IR) and benefit from some classical IR techniques such as inverted files. Among available text retrieval techniques, automatically finding the most relevant words to describe a document has been intensively studied for texts, but not for images. In this paper, we propose to explore the use of term weighting techniques and classical distances from text retrieval in the case of images. These weights are standard VSM weights, weights derived from probabilistic models of IR or new weighting schemes that we propose. Our experiments, performed on several datasets, show that no weighting scheme can improve retrieval on every dataset, but also that choosing weights fitting the properties of the dataset can improve precision and MAP up to 10 percents. This study provides some interesting insights about the semantic and statistical differences between textual and visual words, and about the way visual word-based image retrieval systems can be optimized. It also shows some limits of the bag of visual words model, and the relation existing between Minkowski distances and local weighting. At last, this study questions some experimental habits commonly found in the literature (choice of L1 distance, TF*IDF weights and evaluation using one dataset only).

International audience; In this paper, we propose two ways of improving image classification based on bag-of-words representation. Two shortcomings of this representation are the loss of the spatial information of visual words and the presence of noisy visual words due to the coarseness of the vocabulary building process. On the one hand, we propose a new representation of images that goes further in the analogy with textual data: visual sentences, that allows us to "read" visual words in a certain order, as in the case of text. We can therefore consider simple spatial relations between words. We also present a new image classification scheme that exploits these relations. It is based on the use of language models, a very popular tool from speech and text analysis communities. On the other hand, we propose new techniques to eliminate useless words, one based on geometric properties of the keypoints, the other on the use of probabilistic Latent Semantic Analysis (pLSA). Experiments show that our techniques can significantly improve image classification, compared to a classical Support Vector Machine-based classification.

International audience; Quantifying the quality of a video or audio transmission over the Internet is usually a hard task, as based on the statistical processing of the evaluations made by a panel of humans (the corresponding and standardized area is called subjective testing). In this paper we describe a methodology called Pseudo-Subjective Quality Assessment (PSQA), based on Random Neural Networks, which is able to perform this task automatically, accurately and efficiently. RNN had been chosen here because of their good performances over other possibilities; this is discussed in the paper. Some new insights on PSQA's use and performance are also given. In particular we discuss new results concerning PSQA-based dynamic quality control, and conversational quality assessment.