Search

In this paper, we propose a deep spatio-temporal appearance (DSTA) descriptor for person re-identification (re-ID). The proposed descriptor is based on the deep Fisher vector (FV) encoding of the trajectory spatio-temporal structures. These have the advantage of robustly handling the misalignment in the pedestrian tracklets. The deep encoding exploits the richness of the spatio-temporal structural information around the trajectories. This is achieved by hierarchically encoding the trajectory structures leveraging a larger tracklet neighborhood scale when moving from one layer to the next one. In order to eliminate the noisy background located around the pedestrian and model the uniqueness of its identity, the deep FV encoder is further enriched towards the deep Salient-Gaussian weighted FV (deepSGFV) encoder by integrating the pedestrian Gaussian and saliency templates in the encoding process, respectively. The proposed descriptor produces competitive accuracy with respect to state-of-the art methods and especially the deep CNN ones without necessitating either pre-training or data augmentation on four challenging pedestrian video datasets: PRID2011, i-LIDS-VID, Mars and LPW. The further combination of DSTA with deep CNN boosts the current state-of-the-art methods and demonstrates their complementarity.

This work addresses the problem of human action recognition in videos.We improved the classical Fisher vector by the geometric relationships among features.We added significant information of local and global features to the feature encoding.Better exploiting structural information is important to improve the performance.Our approach obtains the state-of-the-art performance on three challenging datasets. In this paper, we propose a new multi-layer Fisher vector encoding method based on trajectory descriptors for human action recognition. The proposed method aims at improving the classical shallow Fisher vector (FV) encoding method. Our main contribution resides in considering a progressive representation of the geometric relationships among trajectories. In fact, our presentation is based on three nested layers and provides deep and discriminant structures by local spatial pooling and refining the representation from one layer to the next. To preserve more information in feature encoding process, fine and large spatio-temporal structures have been applied. Fine structures aim at exploiting the local spatio-temporal information by building graphs of trajectories, while large structures aim at exploiting the global spatio-temporal information by spatio-temporal video subdivision. Our approach is evaluated on three popular and large human action datasets: Hollywood2, Olympic sports and HMDB51. Experiments show that more layers produce higher action classification accuracy, which proves the capability of our multi-layer Fisher vector encoding method.

Diverse studies have shown the efficiency of sparse coding in feature quantization. However, its major drawback is that it neglects the relationships among features. To reach the spatial context, we proposed in this paper, a novel sparse coding method called Extended Laplacian Sparse Coding. Two successive stages are required in this method. In the first stage, the sparse visual phrases based on Laplacian sparse coding are generated from the local regions in order to represent the geometric information in the image space. The second stage aims to incorporate the spatial relationships among local features in the image space into the objective function of the Laplacian sparse coding. It takes into account the similarity among local regions in the Laplacian sparse coding process. The matching between the local regions is based on the Hungarian method as well as the histogram intersection measure between sparse visual phrases already assigned to the local regions in the first stage. Furthermore, we suggested to improve the pooling step that succeeds the encoding step by introducing the discretized max pooling method that estimates the distribution of the responses of each local feature to the dictionary of basis vectors. Our experimental results prove that our method outperforms the existing background results.

Modelling spatial information of local features is known to improve performance in image categorisation. Compared with simple pairwise features and visual phrases, graphs can capture the structural organisation of local features more adequately. Besides, a dense regular grid can guarantee a more reliable representation than the interest points and give better results for image classification. In this study, the authors introduced a bag of dense local graphs approach that combines the performance of bag of visual words expressing the image classification process with the representational power of graphs. The images were represented with dense local graphs built upon dense scale-invariant feature transform descriptors. The graph-based substructure pattern mining algorithm was applied on the local graphs to discover the frequent local subgraphs, producing a bag of subgraphs representation. The results were reported from experiments conducted on four challenging benchmarks. The findings show that the proposed subgraph histogram improves the categorisation accuracy.

The bag of words approach describes an image as a histogram of visual words. Therefore, the structural relation between words is lost. Since graphs are well adapted to represent these structural relations, we propose, in this paper, an image classification framework which draws benefit from the efficiency of the graph in modeling structural information and the good classification performances given by the bag of words method. For each image in the dataset, a graph is created by modeling the spatial relations between dense local patches. Thus, we obtain a graph dataset. From the graph dataset, we select the most frequent subgraphs to construct the bag of subgraphs (BoSG) and we associate to each image a subgraph histogram that describes its visual content. For experiments, we have used the two challenging datasets: 15 Scenes and Pascal VOC 2007. Experimental results show that the proposed method outperforms the bag of words and the spatial pyramid models in terms of recognition rate.

Person re-identification (re-ID) is a challenging task in the camera surveillance field, since it addresses the problem of re-identifying people across multiple non-overlapping cameras. Most of existing approaches have been concentrated on: 1) achieving a robust and effective feature representation; and 2) enforcing discriminative metric learning to predict if two images represent the same identity. In this context, we present a new approach for person re-ID built upon multi-level descriptors. This is achieved by combining three complementary representations: salient-Gaussian Fisher Vector (SGFV) encoding method, salient-Gaussian BossaNova (SGBN) histogram encoding method and deep Convolutional Neural Network (CNN) features. The two first methods adapt the histogram encoding framework to the person re-ID task. This is achieved by integrating the pedestrian saliency map and the spatial location information, in the histogram encoding process. On one hand, human saliency is reliable and distinctive in the person re-ID task, since it can model the uniqueness of the identity. On the other hand, localizing a person in the image can effectively discard noisy background information. Finally, one of the most advanced metric learning in person re-ID: the Cross-view Quadratic Discriminant Analysis (XQDA) is applied on the top of the resulting description. The proposed method yields promising person re-ID results on two challenging image-based person re-ID benchmarks: CUHK03 and Market-1501.

Due to the exponential growth of the video data stored and uploaded in the Internet websites especially YouTube, an effective analysis of video actions has become very necessary. In this paper, we tackle the challenging problem of human action recognition in realistic video sequences. The proposed system combines the efficiency of the Bag-of-visual-Words strategy and the power of graphs for structural representation of features. It is built upon the commonly used Space-Time Interest Points (STIP) local features followed by a graph-based video representation which models the spatio-temporal relations among these features. The experiments are realized on two challenging datasets: Hollywood2 and UCF YouTube Action. The experimental results show the effectiveness of the proposed method.

Person re-identification is a fundamental challenging task in Computer Vision that consists on recognizing the same person across multiple potentially non-overlapping cameras. This importance is due to the important challenges that it proposes like pose, background clutter and occlusion, illumination changes and low resolution. Also, most of the existing approaches rely on brute-force matching between pedestrian local descriptors and consequently, suffer from low computational efficiency. So, to address this issues, we present a new perspective for person re-identification based on a histogram encoding scheme that assigns a global signature to each pedestrian image and thus, simplifies the matching process. The main contribution of this paper is the design of an extended weighted version of the traditional Fisher vector (FV) encoding scheme. This is achieved by incorporating the Topological location of the encoded descriptors CN, CHS and 15-d in the encoding process and then combining the obtained Topological weighted histograms in order to form our proposed descriptor. The super Fisher vector representation has improved both the rate and the speedup of the person matching process, while weighting the FV encoding scheme by the Topological weight helped out to remove the noisy and busy background clutters surrounding the pedestrians in the images. Besides, Retinex transform was applied in order to handle the problem of illumination variations. Experimental results made on three challenging datasets, the VIPeR dataset, the CUHK03 dataset and the Market-1501 dataset, prove the effectiveness of the proposed method.

The KNN classification algorithm is particularly suited to be used when classifying images described by local features. In this paper, we propose a novel image classification approach, based on local descriptors and the KNN algorithm. The proposed scheme is based on a hierarchical categorization tree that uses both supervised and unsupervised classification techniques. The unsupervised one is based on a hierarchical lattice vector quantization algorithm, while the supervised one is based on both feature vectors labelling and supervised feature selection method. The proposed tree improves the effectiveness of local feature vector classification and outperforms the exact KNN algorithm in terms of categorization accuracy.

One of the challenges in the development of a content-based multimedia indexing and retrieval application is to achieve an efficient indexing scheme. To retrieve a particular image from a large scale image database, users can be frustrated by the long query times. Conventional indexing structures cannot usually cope with the presence of a large amount of feature vectors in high-dimensional space. This paper addresses such problems and presents a novel indexing technique, the embedded lattices tree, which is designed to bring an effective solution especially for realizing the trade off between the retrieval speed up and precision. The embedded lattices tree is based on a lattice vector quantization algorithm that divides the feature vectors progressively into smaller partitions using a finer scaling factor. The efficiency of the similarity queries is significantly improved by using the hierarchy and the good algebraic and geometric properties of the lattice. Furthermore, the dimensionality reduction that we perform on the feature vectors, translating from an upper level to a lower one of the embedded tree, reduces the complexity of measuring similarity between feature vectors. In addition, it enhances the performance on nearest neighbor queries especially for high dimensions. Our experimental results show that the retrieval speed is significantly improved and the indexing structure shows no sign of degradations when the database size is increased.

We propose in this paper a spatio-temporal pyramid representation (STPR) of the video based Accordion image. The Accordion image allows the pixels having a high temporal correlation to be put in space adjacency. The STPR introduces spatial and temporal layout information to the local SIFT features computed on the Accordion image. It consists in applying firstly, a temporal pyramid decomposition on the video to divide it into a sequence of increasingly finer temporal blocks and secondly in performing a spatial pyramid representation on the Accordion images relative to the temporal blocks. The Multiple Kernel Learning approach is used to combine the multi-histograms coming from different SpatioTemporal Pyramid levels. Experiments using the human action recognition datasets (Hollywood2 and Olympic sports) show the effectiveness of the proposed approach.

To generate the visual codebook, a step of quantization process is obligatory. Several works have proved the efficiency of sparse coding in feature quantization process of BoW based image representation. Furthermore, it is an important method which encodes the original signal in a sparse signal space. Yet, this method neglects the relationships among features. To reduce the impact of this issue, we suggest in this paper, a Laplacian Tensor sparse coding method, which will aim to profit from the relationship among the local features. Precisely, we propose to apply the similarity of tensor descriptors to create a Laplacian Tensor similarity matrix, which can better present in the same time the closeness of local features in the data space and the topological relationship among the spatially near local descriptors. Moreover, we integrate statistical analysis applied to the local features assigned to each visual word in the pooling step. Our experimental results prove that our method prevails or exceeds existing background results.

In image classification task, several recent works show that sparse representation plays a basic role in dictionary learning. However, this approach neglects the spatial relationships in the image space during dictionary learning. However, this approach neglects the neighboring relationship in dictionary learning. To alleviate the impact of this problem, we propose a novel dictionary learning based on Laplacian sparse coding method that profits from the neighboring relationship among the local features. For that purpose, we incorporate the matching between local regions in the Laplacian sparse coding formula. Moreover, we integrate statistical analysis of the distribution of the responses of each local feature to the dictionary basis in the final image representation. Our experimental results prove that our method performs existing background results based on sparse representation.

Local space time points detection and description have recently emerged as a major research in based video analysis content in video surveillance and event detection applications. Several works extend tow dimensional descriptor to the temporal dimension. Most of the existing methods consider the video as a spatio-temporal volume and then describe the volumetric region around the salient point in 3D. However, this representation requires a high operational complexity. In this works we propose a new approach to describe motion using a simple 2D representation of the video. Our method is based on tracked feature points in image sequence. The main challenge in motion description is how to detect the local motion information. In this paper we aim to describe motion around moving point without the need to extend them on the 3D dimension. To show the efficiency and accuracy of our approach, we perform action recognition experiments on the KTH and Weizmann databases using the bag of words approach. We have obtained impressive results for action recognition.

In this paper, we propose a novel motion descriptor Seg-SIFT-ACC for human action recognition. The proposed descriptor is based both on the accordion representation of the video and its temporal segmentation into elementary motion segments. The accordion representation aims to put in space adjacency the columns of the video frames having a high temporal correlation. For complex videos containing many different elementary actions, the accordion representation may put in spatial adjacency temporally correlated pixels that belong to different elementary actions. To surmount this problem, we divide the video into elementary motions segments and we apply the accordion representation on each one separately.

Recognizing human action from videos is an active field of research in computer vision and pattern recognition. Human activity recognition has many potential applications such as video surveillance, human machine interaction, sport videos retrieval and robot navigation. Actually, local descriptors and bag of visuals words models achieve state-of-the-art performance for human action recognition. The main challenge in features description is how to represent efficiently the local motion information. Most of the previous works focus on the extension of 2D local descriptors on 3D ones to describe local information around every interest point. In this paper, we propose a new spatio-temporal descriptor based on a spacetime description of moving points. Our description is focused on an Accordion representation of video which is well-suited to recognize human action from 2D local descriptors without the need to 3D extensions. We use the bag of words approach to represent videos. We quantify 2D local descriptor describing both temporal and spatial features with a good compromise between computational complexity and action recognition rates. We have reached impressive results on publicly available action data set, {"references":["D. Weinland, R.Ronfard and E. \" A Survey of Vision-Based Methods\nfor Action Representation, Segmentation and Recognition\", Computer\nVision and Image Understanding 2010","K.Aggarwal and S.Park, \"Human motion: Modeling and recognition of\nactions and interactions\", in 3DPVT-04 Washington, DC, USA: IEEE\nComputer Society, 2004, pp. 640647","T.B.Moeslund, A.Hilton, and V.Kruger, \" A survey of advances in\nvision-based human motion capture and analysis\", CVIU 2006, 90-126","A.F. Bobick and J.W. Davis, \"The recognition of human movement\nusing temporal templates\", IEEE T-PAMI, 257-267, 2001","I. Laptev and T. Lindeberg, \"Space-time interest points\", In ICCV, 2003","I. Laptev, M. Marsza lek, C. Schmid, and B. Rozenfeld, \"Learning\nrealistic human actions from movies\", In CVPR, 2008","M. Mejdoub, L. Fonteles, C. BenAmar, and Marc Antonini. \"Embedded\nlattices tree: An Efficient indexing scheme for content based retrieval on\nimage databases\", Journal of Visual Communication and mage\nRepresentation, Elsevier, 2009.","P. Dollar, V. Rabaud, G. Cottrell, and S. Belongie, Behavior recognition\nvia sparse spatio-temporal features, In VS-PETS, 2005","D. Lowe,\"Distinctive image features from scale-invariant keypoints\",\nIJCV, 91-110,2004\n[10] A. Klaser, M. Marsza lek, and C. Schmid, \"A spatio-temporal descriptor\nbased on 3Dgradients\", In BMVC, 2008\n[11] P. Scovanner, S. Ali, and M. Shah, \"A 3-dimensional SIFT descriptor\nand its application to action recognition\", In MULTIMEDIA, 2007\n[12] A. Klaser, M. Marsza lek, C. Schmid, and A. Zisserman,\"Human\nFocused Action Localization in Video\", in International Workshop on\nSign, Gesture, Activity 2010\n[13] T.Ouni, W.Ayedi and M.Abid, \" New low complexity DCT based video\ncompression method\", In Proceedings of the 16th International\nConference on Telecommunications (ICT-09), 202-207, Piscataway, NJ,\nUSA, 2009, IEEE Press\n[14] T.Ouni, W.Ayedi and Mohamed Abid, \"New Non Predictive\nWaveletBased Video Coder: Performances Analysis\", In Proceedings of\nInternational Conference on Image Analysis and Recognition. Volume\n6111 of LNCS, pages 344-353, Berlin, Heidelberg, 2010. Springer-\nVerlag\n[15] T.Ouni, W.Ayedi et M.Abid, \"A Complete Non predictive\nVideoCompression Scheme Based on a 3D to 2D Geometric\ntransform\",International Journal Signal and Imaging Systems\nEngineering (IJSISE), Inderscience Publisher, 2011\n[16] J.Wang, H. Lu, L.Duan and J.S. Jin, \"Commercial Video Retrieval with\nVideo-based Bag of Words\", Fifth International Conference on\nIntelligent Multimedia Computing and Networking 2007, July.22, 2007.\nSalt Lake City, Utah, USA\n[17] S.Ali, and M.Shah, \"Human action recognition in videos using\nkinematic features and multiple instance learning\", in IEEE Transactions\non Pattern Analysis and Machine Intelligence (TPAMI)28830, 2010\n[18] H. Ning, Y. Hu, T. Huang, \"Searching human behaviors using\nspatialtemporal words\", in Proceedings of IEEE ICIP 07, 2007, pp.\n337340\n[19] A. Fathi and G. Mori. Action recognition by learning mid-level motion\nfeatures, In CVPR, 2008\n[20] R. Messing, C. Pal, and H. Kautz, \"Activity recognition using the\nvelocity histories of tracked keypoints\", In ICCV, 2009\n[21] G. Willems, T. Tuytelaars, and L. Van Gool, \"An effcient dense and\nscale-invariant spatio-temporal interest point detector\", In ECCV, 2008\n[22] A.P.B.Lopes, R.S. Oliveira, J.M. de Almeida, and A.de Albuquerque\nAraujo, Spatio-temporal frames in a bag-of-visual-features approach for\nhuman actions recognition, in SIBGRAPI 09. IEEE Computer Society,\n2009\n[23] Y. Kawai, M. Takahashi, M. Fujii, M. Naemura, S. Sato, \"NHK STRL\nat TRECVID 2010: Semantic Indexing and Surveillance Event\nDetection\", Proc. TRECVID Workshop, Gaithersburg, MD, USA,\nNovember 2010\n[24] Y. Benezeth, P.M. Jodoin, B. Emile, H. Laurent, C.Rosenberger,Review\nand evaluation of commonly-implemented background subtraction\nalgorithms, in Proc. of the International Conference on Pattern\nRecognition, 2008\n[25] C.Stauffer, W. Grimson, \"Learning patterns of activity using real-time\ntracking\", in IEEE Transactions on Pattern Analysis and Machine\nIntelligence 2000, pp. 747757\n[26] C.Tomasi and T.Kanade, Detection and tracking of Point Features,\nCarnegie Mellon University TeChnical Report CMU-CS-91-132, April\n1991\n[27] J.Y Bouguet, \"Pyramidal Implementation of the Lucas Kanade Feature\nTracker Description of the algorithm\", Intel Corporation,\nMicroprocessor Research Labs,1999\n[28] S. Shalev-Shwartz, Y. Singer, and N. Srebro. Pegasos : Primal estimated\nsub-gradient solver for svm, ICML, pages 807814, 2007\n[29] J. Liu, S. Ali, and M. Shah, \"Recognizing human actions using multiple\nfeatures\", In CVPR, 2008\n[30] J. Niebles and L. Fei-Fei, \"A hierarchical model of shape and\nappearance for human action classiffcation\", In CVPR, 2007\n[31] J. Niebles, H. Wang, and L. Fei-Fei, \"Unsupervised learning of human\naction categories using spatial-temporal words\", IJCV, 299-318, 2008\n[32] J. Niebles, H. Wang, and L. Fei-Fei, \"Unsupervised learning of human\naction categories using spatial-temporal words\", In BMVC, 2006"]}

K-nearest neighbor (KNN) classification is an instance-based learning algorithm that has shown to be very effective when classifying images described by local features. In this paper, we present a combined unsupervised and supervised classification tree based on local descriptors and the KNN algorithm. The proposed tree outperforms the classification accuracy of the exact KNN algorithm.

Automatic photo annotation task aims to describe the semantic content by detecting high level concepts in order to further facilitate concept based video retrieval. Most of existing approaches are based on independent semantic concept detectors without considering the contextual correlation between concepts. This drawback has its impact over the efficiency of such systems. Recently, harnessing contextual information to improve the effectiveness of concepts detection becomes a promising direction in such field. In this paper, we propose a new contextbased annotation refinement process. For this purpose, we define a new semantic measure called “Second Order Co-occurence Flickr context similarity” (SOCFCS) which aims to extract the semantic context correlation between two concepts by exploring Flickr resources (Flickr related-tags). Our measure is an extension of FCS measure by taking into consideration the FCS values of common Flickr related-tags of the two target concepts. Our proposed measure is applied to build a concept network which models the semantic context inter-relationships among concepts. A Random Walk with Restart process is performed over this network to refine the annotation results by exploring the contextual correlation among concepts. Experimental studies are conducted on ImageCLEF 2011 Collection containing 10000 images and 99 concepts. The results demonstrate the effectiveness of our proposed approach.

Publication in the conference proceedings of EUSIPCO, Poznan, Poland, 2007