Search

Background In laparoscopic surgery, image quality can be severely degraded by surgical smoke, which not only introduces errors for the image processing algorithms (used in image guided surgery), but also reduces the visibility of the observed organs and tissues. To overcome these drawbacks, this work aims to remove smoke in laparoscopic images using an image preprocessing method based on a variational approach. Methods In this paper, we present the physical smoke model where the degraded image is separated into two parts: direct attenuation and smoke veil and propose an efficient variational-based desmoking method for laparoscopic images. To estimate the smoke veil, the proposed method relies on the observation that smoke veil has low contrast and low inter-channel differences. A cost function is defined based on this prior knowledge and is solved using an augmented Lagrangian method. The obtained smoke veil is then subtracted from the original degraded image, resulting in the direct attenuation part. Finally, the smoke free image is computed using a linear intensity transformation of the direct attenuation part. Results The performance of the proposed method is evaluated quantitatively and qualitatively using three datasets: two public real smoked laparoscopic datasets and one generated synthetic dataset. No-reference and reduced-reference image quality assessment metrics are used with the two real datasets, and show that the proposed method outperforms the state-of-the-art ones. Besides, standard full-reference ones are employed with the synthetic dataset, and indicate also the good performance of the proposed method. Furthermore, the qualitative visual inspection of the results shows that our method removes smoke effectively from the laparoscopic images. Conclusion All the obtained results show that the proposed approach reduces the smoke effectively while preserving the important perceptual information of the image. This allows to provide a better visualization of the operation field for surgeons and improve the image guided laparoscopic surgery procedure.

We propose a novel Hierarchical Feature Model (HFM) for multi-target tracking. The traditional tracking algorithms use handcrafted features that cannot track targets accurately when the target model changes due to articulation, illumination intensity variation or perspective distortions. Our HFM explore deep features to model the appearance of targets. Then, we use an unsupervised dimensionality reduction for sparse representation of the feature vectors to cope with the time-critical nature of the tracking problem. Subsequently, a Bayesian filter is adopted as the tracker and a discrete combinatorial optimization is considered for target association. We compare our proposed HFM against 4 state-of-the-art trackers using 4 benchmark datasets. The experimental results show that our HFM outperforms all the state-of-the-art methods in terms of both Multi Object Tracking Accuracy (MOTA) and Multi Object Tracking Precision (MOTP).

One of the most challenging issues in stereoscopic image quality assessment (SIQA) is how to effectively model the binocular behavior of the human visual system (HVS). The latter has a great impact on the perceptual 3D quality. In this paper, we propose a SIQA metric accounting for binocular combination properties and disparity information. Instead of computing the quality of the left and the right views separately, the proposed metric predicts the quality of a cyclopean image so as to have a good consistency with 3D human perception. The cyclopean image is synthesized based on the local entropy and the visual saliency of each view with the aim to simulate the phenomena of binocular fusion/rivalry. A 2D IQA metric is employed to assess the quality of both the cyclopean image and the disparity map. The obtained scores are used to derive the 3D quality score thanks to a pooling stage. Experimental results on three public 3D IQA databases show that the proposed method outperforms many other state-of-the-art SIQA methods, and achieves high prediction accuracy on these databases.

One of the most challenging issues in stereoscopic image quality assessment (IQA) is how to effectively model the binocular behaviors of the human visual system (HVS). The latter has a great impact on the perceptual stereoscopic 3D (S3D) quality. This paper presents a stereoscopic IQA metric based on the properties of the HVS. Instead of measuring the quality of the left and the right views separately, the proposed method predicts the quality of a cyclopean image to ensure that the overall S3D quality is as close as possible to the binocular vision. The cyclopean image is synthesized based on the local entropy of each view with the aim to simulate the phenomena of the binocular rivalry/suppression. A 2D IQA metric is employed to assess the quality of both the cyclopean image and the disparity map. Additionally, the quality of the cyclopean image is modulated according to the visual importance of each pixel defined by the just noticeable difference (JND). Finally, the 3D quality score is derived by combining the quality estimates of the cyclopean image and disparity map. Experimental results show that the proposed method outperforms many other state-of-the-art SIQA methods in terms of prediction accuracy and computational efficiency.

A new implementation of point level sensor fabricated using the oblique multimode silica fibers is proposed and demonstrated. Based on Fresnel reflection, all the incident lights are reflected back by the oblique sensor header when the sensor is in the air, and a proportion of lights are leaked when the sensor is in the liquid. In order to get the maximum reflected light intensity modulation, the calculation model of the sensor is proposed to optimize the oblique angle. The simulation of light path and light intensity detection by means of finite element analysis demonstrates that the sensor is evidently sensitive to liquid and the reflected lights can be coupled into the receiving fibers, which makes easy detection. Gradient and continuous level experiments indicate that the sensor has the measurement accuracy of ±0.6 mm and good stability.

Background and Objective: For more than a decade, computer-assisted surgical systems have been helping surgeons to plan liver resections. The most widespread strategies to plan liver resections are: drawing traces in individual 2D slices, and using a 3D deformable plane. In this work, we propose a novel method which requires low level of user interaction while keeping high flexibility to specify resections. Methods: Our method is based on the use of Bézier surfaces, which can be deformed using a grid of control points, and distance maps as a base to compute and visualize resection margins (indicators of safety) in realtime. Projection of resections in 2D slices, as well as computation of resection volume statistics are also detailed. Results: The method was evaluated and compared with state-of-the-art methods by a group of surgeons (n = 5, 5–31 years of experience). Our results show that theproposed method presents planning times as low as state-of-the-art methods (174 s median time) with high reproducibility of results in terms of resected volume. In addition, our method not only leads to smooth virtual resections easier to perform surgically compared to other state-of-the-art methods, but also shows superior preservation of resection margins. Conclusions: Our method provides clinicians with a robust and easy-to-use method for planning liver resections with high reproducibility, smoothness of resection and preservation of resection margin. Our results indicate the ability of the method to represent any type of resection and being integrated in real clinical work-flows. © 2017 The Authors. Published by Elsevier Ireland Ltd. This is an open access article under the CC BY-NC-ND license. (http://creativecommons.org/licenses/by-nc-nd/4.0/)

Capsule endoscopy, which uses a wireless camera to take images of the digestive track, is emerging as an alternative to traditional wired colonoscopy. A single examination produces a sequence of approximately 50,000 frames. These sequences are manually reviewed, which is time consuming and typically takes about 45-90 minutes and requires the undivided concentration of the reviewer. In this paper, we propose a novel capsule video summarization framework using sparse coding and dictionary learning in feature space. Video frames are clustered into superframes based on power spectral density, and cluster representative frames are used for video summarization. Handcrafted and deep features that are extracted for representative frames are sparse coded using a learned dictionary. Sparse coded features are later used for training SVM classifier. The proposed method was compared with state-of-the-art methods based on sensitivity and specificity. The achieved results show that our proposed framework provides robust capsule video summarization without losing informative segments. © 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

In this paper, we present a comprehensive analysis and comparison of state-of-the-art Contrast Enhancement (CE) evaluation metrics. In this work, we developed a new database consisting of 182 images. The subjective experiments were performed to obtain a preference rating from different observers for the enhanced images using six CE methods selected from different representative categories. The quality of the enhanced images is measured with most commonly used CE evaluation metrics. We provide the ranking of images based on perceptual preference as well as objective quality metrics scores. We show that some of the metrics used for the enhancement evaluation are not consistent with the human subjective scores. This new database, named as Contrast Enhancement Evaluation Database (CEED2016), is made publicly available to the research community at http://wwwl2ti. univ-paris13.fr/site/index.php/en/CEED2016/ and is expected to be a contribution to the area of Image Quality Assessment (IQA) and particularly for image quality enhancement evaluation.

International audience; The just noticeable difference (JND) notion reflects the maximum tolerable distortion. It has been extensively used for the optimization of 2D applications. For stereoscopic 3D (S3D) content, this notion is different since it relies on different mechanisms linked to our binocular vision. Unlike 2D, 3D-JND models appeared recently and the related literature is rather limited. These models can be used for the sake of compression and quality assessment improvement for S3D content. In this paper, we propose a deep and comparative study of the existing 3D-JND models. Additionally, in order to analyze their performance, the 3D-JND models have been integrated in recent metric dedicated to stereoscopic image quality assessment (SIQA). The results are reported on two widely used S3D image databases.

Multi-target tracking is one of the most challenging tasks in computer vision. Several complex techniques have been proposed in the literature to tackle the problem. The main idea of such approaches is to find an optimal set of trajectories within a temporal window. The performance of such approaches are fairly good but their computational complexity is too high making them unpractical. In this paper, we propose a novel tracking-by-detection approach in a Bayesian filtering framework. The appearance of a target is modeled through HoG descriptor and the critical problem of target association is solved through combinatorial optimization. It is a simple yet very efficient approach and experimental results show that it achieves state-of-the-art performance in real time.

Multispectral face images captured in more than one spectra is known to provide reliable person verification, especially in varying illumination conditions. In this paper, we present an extended multi-spectral face recognition framework by combining the face images captured in six different spectra consisting of 425nm,475nm,525nm,570nm,625nm, and 680nm. We propose a novel image fusion scheme that combines the information from different spectrum of multispectral face images. The proposed image fusion scheme first selects two images from set of all spectral images based on the highest information quantified using entropy measure. Two selected images are combined by decomposing them using Discrete Wavelet Transform (DWT) to get the sub-bands that are fused using weighted sum rule. The weights are computed automatically on each of these sub-bands by measuring the dependency using correlation and wavelet energy. Extensive experiments are carried out on a newly constructed exclusive multispectral face database using a commercial multispectral sensor SpectraCamTM from Pixelteq company. Extensive experiments are carried out on our database to present both qualitative and quantitative results of the proposed image fusion scheme. The comprehensive comparative analysis is performed by comparing the performance of the proposed scheme with four different state-of-the-art schemes. The obtained results have justified the efficacy of the proposed system for robust multispectral face recognition.

Colon cancer and precancerous colon lesions are major health problems. The most frequent precancerous cancer lesions are polyps characterized by abnormal tissue growth in the human bowel. There are also anomalies such as inflammation and bleeding area. Colon capsule endoscopy (CCE) is a recent promising technology which enables to obtain videos of the inside of the intestine via an on board digital camera. The small capsule ingested by the patient is recuperated after it passes through the whole gastrointestinal track. Expert gastroenterologists analyze the video sequence visually in order to find out frames containing abnormalities related to cancer. This process is labor intensive and time consuming. In this paper we propose an algorithm that provides automatic video analysis in order to classify tissue regions in images into two categories: normal and abnormal. In order to achieve high correct classification rates, we first pre-process the image data by removing the noise in it and by normalizing the intensity values of the image pixels. Next, we use the well-known SIFT descriptor algorithm with Bag of Feature (BoF) approach for feature extraction. For training, Support Vector Machine (SVM) is trained on the extracted features using the training dataset. Finally a testing data set is used to assess the performance of the proposed algorithm. The early experimental results are very encouraging and show high correct classification rates, reaching up to 98.25% for images with polyps. The novelty of the proposed algorithm is the combination of using specific SIFT features with Bof and the vignetting correction to capture the local characteristics of the abnormalities in capsule video endoscopy images.

Although face recognition is a widely accepted form of biometrics due to its ease of capture, it still suffers from out-of-focus imaging due to the limited depth-of-field found in conventional 2D imaging systems. In order to overcome such a challenge, light field cameras have been successfully employed in biometrics. The number of depth images obtained from a light field camera can be optimally used by fusing a number of these images or selecting the best focus image from the set of images. In this work, we employ a method to pre-select the images when more than two images corresponding to different focus are present. Further, we evaluate various fusion techniques against the all − in − focus image obtained from light field imaging for the verification performance in face recognition systems. The evaluation of various schemes is performed on one of the largest light field face databases consisting of 80 subjects. The best Equal Error Rate (EER) of 4.14% is observed with the combination of employed pre-selection method and Laplacian Pyramid based fusion approach using both sparse representation and multi-scale transform.

Visual information obtained from endoscopy in laparoscopic surgery plays an important role in surgery navigation and provides a mean for efficient and effective image-based instrument tracking. Instrument tracking based on monocular vision is a well-researched field, in recent years, stereo vision based techniques have attracted the interests of the academic community. In this work, we propose two frameworks for instrument tracking based on stereo vision. We implemented these two frameworks and evaluated the tracking accuracy via comparison with an optical tracking system (OTS) employed regularly in surgery. Our results show the high potential of these methods for tracking instruments in laparoscopic surgical procedures.

This paper presents an objective keyword selection method called visualness with Lesk disambiguation (VLD) for describing educational videos with semantic tags. It extends the work on automatically extracting and associating meaningful keywords carried out in ‘semantic tags for lecture videos’ for efficient indexing and retrieval. VLD uses lecture videos and surrogates documents such as lecture transcripts to extract potential candidate keywords. The candidate keywords undergo a series of selection process extracting fewer but more meaningful keywords based on word sense disambiguation (WSD) and visual similarity. The objective metric then selects top ranking keywords by employing a rank cut-off method. The proposed metric is validated by comparing the automatically selected keywords to those obtained manually, suggesting that the words selected by the proposed objective metric correlate highly with those selected by viewers. The results are further compared to traditional term frequency inverse document frequency (TF-IDF) and state-of-the-art latent Dirichlet allocation (LDA) method, with an improved accuracy of 68.18% on 30 lecture videos.

Many research works have been developed for stereo image compression purpose by focusing on the disparity compensation technique. For this reason, a great attention should be paid to the generation of the disparity-compensated residual image. Generally, the residual image is computed through a simple substruction of the disparity-compensated reference image from the target one. In this paper, we investigate two techniques for optimizing the computation of the residual image. The obtained results confirm the benefits of these optimization approaches in the context of stereo image coding.

Today's eLearning websites are heavily loaded with multimedia contents, which are often unstructured, unedited, unsynchronized, and lack inter-links among different multimedia components. Hyperlinking different media modality may provide a solution for quick navigation and easy retrieval of pedagogical content in media rich eLearning platforms. Creating hyperlinks in media rich platforms often requires video and presentation slides annotation. Manually annotating different media is a challenging, laborious, prone to errors, and a time-consuming task. In this paper, an approach combining state-of-the-art SIFT together with wavelet energy transform is proposed for automatic lecture video annotations for the hyper interactive presenter (HIP). New video annotations are implemented automatically providing the needed random access in lecture videos within the HIP platform. A preliminary study of the usefulness of the system is carried out, and the initial results are suggesting that 70% of the students opted for using HIP as their preferred eLearning platform at Gj⊘vik university college (GUC).

Many research works have been developed for stereo image compression purpose where most of them aim at encoding a reference image, a residual one and a disparity map. While the disparity field is often losslessly encoded, we are mainly interested in this paper in the bit allocation problem between the reference and residual images. Generally, the bit allocation is expressed as an optimization problem which involves the computation of the operational rate-distortion (RD) functions for all the wavelet subbands and for different quantization steps. However, this strategy is computationally intensive. To solve this problem, we consider the uniform scalar quantization of the wavelet subbands of both images modeled by a Generalized Gaussian distribution. Thanks to recent approximations of the entropy and distortion functions, we develop an optimal and fast bit allocation method. The obtained results confirm the efficiency of the proposed bit allocation method in the context of stereo image coding.

Computer vision algorithms such as image compression, image segmentation, context aware image resizing, image quality assessment and target detection, are designed with the aim of replicating our visual system. Human eye fixations recorded by using an eye tracker are typically used as a criterion for the optimisation of such algorithms. In this paper, we propose a method to analyze the spatio-temporal nature of fixations data for different observers. By studying the correlation matrix constructed based on the fixations data of different observers viewing the same image, it was found that 21 percent of the data can be accounted by one eigenvector. A visual inspection of this vector, shows that it represents the time sequence and locations of the objects in the image that observers deem as salient. The eigenvectors are used as a benchmark for evaluating the spatio-temporal performance of Itti's classic visual saliency model. Based on the results obtained from a comprehensive publicly available dataset, we show that the proposed method can be used as ground truth for evaluating the spatio-temporal performance of saliency models. Furthermore, it can be used to provide salient locations and their time sequence for a real-time image compression application with promising results.

This paper presents a novel full-reference Stereo Image Quality Assessment (SIQA) measure based on well understood characteristics of the human visual system (HVS), namely contrast sensitivity and frequency and directional selectivity. Additionally, the proposed metric takes into account the stereo interplay between the two views, where one view may affect our perception of the overall quality of the stereo image pair. Therefore, a Binocular Just Noticeable Difference (BJND) model is used to compute the distortion visibility threshold, and the binocular suppression theory is considered in the proposed metric. The scored 3D LIVE IQA database is used to evaluate the correlation of the proposed metric with the DMOS subjective score provided by the database. The obtained experimental results show that the proposed metric correlates much better with the DMOS score than the state-of-the-art metrics do.

The usage of non-scripted lecture videos as a part of learning material is becoming an everyday activity in most of higher education institutions due to the growing interest in flexible and blended education. Generally these videos are delivered as part of Learning Objects (LO) through various Learning Management Systems (LMS). Currently creating these video learning objects (VLO) is a cumbersome process. Because it requires thorough analyses of the lecture content for meta-data extraction and the extraction of the structural information for indexing and retrieval purposes. Current e-learning systems and libraries (such as libSCORM) lack the functionally for exploiting semantic content for automatic segmentation. Without the additional meta-data and structural information lecture videos thus do not provide the required level of interactivity required for flexible education. As a result, they fail to captivate students' attention for long time and thus their effective use remains a challenge. Exploiting visual actions present in non-scripted lecture videos can be useful for automatically segmenting and extracting the structure of these videos. Such visual cues help identify possible key frames, index points, key events and relevant meta-data useful for e-learning systems, video surrogates and video skims. We therefore, propose a multi-model action classification system for four predefined actions performed by instructor in lecture videos. These actions are writing, erasing, speaking and being idle. The proposed approach is based on human shape and motion analysis using motion history images (MHI) at different temporal resolutions allowing robust action classification. Additionally, it augments the visual features classification based on audio analysis which is shown to improve the overall action classification performance. The initial experimental results using recorded lecture videos gave an overall classification accuracy of 89.06\%. We evaluated the performance of our approch to template matching using correlation and similitude and found nearly 30\% improvement over it. These are very encouraging results that prove the validity of the approach and its potential in extracting structural information from instructional videos.

In this paper, we address the issues pertaining to segmentation and recognition of cursive handwritten text from chalkboard lecture videos. Recognizing handwritten text is a challenging problem in instructor-led lecture video. The task gets even tougher with varying handwriting styles and blackboard type. Unlike handwritten text on whiteboard and electronic boards, chalkboard represents serious challenges such as, lack of uniform edge density, weak chalk contrast against blackboard and leftover chalk dust noise as a result of erasing -- and many others. Moreover, the varying color of boards and the illumination changes within the video makes it impossible to use trivial thresholding techniques, for the extraction of content. Many universities throughout the world still heavily rely on chalkboard as a mode of instruction. Therefore, recognizing these lecture content will not only aid in indexing and retrieval applications but will also help understand high level video semantics, useful for Multi-media Learning Objects (MLO). In order to encounter those adversaries, we here propose a system for segmentation and recognition of cursive handwritten text from chalkboard lecture videos. We first create a foreground model to segment background blackboard. We then segment the text characters using one-dimensional vertical histogram. Later, we extract gradient based features and classify those characters using an SVM classifier. We obtained an encouraging accuracy of 86.28% on 5-fold cross validation.

With the growing interest in flexible and blended education, the usage of recorded lecture videos as a part of learning material, is becoming an everyday activity in most of the higher education institutions. Flexible education provides students with the opportunity to select how, when, and where to study compared to ordinary on-campus education. Additionally, lecture videos contain most of the instructional content. The content of these videos are in general non-scripted and unedited, and thus do not provide the required level of interactivity. Therefore, these videos fail to captivate students' attention for a long time, and thus their effective use remains a challenge. In this regard, Multimedia Learning Object (MLO) can play an important role in future flexible education. MLOs are media rich documents that have pedagogical values encapsulated with all needed features such as navigation structures and surrogates. We propose in this paper a framework for interactive MLOs in distance and blended education. The framework supports creation, storage, distribution, and evaluation of automatically extracted learning objects from digital media. The framework not only conforms to industry standards, but also provide novel methods and techniques to develop Video Learning Objects (VLOs).

In this paper, we address the problem of content classification for chalkboard images. Unlike document images, classifying chalkboard content into different categories is a challenging task. The task gets even tougher with varying handwriting styles and arbitrary drawings. We therefore, propose a tool with a set of functions to distinguish equations from text and figures. A hybrid solution is proposed, consisting of, the state of the art support vector machine (SVM) and optical character recognition (OCR) for this purpose. Prior to feature extraction and classification, some preprocessing steps are performed to remove noise and to enhance the chalk contrast. Our initial experiment shows promising results of above 85% accuracy for chalkboard images. We later on applied our algorithm to MNIST database of handwritten digits, our created handwritten lower-case and upper-case characters and basic mathematical operators and obtained 96% accuracy.

Object tracking is a very active research field nowadays due to its numerous potential applications in video surveillance, e.g. human activity analysis, traffic monitoring. Recently the focus has been on multi-target tracking (MTT). While considering multiple target tracking applications, ‘multi-target occlusion’ is a common and challenging problem that needs to be addressed. Most of the proposed solutions for this problem require camera calibration parameters that make them impractical for outdoor video surveillance applications. To address this problem we propose in this paper a probabilistic approach that does not require camera calibration for multiple camera collaboration. Initial simulation results prove the validity of the proposed approach.

International audience; A video sequence is more than a sequence of still images. It contains a strong spatial-temporal correlation between the regions of consecutive frames. The most important characteristic of videos is the perceived motion foreground objects across the frames. The motion of foreground objects dramatically changes the importance of the objects in a scene and leads to a different saliency map of the frame representing the scene. This makes the saliency analysis of videos much more complicated than that of still images. In this paper, we investigate saliency in video sequences and propose a novel spatiotemporal saliency model devoted for video surveillance applications. Compared to classical saliency models based on still images, such as Itti's model, and space-time saliency models, the proposed model is more correlated to visual saliency perception of surveillance videos. Both bottom-up and top-down attention mechanisms are involved in this model. Stationary saliency and motion saliency are, respectively, analyzed. First, a new method for background subtraction and foreground extraction is developed based on content analysis of the scene in the domain of video surveillance. Then, a stationary saliency model is setup based on multiple features computed from the foreground. Every feature is analyzed with a multi-scale Gaussian pyramid, and all the features conspicuity maps are combined using different weights. The stationary model integrates faces as a supplement feature to other low level features such as color, intensity and orientation. Second, a motion saliency map is calculated using the statistics of the motion vectors field. Third, both motion saliency map and stationary saliency map are merged based on center-surround framework defined by an approximated Gaussian function. The video saliency maps computed from our model have been compared to the gaze maps obtained from subjective experiments with SMI eye tracker for surveillance video sequences. The results show strong correlation between the output of the proposed spatiotemporal saliency model and the experimental gaze maps.

When viewing video sequences, the human visual system (HVS) tends to focus on the active objects. These are perceived as the most salient regions in the scene. Additionally, human observers tend to predict the future positions of moving objects in a dynamic scene and to direct their gaze to these positions. In this paper we propose a saliency detection model that accounts for the motion in the sequence and predicts the positions of the salient objects in future frames. This is a novel technique for attention models that we call Predictive Saliency Map (PSM). PSM improves the consistency of the estimated saliency maps for video sequences. PSM uses both static information provided by static saliency maps (SSM) and motion vectors to predict future salient regions in the next frame. In this paper we focus only on surveillance videos therefore, in addition to low-level features such as intensity, color and orientation we consider high-level features such as faces as salient regions that attract naturally viewers attention. Saliency maps computed based on these static features are combined with motion saliency maps to account for saliency created by the activity in the scene. The predicted saliency map is computed using previous saliency maps and motion information. The PSMs are compared with the experimentally obtained gaze maps and saliency maps obtained using approaches from the literature. The experimental results show that our enhanced model yields higher ability to predict eye fixations in surveillance videos.

We are privileged to have organized the 2nd European Workshop on Visual Information Processing - EUVIP2010, held at the Telecom ParisTech, in Paris, France. This event was created after a discussion we had with Prof. Azeddine Beghdadi in 2008, about the rapidly growing importance of visually inspired techniques for visual media processing. We are hoping this will be one of a long series of successful workshops to come arranged each year in a different country or hosted by a different institution.

The Human Visual System (HVS) tends to focus on specific regions of viewed images or video frames, this is done effortlessly, instantly and unconsciously. These are called salient regions and form a saliency map, which could be used to improve a number of image and video processing techniques. In this paper, we propose a novel non-reference objective video quality metric based on the saliency map to improve the estimation of the perceived video quality. This metric estimates the degree of blur and blockiness in each video frame from the impaired video only, and uses it with the saliency map to derive a weighting function. The latter is used to modulate the contribution of the pixel differences to the final quality score. The salient regions of the videos are automatically computed using our video saliency model. A psychophysical experiment is conducted to estimate the perceived quality of the impaired videos. The results of this subjective test are compared to the scores obtained with the proposed objective metric. The objective and subjective scores are found to be highly correlated, which shows that our metric correctly estimates the perceived quality of a video.

The perception of video is different from that of image because of the motion information in video. Motion objects lead to the difference between two neighboring frames which is usually focused on. By far, most papers have contributed to image saliency but seldom to video saliency. Based on scene understanding, a new video saliency detection model with multi-features is proposed in this paper. First, background is extracted based on binary tree searching, then main features in the foreground is analyzed using a multi-scale perception model. The perception model integrates faces as a high level feature, as a supplement to other low-level features such as color, intensity and orientation. Motion saliency map is calculated using the statistic of the motion vector field. Finally, multi-feature conspicuities are merged with different weights. Compared with the gaze map from subjective experiments, the output of the multi-feature based video saliency detection model is close to gaze map.

Under natural viewing conditions humans tend to fixate on specific parts of images that interest them naturally. A saliency map is the map of regions which are more prominent than other regions in terms of low level image properties such as intensity, color, and orientation. With some modifications it can be used to simulate the natural human fixations also known as gaze. There are numerous applications in the field of engineering, marketing and art that can benefit from understanding the human visual fixation such as image quality evaluation, label design etc. The objective of this research is to understand the factors that influence the saliency map and gaze map and to modify the saliency map in order to make it similar to the gaze map. Eye movements of 20 test subjects were captured using eye tracking equipment. Unsurprisingly it was observed that whenever faces were present in the images, the observers gazed at them. Three different algorithms Saliency by Itti et al., GAFFE by Rajashekara et al., GBVS by Harel et al. were modified by addition of face saliency module to generate modified maps corresponding to each of the algorithms. Experimental gaze maps were analyzed and compared with modified saliency map, modified GAFFE map and modified GBVS map.

This is the copy of journal's version originally published in Proc. SPIE 6811. Reprinted with permission of SPIE: http://spie.org/x10.xml?WT.svl=tn7 There is growing interest in video-based solutions for people monitoring and counting in business and security applications. Compared to classic sensor-based solutions the video-based ones allow for more versatile functionalities, improved performance with lower costs. In this paper, we propose a real-time system for people counting based on single low-end non-calibrated video camera. The two main challenges addressed in this paper are: robust estimation of the scene background and the number of real persons in merge-split scenarios. The latter is likely to occur whenever multiple persons move closely, e.g. in shopping centers. Several persons may be considered to be a single person by automatic segmentation algorithms, due to occlusions or shadows, leading to under-counting. Therefore, to account for noises, illumination and static objects changes, a background substraction is performed using an adaptive background model (updated over time based on motion information) and automatic thresholding. Furthermore, post-processing of the segmentation results is performed, in the HSV color space, to remove shadows. Moving objects are tracked using an adaptive Kalman filter, allowing a robust estimation of the objects future positions even under heavy occlusion. The system is implemented in Matlab, and gives encouraging results even at high frame rates. Experimental results obtained based on the PETS2006 datasets are presented at the end of the paper.

We present a novel and efficient technique for the restoration of color images which are highly corrupted with impulse noise. This is a detection-estimation based approach in which outliers are first detected using a Teager-like operator followed by a locally adaptive threshold. Center pixels whose "energy" exceeds some threshold are replaced with the local marginal median. Simulation results show the superior performance of the proposed filtering algorithm compared to the renowned vector median (VM) and generalized vector directional filter (GVDF), which are commonly used for color image restoration. Monte Carlo simulations show the edge preservation and impulse noise attenuation capabilities of the proposed technique. The efficiency of the algorithm stems from its simple arithmetic operations compared with more demanding ones, e.g. computation of distances and angles in the case of VMF and GVDF, respectively.

In this paper we propose a new approach to shape recognition based on the wavelet transform modulus maxima, and we apply it to the problem of content-based indexing and retrieval of fish contours. The description scheme and the similarity measure developed take into consideration the way our visual system perceives objects and compares them. The proposed scheme is invariant to translation, rotation, scale change and to noise corruption. Moreover, this description scheme allows accurate reconstruction of the shape boundary from the feature vector used to describe it. The experimental results and comparisons show the performance of the proposed technique.

In this paper, we present a novel approach for describing and estimating similarity of shapes. The target application is content-based indexing and retrieval over large image databases. The shape feature vector is based on the efficient indexing of high curvature (HCP) points which are detected at different levels of resolution of the wavelet transform modulus maxima decomposition. The scale information, together with other topological information of those high curvature points are employed in a sophisticated similarity algorithm. The experimental results and comparisons show that the technique isolates efficiently similar shapes from a large database and reflects adequately the human similarity perception. The proposed algorithm also proved efficient in matching heavily occluded contours with their originals and with other shape contours in the database containing similar portions.

In this paper we use the Teager energy driven non-linear diffusion process to reduce the influence of noise in the watershed segmentation of an image. We show that the performance of this non-linear diffusion process is better than the performance of the traditional gradient driven non-linear diffusion process.

There is a finite number of different weighted order statistic (WOS) filters of a fixed length N. However, even for relatively small values of N, one cannot immediately see if two given WOS filters are the same by simply looking at the weights and the thresholds. This problem is addressed in this paper. We define two WOS filters to be equivalent (the same) if they produce the same output for arbitrary inputs. We shall show that the solution requires the use of integer linear programming and next develop a hierarchical heuristical procedure which may provide a much quicker solution to the given problem. The hierarchy starts with simple checks and proceeds to more and more complicated tests. The procedure is exited as soon as a definite conclusion is reached.

A study on the use of vector rational interpolation for the estimation of erroneously received motion fields of MPEG-2 predictively coded frames is undertaken in this paper, aiming further at error concealment (EC). Various rational interpolation schemes have been investigated, some of which are applied to different interpolation directions. One scheme additionally uses the boundary matching error and another one attempts to locate the direction of minimal/maximal change in the local motion field neighborhood. Another one further adopts bilinear interpolation principles, whereas a last one additionally exploits available coding mode information. The methods present temporal EC methods for predictively coded frames or frames for which motion information pre-exists in the video bitstream. Their main advantages are their capability to adapt their behavior with respect to neighboring motion information, by switching from linear to nonlinear behavior, and their real-time implementation capabilities, enabling them for real-time decoding applications. They are easily embedded in the decoder model to achieve concealment along with decoding and avoid post-processing delays. Their performance proves to be satisfactory for packet error rates up to 2% and for video sequences with different content and motion characteristics and surpass that of other state-of-the-art temporal concealment methods that also attempt to estimate unavailable motion information and perform concealment afterwards.

Rational filters are extended to multichannel signal process- ing and applied to image interpolation. Two commonly used decimation schemes are considered: a rectangular grid and a quincunx grid. For each decimation lattice, we propose a number of adaptive resampling algorithms based on the vector rational filter (VRF). These algorithms exhibit desirable properties such as edge and detail preservation and accurate chromaticity estimation. In these approaches, color image pix- els are considered as three-component vectors in the color space. Therefore, the inherent correlation that exists between the different color components is not ignored. This leads to better image quality compared to that obtained by componentwise or marginal processing. Extensive simulations show that multichannel image processing with the proposed algorithms (VRFL) and (VRFd) based on l p-norm and directional pro- cessing, respectively; significantly outperform linear and some nonlinear techniques, e.g., vector FIR median hybrid filters (VFMH). Some images interpolated using VRFL and VRFd are presented for qualitative compari- son. These images are free from blockiness and jaggedness, confirming the quantitative results. © 1999 Society of Photo-Optical Instrumentation Engineers. (S0091-3286(99)00105-1)

In this paper, we present an unsharp masking-based approach for noise smoothing and edge enhancing in multichannel images. The proposed structure is similar to the conventional unsharp masking structure, however, the enhancement is allowed only in the direction of maximal change and the enhancement parameter is computed as a nonlinear function of the rate of change. The proposed scheme enhances the true details, limits the overshoot near sharp edges and attenuates noise in flat areas. Moreover the use of the control function eliminates the need for the subjective coefficient /spl lambda/ used in the conventional unsharp masking technique. Simulations results show that the processed image presents sharp edges which makes it more pleasant to the human eye. Moreover, the amount of noise in the image is clearly reduced.

Content-based image retrieval systems use low-level features like color and texture for image representation. Given these representations as feature vectors, similarity between images is measured by computing distances in the feature space. Unfortunately, these low-level features cannot always capture the high-level concept of similarity in human perception. Relevance feedback tries to improve the performance by allowing iterative retrievals where the feedback information from the user is incorporated into the database search. We present a weighted distance approach where the weights are the ratios of standard deviations of the feature values both for the whole database and also among the images selected as relevant by the user. The feedback is used for both independent and incremental updating of the weights and these weights are used to iteratively refine the effects of different features in the database search. Retrieval performance is evaluated using average precision and progress that are computed on a database of approximately 10,000 images and an average performance improvement of 19% is obtained after the first iteration.

Rational filters are extended to multichannel signal processing and applied to the image interpolation problem. The proposed nonlinear interpolator exhibits desirable properties, such as, edge and details preservation. In this approach the pixels of the color image are considered as 3-component vectors in the color space. Therefore, the inherent correlation which exists between the different color components is not ignored; thus, leading to better image quality than those obtained by component-wise processing. Simulations show that the resulting edges obtained using vector rational filters (VRF) are free from blockiness and jaggedness, which are usually present in images interpolated using especially linear, but also some nonlinear techniques, e.g. vector median hybrid filters (VFMH).

In this paper we propose a new approach to object recognition based on the polygonal approximation of the object contour. The vertices of the polygon are the high curvature points of the contour, selected using the wavelet transform modulus maxima. We associate with this shape description a simple measure to estimate the similarity between objects. The description scheme and the similarity measure proposed take into consideration the way the human visual system perceives objects and compares them. The description scheme and the similarity measure can be used in several applications such as content-based indexing and retrieval, evaluation of segmentation results in the context of MPEG-4, or in classical classification problems. The proposed scheme is invariant to translation, rotation, scale change and noise corruption. Moreover, this description scheme allows accurate reconstruction of the shape boundary from the feature vector used to describe it. The experimental results and comparisons show the performance of the proposed technique.

We present a novel approach to shape similarity estimation based on distance transformation and ordinal correlation. The proposed method operates in three steps: object alignment, contour to multilevel image transformation, and similarity evaluation. This approach is suitable for use in shape classification, content-based image retrieval, and performance evaluation of segmentation algorithms. The two latter applications are addressed in this paper. Simulation results show that in both applications our proposed measure performs quite well in quantifying shape similarity. The scores obtained using this technique reflect well the correspondence between object contours as humans perceive it.

In this paper we propose to incorporate a feedback loop, into the ordinal correlation framework and apply it to shape-based image retrieval. The user's feedback on the relevance of the retrieval results is used to tune the weights of the similarity measure. Statistics from the features of both relevant and irrelevant items are used to estimate the weights. Moreover, the information accumulated from previous retrieval iterations is used in the weights estimation. A simple measure of the discrimination power is proposed and used to show that the relevance feedback increases the capability of the ordinal correlation scheme to discriminate between relevant and irrelevant objects.