Search

International audience; Many objective quality metrics have been proposed over the years to automate the task of subjective quality assessment. However, few of them are designed for 3D graphical contents with appearance attributes; existing ones are based on geometry and color measures, yet they ignore the visual saliency of the objects. In this paper, we combined an optimal subset of geometry-based and color-based features, provided by a state-of-the-art quality metric for 3D colored meshes, with a visual attention complexity feature adapted to 3D graphics. The performance of our proposed new metric is evaluated on a dataset of 80 meshes with diffuse colors, generated from 5 source models corrupted by commonly used geometry and color distortions. With our proposed metric, we showed that the use of the attentional complexity feature brings a significant gain in performance and better stability.

Nowadays, 360�� video/image has been increasingly popular and drawn great attention. The spherical viewing range of 360�� video/image accounts for huge data, which pose the challenges to 360�� video/image processing in solving the bottleneck of storage, transmission, etc. Accordingly, the recent years have witnessed the explosive emergence of works on 360�� video/image processing. In this paper, we review the state-of-the-art works on 360�� video/image processing from the aspects of perception, assessment and compression. First, this paper reviews both datasets and visual attention modelling approaches for 360�� video/image. Second, we survey the related works on both subjective and objective visual quality assessment (VQA) of 360�� video/image. Third, we overview the compression approaches for 360�� video/image, which either utilize the spherical characteristics or visual attention models. Finally, we summarize this overview paper and outlook the future research trends on 360�� video/image processing., Submitted to IEEE J-STSP SI of Perception-driven 360-degree video processing as an Invited Overview Paper

Sonar imagery plays a significant role in oceanic applications since there is little natural light underwater, and light is irrelevant to sonar imaging. Sonar images are very likely to be affected by various distortions during the process of transmission via the underwater acoustic channel for further analysis. At the receiving end, the reference image is unavailable due to the complex and changing underwater environment and our unfamiliarity with it. To the best of our knowledge, one of the important usages of sonar images is target recognition on the basis of contour information. The contour degradation degree for a sonar image is relevant to the distortions contained in it. To this end, we developed a new no-reference contour degradation measurement for perceiving the quality of sonar images. The sparsities of a series of transform coefficient matrices, which are descriptive of contour information, are first extracted as features from the frequency and spatial domains. The contour degradation degree for a sonar image is then measured by calculating the ratios of extracted features before and after filtering this sonar image. Finally, a bootstrap aggregating (bagging)-based support vector regression module is learned to capture the relationship between the contour degradation degree and the sonar image quality. The results of experiments validate that the proposed metric is competitive with the state-of-the-art reference-based quality metrics and outperforms the latest reference-free competitors.

Free-viewpoint video, as the development direction of the next-generation video technologies, uses the depth-image-based rendering (DIBR) technique for the synthesis of video sequences at viewpoints, where real captured videos are missing. As reference videos at multiple viewpoints are not available, a blind reliable real-time quality metric of the synthesized video is needed. Although no-reference quality metrics dedicated to synthesized views successfully evaluate synthesized images, they are not that effective when evaluating synthesized video due to additional temporal flicker distortion typical only for video. In this paper, a new fast no-reference quality metric of synthesized video with synthesis distortions is proposed. It is guided by the fact that the DIBR-synthesized images are characterized by increased high frequency content. The metric is designed under the assumption that the perceived quality of DIBR-synthesized video can be estimated by quantifying the selected areas in the high-high wavelet subband. The threshold is used to select the most important distortion sensitive regions. The proposed No-Reference Morphological Wavelet with Threshold (NR_MWT) metric is computationally extremely efficient, comparable to PSNR, as the morphological wavelet transformation uses very short filters and only integer arithmetic. It is completely blind, without using machine learning techniques. Tested on the publicly available dataset of synthesized video sequences characterized by synthesis distortions, the metric achieves better performances and higher computational efficiency than the state-of-the-art metrics dedicated to DIBR-synthesized images and videos.

Free-Viewpoint Video (FVV) systems allow the viewers to freely change the viewpoints of the scene. In such systems, view synthesis and compression are the two main sources of artifacts influencing the perceived quality. To assess this influence, quality evaluation studies are often carried out using conventional displays and generating predefined navigation trajectories mimicking the possible movement of the viewers when exploring the content. Nevertheless, as different trajectories may lead to different conclusions in terms of visual quality when benchmarking the performance of the systems, methods to identify critical trajectories are needed. This paper aims at exploring the impact of exploration trajectories (defined as Hypothetical Rendering Trajectories: HRT) on perceived quality of FVV subjectively and objectively, providing two main contributions. Firstly, a subjective assessment test including different HRTs was carried out and analyzed. The results demonstrate and quantify the influence of HRT in the perceived quality. Secondly, we propose a new objective video quality assessment measure to objectively predict the impact of HRT. This measure, based on Sketch-Token representation, models how the categories of the contours change spatially and temporally from a higher semantic level. Performance in comparison with existing quality metrics for FVV, highlight promising results for automatic detection of most critical HRTs for the benchmark of immersive systems., 11 pages, 7 figures

Objective image quality metrics try to estimate the perceptual quality of the given image by considering the characteristics of the human visual system. However, it is possible that the metrics produce different quality scores even for two images that are perceptually indistinguishable by human viewers, which have not been considered in the existing studies related to objective quality assessment. In this paper, we address the issue of ambiguity of objective image quality assessment. We propose an approach to obtain an ambiguity interval of an objective metric, within which the quality score difference is not perceptually significant. In particular, we use the visual difference predictor, which can consider viewing conditions that are important for visual quality perception. In order to demonstrate the usefulness of the proposed approach, we conduct experiments with 33 state-of-the-art image quality metrics in the viewpoint of their accuracy and ambiguity for three image quality databases. The results show that the ambiguity intervals can be applied as an additional figure of merit when conventional performance measurement does not determine superiority between the metrics. The effect of the viewing distance on the ambiguity interval is also shown.

International audience

National audience

This paper provides insights on how to perceptually characterize colored 3D Graphical Contents (3DGC). In this study, pre-defined viewpoints were considered to render static graphical objects. For perceptual characterization, we used visual attention complexity (VAC) measures. Considering a view-based approach to exploit the perceived information, an eye-tracking experiment was conducted using colored graphical objects. Based on the collected gaze data, we revised the VAC measure, suggested in 2D imaging context, and adapted it to 3DGC. We also provided an objective predictor that highly mimics the experimental attentional complexity information. This predictor can be useful in Quality of Experience (QoE) studies: to balance content selection when benchmarking 3DGC processing techniques (e.g., rendering, coding, streaming, etc.) for human panel studies or ad hoc key performance indicator, and also to optimize the user's QoE when rendering such contents.

In recent years, the rapid development of diverse media has been evident in disparate fields such as consumer electronics, automotive infotainment and healthcare software. There is a need for innovative methods to assess user perceived Quality of Experience (QoE), as a proxy for consumer satisfaction with such systems and services. Users emotional state plays a key role in QoE; thus, it is necessary to consider it in user experience evaluation and the design process of stereoscopic 3D video content. In the present article we introduce the use of a specially designed model based on a feedforward Multilayer Perception Artificial Neural Network as an appropriate Machine Learning technique for the estimation of human emotional state while viewing various categories of stereoscopic 3D video content. The goal is to design an emotional state estimator based on direct psychophysiological measurements. The considered psychophysiological signals include heart rate (HR) calculated from an echocardiogram (ECG), electro-dermal activity (EDA), and brain activity (BA) in EEG signals. Participants watched a series of 3D video contents varying in terms of visual quality, while the mentioned psychophysiological signals were recorded, and self-reported subjectively experienced emotions using a Self-Assessment Manikin (SAM) questionnaire. The obtained results show that it is possible to construct such a highly precise estimator of emotional states.

National audience

National audience

High-definition 360 videos encoded in fine quality are typically too large in size to stream in its entirety over bandwidth (BW)-constrained networks. One popular remedy is to interactively extract and send a spatial sub-region corresponding to a viewer’s current field-of-view (FoV) in a head-mounted display (HMD) for more BW-efficient streaming. Due to the non-negligible round-trip-time (RTT) delay between server and client, accurate head movement prediction that foretells a viewer’s future FoVs is essential. Existing approaches are either overly simplistic in modelling and predict poorly when RTT is large, or are over-reliant on data-driven learning, resulting in inflexible models that are not robust to RTT heterogeneity. In this paper, we cast the head movement prediction task as a sparse directed graph learning problem, where three sources of relevant information—a 360 image saliency map, collected viewers’ head movement traces, and a biological head rotation model—are aggregated into a unified Markov model. Specifically, we formulate a constrained optimization problem to minimize an l 2 -norm fidelity term and a sparsity term, corresponding to trace data / saliency consistency and a sparse graph model prior respectively. We solve the problem alternately using a hybrid iterative reweighted least square (IRLS) and Frank-Wolfe optimization strategy. Extensive experiments show that our head movement prediction scheme noticeably outperforms existing proposals across a wide range of RTTs.

With the development of immersive media technologies, omnidirectional video service has been popularized at an accelerated speed due to the sense of presence it offered. Accurately measuring the user’s subjective sense of presence on virtual reality (VR) devices are urgently required by video service providers and VR device manufacturers to further improve their quality of service. So far, there is no standardized methodology available for the conduction of the subjective experiment toward the user’s sense of presence when viewing omnidirectional videos. In this paper, we designed a novel subjective assessment method, which evaluates the presence of omnidirectional videos where the scenes have been previously experienced in the real world by subjects. In this setup, subjects can easily and accurately form their sense of presence when viewing omnidirectional videos using previous observations of the real environment as references. The experimental results demonstrated that the proposed method is able to obtain reliable quantitative results toward the sense of presence.

In this work, we develop a novel no-reference (NR) quality assessment metric for stereoscopic images based on monocular and binocular features, motivated by visual perception properties of the human visual system (HVS) named binocular rivalry and binocular integration. To be more specific, we first calculate the normalized intensity feature maps of right- and left-view images through local contrast normalization, where statistical intensity features are extracted by the histogram of the normalized intensity feature map to represent monocular features. Then, we compute the disparity map of stereoscopic image, with which we extract structure feature map of stereoscopic image based on local binary pattern (LBP). We further extract statistical structure features and statistical depth features from structure feature map and disparity map by histogram to represent binocular features. Finally, we adopt support vector regression (SVR) to train the mapping function from the extracted monocular and binocular features to subjective quality scores. Comparison experiments are conducted on four large-scale stereoscopic image databases and the results demonstrate the promising performance of the proposed method in stereoscopic image quality assessment.

Traditionally, the quality of a multimedia system was mainly assessed through the evaluation of its Quality of Service (QoS) that is by evaluating system parameters such as bandwidth, latency, jitter, throughput, transmission delay, availability, etc. However, these metrics often failed to capture the actual end-user perceived quality, which has prompted the development of the construct of Quality of Experience (QoE), widely understood as an interaction of the technical features of multimedia systems with perceptual, and cognitive/emotional factors involved in the interpretation of those features by users. This chapter addresses the open issues in the field of QoS and QoE assessments. First, the perceptual characteristics of the multiview content are analyzed, and then a survey on the existing approaches for QoS and QoE estimation is performed. The analysis is then focused on the subjective aspects of QoE assessment, by describing the standard methodologies currently used and new trends based on human factors research. Finally, the chapter offers a few guidelines for future research directions in the field.

In submarine and underwater detection tasks, conventional optical imaging and analysis methods are not universally applicable due to the limited penetration depth of visible light. Instead, sonar imaging has become a preferred alternative. However, the capture and transmission conditions in complicated and dynamic underwater environments inevitably lead to visual quality degradation of sonar images, which might also impede further recognition, analysis and understanding. To measure this quality decrease and provide a solid quality indicator for sonar image enhancement, we propose a task- and perception-oriented sonar image quality assessment ( TPSIQA ) method, in which a semi-reference (SR) approach is applied to adapt to the limited bandwidth of underwater communication channels. In particular, we exploit reduced visual features that are critical for both human perception of and object recognition in sonar images. The final quality indicator is obtained through ensemble learning, which aggregates an optimal subset of multiple base learners to achieve both high accuracy and a high generalization ability. In this way, we are able to develop a compact but generalized quality metric using a small database of sonar images. Experimental results demonstrate competitive performance, high efficiency, and strong robustness of our method compared to the latest available image quality metrics.

International audience

In image/video systems, contrast adjustment which manages to enhance visual quality is nowadays an important research topic. Yet very limited struggles have been made to the exploration of visual quality assessment for contrast adjustment. To tackle the issue, this paper proposes a novel reduced-reference (RR) quality metric with the integration of bottom-up and top-down strategies. The former one stems from the recently revealed free energy principle that tells that the human visual system seeks to comprehend an input image via uncertainty removal, while the latter one is toward using the symmetric Kullback–Leibler divergence to compare the histogram of the contrast-altered image with that of the pristine image. The bottom-up and top-down strategies are lastly incorporated to derive the RR contrast-altered image quality measure. A comparison using numerous existing IQA models is carried out on five contrast related databases/subsets in CID2013, CCID2014, CSIQ, TID2008, and TID2013, and experimental results validate the superiority of the proposed technique.

Owning to the recorded light ray distributions, light field contains much richer information and provides possibilities of some enlightening applications, and it has becoming more and more popular. To facilitate the relevant applications, many light field processing techniques have been proposed recently. These operations also bring the loss of visual quality, and thus there is need of a light field quality metric to quantify the visual quality loss. To reduce the processing complexity and resource consumption, light fields are generally sparsely sampled, compressed, and finally reconstructed and displayed to the users. We consider the distortions introduced in this typical light field processing chain, and propose a full-reference light field quality metric. Specifically, we measure the light field quality from three aspects: global spatial quality based on view structure matching, local spatial quality based on near-edge mean square error, and angular quality based on multi-view quality analysis. These three aspects have captured the most common distortions introduced in light field processing, including global distortions like blur and blocking, local geometric distortions like ghosting and stretching, and angular distortions like flickering and sampling. Experimental results show that the proposed method can estimate light field quality accurately, and it outperforms the state-of-the-art quality metrics which may be effective for light field.

User expectations have a crucial impact on the levels of quality of experience (QoE) that they consider acceptable or satisfying. Measuring acceptability and annoyance has mainly been performed in separate or multi-step experiments without any control over participants’ expectations. This paper introduces a simple methodology to obtain the information about both of the entities in a single step and compares several data processing strategies useful for results interpretation. A specifically designed subjective experiment, conducted on compressed videos, has shown that the multi-step procedures could be replaced by our proposed single-step approach, regardless of the viewing conditions, while the novel approach is significantly preferred by observers for its low time requirements and higher intuitiveness. The test has simultaneously proven that user expectations can be altered by the instructions and it is, therefore, possible to simulate different user profiles regardless of the participants’ real habits. The acceptability/annoyance experimental results are also used to benchmark the state-of-the-art objective video quality metrics in predicting acceptability/annoyance of QoE. A case study on the determination of the threshold of acceptability/annoyance for objective quality metrics is conducted, which can be served as a guideline for video streaming service providers.

As the immersive multimedia techniques like Free-viewpoint TV (FTV) develop at an astonishing rate, user's demand for high-quality immersive contents increases dramatically. Unlike traditional uniform artifacts, the distortions within immersive contents could be non-uniform structure-related and thus are challenging for commonly used quality metrics. Recent studies have demonstrated that the representation of visual features can be extracted from multiple levels of the hierarchy. Inspired by the hierarchical representation mechanism in the human visual system (HVS), in this paper, we explore to adopt structural representations to quantitatively measure the impact of such structure-related distortion on perceived quality in FTV scenario. More specifically, a bio-inspired full reference image quality metric is proposed based on 1) low-level contour descriptor; 2) mid-level contour category descriptor; and 3) task-oriented non-natural structure descriptor. The experimental results show that the proposed model outperforms significantly the state-of-the-art metrics.

Development and performance evaluation of efficient methods for coding, transmission, and quality assessment of 3D visual content require rich datasets of a suitable test material. The use of these databases allows a fair comparison of systems under test. Moreover, publicly available and widely used datasets are crucial for experimentation leading to reproducible research. This chapter presents an overview of 3D visual content datasets relevant to research in the field of coding, transmission, and quality assessment. Description of regular stereoscopic or multiview image and video datasets is presented. Databases created using emerging technologies, including light-field imaging, are also addressed. Moreover, there are databases of multimedia content annotated with ratings from the subjective experiment, which are a necessary resource for understanding the complex problem of quality of experience while consuming the 3D visual content.

Social difficulties are the hallmark features of Autism Spectrum Disorder (ASD) and can lead to atypical visual attention towards stimuli. Eye movements encode rich information about attention and psychological factors of an individual, which could help to characterize the traits of ASD. Learning atypical eye movements of the individuals with ASD towards various stimuli is important and has many application scenarios. However, due to the lack of open datasets, research in this sense is still limited. In this work, we present an open dataset of eye movements of children with Autism Spectrum Disorder. It consists of 300 natural scene images and the corresponding eye movement data collected from 14 children with ASD and 14 healthy controls. In particular, fixation maps and scanpaths are available in the dataset. Based on this dataset, researchers could analyze the visual traits of children with ASD and design specialized visual attention models to promote research in related fields, as well as design specialized models to identify the individuals with ASD. The dataset can be accessed in http://doi.org/10.5281/zenodo.2647418

Data size is the bottleneck for developing deep saliency models, because collecting eye-movement data is very time-consuming and expensive. Most of current studies on human attention and saliency modeling have used high-quality stereotype stimuli. In real world, however, captured images undergo various types of transformations. Can we use these transformations to augment existing saliency datasets? Here, we first create a novel saliency dataset including fixations of 10 observers over 1900 images degraded by 19 types of transformations. Second, by analyzing eye movements, we find that observers look at different locations over transformed versus original images. Third, we utilize the new data over transformed images, called data augmentation transformation (DAT), to train deep saliency models. We find that label-preserving DATs with negligible impact on human gaze boost saliency prediction, whereas some other DATs that severely impact human gaze degrade the performance. These label-preserving valid augmentation transformations provide a solution to enlarge existing saliency datasets. Finally, we introduce a novel saliency model based on generative adversarial networks (dubbed GazeGAN). A modified U-Net is utilized as the generator of the GazeGAN, which combines classic "skip connection" with a novel "center-surround connection" (CSC) module. Our proposed CSC module mitigates trivial artifacts while emphasizing semantic salient regions, and increases model nonlinearity, thus demonstrating better robustness against transformations. Extensive experiments and comparisons indicate that GazeGAN achieves state-of-the-art performance over multiple datasets. We also provide a comprehensive comparison of 22 saliency models on various transformed scenes, which contributes a new robustness benchmark to saliency community. Our code and dataset are available at.

View synthesis is the most important technique in multi-view and free-viewpoint videos. The whole view synthesis includes the acquisition and processing of texture and depth images, and the virtual view rendering stage. Existing quality metrics for view synthesis have limited ability for the whole synthesis process for the following reasons. First, they are dedicated to a single stage of view synthesis, overlooking the commonality of all the possible distortions introduced in the whole process. Moreover, they only extract low-level features for quality assessment, ignoring the perceptual degradation caused by the mid-level contours that are destructed by heavy distortions in texture/depth images and the imperfect view rendering, which represent the spatial distribution/connection of adjacent contour pixels. Inspired by the above facts, this paper presents a quality metric for view synthesis using both Low-level and Mid-level Structural representation (LMS), aiming to accurately evaluate the distortions in the whole view synthesis process. Specifically, the scale space is first constructed to mimic the hierarchical property of the human visual system. Then, the statistics of gradient orientation is integrated with the statistics of gradient intensity for the low-level structural representation, which is motivated by the importance of the orientation selectivity mechanism to visual perception. Further, the mid-level structure is represented using bag of words for contour description based on the sparse coding of the primary visual cortex. Then the distances of both the low-level and mid-level features between the synthesized and reference images are calculated. Finally, two distances are integrated to generate the whole quality score. Extensive experiments on two public view synthesis databases demonstrate the superiority of the proposed method to the state-of-the-arts in evaluating the quality of the whole view synthesis.

The appropriate characterization of the test material, used for subjective evaluation tests and for benchmarking image and video processing algorithms and quality metrics, can be crucial in order to perform comparative studies that provide useful insights. This paper focuses on the characterisation of 360-degree images. We discuss why it is important to take into account the geometry of the signal and the interactive nature of 360-degree content navigation, for a perceptual characterization of these signals. Particularly, we show that the computation of classical indicators of spatial complexity, commonly used for 2D images, might lead to different conclusions depending on the geometrical domain used to represent the 360-degree signal. Finally, new complexity measures based on the analysis of visual attention and content exploration patterns are proposed.

This paper proposes to blindly evaluate the quality of images synthesized via a depth image-based rendering (DIBR) procedure. As a significant branch of virtual reality (VR), superior DIBR techniques provide free viewpoints in many real applications, including remote surveillance and education; however, limited efforts have been made to measure the performance of DIBR techniques, or equivalently the quality of DIBR-synthesized views, especially in the condition when references are unavailable. To achieve this aim, we develop a novel blind image quality assessment (IQA) method via multiscale natural scene statistical analysis (MNSS). The design principle of our proposed MNSS metric is based on two new natural scene statistics (NSS) models specific to the DBIR-synthesized IQA. First, the DIBR-introduced geometric distortions damage the local self-similarity characteristic of natural images, and the damage degrees of self-similarity present particular variations at different scales. Systematically combining the measurements of the variations mentioned above can gauge the naturalness of the input image and thus indirectly reflect the quality changes of images generated using different DIBR methods. Second, it was found that the degradations in main structures of natural images at different scales remain almost the same, whereas the statistical regularity is destroyed in the DIBR-synthesized views. Estimating the deviation of degradations in main structures at different scales between one DIBR-synthesized image and the statistical model, which is constructed based on a large number of natural images, can quantify how a DIBR method damages the main structures and thus infer the image quality. Via trials, the two NSS-based features extracted above can well predict the quality of DIBR-synthesized images. Further, the two features come from distinct points of view, and we hence integrate them via a straightforward multiplication to derive the proposed blind MNSS metric, which achieves better performance than each component and state-of-the-art quality methods.

Quality of Experience of multimedia content is investigated to understand the overall acceptability of the content as perceived by the end user. Recent perspectives stress the need to focus not only on perceptual aspects of the content but also on the interaction between perception and cognitive–emotional aspects. Our contribution lies on this direction by presenting a contactless approach to estimate heart rate variability by evaluating face skin color variations in a video sequence. The proposed framework allows to analyze long video sequences depicting a subject and it is robust with respect to small movements and illumination changes. The system is validated on a dataset of 64 sequences where users observe 3D movies selected to induce heart rate variations.

Tone-mapping is a crucial step in the task towards displaying high dynamic range (HDR) images on standard displays. Given the number of possible ways to tone-map such images, development of an objective quality criterion, enabling selection of the most suitable tone-mapping operator (TMO) and setting its parameters in order to maximize the quality of the reproduction, is of high interest. In this paper, a new objective metric for natural tone-mapped images is proposed. It is based on a fusion of several perceptually relevant features that have been carefully selected using an appropriate feature selection procedure. The outcome of the selection also provides a valuable insight into the importance of particular perceptual aspects when judging the quality of tone-mapped HDR content. The performance of the resulting combination of features is thoroughly evaluated with respect to three publicly available databases and compared to several relevant state-of-the-art criteria. The proposed approach is shown to significantly outperform the tested metrics and can, therefore, be considered a competitive alternative for tone-mapped images evaluation.

Machine learning (ML) is an essential part of recent advances in computer science. To fully exploit its potential, ML-based algorithms require a considerable amount of annotated data to be used for training. This represents a severe limitation in the field of image and video quality assessment since obtaining large-scale annotated databases is time-consuming and expensive. Moreover, the resulting quality estimators are mainly restricted only to the usecases included in the dataset used for their training. This paper proposes a strategy allowing for combination of multiple databases for training of objective image and video quality assessment algorithms. Using this strategy, the algorithms can be trained using all of the existing relevant databases together which allows to increase the amount of data-points and usecases in orders of magnitude. The potential of the proposed method is demonstrated by re-training the combination of features from Video Multimethod Assessment Fusion (VMAF) algorithm resulting in the significant improvement of its performance with respect to 20 video databases.

The training and performance analysis of objective video quality assessment algorithms is complex due to the huge variety of possible content classes and transmission distortions. Several secondary issues such as free parameters in machine learning algorithms and alignment of subjective datasets put an additional burden on the developer. In this paper, three subsequent steps are presented to address such issues. First, the content and coding parameter space of a large-scale database is used to select dedicated subsets for training objective algorithms. This aims at providing a method for selecting the most significant contents and coding parameters from all imaginable combinations. In the practical case where only a limited set is available, it also helps us to avoid redundancy in the training subset selection. The second step is a discussion on performance measures for algorithms that employ machine-learning methods. The particularity of the performance measures is that the quality of the training and verification datasets is taken into consideration. Common issues that often use existing measures are presented, and improved or complementary methods are proposed. The measures are applied to two examples of no-reference objective assessment algorithms using the aforementioned subsets of the large-scale database. While limited in terms of practical applications, this sandbox approach of objectively predicting an objectively evaluated video sequences allows for eliminating additional influence factors from subjective studies. In the third step, the proposed performance measures are applied to the practical case of training and analyzing assessment algorithms on readily available subjectively annotated image datasets. The presentation method in this part of the paper can also be used as an exemplified recommendation for reporting in-depth information on the performance. Using this presentation method, future publications presenting newly developed quality assessment algorithms may be significantly improved.

Research on understanding how people watch/explore Virtual Reality (VR) and 360° content is crucial to develop appropriate technologies for processing, encoding, delivering and rendering media content in order to provide high-quality immersive experiences to users. In this sense, the work described in this paper aims at supporting the research on visual attention modeling for omnidirectional content, by presenting a dataset and toolbox that were created for the Grand Challenge “Salient360!” organized at ICME’17. In particular, the dataset includes 360° images and their corresponding saliency maps and scanpaths, obtained from a subjective experiment in which eye gazes and head movements of observers were recorded during a free-viewing task. In addition, guidelines to obtain saliency maps and scanpaths from eye and head tracking data for omnidirectional images are provided. Finally, together with the dataset, a toolbox has been made publicly available, containing scripts for comparing saliency maps and scanpaths to benchmark developed models.

The present article describes an experimental study towards the examination of human visual behavior during the observation of unmanned aerial vehicles (UAVs) videos. Experimental performance is based on the collection and the quantitative &amp, qualitative analysis of eye tracking data. The results highlight that UAV flight altitude serves as a dominant specification that affects the visual attention process, while the presence of sky in the video background seems to be the less affecting factor in this procedure. Additionally, the main surrounding environment, the main size of the observed object as well as the main perceived angle between UAV’s flight plain and ground appear to have an equivalent influence in observers’ visual reaction during the exploration of such stimuli. Moreover, the provided heatmap visualizations indicate the most salient locations in the used UAVs videos. All produced data (raw gaze data, fixation and saccade events, and heatmap visualizations) are freely distributed to the scientific community as a new dataset (EyeTrackUAV) that can be served as an objective ground truth in future studies.

Light Field imaging is a promising technology that allows to capture the whole set of light rays in a scene thus enabling the generation of perspective views from any position. This possibility can be exploited in several application scenarios, such as virtual and augmented reality or depth estimation. In this framework many issues arise due different aspects such as the large amount of generated data or to the need of dedicated and expensive hardware for Light Field capturing. Moreover, the Light Field carries information about the entire scene and the data that is delivered to the users largely differs from the traditional 2D and 3D media in terms of content and way of fruition. Dedicated rendering technology and devices for the Light Field are nowadays still not mature or quite expensive and the best option is to render the Light Field data on a conventional 2D screen. Consequently, there is the need for finding the best visualization technique that allows to exploit the information in the Light Field while being accepted by the viewers. In this paper we address this issue by considering six visualization options and by running experimental tests to study which is the technique preferred by the users.

Assessment of multimedia quality relies heavily on subjective assessment, and is typically done by human subjects in the form of preferences or continuous ratings. Such data are crucial for analysis of different multimedia-processing algorithms as well as validation of objective (computational) methods for the said purpose. To that end, statistical testing provides a theoretical framework toward drawing meaningful inferences, and making well-grounded conclusions and recommendations. While parametric tests (such as $t$ test, ANOVA, and error estimates like confidence intervals) are popular and widely used in the community, there appears to be a certain degree of confusion in the application of such tests. Specifically, the assumptions of normality and homogeneity of variance are often not well understood, leading to incorrect application and/or interpretation of the statistical test results. Therefore, the main goal of this paper is to present new guidelines toward proper use of statistical tests and, hence, fix some of the issues in multimedia quality assessment. The said guidelines are derived based on theoretical analysis of sampling distribution of test statistics, and consider practical aspects of data analysis in the said domain. Experimental results on both simulated and real data are presented to support the arguments made. Software that implements the said recommendations is also made publicly available, in order to help researchers and practitioners perform correct statistical comparison of models.

Image stitching-composition of different viewpoint images to form a 360-degree panoramic image-is an essential component towards immersive applications like VR and AR. A no-reference (NR) quality metric specifically designed to evaluate stitched panoramic images is highly desirable when ground-truth reference images are not available. In this paper, we use Convolutional Sparse Coding (CSC) with a set of convolutional filters to locate stitching-specific distortions in a target image, and design trained kernels to quantify the compound effects of multiple distortion types in a local region. Specifically, our contributions are: i) a training database labeled with location information of the distortion regions is released; ii) a NR metric is proposed to accurately assess stitching-specific artifacts like ghosting using convolutional sparse coding; and iii) a novel sequential feature selection algorithm is proposed to quantify the aforementioned compound distortion effects. In extensive experiments, we show that the performance of our proposed NR metric is comparable to the state-of-the-art full-reference metrics designed for stitched images.

International audience

Virtual Reality (VR) provides the users with new immersive media experiences, offering the possibility to freely explore 360° content. Understanding these new exploration behaviors is crucial for the development of efficient techniques for processing, coding, delivering and rendering omnidirectional content to offer the highest possible Quality of Experience (QoE). Progress has already been made on visual attention (VA) modeling for 360° content. In this paper we briefly review the current status of research on this topic that led us to propose a benchmarking platform for evaluating and comparing the performance of models for saliency and scanpath prediction for 360° content. This paper introduces the ‘UN Salient360! benchmark” platform featuring a dataset, a toolbox and a framework for evaluation of different class of models. This online platform can be found in httns://salient360.ls2n.fr/.

International audience

International audience

International audience

International audience

Purpose Abnormal torsion could be associated with cyclovertical strabismus, but torsion measurements are not reliable in children. To assess an objective fundus torsion evaluation in a paediatric population, we used Non-Mydriatic Fundus photography (NMFP) in healthy and cyclovertical strabismus patients to evaluate the disc-foveal angle over time and observers. Methods We used a retrospective set of NMFP including 24 A or V-pattern strabismus and 27 age-matched normal children (mean age 6.4 and 6.7 years respectively), taken during 2 distinct follow-up consultations (separated by 251 and 479 days respectively). Each disc-foveal angle measurement (from which the ocular torsion can be assessed) was performed by 5 different observers, using graphical software and based on reproducible fundus anatomical marks. Statistical analysis was performed with a multivariate ANOVA using group, time and observers as factors, in addition to intraclass coefficient correlation (ICC) to assess measurement reproducibility. Results A significant difference of disc-foveal angle measures was observed between groups (p0,97). Conclusion Abnormal amount of objective torsion could be associated with alphabet-pattern strabismus. Disc-foveal angle evaluation by NMFP in a children population appears as a non-invasive, reliable and reproducible method.

Subjective assessment for image or video qualities is considered as the most reliable way to obtain the ground truth for the development of objective quality metrics, especially when leaded by Mean Opinion Score (MOS approaches). However, obtained MOS with standard protocols are noisy due to subject's personal characteristics, such as viewing experience, gender or profession, leading to uncertain ground truth driven by the number of panelists/subjects. The usual way to reduce uncertainty relies on raising this number. In this paper, we demonstrate how a recently introduced Maximum Likelihood Estimation (MLE) based quality recovery model can improve the discriminability of standard subjective quality assessment. Compared to straightforward MOS computation, we present a case study where one can save between 26% to 39% in terms of numbers of subjects at the same discriminability.

Machine learning has been attached greater attention in the field of quality assessment and related models have been developed by assuming that any region within images share the same quality score. However, this assumption may no longer stand when non-uniform distortions exist, e.g. geometric distortion within synthesized views in the case of free- view point video TV (FTV). In this paper, we explore to use Convolutional Sparse Coding (CSC), which computes a sparse representation for an entire image with the sum of a set of convolutions with dictionary filters instead of a linear combination of a set of dictionary atoms, to learn a visible codebook and propose a methodology to quantify the geometric distortion without using the reference. Experimental results show that the proposed no reference metric is the most visual friendly and reliable metric among the compared blind image metrics designed for synthesized images.

The assessment of Quality of Experience (QoE) for stereoscopic 3-D video is a challenging task, especially in 3-D video compression and transmission applications. The focus of this contribution is the development of a QoE assessment framework, in line with the latest standardization progress in the field of QoE assessment, for understanding the visual effect of asymmetric and symmetric encoding for immersive media. Asymmetric stereoscopic video coding exploits the binocular suppression of the human vision system by representing one of the two views with a lower quality. This processing, while of limited effects on image quality, may influence the overall QoE. Many studies show that the QoE of immersive media such as 3-DTV can be thought as the combination of the perceived visual quality, the perceived depth quality, the visual fatigue, and visual discomfort. In this paper, we aim at: 1) exploiting the concept of preference of experience and protocols recently standardized for characterizing QoE; 2) conducting a case study using these standardized protocols to investigate the factors involving visual discomfort in stereoscopic video sequences with a focus on binocular rivalry; and 3) presenting the results of subjective experiments performed, by using the perceptual quality and preference of experience assessment protocols, for evaluating the impact of symmetrical, asymmetrical, and alternate coding schemes.

This chapter is related to the concept of Quality of Experience (QoE) in multimedia applications. Throughout the chapter, QoE mostly has to be considered in such applied context, even if the term multimedia is not always mentioned. After reminding the concept of QoE, the chapter provides guidelines on: (1) how to design, conduct, and interpret QoE experiments with humans and (2) how to use the results obtained from such experiments for evaluating the performance of objective QoE estimators. The main factors influencing QoE are first summarized and the possibilities of involving particular aspects of QoE into the subjective tests are discussed. The most popular subjective procedures are introduced in detail, together with their advantages and weaknesses. Thorough instructions regarding processing the outcomes of the particular procedures are provided, as well as advanced methodologies for determining the significance of influence of the specific dimensions on the measured overall QoE. Finally, the possibilities of utilizing subjective data for evaluating the performance of objective QoE estimators are introduced in detail. Such methods are useful for comparison, training, and validation of the estimators.

The Internet streaming is changing the way of watching videos for people. Traditional quality assessment on the cable/satellite broadcasting system mainly focused on the perceptual quality. Nowadays, this concept has been extended to Quality of Experience (QoE) which considers also the contextual factors, such as the environment, the display devices, etc. In this study, we focus on the influence of devices on QoE. A subjective experiment was conducted by using our proposed AccAnn methodology. The observers evaluated the QoE of the video sequences by considering their Acceptance and Annoyance. Two devices were used in this study, TV and Tablet. The experimental results showed that the device was a significant influence factor on QoE. In addition, we found that this influence varied with the QoE of the video sequences. To quantify this influence, the Eliminated-By-Aspects model was used. The results could be used for the training of a device-neutral objective QoE metric. For video streaming providers, the quantification results of the influence from devices could be used to optimize the selection of streaming content. On one hand it could satisfy the QoE expectations of the observers according to the used devices, on the other hand it could help to save the bitrates.

The overarching goal of this workshop is to bring together practices and research in medical fields with media contents developers, designers, and UI/UX/QoE researchers, as well as hospital practitioners as ophthalmologists and psychiatrists. Discussions will relate to:how new visual experience and media contents can improve the in-hospital experience of in-patients, caregivers, and medical staffs;how better understanding of visual impairments (e.g. for elderly) can help to design more inclusive TV experience. The starting point will be the cases and experiences of Care TVX, followed by a multidisciplinary discussion on "challenges and design considerations to adjusted or new hospital and care practices", led by workshop participants. The outcome of the workshop will be a collection of best practices in the form of position papers and online content.