Search

Probability density functions for the components of the first and second derivatives of localized phase are derived.

The derivation of bounds on the structural similarity (SSIM) index as a function of quantization rate for fixed-rate uniform quantization is discussed. Findings reveal the importance of the proposed bounds for rate allocation problems in practical image and video coding applications.

An algorithm for designing linear equalizers, which maximizes the structural similarity (SSIM) index between the reference and restored signals, is proposed. Findings reveal the achievement of higher SSIM index and superior perceptual quality.

A foveated, multifixating strategy is presented for locating corners in natural images. The results have shown that the algorithm performs well on strong edges with sharp corners and less well in areas of fine detail.

A performance evaluation study of ten image quality assessment (QA) algorithms are presented, which involved 779 distorted images derived from 29 source images using five different distortion types. The study was diverse in terms of image content, distortion types, distortion strength and the number of human subjects ranking each image, and results and data from the study are made publicly available to facilitate future research in image quality assessment.

The concept of extracting features of the original high-quality images and embedding into the image data as invisible hidden messages and can provide an objective measure of the quality of the distorted images known as quality-aware image is described. A practical quality-aware image encoding, decoding and quality analysis system is formulated which uses a unique reduced-reference image quality assessment technique based on a statistical model of natural images and quantization watermarking-based data hiding method in wavelet transform domain.

The goal of quality assessment (QA) research is to design algorithms that can automatically assess the quality of images or videos in a perceptually consistent manner. The problem of image QA is approached by proposing a novel information fidelity criterion that is based on natural scene statistics.

A model that accounts for the properties of the imaging system and of the tissues is developed to get a better understanding of confocal reflectance microscopy images of cervical tissues, and to supply a basis for parameter selection in a classification algorithm. The algorithm was applied to an extensive set of cervical images and it was found that it detects 90% of hand-segmented nuclei with an average of 6 false positive per frame.

Theorems are developed that place limits on the point-wise approximation of the responses of filters, both linear shift invariant (LSI) and linear shift variant (LSV) to input signals and images that are LSV. The application of the theory was found to defoveation of images, deblurring of shift-variant blurs, and shift-variant edge detection.

A study of the objective method for assessing perceptual image quality that attempted to quantify the visibility of errors between distorted and reference images according to known characteristics of the human visual system (HVS) is presented. The development of the structural similarity index (SSIM) for quality assessment based on the assumption that HVS is highly adapted to extract structural information is discussed.

A wavelet-based, scalable video coding system, foveation scalable video coding (FSVC, is described and discussed.

It is shown that formulating the nullspace-based approach to eigenstructure-based direct multichannel blind image restoration methods implies a subspace-based approach that relies on matrix operations. The new approach has the same advantages as the nullspace-based one but requires less computational complexity.

An embedded foveation image coding (EFIC) algorithm is proposed. The algorithm orders the encoded bitstream to optimize foveated visual quality at arbitrary bit rates.

Multidimensional multicomponent and quasi-Eigenfunction approximation AM-FM models are discussed. Techniques on several images of general interest in practical applications have been demonstrated. Reconstructions that establish validity of characterizing the type of images as sums of locally narrowband modulated components have been made. Two discrete computational paradigms were used. Dominant component analysis does one thing and channelized components analysis does another.

We develop a general formalism for computing high quality, low-cost solutions to nonconvex combinatorial optimization problems expressible as distributed interacting local constraints. For problems of this type, generalized deterministic annealing (GDA) avoids the performance-related sacrifices of current techniques. GDA exploits the localized structure of such problems by assigning K-state neurons to each optimization variable. The neuron values correspond to the probability densities of K-state local Markov chains and may be updated serially or in parallel; the Markov model is derived from the Markov model of simulated annealing (SA), although it is greatly simplified. Theorems are presented that firmly establish the convergence properties of GDA, as well as supplying practical guidelines for selecting the initial and final temperatures in the annealing process. A benchmark image enhancement application is provided where the performance of GDA is compared to other optimization methods. The empirical data taken in conjunction with the formal analytical results suggest that GDA enjoys significant performance advantages relative to current methods for combinatorial optimization.

Although it is well-known that the negative effects of VR sickness, and the desirable sense of presence are important determinants of a user’s immersive VR experience, there remains a lack of definitive research outcomes to enable the creation of methods to predict and/or optimize the trade-offs between them. Most VR sickness assessment (VRSA) and VR presence assessment (VRPA) studies reported to date have utilized simple image patterns as probes, hence their results are difficult to apply to the highly diverse contents encountered in general, real-world VR environments. To help fill this void, we have constructed a large, dedicated VR sickness/presence (VR-SP) database, which contains 100 VR videos with associated human subjective ratings. Using this new resource, we developed a statistical model of spatio-temporal and rotational frame difference maps to predict VR sickness. We also designed an exceptional motion feature, which is expressed as the correlation between an instantaneous change feature and averaged temporal features. By adding additional features (visual activity, content features) to capture the sense of presence, we use the new data resource to explore the relationship between VRSA and VRPA. We also show the aggregate VR-SP model is able to predict VR sickness with an accuracy of 90% and VR presence with an accuracy of 75% using the new VR-SP dataset. [ABSTRACT FROM AUTHOR]

The topics of visual and audio quality assessment (QA) have been widely researched for decades, yet nearly all of this prior work has focused only on single-mode visual or audio signals. However, visual signals rarely are presented without accompanying audio, including heavy-bandwidth video streaming applications. Moreover, the distortions that may separately (or conjointly) afflict the visual and audio signals collectively shape user-perceived quality of experience (QoE). This motivated us to conduct a subjective study of audio and video (A/V) quality, which we then used to compare and develop A/V quality measurement models and algorithms. The new LIVE-SJTU Audio and Video Quality Assessment (A/V-QA) Database includes 336 A/V sequences that were generated from 14 original source contents by applying 24 different A/V distortion combinations on them. We then conducted a subjective A/V quality perception study on the database towards attaining a better understanding of how humans perceive the overall combined quality of A/V signals. We also designed four different families of objective A/V quality prediction models, using a multimodal fusion strategy. The different types of A/V quality models differ in both the unimodal audio and video quality prediction models comprising the direct signal measurements and in the way that the two perceptual signal modes are combined. The objective models are built using both existing state-of-the-art audio and video quality prediction models and some new prediction models, as well as quality-predictive features delivered by a deep neural network. The methods of fusing audio and video quality predictions that are considered include simple product combinations as well as learned mappings. Using the new subjective A/V database as a tool, we validated and tested all of the objective A/V quality prediction models. We will make the database publicly available to facilitate further research. [ABSTRACT FROM AUTHOR]

Most full-reference image quality assessment (FR-IQA) methods advanced to date have been holistically designed without regard to the type of distortion impairing the image. However, the perception of distortion depends nonlinearly on the distortion type. Here we propose a novel FR-IQA framework that dynamically generates receptive fields responsive to distortion type. Our proposed method- dynamic receptive field generation based image quality assessor (DRF-IQA)-separates the process of FR-IQA into two streams: 1) dynamic error representation and 2) visual sensitivity-based quality pooling. The first stream generates dynamic receptive fields on the input distorted image, implemented by a trained convolutional neural network (CNN), then the generated receptive field profiles are convolved with the distorted and reference images, and differenced to produce spatial error maps. In the second stream, a visual sensitivity map is generated. The visual sensitivity map is used to weight the spatial error map. The experimental results show that the proposed model achieves state-of-the-art prediction accuracy on various open IQA databases. [ABSTRACT FROM AUTHOR]

Virtual Reality (VR) and its applications have attracted significant and increasing attention. However, the requirements of much larger file sizes, different storage formats, and immersive viewing conditions pose significant challenges to the goals of acquiring, transmitting, compressing and displaying high quality VR content. Towards meeting these challenges, it is important to be able to understand the distortions that arise and that can affect the perceived quality of displayed VR content. It is also important to develop ways to automatically predict VR picture quality. Meeting these challenges requires basic tools in the form of large, representative subjective VR quality databases on which VR quality models can be developed and which can be used to benchmark VR quality prediction algorithms. Towards making progress in this direction, here we present the results of an immersive 3D subjective image quality assessment study. In the study, 450 distorted images obtained from 15 pristine 3D VR images modified by 6 types of distortion of varying severities were evaluated by 42 subjects in a controlled VR setting. Both the subject ratings as well as eye tracking data were recorded and made available as part of the new database, in hopes that the relationships between gaze direction and perceived quality might be better understood. We also evaluated several publicly available IQA models on the new database, and also report a statistical evaluation of the performances of the compared IQA models. [ABSTRACT FROM AUTHOR]

In a typical communication pipeline, images undergo a series of processing steps that can cause visual distortions before being viewed. Given a high quality reference image, a reference (R) image quality assessment (IQA) algorithm can be applied after compression or transmission. However, the assumption of a high quality reference image is often not fulfilled in practice, thus contributing to less accurate quality predictions when using stand-alone R IQA models. This is particularly common on social media, where hundreds of billions of user-generated photos and videos containing diverse, mixed distortions are uploaded, compressed, and shared annually on sites like Facebook, YouTube, and Snapchat. The qualities of the pictures that are uploaded to these sites vary over a very wide range. While this is an extremely common situation, the problem of assessing the qualities of compressed images against their pre-compressed, but often severely distorted (reference) pictures has been little studied. Towards ameliorating this problem, we propose a novel two-step image quality prediction concept that combines NR with R quality measurements. Applying a first stage of NR IQA to determine the possibly degraded quality of the source image yields information that can be used to quality-modulate the R prediction to improve its accuracy. We devise a simple and efficient weighted product model of R and NR stages, which combines a pre-compression NR measurement with a post-compression R measurement. This first-of-a-kind two-step approach produces more reliable objective prediction scores. We also constructed a new, first-of-a-kind dedicated database specialized for the design and testing of two-step IQA models. Using this new resource, we show that two-step approaches yield outstanding performance when applied to compressed images whose original, pre-compression quality covers a wide range of realistic distortion types and severities. The two-step concept is versatile as it can use any desired R and NR components. We are making the source code of a particularly efficient model that we call 2stepQA publicly available at https://github.com/xiangxuyu/2stepQA. We are also providing the dedicated new two-step database free of charge at http://live.ece.utexas.edu/research/twostep/index.html. [ABSTRACT FROM AUTHOR]

Natural scene statistics (NSS) is used for blindly measuring the quality of images compressed by JPEG2000 image coder. An average error of 8.05 is achieved from the mean opinion scores.

The great variations of videographic skills in videography, camera designs, compression and processing protocols, communication and bandwidth environments, and displays leads to an enormous variety of video impairments. Current no-reference (NR) video quality models are unable to handle this diversity of distortions. This is true in part because available video quality assessment databases contain very limited content, fixed resolutions, were captured using a small number of camera devices by a few videographers and have been subjected to a modest number of distortions. As such, these databases fail to adequately represent real world videos, which contain very different kinds of content obtained under highly diverse imaging conditions and are subject to authentic, complex, and often commingled distortions that are difficult or impossible to simulate. As a result, NR video quality predictors tested on real-world video data often perform poorly. Toward advancing NR video quality prediction, we have constructed a large-scale video quality assessment database containing 585 videos of unique content, captured by a large number of users, with wide ranges of levels of complex, authentic distortions. We collected a large number of subjective video quality scores via crowdsourcing. A total of 4776 unique participants took part in the study, yielding over 205 000 opinion scores, resulting in an average of 240 recorded human opinions per video. We demonstrate the value of the new resource, which we call the live video quality challenge database (LIVE-VQC), by conducting a comparison with leading NR video quality predictors on it. This paper is the largest video quality assessment study ever conducted along several key dimensions: number of unique contents, capture devices, distortion types and combinations of distortions, study participants, and recorded subjective scores. The database is available for download on this link: http://live.ece.utexas.edu/research/LIVEVQC/index.html. [ABSTRACT FROM AUTHOR]

We develop a model that expresses the joint impact of spatial resolution $s$ and JPEG compression quality factor $q^{f}$ on immersive image quality. The model is expressed as the product of optimized exponential functions of these factors. The model is tested on a subjective database of immersive image contents rendered on a head mounted display. High Pearson correlation and Spearman correlation (>0.95) and small relative root mean squared error (<5.6%) are achieved between the model predictions and the subjective quality judgements. The immersive ground-truth images along with the rest of the database are made available for future research and comparisons. [ABSTRACT FROM AUTHOR]

HTTP adaptive streaming is being increasingly deployed by network content providers, such as Netflix and YouTube. By dividing video content into data chunks encoded at different bitrates, a client is able to request the appropriate bitrate for the segment to be played next based on the estimated network conditions. However, this can introduce a number of impairments, including compression artifacts and rebuffering events, which can severely impact an end-user’s quality of experience (QoE). We have recently created a new video quality database, which simulates a typical video streaming application, using long video sequences and interesting Netflix content. Going beyond previous efforts, the new database contains highly diverse and contemporary content, and it includes the subjective opinions of a sizable number of human subjects regarding the effects on QoE of both rebuffering and compression distortions. We observed that rebuffering is always obvious and unpleasant to subjects, while bitrate changes may be less obvious due to content-related dependencies. Transient bitrate drops were preferable over rebuffering only on low complexity video content, while consistently low bitrates were poorly tolerated. We evaluated different objective video quality assessment algorithms on our database and found that objective video quality models are unreliable for QoE prediction on videos suffering from both rebuffering events and bitrate changes. This implies the need for more general QoE models that take into account objective quality models, rebuffering-aware information, and memory. The publicly available video content as well as metadata for all of the videos in the new database can be found at http://live.ece.utexas.edu/research/LIVE_NFLXStudy/nflx_index.html. [ABSTRACT FROM AUTHOR]

The human visual system perceives 3D depth following sensing via its binocular optical system, a series of massively parallel processing units, and a feedback system that controls the mechanical dynamics of eye movements and the crystalline lens. The process of accommodation (focusing of the crystalline lens) and binocular vergence is controlled simultaneously and symbiotically via cross-coupled communication between the two critical depth computation modalities. The output responses of these two subsystems, which are induced by oculomotor control, are used in the computation of a clear and stable cyclopean 3D image from the input stimuli. These subsystems operate in smooth synchronicity when one is viewing the natural world; however, conflicting responses can occur when viewing stereoscopic 3D (S3D) content on fixed displays, causing physiological discomfort. If such occurrences could be predicted, then they might also be avoided (by modifying the acquisition process) or ameliorated (by changing the relative scene depth). Toward this end, we have developed a dynamic accommodation and vergence interaction (DAVI) model that successfully predicts visual discomfort on S3D images. The DAVI model is based on the phasic and reflex responses of the fast fusional vergence mechanism. Quantitative models of accommodation and vergence mismatches are used to conduct visual discomfort prediction. Other 3D perceptual elements are included in the proposed method, including sharpness limits imposed by the depth of focus and fusion limits implied by Panum’s fusional area. The DAVI predictor is created by training a support vector machine on features derived from the proposed model and on recorded subjective assessment results. The experimental results are shown to produce accurate predictions of experienced visual discomfort. [ABSTRACT FROM PUBLISHER]

Renewable solar energy harvesting systems have received considerable attention as a possible substitute for conventional chemical batteries in sensor networks. However, it is difficult to optimize the use of solar energy based only on empirical power acquisition patterns in sensor networks. We apply acquisition patterns from actual solar energy harvesting systems and build a framework to maximize the utilization of solar energy in general sensor networks. To achieve this goal, we develop a cross-layer optimization-based scheduling scheme called binding optimization of duty cycling and networking through energy tracking (BUCKET), which is formulated in four-stages: 1) prediction of energy harvesting and arriving traffic; 2) internode optimization at the transport and network layers; 3) intranode optimization at the medium access control layer; and 4) flow control of generated communication task sets using a token-bucket algorithm. Monitoring of the structural health of bridges is shown to be a potential application of an energy-harvesting sensor network. The example network deploys five sensor types: 1) temperature; 2) strain gauge; 3) accelerometer; 4) pressure; and 5) humidity. In the simulations, the BUCKET algorithm displays performance enhancements of ~12-15% over those of conventional methods in terms of the average service rate. [ABSTRACT FROM PUBLISHER]

We announce a new video quality model (VQM) that accounts for the perceptual impact of variable frame delays (VFD) in videos with demonstrated top performance on the laboratory for image and video engineering (LIVE) mobile video quality assessment (VQA) database. This model, called VQM_VFD, uses perceptual features extracted from spatial-temporal blocks spanning fixed angular extents and a long edge detection filter. VQM_VFD predicts video quality by measuring multiple frame delays using perception based parameters to track subjective quality over time. In the performance analysis of VQM_VFD, we evaluated its efficacy at predicting human opinions of visual quality. A detailed correlation analysis and statistical hypothesis testing show that VQM_VFD accurately predicts human subjective judgments and substantially outperforms top-performing image quality assessment and VQA models previously tested on the LIVE mobile VQA database. VQM_VFD achieved the best performance on the mobile and tablet studies of the LIVE mobile VQA database for simulated compression, wireless packet-loss, and rate adaptation, but not for temporal dynamics. These results validate the new model and warrant a hard release of the VQM_VFD algorithm. It is freely available for any purpose, commercial, or noncommercial at http://www.its.bldrdoc.gov/vqm/. [ABSTRACT FROM PUBLISHER]

Foveation has been investigated as a tool for exploring techniques for optimal rate control of foveated video. It is concluded that foveated compression and rate control offers significant potential with regard to compression performance, speed and visual quality.

Partial differential equations and AM-FM image modeling was used to achieve an automated method for repairing broken, occluded oriented image textures.

To achieve clear binocular vision, neural processes that accomplish accommodation and vergence are performed via two collaborative, cross-coupled processes: accommodation-vergence (AV) and vergence-accommodation (VA). However, when people watch stereo images on stereoscopic displays, normal neural functioning may be disturbed owing to anomalies of the cross-link gains. These anomalies are likely the main cause of visual discomfort experienced when viewing stereo images, and are called Accommodation-Vergence Mismatches (AVM). Moreover, the absence of any useful accommodation depth cues when viewing 3D content on a flat panel (planar) display induces anomalous demands on binocular fusion, resulting in possible additional visual discomfort. Most prior efforts in this direction have focused on predicting anomalies in the AV cross-link using measurements on a computed disparity map. We further these contributions by developing a model that accounts for both accommodation and vergence, resulting in a new visual discomfort prediction algorithm dubbed the 3D-AVM Predictor. The 3D-AVM model and algorithm make use of a new concept we call local 3D bandwidth (BW) which is defined in terms of the physiological optics of binocular vision and foveation. The 3D-AVM Predictor accounts for anomalous motor responses of both accommodation and vergence, yielding predictive power that is statistically superior to prior models that rely on a computed disparity distribution only. [ABSTRACT FROM PUBLISHER]

Owing to the increasingly large volume and complexity of captured videos, renewable energy systems based on solar energy are of particular interest in the design of energy harvesting (EH) wireless visual sensor networks (WVSNs). Since additional energy consumption following image capture occurs owing to image processing, mote operation, data transmission, and reception, the capture rate significantly affects the lifetime of a node. To this end, we explore a novel energy-efficient framework for EH-WVSN design by developing an optimal algorithm named capture rate and pervasive network control for multicamera networks where the quality of service is maximized by obtaining optimal values for the capture rate, allocated energy, and transmit power, based on field of view-based networking in the presence of event and power acquisition patterns. Through simulations, we demonstrate the feasibility of EH-WVSNs in terms of energy consumption, energy allocation, and capture rate in a realistic scenario (parking surveillance). [ABSTRACT FROM PUBLISHER]

An AM-FM surface model was developed to study the projection of surface patterns seen through a binocular camera system. The proposed model also proved to be useful in the design of a modulation-based stereo matching framework where local image phase was used to compute stereo disparities. The combined concepts of the phase-based matching and AM-FM image model and use powerful multichannel Gabor image demodulation techniques led to the development of a systematic approach for coarse-to-coarse disparity computation of disparities in stereoscopic images.

Novel smoothing constraints have been developed for use in the regularized image restoration problem with the aid of two broad classes of physically significant models. The models, called piecewise image models and local image models, capture distinct image properties that can be adapted to the image and that reflect structurally important surface properties. Numerical results are presented to illustrate the mechanism of model selection, image restoration and parameter selection.

Finding ways to monitor and control the perceptual quality of digital visual media has become a pressing concern as the volume being transported and viewed continues to increase exponentially. This paper discusses the principles and methods of modern algorithms for automatically predicting the quality of visual signals. By casting the problem as analogous to assessing the efficacy of a visual communication system, it is possible to divide the quality assessment problem into understandable modeling subproblems. Along the way, we will visit models of natural images and videos, of visual perception, and a broad spectrum of applications. [ABSTRACT FROM PUBLISHER]