Search

Abstract Conventional photography can only provide a two-dimensional image of the scene, whereas emerging imaging modalities such as light field enable the representation of higher dimensional visual information by capturing light rays from different directions. Light fields provide immersive experiences, a sense of presence in the scene, and can enhance different vision tasks. Hence, research into light field processing methods has become increasingly popular. It does, however, come at the cost of higher data volume and computational complexity. With the growing deployment of machine-learning and deep architectures in image processing applications, a paradigm shift toward learning-based approaches has also been observed in the design of light field processing methods. Various learning-based approaches are developed to process the high volume of light field data efficiently for different vision tasks while improving performance. Taking into account the diversity of light field vision tasks and the deployed learning-based frameworks, it is necessary to survey the scattered learning-based works in the domain to gain insight into the current trends and challenges. This paper aims to review the existing learning-based solutions for light field imaging and to summarize the most promising frameworks. Moreover, evaluation methods and available light field datasets are highlighted. Lastly, the review concludes with a brief outlook for future research directions.

Light fields are one of the emerging 3D representation formats with an effective potential to offer very realistic and immersive visual experiences. This capability comes at the cost of a very large amount of acquired data which practical use requires efficient coding solutions. This need was already addressed by the JPEG Pleno Light Field Coding standard for static light fields, which has specified two coding modes, named 4D-Transform and 4D-Prediction. While the first offers better compression performance for smaller baseline light fields, the second excels for larger baseline light fields. This paper intends to propose a novel light field coding mode, the Slanted 4D-Transform coding mode, which extends the 4D-Transform coding mode based on the conventional 4D-DCT, to offer better compression performance than both the available JPEG Pleno coding modes, independently of the baseline. The key idea is to apply first to each 4D block in the light field an adaptive, hierarchical geometric transformation, which makes the data in the block more energy-compaction friendly for the following 4D-DCT. The rate-distortion performance results show that the proposed Slanted 4D-Transform codec is able to outperform both the already standardized JPEG Pleno coding modes, showing BD-Rates gains of 31.03% and 28.30% for the 4D-Transform and 4D-Prediction modes, respectively, thus implying that a single coding mode can efficiently code all types of light fields.

While 60 years of successful application of holography is celebrated in this special issue, efficient representation and compression of holographic data has received relatively little attention in research. Notwithstanding this observation, and particularly due to the digitization that is also penetrating the holographic domain, interest is growing on how to efficiently compress holographic data such that interactive exchange of content, as well as digital storage can be facilitated proficiently. This is a particular challenge, not only because of its interferometric nature and the various representation formats, but also the often extremely large data volumes involved in pathological, tomographic, or high-end visualization applications. In this paper, we provide an overview of the state of the art in compression techniques and corresponding quality metrics for various practical applications in digital holography. We also consider the future by analyzing the emerging trends for addressing the key challenges in this domain.

Multi-view 3D content is subject to distortions during the process of depth image-based rendering (DIBR). Studies have shown the unreliable performance of the well-established image quality assessment (IQA) models for evaluation of DIBR-synthesized views which surge the need for more effective IQA methods. Existing objective methods generally rely on the pixel-wise correspondences between the reference and distorted images, while view synthesis can introduce pixel shifts. DIBR distortions such as stretching and local hole-filling errors have different visual impacts from conventional distortions, challenging the existing IQA models. Here, we developed a Full-Reference (FR) objective IQA metric for synthesized views that significantly outperforms 2D IQA and the state-of-the-art DIBR IQA approaches. While the pixel misalignment between the reference and synthesized views is a big challenge for quality assessment, we deployed a Convolutional Neural Network (CNN) model to acquire a feature representation that inherently offers resilience to the imperceptible pixel shift between the compared images. Therefore, our model does not need accurate shift compensation. We deployed a set of quality-aware CNN features representing high-order statistics, to measure the structural similarity which is combined with a semantic similarity measure for accurate quality assessment. Moreover, prediction accuracy is improved by incorporating a visual saliency model acquired using the activations of the higher CNN layers. Experimental results indicate a significant performance gain (14.6% in terms of Spearman’s rank-order correlation) compared to the top existing IQA model. The source code of the proposed metric is available at: https://gitlab.com/saeedmp/sequss.

Guaranteeing interoperability between devices and applications is the core role of standards organizations. Since its first JPEG standard in 1992, the Joint Photographic Experts Group (JPEG) has published several image coding standards that have been successful in a plethora of imaging markets. Recently, these markets have become subject to potentially disruptive innovations owing to the rise of new imaging modalities such as light fields, point clouds, and holography. These so‐called plenoptic modalities hold the promise of facilitating a more efficient and complete representation of 3D scenes when compared to classic 2D modalities. However, due to the heterogeneity of plenoptic products that will hit the market, serious interoperability concerns have arisen. In this paper, we particularly focus on the holographic modality and outline how the JPEG committee has addressed these tremendous challenges. We discuss the main use cases and provide a preliminary list of requirements. In addition, based on the discussion of real‐valued and complex data representations, we elaborate on potential coding technologies that range from approaches utilizing classical 2D coding technologies to holographic content‐aware coding solutions. Finally, we address the problem of visual quality assessment of holographic data covering both visual quality metrics and subjective assessment methodologies.

Abstract Current image coding standards provide limited support for privacy and security features. An exception is the JPSEC standard, which defines security extensions in JPEG 2000 specifications (part 8). Notwithstanding this shortcoming, the JPEG committee is currently defining a new JPEG Systems standard, which envisages privacy and security support across JPEG family of standards. In this manuscript, the main philosophy of this emerging specification is outlined along with typical use cases, main requirements as well as examples of potential technological solutions. The upcoming specification guarantees backward and forward compatibility with earlier standards and legacy implementations. Finally, we illustrate the introduced framework by two applications targeting secure photo sharing on social networks and IPR management in the GLAM sector.

Computer-generated holography (CGH) is a notoriously difficult computation problem, simulating numerical diffraction, where every scene point can affect every hologram pixel. To tackle this challenge, specialized software instructions and hardware solutions are developed to significantly reduce calculation time and power consumption. In this work, we propose a novel algorithm for high-performance point-based CGH, leveraging fixed-point integer representations, the separability of the Fresnel transform and using new look-up table free cosine representation. We report up to a 3-fold speed up over an optimized floating-point GPU implementation, as well as a 15 dB increase in quality over a state-of-the-art FPGA-based fixed-point integer solution.

Digital holograms are a prime example for signals, which are best understood in phase space—the joint space of spatial coordinates and spatial frequencies. Many characteristics, as well as optical operations can be visualized therein with so called phase space representations (PSRs). However, literature relies often only on symbolic PSRs or on, in practice, visually insufficient PSRs like the Wigner–Ville representation. In this tutorial-style paper, we will showcase the S-method, which is both a PSR that can be calculated directly from any given signal, and that allows for a clear visual interpretation. We will highlight the power of space-frequency analysis in digital holography, explain why this specific PSR is recommended, discuss a broad range of basic operations, and briefly overview several interesting practical questions in digital holography.

Realizing high-quality digital image transmission via a satellite link, while optimizing resource distribution and minimizing battery consumption, is a challenging task. This paper describes a methodology to optimize a turbo-encoded wavelet-based satellite downlink progressive image transmission system with unequal error protection (UEP) techniques. To achieve that goal, we instantiate a generic UEP methodology onto the system, and demonstrate that the proposed solution has little impact on the average performance, while greatly reducing the run-time complexity. Based on a simple design-time distortion model and a low-complexity run-time algorithm, the provided solution can dynamically tune the system's configuration to any bitrate constraint or channel condition. The resulting system outperforms in terms of peak signal-to-noise ratio (PSNR), a state-of-the-art, fine-tuned equal error protection (EEP) solution by as much as 2 dB.

Scalable multiple-description (MD) coding allows for fine-grain rate adaptation as well as robust coding of the input source. In this paper, we present a new approach for scalable MD coding of images, which couples the multiresolution nature of the wavelet transform with the robustness and scalability features provided by embedded multiple-description scalar quantization (EMDSQ). Two coding systems are proposed that rely on quadtree coding to compress the side descriptions produced by EMDSQ. The proposed systems are capable of dynamically adapting the bitrate to the available bandwidth while providing robustness to data losses. Experiments performed under different simulated network conditions demonstrate the effectiveness of the proposed scalable MD approach for image streaming over error-prone channels.