Search

Unmanned aerial vehicles (UAV) can capture multiview images of large-scale scenes, and then using aerial light field (ALF)-rendering technology, they can render high-quality, large-scale 3D scenes. However, the reconstruction of large-scale scenes poses challenges, such as rendering distortion and high memory consumption. In this article, we study the multiview image-capturing and novel view-rendering method of UAV sampling to address these issues. First, we present an ALF sampling model using the spectral analysis of light field and obtain an exact spectrum expression of the ALF. Through the spectral support of ALF, we determine the bandwidth and calculate the minimum required UAV sampling rate. Finally, we demonstrate that our sampling and rendering methods can improve the rendering quality of UAV 3D reconstruction and reduce the minimum sampling rate. [ABSTRACT FROM AUTHOR]

Depth estimation tries to obtain 3D scene geometry from low-dimensional data like 2D images. This is a vital operation in computer vision and any general solution must preserve all depth information of potential relevance to support higher-level tasks. For scenes with well-defined depth, this work shows that multi-view edges can encode all relevant information—that multi-view edges are complete. For this, we follow Elder's complementary work on the completeness of 2D edges for image reconstruction. We deploy an image-space geometric representation: an encoding of multi-view scene edges as constraints and a diffusion reconstruction method for inverting this code into depth maps. Due to inaccurate constraints, diffusion-based methods have previously underperformed against deep learning methods; however, we will reassess the value of diffusion-based methods and show their competitiveness without requiring training data. To begin, we work with structured light fields and epipolar plane images (EPIs). EPIs present high-gradient edges in the angular domain: with correct processing, EPIs provide depth constraints with accurate occlusion boundaries and view consistency. Then, we present a differentiable representation form that allows the constraints and the diffusion reconstruction to be optimized in an unsupervised way via a multi-view reconstruction loss. This is based around point splatting via radiative transport, and extends to unstructured multi-view images. We evaluate our reconstructions for accuracy, occlusion handling, view consistency, and sparsity to show that they retain the geometric information required for higher-level tasks. [ABSTRACT FROM AUTHOR]

Abstract Conventional photography can only provide a two-dimensional image of the scene, whereas emerging imaging modalities such as light field enable the representation of higher dimensional visual information by capturing light rays from different directions. Light fields provide immersive experiences, a sense of presence in the scene, and can enhance different vision tasks. Hence, research into light field processing methods has become increasingly popular. It does, however, come at the cost of higher data volume and computational complexity. With the growing deployment of machine-learning and deep architectures in image processing applications, a paradigm shift toward learning-based approaches has also been observed in the design of light field processing methods. Various learning-based approaches are developed to process the high volume of light field data efficiently for different vision tasks while improving performance. Taking into account the diversity of light field vision tasks and the deployed learning-based frameworks, it is necessary to survey the scattered learning-based works in the domain to gain insight into the current trends and challenges. This paper aims to review the existing learning-based solutions for light field imaging and to summarize the most promising frameworks. Moreover, evaluation methods and available light field datasets are highlighted. Lastly, the review concludes with a brief outlook for future research directions.

Conventional photography can only provide a two-dimensional image of the scene, whereas emerging imaging modalities such as light field enable the representation of higher dimensional visual information by capturing light rays from different directions. Light fields provide immersive experiences, a sense of presence in the scene, and can enhance different vision tasks. Hence, research into light field processing methods has become increasingly popular. It does, however, come at the cost of higher data volume and computational complexity. With the growing deployment of machine-learning and deep architectures in image processing applications, a paradigm shift toward learning-based approaches has also been observed in the design of light field processing methods. Various learning-based approaches are developed to process the high volume of light field data efficiently for different vision tasks while improving performance. Taking into account the diversity of light field vision tasks and the deployed learning-based frameworks, it is necessary to survey the scattered learning-based works in the domain to gain insight into the current trends and challenges. This paper aims to review the existing learning-based solutions for light field imaging and to summarize the most promising frameworks. Moreover, evaluation methods and available light field datasets are highlighted. Lastly, the review concludes with a brief outlook for future research directions. [ABSTRACT FROM AUTHOR]

Nematic liquid‐crystal microlens arrays (LC‐MLAs) often exhibit chromatic aberration and low resolution, severely compromising their optical imaging quality. This study proposes an achromatic and resolution enhancement light field (ARELF) deep neural network (DNN) to address these issues. The training set is constructed by incorporating LC‐MLA characteristics' degradation, retrofitting the vimeo90k dataset. The network's hidden layer is trained to learn about chromatic aberration and low resolution of LC‐MLA while extracting imaging features and fusing the information of complementary features of a light field under varying voltages. The loss function includes both chromatic aberration and overall resolution. The light field images of ZnO LC‐MLA under seven consecutive voltages are used as input to test the proposed DNN model. After experimental verification, the proposed model effectively eliminates chromatic aberration while enhancing the spatial and temporal resolution of LC‐MLA. This novel network can be utilized to optimize the design process of LC‐MLA and significantly improve its imaging performance. [ABSTRACT FROM AUTHOR]

Nematic liquid‐crystal microlens arrays (LC‐MLAs) often exhibit chromatic aberration and low resolution, severely compromising their optical imaging quality. This study proposes an achromatic and resolution enhancement light field (ARELF) deep neural network (DNN) to address these issues. The training set is constructed by incorporating LC‐MLA characteristics’ degradation, retrofitting the vimeo90k dataset. The network's hidden layer is trained to learn about chromatic aberration and low resolution of LC‐MLA while extracting imaging features and fusing the information of complementary features of a light field under varying voltages. The loss function includes both chromatic aberration and overall resolution. The light field images of ZnO LC‐MLA under seven consecutive voltages are used as input to test the proposed DNN model. After experimental verification, the proposed model effectively eliminates chromatic aberration while enhancing the spatial and temporal resolution of LC‐MLA. This novel network can be utilized to optimize the design process of LC‐MLA and significantly improve its imaging performance.

Conventional stereoscopic displays suffer from vergence-accommodation conflict and cause visual fatigue. Integral-imaging-based displays resolve the problem by directly projecting the sub-aperture views of a light field into the eyes using a microlens array or a similar structure. However, such displays have an inherent trade-off between angular and spatial resolutions. In this paper, we propose a novel coded time-division multiplexing technique that projects encoded sub-aperture views to the eyes of a viewer with correct cues for vergence-accommodation reflex. Given sparse light field sub-aperture views, our pipeline can provide a perception of high-resolution refocused images with minimal aliasing by jointly optimizing the sub-aperture views for display and the coded aperture pattern. This is achieved via deep learning in an end-to-end fashion by simulating light transport and image formation with Fourier optics. To our knowledge, this work is among the first that optimize the light field display pipeline with deep learning. We verify our idea with objective image quality metrics (PSNR, SSIM, and LPIPS) and perform an extensive study on various customizable design variables in our display pipeline. Experimental results show that light fields displayed using the proposed technique indeed have higher quality than that of baseline display designs. [ABSTRACT FROM AUTHOR]

As one of the fundamental technologies for scene understanding, semantic segmentation has been widely explored in the last few years. Light field cameras encode the geometric information by simultaneously recording the spatial information and angular information of light rays, which provides us with a new way to solve this issue. In this paper, we propose a high-quality and challenging urban scene dataset, containing 1074 samples composed of real-world and synthetic light field images as well as pixel-wise annotations for 14 semantic classes. To the best of our knowledge, it is the largest and the most diverse light field dataset for semantic segmentation. We further design two new semantic segmentation baselines tailored for light field and compare them with state-of-the-art RGB, video and RGB-D-based methods using the proposed dataset. The outperforming results of our baselines demonstrate the advantages of the geometric information in light field for this task. We also provide evaluations of super-resolution and depth estimation methods, showing that the proposed dataset presents new challenges and supports detailed comparisons among different methods. We expect this work inspires new research direction and stimulates scientific progress in related fields. The complete dataset is available at https://github.com/HAWKEYE-Group/UrbanLF. [ABSTRACT FROM AUTHOR]

Highly realistic 3-D displays that can reproduce object images to look like physical objects are utilized for natural and correct remote operation in industrial scenes. Therefore, we developed a visually equivalent light field 3-D (VELF3D) display that can produce highly realistic, accurate images with a high resolution and a smooth, accurate motion parallax. However, the observation distance is slightly long, and users cannot reach the displayed images. Therefore, we aim to develop a tablet-computer-type VELF3D display that enables users to touch the displayed objects. The display viewpoint density has been increased to achieve a shorter observation distance, while maintaining the display depth range. Because higher resolutions are required for a close observation distance and increased display viewpoints, we aimed to improve the effective resolution using almost the same pixel pitch display panel. Therefore, we built a prototype that combines a vertical red, green, blue stripe display panel and a parallax barrier with subpixel width slits. We confirmed effective resolution improvement by tiny subjective tests. This method also helps increase the depth range of the display when it is observed from a normal distance. [ABSTRACT FROM AUTHOR]

Deep generative models have proven to be effective priors for solving a variety of image processing problems. However, the learning of realistic image priors, based on a large number of parameters, requires a large amount of training data. It has been shown recently, with the so-called deep image prior (DIP), that randomly initialized neural networks can act as good image priors without learning. In this paper, we propose a deep generative model for light fields, which is compact and which does not require any training data other than the light field itself. To show the potential of the proposed generative model, we develop a complete light field compression scheme with quantization-aware learning and entropy coding of the quantized weights. Experimental results show that the proposed method yields very competitive results compared with state-of-the-art light field compression methods, both in terms of PSNR and MS-SSIM metrics. [ABSTRACT FROM AUTHOR]

Many computer vision applications rely on feature detection and description, hence the need for computationally efficient and robust 4D light field (LF) feature detectors and descriptors. In this paper, we propose a novel light field feature descriptor based on the Fourier disparity layer representation, for light field imaging applications. After the Harris feature detection in a scale-disparity space, the proposed feature descriptor is then extracted using a circular neighborhood rather than a square neighborhood. It is shown to yield more accurate feature matching, compared with the LiFF LF feature, with a lower computational complexity. In order to evaluate the feature matching performance with the proposed descriptor, we generated a synthetic stereo LF dataset with ground truth matching points. Experimental results with synthetic and real-world dataset show that our solution outperforms existing methods in terms of both feature detection robustness and feature matching accuracy. [ABSTRACT FROM AUTHOR]

Most existing salient object detection (SOD) methods are designed for RGB images and do not take advantage of the abundant information provided by light fields. Hence, they may fail to detect salient objects of complex structures and delineate their boundaries. Although some methods have explored multi-view information of light field images for saliency detection, they require tedious pixel-level manual annotations of ground truths. In this paper, we propose a novel weakly-supervised learning framework for salient object detection on light field images based on bounding box annotations. Our method has two major novelties. First, given an input light field image and a bounding-box annotation indicating the salient object, we propose a ground truth label hallucination method to generate a pixel-level pseudo saliency map, to avoid heavy cost of pixel-level annotations. This method generates high quality pseudo ground truth saliency maps to help supervise the training, by exploiting information obtained from the light field (including depths and RGB images). Second, to exploit the multi-view nature of the light field data in learning, we propose a fusion attention module to calibrate the spatial and channel-wise light field representations. It learns to focus on informative features and suppress redundant information from the multi-view inputs. Based on these two novelties, we are able to train a new salient object detector with two branches in a weakly-supervised manner. While the RGB branch focuses on modeling the color contrast in the all-in-focus image for locating the salient objects, the Focal branch exploits the depth and the background spatial redundancy of focal slices for eliminating background distractions. Extensive experiments show that our method outperforms existing weakly-supervised methods and most fully supervised methods. [ABSTRACT FROM AUTHOR]

Recently, research on six-degree-of-freedom virtual reality (VR) systems based on a light field (LF) has been actively conducted. The LF-based approach is photorealistic and less prone to errors than the existing three-dimensional (3D) modeling-based approach. On the other hand, because the amount of data is very large for LF-based immersive virtual reality, it is important to manage the data efficiently. This paper makes two major contributions to data management for rendering acceleration in this context. First, the GPUs commonly used for high-speed rendering require optimization of the data transfer process between the host memory and the device memory. This paper proposes a simple but effective way to maximize data reuse by finding the proper size of a transfer unit depending on the target applications and the system capability constraints. Also, view synthesis and data transfer are performed in a pipelined manner to hide the data movement time. Second, for the LF rendering of a wide space, data management in the storage-host-GPU hierarchy is attempted for the first time. As the size of the virtual space increases, the required amount of LF data grows rapidly and exceeds the allocatable host memory capacity. Because the loading of data stored in storage is a very slow process, a system-level LF rendering structure that carefully considers the characteristics of the storage-host-GPU memory hierarchy should be designed. In this paper, progressive LF updating is proposed, and it prevents rendering stalls caused by slow storage data loading. Experimental results show that a 360-degree view with a $9000 \times 2048$ resolution can be rendered in an average of 2.57 ms from LF data with a $4096 \times 2048$ resolution, achieving performance close to 400 frames per second. [ABSTRACT FROM AUTHOR]

The presence of monocular depth sense is identified with a light field imaging system which can project up to 8 different view images simultaneously to a viewer’s each eye. The depth of field of subjects’ eyes increases further as the number of simultaneously projected images to subjects’ each eye increases more, though the increasing rate is somewhat different for different subjects. The diopter values exceed more than those of real object as the number exceeds more than six for the binocular viewing while it is eight for monocular viewing. The increasing rate of the diopter values for the binocular viewing is more than that for monocular viewing. These results assure that a natural viewing condition can be incorporated in light field imaging systems. These results are derived from 7 subjects under age 35, having eye sight greater than 1.0. [ABSTRACT FROM AUTHOR]

The Degree of Polarization (DoP) of a light beam inside a medium contains unique information about the medium. 3D imaging techniques constitute an optimal procedure for determining the DoP under low light conditions, as the computational reconstruction process can increase the signal-to-noise ratio of the detected light. The definition of the DoP contains a division by the total number of detected photons from the sensor. However, under photon starved conditions, the number of detected photons at a single time period may be equal to zero. This may pose a division by zero problem for the computation of DoP. In this work, we consider a truncated Poisson distribution to overcome this problem and show that the mean value of the computed DoP goes to zero independently of the state of polarization of the light. The validity of our approach is verified by capturing the light fields of a test object to compute its DoP under low light conditions. The formulae derived in this work can be used to correct the deviation of the mean value of the DoP with respect to the ideal measurements. [ABSTRACT FROM AUTHOR]

In the area of neural rendering-based novel view synthesis, illumination is important since shadows cast by objects under various light sources provide indications about their geometries and materials. However, due to high physical device complexity and simulation distortion, large-scale photorealistic multiple illumination multi-view datasets are difficult to obtain. In order to address this problem, a physical-virtual interactive parallel light fields based collection method is proposed in this paper. The physical part of parallel light fields is firstly used to capture 3D models and 2D images of objects under different lights. Then a Reak-to-Sim adaptation module was proposed to enhance realism by estimating material characteristic. Instead of manually setting, the learned resulting material parameters are then utilized to initialize virtual engine blender for subsequent rendering and data collection. Besides, to better handle self-occlusion problem in the acquired parallel light fields dataset, a conditional visibility module is designed in modeling visibility of each sampling point along a sampling ray. Compared with the Neuray, by introducing Conditional Normalizing Flow, visibility are assumed as samples from some distribution due to the fact that visibilities along the ray should be monotonically decreasing and are within the range of [ 0 , 1 ]. The visibility are calculated in a data driven manner, which brings more flexibility. By pretraining the conditional visibility network in parallel light field dataset, experiments demonstrate that more photorealistic inputs improve Peak-Signal-Noise Ratio (PSNR) and Structure Similarity Index Measure (SSIM) by 0.11% and 0.68% in validation dataset NeRF synthesis and LLFF. Besides, compared to Neuray, the proposed conditional visibility module is more flexible and get a PSNR improvement of 0.55 and 0.5 in NeRF synthesis and LLFF dataset, respectively. [ABSTRACT FROM AUTHOR]

Light Field (LF) video technology is a step towards offering a better immersive experience through on-demand refocusing and perspective viewing. However, the significant increase in captured data makes the need for efficient compression of paramount importance. In this paper, we proposed two prediction structures and coding orders that efficiently compress LF video content using the existing HEVC standard. This is achieved by utilizing horizontal and vertical correlation among the views for better inter-view prediction. To assess the performance of the schemes, ten publicly available and widely used LF video sequences were used. Our first method is highly suitable for applications demanding high compression efficiency. It outperforms the best pseudo-sequence-based compression technique to date by up to 17% in bitrate reduction while being scalable in the number of views. The second method is proficient for low computational and random-access complexity to any arbitrary view in the light field video. It offers 10% faster decoding and 20% lower random-access complexity compared to the best existing technique. [ABSTRACT FROM AUTHOR]

Light field images have emerged as an important advance in the representation of visual data; at the same time, dealing with such images has brought new challenges related to multimedia transmission and security. In this article, we introduce a robust and blind scheme to watermark lenslet light field images. The method, which is based on quantization index modulation, allows to recover the watermark even after the image is submitted to compression in the 4D-DCT (four-dimensional discrete cosine transform) domain or other attacks. To achieve this, an empirical strategy for scanning the coefficients in the referred domain is also proposed; we demonstrate that, in general, such a strategy outperforms state-of-the-art methods whose purpose is to exploit the energy compacting property of the 4D-DCT. The introduced watermarking scheme has been evaluated using 30 light field images and several embedding / extracting parameters. The obtained results indicate that the watermark is satisfactorily imperceptible and confirm its robustness against various attacks.

We present twisted spatiotemporal optical vortex (STOV) beams, which are partially coherent light sources that possess a coherent optical vortex and a random twist coupling their space and time dimensions. These beams have controllable partial coherence and transverse orbital angular momentum (OAM), which distinguishes them from the more common spatial vortex and twisted beams (known to carry longitudinal OAM) in the literature and should ultimately make them useful in applications such as optical communications and optical tweezing. We present the mathematical analysis of twisted STOV beams, deriving the mutual coherence function and linear and angular momentum densities. We simulate the synthesis of twisted STOV beams and investigate their free-space propagation characteristics. We discuss how to physically generate twisted STOV fields and lastly conclude with a summary and brief discussion of future research.

Light field (LF) image super-resolution (SR) aims at reconstructing high-resolution LF images from their low-resolution counterparts. Although CNN-based methods have achieved remarkable performance in LF image SR, these methods cannot fully model the non-local properties of the 4D LF data. In this paper, we propose a simple but effective Transformer-based method for LF image SR. In our method, an angular Transformer is designed to incorporate complementary information among different views, and a spatial Transformer is developed to capture both local and long-range dependencies within each sub-aperture image. With the proposed angular and spatial Transformers, the beneficial information in an LF can be fully exploited and the SR performance is boosted. We validate the effectiveness of our angular and spatial Transformers through extensive ablation studies, and compare our method to recent state-of-the-art methods on five public LF datasets. Our method achieves superior SR performance with a small model size and low computational cost. Code is available at. [ABSTRACT FROM AUTHOR]

Phase-only holograms are more attractive than the amplitude holograms for the higher energy utilization and the possibility to realize on-axis holographic 3D display without conjugate image. In this study, we propose a high-resolution phase-only holographic 3D display using the light field images rendered in the frequency domain. The high-resolution phase-only hologram contains a large number of elemental phase-only holograms (EPHs), and each EPH is calculated from the light field image of the corresponding viewing point in frequency domain through weighted GS (Gerchberg-Saxton) algorithm. Parallel calculation is performed to speed up the calculation in the row direction and the previous EPH is used as the initial phase for the current EPH calculation in order to improve the display quality due to the great similarity between adjacent light field images. Two high-resolution phase-only holograms both with the size of 64 mm×64 mm and the resolution of 200k×200k pixels were calculated and printed by our homemade holographic printer. The full-parallax and high quality 3D displays were verified by optical experiments, which have the potential to be applied in holographic advertising and other fields. [ABSTRACT FROM AUTHOR]

The emerging light field images (LFIs) support 6 degrees of freedom (6DoF) user interaction, which is the key feature for future virtual reality (VR) media experiences. Compared to regular 2-D images, LFIs are characterized by particular image structure with both spatial and angular information. In practice, it is infeasible for the user to manually edit each subaperture of the LFI, respectively, and the user cannot guarantee the parallax consistency between different subapertures. To address this problem, we propose a deep-learning-based LFI editing scheme named central view augmentation propagation (CVAP), which employs interleaved spatial-angular convolutional neural networks (4-D CNN) for effective learning of both spatial and angular features from the input LFI. Moreover, for comparison purposes, we also implemented a "direct editing" scheme based on the geometry correspondence between subviews, and another benchmark method based on light field super resolution (LFSR). The experimental results show that CVAP achieved higher PSNR and overall more pleasant visual quality than direct editing and LFSR. [ABSTRACT FROM AUTHOR]

Efficient and accurate segmentation of full 4D light fields is an important task in computer vision and computer graphics. The massive volume and the redundancy of light fields make it an open challenge. In this article, we propose a novel light field hypergraph (LFHG) representation using the light field super-pixel (LFSP) for interactive light field segmentation. The LFSPs not only maintain the light field spatio-angular consistency, but also greatly contribute to the hypergraph coarsening. These advantages make LFSPs useful to improve segmentation performance. Based on the LFHG representation, we present an efficient light field segmentation algorithm via graph-cut optimization. Experimental results on both synthetic and real scene data demonstrate that our method outperforms state-of-the-art methods on the light field segmentation task with respect to both accuracy and efficiency. [ABSTRACT FROM AUTHOR]

We present a novel learning-based approach to synthesize new views of a light field image. In particular, given the four corner views of a light field, the presented method estimates any in-between view. We use three sequential convolutional neural networks for feature extraction, scene geometry estimation and view selection. Compared to state-of-the-art approaches, in order to handle occlusions we propose to estimate a different disparity map per view. Jointly with the view selection network, this strategy shows to be the most important to have proper reconstructions near object boundaries. Ablation studies and comparison against the state of the art on Lytro light fields show the superior performance of the proposed method. Furthermore, the method is adapted and tested on light fields with wide baselines acquired with a camera array and, in spite of having to deal with large occluded areas, the proposed approach yields very promising results. [ABSTRACT FROM AUTHOR]

Light fields capture both the spatial and angular rays, thus enabling free-viewpoint rendering and custom selection of the focal plane. Scientists can interactively explore pre-recorded microscopic light fields of organs, microbes, and neurons using virtual reality headsets. However, rendering high-resolution light fields at interactive frame rates requires a very high rate of texture sampling, which is challenging as the resolutions of light fields and displays continue to increase. In this article, we present an efficient algorithm to visualize $4D$ 4 D light fields with 3D-kernel foveated rendering (3D-KFR). The 3D-KFR scheme coupled with eye-tracking has the potential to accelerate the rendering of $4D$ 4 D depth-cued light fields dramatically. We have developed a perceptual model for foveated light fields by extending the KFR for the rendering of $3D$ 3 D meshes. On datasets of high-resolution microscopic light fields, we observe $3.47\times -7.28\times$ 3. 47 × - 7. 28 × speedup in light field rendering with minimal perceptual loss of detail. We envision that 3D-KFR will reconcile the mutually conflicting goals of visual fidelity and rendering speed for interactive visualization of light fields. [ABSTRACT FROM AUTHOR]

The commercial light field camera is able to capture four-dimensional Light Field Image (LFI), which can be visualized to LFI contents on 2D displays by means of the Pseudo Video (PV) or the Refocused Images (RIs) generated with the refocusing function of LFI. However, the quality degradation of LFI will affect user’s visual experience of LFI contents. Hence, it is crucial to develop an effective LFI quality assessment method to monitor the LFI quality. Most existing subjective databases of LFI use PV and RIs visualization techniques to assess the quality of LFI. Therefore, as the way of presenting LFI on 2D display, PV and RIs are closely related to the subjective perception of LFI by human eyes. Based on these two visualization techniques, this article proposes a novel PV and RIs based blind LFI quality assessment method, in which the feature extraction is divided into two parts. In the first part, the PV’s structure, motion and disparity information are extracted with multi-scale and multi-directional Shearlet transform. In the other part, the spatial structure, depth and semantic information of the RIs are obtained. Finally, support vector regression is used to nonlinear map the perceptual features to quality score of LFI. The experimental results on four LFI databases show that the proposed method has better correlation with human visual perception, compared with the classical 2D image quality assessment methods as well as the state-of-the-art LFI quality assessment methods. [ABSTRACT FROM AUTHOR]

Semantic segmentation is of great importance and a challenge in computer vision. One of its main problems is how to efficiently obtain rich information (geometric structure) and identify useful features from higher dimensions. A light field camera, due to its special microlens array structure, can completely record the angular-spatial information of scenes, which is attractive and has great potential to improve the performance of semantic segmentation tasks. Inspired by this, we propose an end-to-end semantic segmentation network that can process light field macropixel images robustly and extract their features. In addition, this network can flexibly and efficiently load the different popular deep learning backbones. Furthermore, we propose an efficient angular model, which, to learn the angular features between the different viewpoints of the macropixel image, improves the nonlinearity of angular-spatial features and enhances multichannel semantic correlations. To evaluate the network, we construct a new real scene light field dataset comprising 800 high-quality samples. The quantitative and qualitative results show that the highest mean intersection over union (mIoU) based on our algorithm is greater than 57%. Our algorithm achieves a 10.30% increase compared with state-of-the-art semantic segmentation algorithms. In combination with different backbones or multiscale light field macropixel images, the network can also achieve comparable results. This preliminary work demonstrates that the combination of light field imaging and deep learning technology has potential applications in the future study of semantic segmentation. [ABSTRACT FROM AUTHOR]

The phase measuring deflectometry is a powerful measuring method for complex optical surfaces, which captures the reflected fringe images associated with a displaying screen and calculates the normal vectors of the surface under test (SUT) accordingly. The captured images are usually set conjugate to the SUT, which in turn makes the screen defocused. As a result, the blurring effect caused by the point spread function (PSF) of the off-axis catadioptric imaging system can bias the solved phases. In order to correct the phase errors, a light field is constructed based on the Fourier compressive sensing method to describe the light transmission between the screen and camera pixels. Fringe modulation is conducted to enhance the robustness against noise, and then, space-variant PSFs can be extracted from the light field. The true phases are obtained by solving a Wiener deconvolution problem, with the merit function adaptively regularized by adjusting the damping parameter. The proposed method can correct adaptively the phase errors induced by the complex aberrations. Compared to the reference measurements, the form accuracy can be improved by four times. [ABSTRACT FROM AUTHOR]

Light fields (LFs) and light field videos (LFVs) capture both angular and spatial variation of light rays emanating from scenes. This richness of information leads to novel applications such as post-capture refocusing, depth estimation and depth-velocity filtering which are not possible with images and videos. These capabilities come, however, with a significant increase in data to be processed. In order to fully exploit opportunities provided by LFs and LFVs, low-complexity signal processing algorithms that process LF and LFV data in real-time are required. In this brief, we survey such state-of-the-art algorithms, in particular for depth filtering, refocusing and denoising of LFs and depth-velcoty filtering for LFVs, and future directions for these real-time LF an LFV processing algorithms. [ABSTRACT FROM AUTHOR]

The image quality might be degraded severely due to water scatterings, especially near-shore water scatterings. In addition, the degraded image reduces the accuracy of underwater 3D reconstruction. We propose an underwater imaging system based on light field camera array to obtain enhanced high-quality images and accurate three-dimensional reconstruction from harsh underwater environments. Both 2D-imaging and 3D-reconstruction using light-field imaging were studied by experiments. By refocusing and analyzing the light field images of the target through turbid water, we obtained accurate three-dimensional estimations in the underwater scene. These experiments confirm the reliability of our method, showing that the SSIM (structural similarity) value of the system in turbid water is 0.378, compared to 0.197, the SSIM value of conventional single-camera imaging in the experiments. Therefore, we verify that the light field imaging system is less affected by forward scattering of water and has a larger detective range than conventional single-camera imaging with any specific resolution. [ABSTRACT FROM AUTHOR]

The three-dimensional reconstruction of bubble flow field is of great significance to study the motion of gas-liquid two phase flow. Combined with the abundant information of bubbles in the light field images, a fast and accurate 3D reconstruction method for dilute bubbly flow based on the deep learning algorithm DIF-LeNet (Double Information Fusion with LeNet5 net) is proposed in this work. The calibration method and bubble segmentation algorithm of light field image for data processing is detailed. DIF-LeNet realizes the fusion of different dimension data and regression prediction. By DIF-LeNet, bubble depth could be predicted by fusing the refocused bubble image with the focal distance of the refocused image. Then, the three-dimensional bubble field could be reconstructed with the bubble depth, bubble center and bubble diameter. Comparing with the reconstruction method based on statistics and LeNet5, the reconstruction accuracy and speed of the proposed method based on DIF-LeNet are improved. Especially for the bubble without focused image in the refocused image sequence, the effect of 3D reconstruction based on DIF-LeNet is much prominent. [ABSTRACT FROM AUTHOR]

We present twisted spatiotemporal optical vortex (STOV) beams, which are partially coherent light sources that possess a coherent optical vortex and a random twist coupling their space and time dimensions. These beams have controllable partial coherence and transverse orbital angular momentum (OAM), which distinguishes them from the more common spatial vortex and twisted beams (known to carry longitudinal OAM) in the literature and should ultimately make them useful in applications such as optical communications and optical tweezing. We present the mathematical analysis of twisted STOV beams, deriving the mutual coherence function and linear and angular momentum densities. We simulate the synthesis of twisted STOV beams and investigate their free-space propagation characteristics. We discuss how to physically generate twisted STOV fields and lastly conclude with a summary and brief discussion of future research. [ABSTRACT FROM AUTHOR]

A microfiber Mach-Zehnder interferometer (MMZI) with two spindle-shaped structures was reported for measuring temperature and curvature. Due to the core-separated structure, the MMZI could excite more high-order modes, which improves the sensitivity more efficiently. When the central waist region is 4 μm, the sensitivities for temperature and curvature were 1290.4pm /°C and 56.1947 nm/m−1, respectively. Furthermore, the sensitivity could be improved effectively by increasing the length of the central waist region. Increasing the length of central waist region to 6 μm, the temperature sensitivity of the sensor could up to 1756.3 pm/°C, which was the highest temperature sensitivity we have known in MMZI. [ABSTRACT FROM AUTHOR]

In four-dimensional (4-D) light field (LF) processing, 4-D linear shift-invariant filters having hyperplanar passbands are used for depth-based scene enhancement. In this paper, two low-sensitivity and low-complexity field programmable gate array (FPGA)-based digital hardware architectures for 4-D hyperplanar filters are proposed for on-chip real-time processing of LFs. Both 4-D filters are designed exploiting resonant properties of multi-dimensional passive prototype networks, and are realized as wave digital filters (WDFs). The two 4-D WDF realizations are implemented as raster-scanned processing architectures on a Xilinx Virtex 6 Sx35 FPGA with a real-time clock speed of up to 33 MHz. This corresponds to a real-time throughout of 16.8 LFs/s for an LF of size 11 × 11 × 128 × 128 . [ABSTRACT FROM AUTHOR]

Light field image (LFI) collects radiance from rays in different directions, offers powerful capabilities for immersive media and computer vision. As the high-dimensional data, LFI suffers from spatial as well as angular information distortions in its processing, which brings new challenges to image quality assessment (IQA). Based on the strong ability of tensor about representing high-dimensional data and distortion characteristics of LFI, this paper proposes a method of combining tensor slice and singular value for blind light field image quality assessment (TSSV-LFIQA) to effectively evaluate the quality of LFI content. Specifically, five-order tensor representation of LFI is firstly defined which contains light ray intensity, angular information and color information of the LFI. Secondly, the first slice sharpness measurement and the other slice information distribution are used to describe the tensor slice spatial feature (TSSF) of the LFI. Moreover, singular value angular feature (SVAF) is also proposed to measure the angular consistency of LFI by further unfolding the five-order tensor of LFI and analyzing the percentage of singular values. The experimental results show that benefiting from the combination of TSSF and SVAF, the proposed TSSV-LFIQA method is statistically superior to the existing IQA methods, and matches well with human subjective opinions. [ABSTRACT FROM AUTHOR]

Salient object detection, which simulates human visual perception in locating the most significant object(s) in a scene, has been widely applied to various computer vision tasks. Now, the advent of depth sensors means that depth maps can easily be captured; this additional spatial information can boost the performance of salient object detection. Although various RGB-D based salient object detection models with promising performance have been proposed over the past several years, an in-depth understanding of these models and the challenges in this field remains lacking. In this paper, we provide a comprehensive survey of RGB-D based salient object detection models from various perspectives, and review related benchmark datasets in detail. Further, as light fields can also provide depth maps, we review salient object detection models and popular benchmark datasets from this domain too. Moreover, to investigate the ability of existing models to detect salient objects, we have carried out a comprehensive attribute-based evaluation of several representative RGB-D based salient object detection models. Finally, we discuss several challenges and open directions of RGB-D based salient object detection for future research. All collected models, benchmark datasets, datasets constructed for attribute-based evaluation, and related code are publicly available at https://github.com/taozh2017/RGBD-SODsurvey. [ABSTRACT FROM AUTHOR]

Light field manipulation is an important technology for improving system-level efficiency of optical wireless power transfer (OWPT). In this paper, we propose and compare the Gerchberg-Saxton algorithm, direct binary search and genetic algorithm enabled light field optimization for adapting photovoltaic cells in different geometric arrangements. Under static condition, photoelectric conversion efficiency can reach 33.14% through optimizing light field using Gerchberg-Saxton algorithm, almost 200% improvement over the case with Gaussian beams. Moreover, the resilience of power delivery can be improved by employing direct binary search and genetic algorithm in a dynamic fashion. By utilizing genetic algorithm optimization, initial photoelectric conversion efficiency is 14.11% with the help of random optimization process. The dynamic light field optimization technology will be an enabling technology of high efficient power delivery for mobile objects. [ABSTRACT FROM AUTHOR]

A novel method is described for adaptive filtering of light fields to enhance objects at a given depth. Using the frequency domain of an epipolar-plane image (EPI) to select the minimum and maximum depths of an object of interest (OOI) allows greater selectivity over traditional methods and the ability to re-focus a light field as the scene changes. This method is executed on real light fields, where depth information is extracted and used for depth filtering. A light field video is used to show that as an OOI moves to varying depths in a scene, the performance of a fixed-depth filter decreases when compared to an adaptive-depth filter. This method is shown to be robust in an environment where an OOI is moving to different depths in relation to the camera, and has implications in tasks where objects must be identified by their depth, such as in robotics or autonomous vehicles. [ABSTRACT FROM AUTHOR]

Geometric information of scenes available with four-dimensional (4-D) light fields (LFs) paves the way for post-capture refocusing. Light field refocusing methods proposed so far have been limited to a single planar or a volumetric region of a scene. In this letter, we demonstrate simultaneous refocusing of multiple volumetric regions in LFs. To this end, we employ a 4-D sparse finite-extent impulse response (FIR) filter consisting of multiple hyperfan-shaped passbands. We design the 4-D sparse FIR filter as an optimal filter in the least-squares sense. Experimental results confirm that the proposed filter provides 63% average reduction in computational complexity with negligible degradation in the fidelity of multi-volumetric refocused LFs compared to a 4-D nonsparse FIR filter. [ABSTRACT FROM AUTHOR]

Face recognition has attracted increasing attention due to its wide range of applications, but it is still challenging when facing large variations in the biometric data characteristics. Lenslet light field cameras have recently come into prominence to capture rich spatio-angular information, thus offering new possibilities for advanced biometric recognition systems. This paper proposes a double-deep spatio-angular learning framework for light field-based face recognition, which is able to model both the intra-view/spatial and inter-view/angular information using two deep networks in sequence. This is a novel recognition framework that has never been proposed in the literature for face recognition or any other visual recognition task. The proposed double-deep learning framework includes a long short-term memory (LSTM) recurrent network, whose inputs are VGG-Face descriptions, computed using a VGG-16 convolutional neural network (CNN). The VGG-Face spatial descriptions are extracted from a selected set of 2D sub-aperture (SA) images rendered from the light field image, corresponding to different observation angles. A sequence of the VGG-Face spatial descriptions is then analyzed by the LSTM network. A comprehensive set of experiments has been conducted using the IST-EURECOM light field face database, addressing varied and challenging recognition tasks. The results show that the proposed framework achieves superior face recognition performance when compared to the state of the art. [ABSTRACT FROM AUTHOR]

Existing light field depth map estimation approaches only utilize partial angular views in occlusion areas and local spatial dependencies in the optimization. This paper proposes a novel two-stage light field depth estimation method via graph spectral analysis to exploit the complete correlations and dependencies within angular patches and spatial images. The initial depth map estimation leverages the undirected graph to jointly consider occluded and unoccluded views within each angular patch. The estimated depth minimizes the structural incoherence of its corresponding angular patch with the focused one by evaluating the highest graph frequency component. Subsequently, depth map refinement optimizes the initial depth map with the color consistency and smoothness formulated by weighted adjacency matrix. The structural constraints are efficiently employed using low-pass graph filtering with Chebyshev polynomial approximation. Experimental results demonstrate that the proposed method improves the depth map estimation, especially in the edge regions. [ABSTRACT FROM AUTHOR]

Conventionally, a digital micromirror device (DMD) can only perform binary amplitude modulation of a light field. Simultaneous amplitude and phase modulation with a DMD is achieved by our proposed error diffusion scheme with a 4f double-lens setup for the first time. The DMD pixels are encoded by adaptive global optimization of binarization errors. In holographic projection, the object image can be optically reconstructed from a complex-amplitude hologram with this scheme. Experimental results show that our proposed error diffusion scheme outperforms the previous superpixel scheme in terms of the image quality and light efficiency of holographic reconstruction results under the same conditions. [ABSTRACT FROM AUTHOR]