Search

A three-dimensional (3D) metrology based on light field imaging is developed to study the behavior of rising bubbles in a cylindrical column. A multi-focus plenoptic camera is used to obtain the depth location of bubbles and their size. The all-in-focus images obtained from the light field have a different magnification along the x or y axis due to the astigmatism of the cylinder's window. This phenomenon also affects depth estimation and makes the measured depth dependent on the orientation of the bubble image's interface. We developed a new 3D calibration that takes into account these dependencies. We tested our method for two configurations and two regimes by changing the gaseous Reynolds number. Results show that the far configuration is more adapted as it provides a larger depth of field and field of view at the cost of a lower spatial resolution. We were able to obtain the 3D location and size distribution of bubbles for the studied regimes. Results show that light field imaging can be adapted to in-cylinder flow visualization. [ABSTRACT FROM AUTHOR]

In this paper, we study facial expression recognition (FER) using three modalities obtained from a light field camera: sub-aperture (SA), depth map, and all-in-focus (AiF) images. Our objective is to construct a more comprehensive and effective FER system by investigating multimodal fusion strategies. For this purpose, we employ EfficientNetV2-S, pre-trained on AffectNet, as our primary convolutional neural network. This model, combined with a BiGRU, is used to process SA images. We evaluate various fusion techniques at both decision and feature levels to assess their effectiveness in enhancing FER accuracy. Our findings show that the model using SA images surpasses state-of-the-art performance, achieving 88.13% ± 7.42% accuracy under the subject-specific evaluation protocol and 91.88% ± 3.25% under the subject-independent evaluation protocol. These results highlight our model's potential in enhancing FER accuracy and robustness, outperforming existing methods. Furthermore, our multimodal fusion approach, integrating SA, AiF, and depth images, demonstrates substantial improvements over unimodal models. The decision-level fusion strategy, particularly using average weights, proved most effective, achieving 90.13% ± 4.95% accuracy under the subject-specific evaluation protocol and 93.33% ± 4.92% under the subject-independent evaluation protocol. This approach leverages the complementary strengths of each modality, resulting in a more comprehensive and accurate FER system. [ABSTRACT FROM AUTHOR]

Light field cameras are unsuitable for further acquisition of high-quality images due to their small depth of field, insufficient spatial resolution, and poor imaging quality. To address these issues, we proposed a novel four-focal-square microlens and light field system. A square aspheric microlens array with four orthogonal focal lengths was designed, in which the aperture of a single lens was 100 μm. The square arrangement improves pixel utilization, the four focal lengths increase the depth of field, and the aspheric improves image quality. The simulations demonstrate pixel utilization rates exceeding 90%, depth-of-field ranges 6.57 times that of a single focal length, and image quality is significantly improved. We have provided a potential solution for improving the depth of field and image quality of the light field imaging system. [ABSTRACT FROM AUTHOR]

Camera arrays can enhance the signal-to-noise ratio (SNR) between dim targets and backgrounds through multi-view synthesis. This is crucial for the detection of dim targets. To this end, we design and develop an infrared camera array system with a large baseline. The multi-view synthesis of camera arrays relies heavily on the calibration accuracy of relative poses in the sub-cameras. However, the sub-cameras within a camera array lack strict geometric constraints. Therefore, most current calibration methods still consider the camera array as multiple pinhole cameras for calibration. Moreover, when detecting distant targets, the camera array usually needs to adjust the focal length to maintain a larger depth of field (DoF), so that the distant targets are located on the camera's focal plane. This means that the calibration scene should be selected within this DoF range to obtain clear images. Nevertheless, the small parallax between the distant sub-aperture views limits the calibration. To address these issues, we propose a calibration model for camera arrays in distant scenes. In this model, we first extend the parallax by employing dual-array frames (i.e., recording a scene at two spatial locations). Secondly, we investigate the linear constraints between the dual-array frames, to maintain the minimum degrees of freedom of the model. We develop a real-world light field dataset called NUDT-Dual-Array using an infrared camera array to evaluate our method. Experimental results on our self-developed datasets demonstrate the effectiveness of our method. Using the calibrated model, we improve the SNR of distant dim targets, which ultimately enhances the detection and perception of dim targets. [ABSTRACT FROM AUTHOR]

Among the novel 3D visualization technologies of our era, light field displays provide the complete 3D visual experience without the need for any personal viewing device. Due to the lack of such constraint, these displays may be viewed by any number of observers simultaneously, and the corresponding use case contexts may also involve a virtually unlimited numbers of users; any number that the valid viewing area of the display may accommodate. While many instances of the utilization of this technology operate with static contents, camera animation may also be relevant. While the topic of light field camera animation has already been addressed on an initial level, there are still numerous research efforts to be carried out. In this paper, we elaborate on the lessons learned from implementing light field camera animation. The paper discusses the associated implications, limitations, potentials, and future research efforts. Each of these areas are approached from the perspectives of use cases, visual content, and quality assessment, as well as capture and display hardware. Our work highlights the existing research gaps in the investigated topic, the severe issues related to visualization sharpness, and the lack of appropriate datasets, as well as the constraints due to which novel contents may be captured by virtual cameras instead of real capture systems. [ABSTRACT FROM AUTHOR]

In water electrolysis, which converts electricity into hydrogen, the distribution of bubbles affects ion transport in the electrolyte. Therefore, measuring the three‐dimensional distribution of bubbles in the electrolysis cell can be effective in improving efficiency. However, the electrolysis cell contains electrodes and membranes, making it difficult to use conventional stereo photography to measure the three‐dimensional bubble distribution. Therefore, in this study, we investigated the measurement of the three‐dimensional distribution of bubbles in liquids using a light field camera that enables three‐dimensional imaging from one direction by using a microlens array. The study was conducted by performing imaging experiments of bubbles ejected from an axisymmetric nozzle using a light field camera and determining whether the measurement of the three‐dimensional distribution of bubbles was possible. As a result, we found that the axisymmetric bubble distribution can be measured in three dimensions and that a light field camera can be used for this measurement. © 2024 Institute of Electrical Engineer of Japan and Wiley Periodicals LLC. [ABSTRACT FROM AUTHOR]

Under turbulent conditions, the images of objects may be severely blurred and become unrecognizable because of the complex aberrations introduced by turbulence. Using adaptive optics (AO) to compensate for these wavefront aberrations can theoretically solve the problem of image blurring under turbulent conditions. However, due to its small field of view and high cost, the AO system is difficult to widely use. In recent years, using the technology of light field photography to correct turbulence images has been proposed. However, the current studies are either based on the structure of the non-focused light field camera, which makes it hard to achieve good image quality, or on the need to redesign the core sensor of the non-focused light field camera, which greatly increases the cost and complexity. We believe that the structure of a focused light field camera has a good potential for turbulence image correction and propose a new method of achieving a corrected focused image using a focused light field camera in this paper. We propose to use phase estimation instead of depth estimation for a focused light field camera and phase maps instead of a depth map as the basis for achieving a focused image. By the proposed method, a corrected focused image with a large field of view under turbulent conditions is achieved without using independent wavefront correction devices or redesigning the light field camera. The experiments are given to verify the effectiveness of the proposed method under strong turbulent conditions in the laboratory and real turbulent conditions in the outdoors. The proposed method in this paper provides a new and low-cost solution for image correction for optical imaging systems under turbulent conditions, so it has a high application value. [ABSTRACT FROM AUTHOR]

Light-field video provides a detailed representation of scenes captured from different perspectives. This results in a visualisation modality that enhances the immersion and engagement of the viewers with the depicted environment. In order to perform research on compression, transmission and signal processing of light field data, datasets with light-field contents of different categories and acquired with different modalities are required. In particular, the development of machine learning models for quality assessment and for light-field processing, including the generation of new views, require large amounts of data. Most existing datasets consist of static scenes and, in many cases, synthetic contents. This paper presents a novel light-field plenoptic video dataset, KULFR8, involving six real-world scenes with moving objects and 336 distorted light-field videos derived from the original contents; in total, the original scenes in the dataset contain 1800 distinctive frames, with angular resolution of 5 × 5 with and total spatial resolution of 9600 × 5400 pixels (considering all the views); overall, the dataset consists of 45,000 different views with spatial resolution of 1920 × 1080 pixels. We analyse the content characteristics based on the dimensions of the captured objects and via the acquired videos using the central views extracted from each quilted frame. Additionally, we encode and decode the contents using various video encoders across different bitrate ranges. For quality assessments, we consider all the views, utilising frames measuring 9600 × 5400 pixels, and employ two objective quality metrics: PSNR and SSIM. [ABSTRACT FROM AUTHOR]

Among the common applications of plenoptic cameras are depth reconstruction and post-shot refocusing. These require a calibration relating the camera-side light field to that of the scene. Numerous methods with this goal have been developed based on thin lens models for the plenoptic camera's main lens and microlenses. Our work addresses the often-overlooked role of the main lens exit pupil in these models, specifically in the decoding process of standard plenoptic camera (SPC) images. We formally deduce the connection between the refocusing distance and the resampling parameter for the decoded light field and provide an analysis of the errors that arise when the exit pupil is not considered. In addition, previous work is revisited with respect to the exit pupil's role, and all theoretical results are validated through a ray tracing-based simulation. With the public release of the evaluated SPC designs alongside our simulation and experimental data, we aim to contribute to a more accurate and nuanced understanding of plenoptic camera optics. [ABSTRACT FROM AUTHOR]

This work describes the development of a particle tracking velocimetry (PTV) algorithm designed to improve three-dimensional (3D), three-component velocity field measurements using a single plenoptic camera. Particular focus is on mitigating the longstanding depth uncertainty issues that have traditionally plagued plenoptic particle image velocimetry (PIV) experiments by leveraging the camera's ability to generate multiple perspective views of a scene in order to assist both particle triangulation and tracking. 3D positions are first estimated via light field ray bundling (LFRB) whereby particle rays are projected into the measurement volume using image-to-object space mapping. Tracking is subsequently performed independently within each perspective view, providing a statistical amalgamation of each particle's predicted motion through time in order to help guide 3D trajectory estimation while simultaneously protecting the tracking algorithm from physically unreasonable fluctuations in particle depth positions. A synthetic performance assessment revealed a reduction in the average depth errors obtained by LFRB as compared to the conventional multiplicative algebraic reconstruction technique when estimating particle locations. Further analysis using a synthetic vortex ring at a magnification of − 0.6 demonstrated plenoptic-PIV capable of maintaining the equivalent of 0.1–0.15 voxel accuracy in the depth domain at a spacing to displacement ratio of 5.3–10.5, an improvement of 84–89% compared to plenoptic-PIV. Experiments were conducted at a spacing to displacement ratio of approximately 5.8 to capture the 3D flow field around a rotor within the rotating reference frame. The resulting plenoptic-PIV/PTV vector fields were evaluated with reference to a fixed frame stereoscopic-PIV (stereo-PIV) validation experiment. A systematic depth-wise (radial) component of velocity directed toward the wingtip, consistent with observations from prior literature and stereo-PIV experiments, was captured by plenoptic-PTV at magnitudes similar to the validation data. In contrast, the plenoptic-PIV did not discern any coherent indication of radial motion. Our algorithm constitutes a significant advancement in enhancing the functionality and versatility of single-plenoptic camera flow diagnostics by directly addressing the primary limitation associated with plenoptic imaging. [ABSTRACT FROM AUTHOR]

To uncover the underlying fluid mechanisms, it is crucial to explore imaging techniques for high-resolution and large-scale three-dimensional (3D) measurements of the flow field. Plenoptic background-oriented schlieren (Plenoptic BOS), an emerging volumetric method in recent years, has demonstrated being able to resolve volumetric flow dynamics with a single plenoptic camera. The focus-stack-based plenoptic BOS system can qualitatively infer the position of the density gradient in 3D space based on the relative sharpness of the refocused BOS image. Plenoptic BOS systems based on tomography or specular enhancement techniques are realized for use in high-fidelity 3D flow measurements due to the increased number of acquisition views. Here, we first review the fundamentals of plenoptic BOS, and then discuss the system configuration and typical application of single-view and multi-view plenoptic BOS. We also discuss the related challenges and outlook on the potential development of plenoptic BOS in the future. [ABSTRACT FROM AUTHOR]

Due to the presence of an additional glass pane between the camera and the scene, an additional reflection scene is captured in the image apart from the desired object sometimes. Images are more often captured from mobile handsets these days which have multiple cameras. This paper gives the advantage of multiple cameras. There exists a disparity and varied field of view when images are captured with multiple cameras. We use these two factors to act as a cue to remove reflection, as reflection intensity across the image pairs change with different field-of-view. The proposed method is robust and convenient to implement as it does not require an additional hardware, for example, light field camera for stereo images. Also, it does not make assumptions about the appearance or intensity of reflection. [ABSTRACT FROM AUTHOR]

Underwater image restoration has been a challenging problem for decades since the advent of underwater photography. Most solutions focus on shallow water scenarios, where the scene is uniformly illuminated by the sunlight. However, the vast majority of uncharted underwater terrain is located beyond 200 meters depth where natural light is scarce and artificial illumination is needed. In such cases, light sources co-moving with the camera, dynamically change the scene appearance, which make shallow water restoration methods inadequate. In particular for multi-light source systems (composed of dozens of LEDs nowadays), calibrating each light is time-consuming, error-prone and tedious, and we observe that only the integrated illumination within the viewing volume of the camera is critical, rather than the individual light sources. The key idea of this paper is therefore to exploit the appearance changes of objects or the seafloor, when traversing the viewing frustum of the camera. Through new constraints assuming Lambertian surfaces, corresponding image pixels constrain the light field in front of the camera, and for each voxel a signal factor and a backscatter value are stored in a volumetric grid that can be used for very efficient image restoration of camera-light platforms, which facilitates consistently texturing large 3D models and maps that would otherwise be dominated by lighting and medium artifacts. To validate the effectiveness of our approach, we conducted extensive experiments on simulated and real-world datasets. The results of these experiments demonstrate the robustness of our approach in restoring the true albedo of objects, while mitigating the influence of lighting and medium effects. Furthermore, we demonstrate our approach can be readily extended to other scenarios, including in-air imaging with artificial illumination or other similar cases. [ABSTRACT FROM AUTHOR]

This paper presents a hierarchical baseline stereo-matching framework for depth estimation using a novelly developed light field camera. The imaging process of a micro-lens array-based light field camera is derived. A macro-pixel map is constructed by treating each micro-lens as one macro-pixel in the light field's raw image. For each macro-pixel, a feature vector is represented by leveraging texture and gradient cues over the surrounding ring of neighboring macro-pixels. Next, the micro-lenses containing edges are detected on the macro-pixel map. Hierarchical baseline stereo-matching is performed by macro-pixel-wise coarse matching and pixel-wise fine matching, effectively eliminating matching ambiguities. Finally, a post-processing step is applied to improve accuracy. The lab-designed light field camera's imaging performance is evaluated in terms of accuracy and processing speed by capturing real-world scenes under studio lighting conditions. And an experiment using rendered synthetic samples is conducted for quantitative evaluation, showing that depth maps with local details can be accurately recovered. [ABSTRACT FROM AUTHOR]

In the last decade, many neural network algorithms have been proposed to solve depth reconstruction. Our focus is on reconstruction from images captured by multi-camera arrays which are a grid of vertically and horizontally aligned cameras that are uniformly spaced. Training these networks using supervised learning requires data with ground truth. Existing datasets are simulating specific configurations. For example, they represent a fixed-size camera array or a fixed space between cameras. When the distance between cameras is small, the array is said to be with a short baseline. Light-field cameras, with a baseline of less than a centimeter, are for instance in this category. On the contrary, an array with large space between cameras is said to be of a wide baseline. In this paper, we present a purely virtual data generator to create large training datasets: this generator can adapt to any camera array configuration. Parameters are for instance the size (number of cameras) and the distance between two cameras. The generator creates virtual scenes by randomly selecting objects and textures and following user-defined parameters like the disparity range or image parameters (resolution, color space). Generated data are used only for the learning phase. They are unrealistic but can present concrete challenges for disparity reconstruction such as thin elements and the random assignment of textures to objects to avoid color bias. Our experiments focus on wide-baseline configuration which requires more datasets. We validate the generator by testing the generated datasets with known deep-learning approaches as well as depth reconstruction algorithms in order to validate them. The validation experiments have proven successful. [ABSTRACT FROM AUTHOR]

In this study, a multispectral light-field imaging system is developed using nematic liquid-crystal microlens arrays (LC-MLAs). The system solves the spectral under-sampling and complex system issues in existing light-field and spectral cameras. The LC-MLA is the core optical element of the snapshot imaging system. No dispersion or customized scanning machinery is required to obtain multispectral light fields. Deep learning technology is utilized to reconstruct spectral information from raw data. In the experiments, it is demonstrated that the proposed method efficiently reconstructs multispectral data with 10 nm accuracy in the visible wavelength range of 400-700 nm. The system offers high precision, simple equipment, and relatively low cost without additional dispersion and filtering devices. [ABSTRACT FROM AUTHOR]

A light field camera can capture light information from various directions within a scene, allowing for the reconstruction of the scene. The light field image inherently contains the depth information of the scene, and depth estimations of light field images have become a popular research topic. This paper proposes a depth estimation network of light field images with occlusion awareness. Since light field images contain many views from different viewpoints, identifying the combinations that contribute the most to the depth estimation of the center view is critical to improving the depth estimation accuracy. Current methods typically rely on a fixed set of views, such as vertical, horizontal, and diagonal, which may not be optimal for all scenes. To address this limitation, we propose a novel approach that considers all available views during depth estimation while leveraging an attention mechanism to assign weights to each view dynamically. By inputting all views into the network and employing the attention mechanism, we enable the model to adaptively determine the most informative views for each scene, thus achieving more accurate depth estimation. Furthermore, we introduce a multi-scale feature fusion strategy that amalgamates contextual information and expands the receptive field to enhance the network's performance in handling challenging scenarios, such as textureless and occluded regions. [ABSTRACT FROM AUTHOR]

3D in vivo imaging techniques facilitate disease tracking and treatment, but bulky configurations and motion artifacts limit practical clinical applications. Compact light-field cameras with microlens arrays offer a feasible option for rapid volumetric imaging, yet their utilization in clinical practice necessitates an increased depth-of-field for handheld operation. Here, we report deep focus light-field camera (DF-LFC) with crosstalk-free solid immersion microlens arrays (siMLAs), allowing large depth-of-field and high-resolution imaging for handheld 3D intraoral scanning. The siMLAs consist of thin PDMS-coated microlens arrays and a metal–insulator–metal absorber to extend the focal length with low optical crosstalk and specular reflection. The experimental results show that the immersion of MLAs in PDMS increases the focal length by a factor of 2.7 and the transmittance by 5.6%–27%. Unlike conventional MLAs, the siMLAs exhibit exceptionally high f-numbers up to f/6, resulting in a large depth-of-field for light-field imaging. The siMLAs were fully integrated into an intraoral scanner to reconstruct a 3D dental phantom with a distance measurement error of 82 ± 41 μm during handheld operation. The DF-LFC offers a new direction not only for digital dental impressions with high accuracy, simplified workflow, reduced waste, and digital compatibility but also for assorted clinical endoscopy and microscopy. [ABSTRACT FROM AUTHOR]

Visual measurement methods are extensively used in various fields, such as aerospace, biomedicine, agricultural production, and social life, owing to their advantages of high speed, high accuracy, and non-contact. However, traditional camera-based measurement systems, relying on the pinhole imaging model, face challenges in achieving three-dimensional measurements using a single camera by one shot. Moreover, traditional visual systems struggle to meet the requirements of high precision, efficiency, and compact size simultaneously. With the development of light field theory, the light field camera has garnered significant attention as a novel measurement method. Due to its special structure, the light field camera enables high-precision three-dimensional measurements with a single camera through only one shot. This paper presents a comprehensive overview of light field camera measurement technologies, including the imaging principles, calibration methods, reconstruction algorithms, and measurement applications. Additionally, we explored future research directions and the potential application prospects of the light field camera. [ABSTRACT FROM AUTHOR]

Compound eye cameras are a vital component of bionics. Compound eye lenses are currently used in light field cameras, monitoring imaging, medical endoscopes, and other fields. However, the resolution of the compound eye lens is still low at the moment, which has an impact on the application scene. Photolithography and negative pressure molding were used to create a double-glued multi-focal bionic compound eye camera in this study. The compound eye camera has 83 microlenses, with ommatidium diameters ranging from 400 μm to 660 μm, and a 92.3 degree field-of-view angle. The double-gluing structure significantly improves the optical performance of the compound eye lens, and the spatial resolution of the ommatidium is 57.00 lp mm−1. Additionally, the measurement of speed is investigated. This double-glue compound eye camera has numerous potential applications in the military, machine vision, and other fields. [ABSTRACT FROM AUTHOR]

A polarized light sensor is applied to the front-end detection of a biomimetic polarized light navigation system, which is an important part of analyzing the atmospheric polarization mode and realizing biomimetic polarized light navigation, having received extensive attention in recent years. In this paper, biomimetic polarized light navigation in nature, the mechanism of polarized light navigation, point source sensor, imaging sensor, and a sensor based on micro nano machining technology are compared and analyzed, which provides a basis for the optimal selection of different polarized light sensors. The comparison results show that the point source sensor can be divided into basic point source sensor with simple structure and a point source sensor applied to integrated navigation. The imaging sensor can be divided into a simple time-sharing imaging sensor, a real-time amplitude splitting sensor that can detect images of multi-directional polarization angles, a real-time aperture splitting sensor that uses a light field camera, and a real-time focal plane light splitting sensor with high integration. In recent years, with the development of micro and nano machining technology, polarized light sensors are developing towards miniaturization and integration. In view of this, this paper also summarizes the latest progress of polarized light sensors based on micro and nano machining technology. Finally, this paper summarizes the possible future prospects and current challenges of polarized light sensor design, providing a reference for the feasibility selection of different polarized light sensors. [ABSTRACT FROM AUTHOR]

This paper proposes a novel method for three-dimensional (3D) temperature measurement using light field colorimetric thermometry, aiming to overcome the challenges associated with the intricate system structure and the limited availability of 3D information in traditional radiation temperature measurement methods. Firstly, the correlation between corresponding image points and the positions of 3D object in the light field image system is analyzed using the ray tracing method. The 3D position acquisition model and the light field colorimetric thermometry model are established, enabling simultaneous acquisition of the spatial coordinates and radiometric information of the 3D object. Then, the light field camera radiation calibration experiment was conducted, and the 3D temperature field will be obtained by employing colorimetric thermometry for each corresponding image point of the same object point. Finally, the experiment employed a light field camera for temperature measurement and reconstruction of candle flames. The accuracy of the temperature measurement is 3.31%, thus confirming the feasibility and effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Recording interference holograms using coherent light is a complex process for real-world objects. Alternatively, holography based on integral imaging with natural illumination is considered an option. However, the field of view (FOV) of this technology is limited by the lenslet array of the integral imaging system. This paper proposes an integral imaging-based hologram recording method via a commercial plenoptic camera to increase the FOV for a conventional II system. Meanwhile, to address the problem of depth non-uniform compression caused by the main lens of the plenoptic camera, this paper proposed a depth recovery method combined with the camera's metadata parameters for depth retargeting. Finally, we introduce a reflective Fourier-ptychography-based algorithm via an optimum overlap rate to acquire high-performance holograms. Compared to stereotypical integral imaging holography, this method produces a higher resolution reconstruction result. In addition, we conduct three sets of experiments on real scenes whose results also prove the proposed method embodies finer reconstruction results, and depth is recovered up to twice as much as the original outcome. [ABSTRACT FROM AUTHOR]

A light-field camera is an optical system that provides multi-field-angle information in one shot. The microlens array (MLA) in a light-field camera system is the key to provide multi-field-angle information. However, its numerical aperture limits the microlens' spatial resolution and the image's quality. We proposed a new microlens array for the light-field camera in this work. This MLA consists of one larger primary microlens and 72 satellite microlenses, simultaneously providing high-resolution image information from different angles. Moreover, this MLA can be easily fabricated by nanoimprint at a low cost. The measured spatial resolution of the primary microlens is 53.74 lp/mm higher than satellite microlens when MTF = 0.3. Finally, a one-shot photograph of a light-field camera shows one high-resolution and other views from different angles of the bear doll. The high-resolution light-field camera has great potential to apply in object ranging and face recognition in the future. [ABSTRACT FROM AUTHOR]

Light field (LF) images taken by plenoptic cameras can record spatial and angular information from real-world scenes, and it is beneficial to fully integrate these two pieces of information to improve image super-resolution (SR). However, most of the existing approaches to LF image SR cannot fully fuse the information at the spatial and angular levels. Moreover, the performance of SR is hindered by the ability to incorporate distinctive information from different views and extract informative features from each view. To solve these core issues, we propose a fusion and allocation network (LF-FANet) for LF image SR. Specifically, we have designed an angular fusion operator (AFO) to fuse distinctive features among different views, and a spatial fusion operator (SFO) to extract deep representation features for each view. Following these two operators, we further propose a fusion and allocation strategy to incorporate and propagate the fusion features. In the fusion stage, the interaction information fusion block (IIFB) can fully supplement distinctive and informative features among all views. For the allocation stage, the fusion output features are allocated to the next AFO and SFO for further distilling the valid information. Experimental results on both synthetic and real-world datasets demonstrate that our method has achieved the same performance as state-of-the-art methods. Moreover, our method can preserve the parallax structure of LF and generate faithful details of LF images. [ABSTRACT FROM AUTHOR]

Compressive light field cameras have attracted notable attention over the past few years because they can efficiently determine redundancy from light fields. However, much of the research has only concentrated on reconstructing the entire light field from compressed sampling, which ignores the possibility of directly extracting information such as depth from it. In this paper, we introduce a light field camera configuration with a random color-coded microlens array. Considering the color-coded light fields, we propose a novel attention-based encoder–decoder network. Specifically, the encoder part compresses the coded measurement into a low-dimensional representation that removes most redundancy, and the decoder part constructs the depth map directly from the latent representation. The attention mechanism enables the network to process spatial and angular features dynamically and effectively, thus significantly improving performance. Extensive experiments on synthetic and real-world datasets show that our method outperforms the state-of-the-art light field depth estimation method designed for non-coded light fields. To our knowledge, this is the first study that combines the color-coded light field with the attention-based deep learning approach, which provides a crucial insight into the design of enhanced light field photography systems. [ABSTRACT FROM AUTHOR]

Rice seed variety purity, an important index for measuring rice seed quality, has a great impact on the germination rate, yield, and quality of the final agricultural products. To classify rice varieties more efficiently and accurately, this study proposes a multimodal l fusion detection method based on an improved voting method. The experiment collected eight common rice seed types. Raytrix light field cameras were used to collect 2D images and 3D point cloud datasets, with a total of 3194 samples. The training and test sets were divided according to an 8:2 ratio. The experiment improved the traditional voting method. First, multiple models were used to predict the rice seed varieties. Then, the predicted probabilities were used as the late fusion input data. Next, a comprehensive score vector was calculated based on the performance of different models. In late fusion, the predicted probabilities from 2D and 3D were jointly weighted to obtain the final predicted probability. Finally, the predicted value with the highest probability was selected as the final value. In the experimental results, after late fusion of the predicted probabilities, the average accuracy rate reached 97.4%. Compared with the single support vector machine (SVM), k-nearest neighbors (kNN), convolutional neural network (CNN), MobileNet, and PointNet, the accuracy rates increased by 4.9%, 8.3%, 18.1%, 8.3%, and 9%, respectively. Among the eight varieties, the recognition accuracy of two rice varieties, Hannuo35 and Yuanhan35, by applying the voting method improved most significantly, from 73.9% and 77.7% in two dimensions to 92.4% and 96.3%, respectively. Thus, the improved voting method can combine the advantages of different data modalities and significantly improve the final prediction results. [ABSTRACT FROM AUTHOR]

Light fields contain a wealth of information about real-world scenes and can be easily acquired by commercial light field cameras. However, the trade-off between angular and spatial resolution is inevitable. This paper proposes an end-to-end light field angular super-resolution network by exploiting structure and scene information to mitigate this problem. First, a Light Field ResBlock is proposed to explicitly extract epipolar plane image features and scene features. The epipolar plane image features with obvious directionality represent light field structure information that contains correlations among different views, such as the intensity and depth consistencies. The scene features express scene information that depicts local details and global information and describes individual objects in a scene. The exploitation of light field structure and scene information can gather the intrinsic characteristics of light fields. Then, 4D deconvolution is employed to upsample the angular resolution based on the extracted features related to light field structure and scene. The utilization of 4D deconvolution preserves the connections among sparse views and generates new views with high correlations. Last but not least, the refinement network further exploiting LF structure and scene information is employed to alleviate the artifacts induced by too little angular information during the early stage of the model. Experiments on two different angular super-resolution tasks demonstrate that our method can achieve the best angular resolution enhancement performance on different datasets. [ABSTRACT FROM AUTHOR]

Light field reconstruction and synthesis algorithms are essential for improving the lower spatial resolution for hand-held plenoptic cameras. Previous light field synthesis algorithms produce blurred regions around depth discontinuities, especially for stereo-based algorithms, where no information is available to fill the occluded areas in the light field image. In this paper, we propose a light field synthesis algorithm that uses the focal stack images and the all-in-focus image to synthesize a 9 × 9 sub-aperture view light field image. Our approach uses depth from defocus to estimate a depth map. Then, we use the depth map and the all-in-focus image to synthesize the sub-aperture views, and their corresponding depth maps by mimicking the apparent shifting of the central image according to the depth values. We handle the occluded regions in the synthesized sub-aperture views by filling them with the information recovered from the focal stack images. We also show that, if the depth levels in the image are known, we can synthesize a high-accuracy light field image with just five focal stack images. The accuracy of our approach is compared with three state-of-the-art algorithms: one non-learning and two CNN-based approaches, and the results show that our algorithm outperforms all three in terms of PSNR and SSIM metrics. [ABSTRACT FROM AUTHOR]

In this paper, a novel method based on edge spread information fitting (ESIF) is proposed to accurately calibrate the geometric parameters of a focused light field camera. A focused light field camera with flexible adjustment of spatial resolution and angular resolution is designed and built to capture the four-dimensional light field information of the scenes, and the geometric relationship between the focus plane of the camera and its internal parameters is derived to establish and simplify the calibration model. After that, a new algorithm based on sub-pixel edge fitting is designed to accurately detect corners, and the minimum corner size can be calculated to confirm the focus plane. In the simulation experiments, the error is 0.083% between the ground truth and the calibration result. The physical experiments show that our method is effective and reliable for the geometric calibration of a focused light field camera. Our method cleverly utilizes the special imaging geometric relationship of the focused light field camera to ensure the better calibration accuracy, which makes the calibration process more reliable. [ABSTRACT FROM AUTHOR]

Light field imaging can record the intensity and direction information of light rays in space, which attracts extensive attention. However, the trade-off between angular and spatial resolution cannot be avoided due to the limitation of sensors in commercial light field cameras. To mitigate this problem, this paper proposes the view position prior-supervised light field angular super-resolution network with asymmetric feature extraction and spatial-angular interaction. First, there is a severe information asymmetry between the spatial and angular dimensions in light fields with sparse views. The asymmetric feature extraction block is proposed to extract spatial and angular features with different receptive fields in an asymmetric manner. As a result, more light field intrinsic features are extracted, which improves the utilization rate of the limited light field information. Second, the existing methods usually ignore the correlations among the newly synthesized views. The spatial-angular interaction module is proposed to collect the local and global information, build relations between any two points in the feature space, and reconstruct light field consistencies. Thus, the complete view correlations can be established. Last but not least, we investigate the impact of the given views on each viewpoint and propose a loss function based on the view position prior, which reduces the quality difference among the synthesized sub-aperture images and further improves the network performance. Comprehensive experiments demonstrate that our method can perform best on all datasets, and the depth estimation results present a new perspective to show the superiority of the proposed method. [ABSTRACT FROM AUTHOR]

Most of the traditional image feature point extraction and matching methods are based on a series of light properties of images. These light properties easily conflict with the distinguishability of the image features. The traditional light imaging methods focus only on a fixed depth of the target scene, and subjects at other depths are often easily blurred. This makes the traditional image feature point extraction and matching methods suffer from a low accuracy and a poor robustness. Therefore, in this paper, a light field camera is used as a sensor to acquire image data and to generate a full-focus image with the help of the rich depth information inherent in the original image of the light field. The traditional ORB feature point extraction and matching algorithm is enhanced with the goal of improving the number and accuracy of the feature point extraction for the light field full-focus images. The results show that the improved ORB algorithm extracts not only most of the features in the target scene but also covers the edge part of the image to a greater extent and produces extracted feature points which are evenly distributed for the light field full-focus image. Moreover, the extracted feature points are not repeated in a large number in a certain part of the image, eliminating the aggregation phenomenon that exists in traditional ORB algorithms. [ABSTRACT FROM AUTHOR]

Aiming at the realization of fast and high-precision detection of the workpiece, an adaptive bidirectional gray-scale center of gravity extraction algorithm for laser stripes is proposed in this paper. The algorithm is processed in the following steps. Firstly, the initial image processing area is set according to the floating field of the camera's light stripe, followed by setting the adaptive image processing area according to the actual position of the light stripe. Secondly, the center of light stripe is obtained by using the method of combining the upper contour with the barycenter of the bidirectional gray-scale. The obtained center of the light stripe is optimized by reducing the deviation from adjacent center points. Finally, the slope and intercept are used to complete the breakpoint. The experimental results show that the algorithm has the advantages of high speed and precision and has specific adaptability to the laser stripes of the complex environment. Compared with other conventional algorithms, it greatly improves and can be used in industrial detection. [ABSTRACT FROM AUTHOR]

Light-field cameras allow the acquisition of both the spatial and angular components of the light-field. The conventional way to perform such acquisitions leads to a strong spatio-angular resolution limitation but correlation-enabled plenoptic cameras have been introduced recently that relax this constraint. Here we use a computational version of this concept to acquire realistic light-fields images using a commercial DSLR Camera lens as an imaging system. By placing the image sensor in the focal plane of a lens, within the camera we ensure the acquisition of pure angular components together with the spatial information. We perform an acquisition presenting a high spatio-angular rays resolution obtained through a trade off of the temporal resolution. The acquisition reported is photo-realistic and the acquisition of diffraction limited features is observed with the setup. Finally, we demonstrate the refocusing abilities of the camera. [ABSTRACT FROM AUTHOR]

This paper describes a benchmark dataset for the detection of fuel particles in 2D and 3D image data in a rotary kiln combustion chamber. The specific challenges of detecting the small particles under demanding environmental conditions allows for the performance of existing and new particle detection techniques to be evaluated. The data set includes a classification of burning and non-burning particles, which can be in the air but also on the rotary kiln wall. The light-field camera used for data generation offers the potential to develop and objectively evaluate new advanced particle detection methods due to the additional 3D information. Besides explanations of the data set and the contained ground truth, an evaluation procedure of the particle detection based on the ground truth and results for an own particle detection procedure for the data set are presented. Dataset: 10.5281/zenodo.6358536. Dataset License: Creative Commons Attribution 4.0 International [ABSTRACT FROM AUTHOR]

Tomographic particle image velocimetry (Tomo-PIV) is a powerful tool for measuring fluid velocity. However, its use in deep and narrow spaces presents challenges due to the limited optical observation window. As a result, the spatial resolution along the depth direction is significantly constrained by the short base-line. An illustrative example is the measurement of three-dimensional (3D) fluid velocity within a functioning turbine engine. To mitigate the problem, this work proposes a novel Tomo-PIV technique called kaleidoscopic light field (KLF) camera-based PIV (referred to as KLF-PIV). This imaging system divides the captured scene into nine distinct perspectives using a single sensor, thereby enabling 3D resolution through multi-ocular vision. The study investigates the ultimate volumetric measurement performance of the KLF-PIV technique through Gaussian optics simulation, and compares it with the micro-lens array (MLA) based light field (LF) camera. The results demonstrate that KLF-PIV achieves significantly improved axial resolution and enlarged depth of field (DOF). It is also more suitable for scenarios with denser illuminated particles. Compared to LF-PIV, the proposed method enables more accurate measurements with notably reduced computational load (> 50 %). Overall, the KLF-PIV provides a precise and compact single-sensor Tomo-PIV technique adaptable to the limited optical window, making it a valuable diagnostics tool in challenging scenarios. • We propose a compact single-sensor Tomo-PIV based on kaleidoscopic light field camera, termed KLF-PIV. • The performance is numerically tested and compared to the micro-lens array based light field PIV (LF-PIV). • The results demonstrate that KLF-PIV achieves significantly improved axial resolution and enlarged depth of field. • KLF-PIV is more adaptable with denser seeding particles and less vulnerable to image noise, compared to LF-PIV. • The proposed method enables more accurate velocity measurements with computational load drastically reduced over 50%. [ABSTRACT FROM AUTHOR]

As one of the fundamental technologies for scene understanding, semantic segmentation has been widely explored in the last few years. Light field cameras encode the geometric information by simultaneously recording the spatial information and angular information of light rays, which provides us with a new way to solve this issue. In this paper, we propose a high-quality and challenging urban scene dataset, containing 1074 samples composed of real-world and synthetic light field images as well as pixel-wise annotations for 14 semantic classes. To the best of our knowledge, it is the largest and the most diverse light field dataset for semantic segmentation. We further design two new semantic segmentation baselines tailored for light field and compare them with state-of-the-art RGB, video and RGB-D-based methods using the proposed dataset. The outperforming results of our baselines demonstrate the advantages of the geometric information in light field for this task. We also provide evaluations of super-resolution and depth estimation methods, showing that the proposed dataset presents new challenges and supports detailed comparisons among different methods. We expect this work inspires new research direction and stimulates scientific progress in related fields. The complete dataset is available at https://github.com/HAWKEYE-Group/UrbanLF. [ABSTRACT FROM AUTHOR]

Capturing the directions of light by light field cameras powers next-generation immersive multimedia applications. A critical problem in taking advantage of the rich visual information in light field images is depth estimation. Conventional light field depth estimation methods build a cost volume that measures the photo-consistency of pixels refocused to a range of depths, and the highest consistency indicates the correct depth. This strategy works well in most regions but usually generates blurry edges in the estimated depth map due to occlusions. Recent work shows that integrating occlusion models to light field depth estimation can largely reduce blurry edges. However, existing occlusion handling methods rely on complex edge-aided processing and post-refinement, and this reliance limits the resultant depth accuracy and impacts on the computational performance. In this paper, we propose a novel occlusion-aware vote cost (OAVC) which is able to accurately preserve edges in the depth map. Instead of using photo-consistency as an indicator of the correct depth, we construct a novel cost from a new perspective that counts the number of refocused pixels whose deviations from the central-view pixel are less than a small threshold, and utilizes that number to select the correct depth. The pixels from occluders are thus excluded in determining the correct depth. Without the use of any explicit occlusion handling methods, the proposed method can inherently preserve edges and produces high-quality depth estimates. Experimental results show that the proposed OAVC outperforms state-of-the-art light field depth estimation methods in terms of depth estimation accuracy and computational complexity. [ABSTRACT FROM AUTHOR]

This letter describes the design and implementation of a multi-wavelength light-field pyrometer, where six-channel radiation images were captured with one CMOS sensor. Such capability is achieved by placing a 2 × 3 filter array in front of the main lens of an unfocused light-field camera, such that discrete wavelength and radiation intensity can be simultaneously recorded. It demonstrates that through black-body furnace experiments, how multi-channel radiation images can be extracted from one raw light-field multispectral image, and how accurate 2D temperature distribution can be recovered by optimization algorithms. [ABSTRACT FROM AUTHOR]

Light field cameras have been used for 3-dimensional geometrical measurement or refocusing of captured photo. In this paper, we propose the light field acquiring method using a spherical mirror array. By employing a mirror array and two cameras, a virtual camera array that captures an object from around it can be generated. Since large number of virtual cameras can be constructed from two real cameras, an affordable high-density camera array can be achieved using this method. Furthermore, the spherical mirrors enable the capturing of large objects as compared to the previous methods. We conducted simulations to capture the light field, and synthesized arbitrary viewpoint images of the object with observation from 360 degrees around it. The ability of this system to refocus assuming a large aperture is also confirmed. We have also built a prototype which approximates the proposal to conduct a capturing experiment in order to ensure the system's feasibility. [ABSTRACT FROM AUTHOR]

Among novel capture and visualization technologies, light field has made significant progress in the current decade, bringing closer its emergence in everyday use cases. Unlike many other forms of 3D displays and devices, light field visualization does not depend on any viewing equipment. Regarding its potential use cases, light field is applicable to both cinematic and interactive contents. Such contents often rely on camera animation, which is a frequent tool for the creation and presentation of 2D contents. However, while common 3D camera animation is often rather straightforward, light field visualization has certain constraints that must be considered before implementing any variation of such techniques. In this paper, we introduce our work on camera animation for light field visualization. Different types of conventional camera animation were applied to light field contents, which produced an interactive simulation. The simulation was visualized and assessed on a real light field display, the results of which are presented and discussed in this paper. Additionally, we tested different forms of realistic physical camera motion in our study, and based on our findings, we propose multiple metrics for the quality evaluation of light field visualization in the investigated context and for the assessment of plausibility. [ABSTRACT FROM AUTHOR]

Conventional light-field cameras with a micro-lens array suffer from resolution trade-off and shallow depth of field. Here we develop a full-resolution light-field camera based on dual photography. We extend the principle of dual photography from real space to Fourier space for obtaining two-dimensional (2D) angular information of the light-field. It uses a spatial light modulator at the image plane as a virtual 2D detector to record the 2D spatial distribution of the image, and a real 2D detector at the Fourier plane of the image to record the angles of the light rays. The Fourier-spectrum signals recorded by each pixel of the real 2D detector can be used to reconstruct a perspective image through single-pixel imaging. Based on the perspective images reconstructed by different pixels, we experimentally demonstrated that the camera can digitally refocus on objects at different depths. The camera can achieve light-field imaging with full resolution and provide an extreme depth of field. The method provides a new idea for developing full-resolution light-field cameras. [ABSTRACT FROM AUTHOR]

Light field essentially represents rays in space. The epipolar geometry between two light fields is an important relationship that captures ray-ray correspondences and relative configuration of two views. Unfortunately, so far little work has been done in deriving a formal epipolar geometry model that is specifically tailored for light field cameras. This is primarily due to the high-dimensional nature of the ray sampling process with a light field camera. This paper fills in this gap by developing a novel ray-space epipolar geometry which intrinsically encapsulates the complete projective relationship between two light fields, while the generalized epipolar geometry which describes relationship of normalized light fields is the specialization of the proposed model to calibrated cameras. With Plücker parameterization, we propose the ray-space projection model involving a $6\!\times \!6$ 6 × 6 ray-space intrinsic matrix for ray sampling of light field camera. Ray-space fundamental matrix and its properties are then derived to constrain ray-ray correspondences for general and special motions. Finally, based on ray-space epipolar geometry, we present two novel algorithms, one for fundamental matrix estimation, and the other for calibration. Experiments on synthetic and real data have validated the effectiveness of ray-space epipolar geometry in solving 3D computer vision tasks with light field cameras. [ABSTRACT FROM AUTHOR]

In this paper, we propose a relative pose estimation algorithm for micro-lens array (MLA)-based conventional light field (LF) cameras. First, by employing the matched LF-point pairs, we establish the LF-point-LF-point correspondence model to represent the correlation between LF features of the same 3D scene point in a pair of LFs. Then, we employ the proposed correspondence model to estimate the relative camera pose, which includes a linear solution and a non-linear optimization on manifold. Unlike prior related algorithms, which estimated relative poses based on the recovered depths of scene points, we adopt the estimated disparities to avoid the inaccuracy in recovering depths due to the ultra-small baseline between sub-aperture images of LF cameras. Experimental results on both simulated and real scene data have demonstrated the effectiveness of the proposed algorithm compared with classical as well as state-of-art relative pose estimation algorithms. [ABSTRACT FROM AUTHOR]

This paper presents a novel calibration algorithm for plenoptic cameras, especially the multi-focus configuration, where several types of micro-lenses are used, using raw images only. Current calibration methods rely on simplified projection models, use features from reconstructed images, or require separated calibrations for each type of micro-lens. In the multi-focus configuration, the same part of a scene will demonstrate different amounts of blur according to the micro-lens focal length. Usually, only micro-images with the smallest amount of blur are used. In order to exploit all available data, we propose to explicitly model the defocus blur in a new camera model with the help of our newly introduced Blur Aware Plenoptic (BAP) feature. First, it is used in a pre-calibration step that retrieves initial camera parameters, and second, to express a new cost function to be minimized in our single optimization process. Third, it is exploited to calibrate the relative blur between micro-images. It links the geometric blur, i.e., the blur circle, to the physical blur, i.e., the point spread function. Finally, we use the resulting blur profile to characterize the camera's depth of field. Quantitative evaluations in controlled environment on real-world data demonstrate the effectiveness of our calibrations. [ABSTRACT FROM AUTHOR]