Search

In recent years, the development of holographic near-eye displays (HNED) has surpassed the progress of digital hologram recording systems, especially in terms of wide-angle viewing capabilities. Thus, there is capture-display parameters incompatibility, which makes it impossible to reconstruct recorded objects in wide-angle display. This paper presents a complete imaging chain extending the available content for wide-angle HNED of pupil and non-pupil configuration with narrow-angle digital holograms of real objects. To this end, a new framework based on the phase-space approach is proposed that includes a set of affine transformations required to account for all differences in capture-display cases. The developed method allows free manipulation of the geometry of reconstructed objects, including axial and lateral positioning and size scaling. At the same time, it has a low computational effort. The presented work is supported with non-paraxial formulas developed using the phase-space approach, enabling accurate tracing of the holographic signal, its reconstruction, and measuring appearing deformations. The applicability of the proposed hologram manipulation method is proven with experimental results of digital hologram reconstruction in wide-angle HNED.

In this study, we investigate the application of generative models to assist artificial agents, such as delivery drones or service robots, in visualising unfamiliar destinations solely based on textual descriptions. We explore the use of generative models, such as Stable Diffusion, and embedding representations, such as CLIP and VisualBERT, to compare generated images obtained from textual descriptions of target scenes with images of those scenes. Our research encompasses three key strategies: image generation, text generation, and text enhancement, the latter involving tools such as ChatGPT to create concise textual descriptions for evaluation. The findings of this study contribute to an understanding of the impact of combining generative tools with multi-modal embedding representations to enhance the artificial agent's ability to recognise unknown scenes. Consequently, we assert that this research holds broad applications, particularly in drone parcel delivery, where an aerial robot can employ text descriptions to identify a destination. Furthermore, this concept can also be applied to other service robots tasked with delivering to unfamiliar locations, relying exclusively on user-provided textual descriptions.

Small eyebox in wide-angle holographic near-eye display is a severe limitation for 3D visual immersion of the device. In this paper, an opto-numerical solution for extending the eyebox size in these types of devices is presented. The hardware part of our solution expands the eyebox by inserting a grating of frequency fg within a non-pupil forming display configuration. The grating multiplies eyebox, increasing the possible eye motion. The numerical part of our solution is an algorithm that enables proper coding of wide-angle holographic information for projecting correct object reconstruction at arbitrary eye position within the extended eyebox. The algorithm is developed through the employment of the phase-space representation, which facilitates the analysis of the holographic information and the impact of the diffraction grating in the wide-angle display system. It is shown that accurate encoding of the wavefront information components for the eyebox replicas is possible. In this way, the problem of missing or incorrect views in wide angle near-eye display with multiplied eyeboxes is efficiently solved. Moreover, this study investigates the space-frequency relation between the object and the eyebox and how the hologram information is shared between eyebox replicas. The functionality of our solution is tested experimentally in an augmented reality holographic near-eye display that has maximum field of view of 25.89°. Obtained optical reconstructions demonstrate that correct object view is obtained for arbitrary eye position within extended eyebox.

In this paper, two solutions are proposed to improve the quality of a large image that is reconstructed in front of the observer in a near-eye holographic display. One of the proposed techniques, to the best of our knowledge, is the first wide-angle solution that successfully uses a non-coherent LED source. It is shown that the resulting image when employing these types of sources has less speckle noise but a resolution comparable to that obtained with coherent light. These results are explained by the developed theory, which also shows that the coherence effect is angle varying. Furthermore, for the used pupil forming display architecture, it is necessary to compute a large virtual nonparaxial hologram. We demonstrate that for this hologram there exists a small support region that has a frequency range capable of encoding information generated by a single point of the object. This small support region is beneficial since it enables to propose a wide-angle rigorous CGH computational method, which allows processing very dense cloud of points that represents three-dimensional objects. This is our second proposed key development. To determine the corresponding support region, the concept of local wavefront spatial curvature is introduced, which is proportional to the tangent line to the local spatial frequency of the spherical wavefront. The proposed analytical solution shows that the size of this area strongly depends on the transverse and longitudinal coordinate of the corresponding object point.

A method for reconstruction of partial off-axis areas of arbitrary size for wide-angle viewing computer generated holograms is presented. Proposed method employs paraxial spherical phase factors and modified propagation kernels. This significantly reduces the numerical space-bandwidth product needed for off-axis wave field calculations, which makes it an efficient propagation method. As a result, propagated wavefields of high-off axis and large size output windows can be obtained. To that end, a phase-space analysis for obtaining the proper condition for implementing spatial-frequency zero-padding for accurate wavefield propagation is carried out. Hence, suppression of aliased components and high spatial resolution is possible. Nevertheless, proposed algorithm faces a computer memory bottleneck when reconstructing very large off-axis areas due to too extensive zero-padding needed. To solve this problem, a memory optimized tiling implementation is introduced. Utility of the developed propagation tools are proven by partial reconstructions from a high-resolution hologram. The size of the reconstructions areas ranges from 100 × 100 mm 2 up to 550 × 550 mm 2 .

Surface reconstruction for micro-samples with large discontinuities using digital holography is a challenge. To overcome this problem, multi-incidence digital holographic profilometry (MIDHP) has been proposed. MIDHP relies on the numerical generation of the longitudinal scanning function (LSF) for reconstructing the topography of the sample with large depth and high axial resolution. Nevertheless, the method is unable to reconstruct surfaces with large gradients due to the need of: (i) high precision focusing that manual adjustment cannot fulfill and (ii) preserving the functionality of the LSF that requires capturing and processing many digital holograms. In this work, we propose a novel MIDHP method to solve these limitations. First, an autofocusing algorithm based on the comparison of shapes obtained by the LSF and the thin tilted element approximation is proposed. It is proven that this autofocusing algorithm is capable to deliver in-focus plane localization with submicron resolution. Second, we propose that wavefield summation for the generation of the LSF is carried out in Fourier space. It is shown that this scheme enables a significant reduction of arithmetic operations and can minimize the number of Fourier transforms needed. Hence, a fast generation of the LSF is possible without compromising its accuracy. The functionality of MIDHP for measuring surfaces with large gradients is supported by numerical and experimental results.

Recent advances have shown for the first time that it is possible to beat a human with an autonomous drone in a drone race. However, this solution relies heavily on external sensors, specifically on the use of a motion capture system. Thus, a truly autonomous solution demands performing computationally intensive tasks such as gate detection, drone localisation, and state estimation. To this end, other solutions rely on specialised hardware such as graphics processing units (GPUs) whose onboard hardware versions are not as powerful as those available for desktop and server computers. An alternative is to combine specialised hardware with smart sensors capable of processing specific tasks on the chip, alleviating the need for the onboard processor to perform these computations. Motivated by this, we present the initial results of adapting a novel smart camera, known as the OpenCV AI Kit or OAK-D, as part of a solution for the ADR running entirely on board. This smart camera performs neural inference on the chip that does not use a GPU. It can also perform depth estimation with a stereo rig and run neural network models using images from a 4K colour camera as the input. Additionally, seeking to limit the payload to 200 g, we present a new 3D-printed design of the camera's back case, reducing the original weight 40%, thus enabling the drone to carry it in tandem with a host onboard computer, the Intel Stick compute, where we run a controller based on gate detection. The latter is performed with a neural model running on an OAK-D at an operation frequency of 40 Hz, enabling the drone to fly at a speed of 2 m/s. We deem these initial results promising toward the development of a truly autonomous solution that will run intensive computational tasks fully on board.

In [Appl. Opt.56, 7079 (2017)APOPAI0003-693510.1364/AO.56.007079], a phase-unwrapping algorithm has been reported that is based on the transport of intensity (TIE) equation. Section 3 of that manuscript presented one way to derive an analytical expression for the axial intensity derivative using the paraxial angular spectrum (AS) method. In the recent comment by Yoneda et al. [Appl. Opt.60, 7500 (2021)APOPAI0003-693510.1364/AO.417146], the authors misunderstand the utility of the angular spectrum method and conclude that the corresponding derivation is ambiguous. In this response, we take the opportunity to correct a typo that clarifies the calculation of the axial derivative of the intensity.

This work develops a single-shot holographic profilometer that enables shape characterization of discontinuous deep surfaces. This is achieved by combining hologram frequency multiplexing and an illumination technique of complex amplitude in multi-incidence angle profilometer. Object illumination is carried out from seven directions simultaneously, where the radial angular coordinates of illumination plane waves obey the geometric series. It is shown that: (i) the illumination pattern provides the required frequency separation of all object wavefronts in transverse frequency space, which is necessary for hologram demultiplexing, and (ii) numerical generation of longitudinal scanning function (LSF) is possible, which has large measurement range, high axial resolution, and small side lobes. Low side lobes of LSF and the developed multiplexed field dependent aberration compensation method are essential to minimize the negative influence of speckle noise of single-shot capture on the measurement result. The utility of the proposed method is demonstrated with experimental measurement of heights of two step-like objects.

Registration and reconstruction of high-quality digital holograms with a large view angle are intensive computer tasks since they require the space-bandwidth product (SBP) of the order of tens of gigapixels or more. This massive use of SBP severely affects the storing and manipulation of digital holograms. In order to reduce the computer burden, this work focuses on the generation and reconstruction of very large horizontal parallax only digital holograms (HPO-DHs). It is shown that these types of holograms can preserve high quality and large view angle in x direction while keeping a low use of SBP. This work first proposes a numerical technique that allows calculating very large HPO-DHs with large pixel size by merging the Fourier holography and phase added stereogram algorithm. The generated Fourier HPO-DHs enable accurate storing of holographic data from 3D objects. To decode the information contained in these Fourier HPO-DHs (FHPO-DHs), a novel angular spectrum (AS) technique that provides an efficient use of the SBP for reconstruction is proposed. Our reconstruction technique, which is called compact space bandwidth AS (CSW-AS), makes use of cylindrical parabolic waves that solve sampling issues of FHPO-DHs and AS. Moreover, the CSW-AS allows for implementing zero-padding for accurate wavefield reconstructions. Hence, suppression of aliased components and high spatial resolution is possible. Notably, the imaging chain of Fourier HPO-DH enables efficient calculation, reconstruction and storing of HPO holograms of large size. Finally, the accuracy and utility of the developed technique is proved by both numerical and optical reconstructions.

Although a significant amount of work has been carried out for visual perception in the context of unmanned aerial vehicles (UAVs), not so much has been done regarding auditory perception. The latter can complement the observation of the environment that surrounds a UAV by providing additional information that can be used to detect, classify, and localize audio sources of interest. Motivated by the usefulness of auditory perception for UAVs, we present a literature review that discusses the audio techniques and microphone configurations reported in the literature. A categorization of techniques is proposed based on the role a UAV plays in the auditory perception (is it the one being perceived or is it the perceiver?), as well as a set of objectives that are more popularly aimed to be accomplished in the current literature (detection, classification, and localization). This literature review aims to provide a concise landscape of the most relevant works on auditory perception in the context of UAVs to date and provides insights into future avenues of research as a guide to those who are beginning to work in this field.

Accurate reconstruction of digital holograms that are large in the x direction and small in the y direction, known as horizontal parallax only digital hologram (HPO-DH), must be carried out by non-paraxial propagation approaches such as the classical angular spectrum (AS) method. However, the required space-bandwidth product (SBP) for reconstruction of HPO-DHs requires billions of pixels, which is computationally intensive. Moreover, application of zero-padding for removing aliasing components would generate an unbearable computational burden. In this work, a novel AS technique that reconstructs non-paraxial HPO-DHs with low SBP is proposed. The proposed technique first employs the multi-Fourier transform plane propagation method, which avoids the increase of size in the vertical direction of the HPO-DH to be processed. The second ingredient for field calculation is coherent superposition of vertical tiles formed from the multi-Fourier transform calculations. The described methodology enables reconstruction of HPO-DHs with the AS method and reduced SBP. Efficient managing of the SBP allows implementing zero-padding strategies in the x direction. It is shown that the padding strategies can be implemented in the frequency, space, and space-frequency domains. Hence, suppression of aliased components and increase of the spatial resolution is possible at the same time. Finally, the accuracy and utility of the developed technique is proved by both numerical simulations and experiments.

Autonomous Drone Racing (ADR) was first proposed in IROS 2016. It called for the development of an autonomous drone capable of beating a human in a drone race. After almost five years, several teams have proposed different solutions with a common pipeline: gate detection; drone localization; and stable flight control. Recently, Deep Learning (DL) has been used for gate detection and localization of the drone regarding the gate. However, recent competitions such as the Game of Drones, held at NeurIPS 2019, called for solutions where DL played a more significant role. Motivated by the latter, in this work, we propose a CNN approach called DeepPilot that takes camera images as input and predicts flight commands as output. These flight commands represent: the angular position of the drone's body frame in the roll and pitch angles, thus producing translation motion in those angles; rotational speed in the yaw angle; and vertical speed referred as altitude h . Values for these 4 flight commands, predicted by DeepPilot, are passed to the drone's inner controller, thus enabling the drone to navigate autonomously through the gates in the racetrack. For this, we assume that the next gate becomes visible immediately after the current gate has been crossed. We present evaluations in simulated racetrack environments where DeepPilot is run several times successfully to prove repeatability. In average, DeepPilot runs at 25 frames per second (fps). We also present a thorough evaluation of what we called a temporal approach, which consists of creating a mosaic image, with consecutive camera frames, that is passed as input to the DeepPilot. We argue that this helps to learn the drone's motion trend regarding the gate, thus acting as a local memory that leverages the prediction of the flight commands. Our results indicate that this purely DL-based artificial pilot is feasible to be used for the ADR challenge.

In this work, multi-incident digital holographic profilometry for microscale measurements is presented. This technique assembles the set of object fields from captured holograms for generation of the longitudinal scanning function (LSF). Numerical propagation is used for refocusing, and thus, the LSF can be determined at any given plane along the optical axis. The LSF takes maximum value for in focus object points, which are used to obtain full-field height distribution of the sample. This principle is the base of proposed measurement technique. Three capturing holograms strategies, which give control over the shape of the LSF, unambiguous measurement range, axial resolution, and noise immunity, are discussed. The conclusions of this work are supported by numerical and experimental results.

Multirotor micro air vehicles can operate in complex and confined environments that are otherwise inaccessible to larger drones. Operation in such environments results in airflow interactions between the propellers and proximate surfaces. The most common of these interactions is the ground effect. In addition to the increment in thrust efficiency, this effect disturbs the onboard sensors of the drone. In this paper, we present a fault-tolerant scheme for a multirotor with altitude sensor faults caused by the ground effect. We assume a hierarchical control structure for trajectory tracking. The structure consists of an external Proportional-Derivative controller and an internal Proportional-Integral controller. We consider that the sensor faults occur on the inner loop and counteract them in the outer loop. In a novel approach, we use a metric monocular Simultaneous Localization and Mapping algorithm for detecting internal faults. We design the fault diagnosis scheme as a logical process which depends on the weighted residual. Furthermore, we propose two control strategies for fault mitigation. The first combines the external PD controller and a function of the residual. The second treats the sensor fault as an actuator fault and compensates with a sliding mode action. In either case, we utilize onboard sensors only. Finally, we evaluate the effectiveness of the strategies in simulations and experiments.

Audio analysis over an Unmanned Aerial Systems (UAS) is of interest it is an essential step for on-board sound source localization and separation. This could be useful for search & rescue operations, as well as for detection of unauthorized drone operations. In this paper, an analysis of the previously introduced Acoustic Interactions for Robot Audition (AIRA)-UAS corpus is presented, which is a set of recordings produced by the ego-noise of a drone performing different aerial maneuvers and by other drones flying nearby. It was found that the recordings have a very low Signal-to-Noise Ratio (SNR), that the noise is dynamic depending of the drone's movements, and that their noise signatures are highly correlated. Three popular filtering techniques were evaluated in this work in terms of noise reduction and signature extraction, which are: Berouti's Non-Linear Noise Subtraction, Adaptive Quantile Based Noise Estimation, and Improved Minima Controlled Recursive Averaging. Although there was moderate success in noise reduction, no filter was able to keep intact the signature of the drone flying in parallel. These results are evidence of the challenge in audio processing over drones, implying that this is a field prime for further research.

High-Level Structure (HLS) extraction in a set of images consists of recognizing 3D elements with useful information to the user or application. There are several approaches to HLS extraction. However, most of these approaches are based on processing two or more images captured from different camera views or on processing 3D data in the form of point clouds extracted from the camera images. In contrast and motivated by the extensive work developed for the problem of depth estimation in a single image, where parallax constraints are not required, in this work, we propose a novel methodology towards HLS extraction from a single image with promising results. For that, our method has four steps. First, we use a CNN to predict the depth for a single image. Second, we propose a region-wise analysis to refine depth estimates. Third, we introduce a graph analysis to segment the depth in semantic orientations aiming at identifying potential HLS. Finally, the depth sections are provided to a new CNN architecture that predicts HLS in the shape of cubes and rectangular parallelepipeds.

The phase information of a complex field is routinely obtained using coherent measurement techniques as, e.g., interferometry or holography. The obtained measurement result is subject to a 2π ambiguity and is often referred to as wrapped phase. Phase-unwrapping algorithms (PUAs) are commonly employed to remove this ambiguity and, hence, obtain the absolute phase. However, implementing PUAs can be computationally intensive, and the accuracy of those algorithms may be low. Recently, the transport of intensity equation (TIE) has been proposed as a simple and practical alternative for obtaining the absolute phase map. Nevertheless, an efficient implementation of this technique has not yet been made. In this work, we propose an accurate solution for the TIE-based PUA that does not require the use of wave-propagation techniques, as previously reported TIE-based approaches. The proposed method calculates directly the axial derivative of the intensity from the wrapped phase when considering the correct propagation method. This is done in order to bypass the time-consuming wave-propagation techniques employed in similar methods. The analytical evaluation of this parameter allows obtaining an accurate solution when unwrapping the phase map with low computational effort. This work further introduces the use of the iterative TIE-PUA that, in a few steps, improves significantly the accuracy of the final absolute phase map, even in the presence of noise or aliasing of the wrapped data. The high accuracy and utility of the developed TIE-PUA technique is proven by both numerical simulations and experiments for various objects.