Search

Time of flight is promising technology in machine vision and sensing, with an emerging need for low power consumption, a high image resolution, and reliable operation in high ambient light conditions. Therefore, we propose a novel direct time-of-flight pixel using the single-photon avalanche diode (SPAD) sensor, with an in-pixel averaging method to suppress ambient light and detect the laser pulse arrival time. The system utilizes two orthogonal sinusoidal signals applied to the pixel as inputs, which are synchronized with a pulsed laser source. The detected signal phase indicates the arrival time. To evaluate the proposed system's potential, we developed analytical and statistical models for assessing the phase error and precision of the arrival time under varying ambient light levels. The pixel simulation showed that the phase precision is less than 1% of the detection range when the ambient-to-signal ratio is 120. A proof-of-concept pixel array prototype was fabricated and characterized to validate the system's performance. The pixel consumed, on average, 40 μ W of power in operation with ambient light. The results demonstrate that the system can operate effectively under varying ambient light conditions and its potential for customization based on specific application requirements. This paper concludes by discussing the system's performance relative to the existing direct time-of-flight technologies, identifying their strengths and limitations. [ABSTRACT FROM AUTHOR]

Significance: Fluorescence guidance is used clinically by surgeons to visualize anatomical and/or physiological phenomena in the surgical field that are difficult or impossible to detect by the naked eye. Such phenomena include tissue perfusion or molecular phenotypic information about the disease being resected. Conventional fluorescence-guided surgery relies on long, microsecond scale laser pulses to excite fluorescent probes. However, this technique only provides two-dimensional information; crucial depth information, such as the location of malignancy below the tissue surface, is not provided. Aim: We developed a depth sensing imaging technique using light detection and ranging (LiDAR) time-of-flight (TOF) technology to sense the depth of target tissue while overcoming the influence of tissue optical properties and fluorescent probe concentration. Approach: The technology is based on a large-format (512 × 512 pixel), binary, gated, single-photon avalanche diode (SPAD) sensor with an 18 ps time-gate step, synchronized with a picosecond pulsed laser. The fast response of the sensor was developed and tested for its ability to quantify fluorescent inclusions at depth and optical properties in tissue-like phantoms through analytical model fitting of the fast temporal remission data. Results: After calibration and algorithmic extraction of the data, the SPAD LiDAR technique allowed for sub-mm resolution depth sensing of fluorescent inclusions embedded in tissue-like phantoms, up to a maximum of 5 mm in depth. The approach provides robust depth sensing even in the presence of variable tissue optical properties and separates the effects of fluorescence depth from absorption and scattering variations. Conclusions: LiDAR TOF fluorescence imaging using an SPAD camera provides both fluorescence intensity images and the temporal profile of fluorescence, which can be used to determine the depth at which the signal is emitted over a wide field of view. The proposed tool enables fluorescence imaging at a higher depth in tissue and with higher spatial precision than standard, steady-state fluorescence imaging tools, such as intensity-based near-infrared fluorescence imaging, optical coherence tomography, Raman spectroscopy, or confocal microscopy. Integration of this technique into a standard surgical tool could enable rapid, more accurate estimation of resection boundaries, thereby improving the surgeon's efficacy and efficiency, and ultimately improving patient outcomes. [ABSTRACT FROM AUTHOR]

Time of flight is promising technology in machine vision and sensing, with an emerging need for low power consumption, a high image resolution, and reliable operation in high ambient light conditions. Therefore, we propose a novel direct time-of-flight pixel using the single-photon avalanche diode (SPAD) sensor, with an in-pixel averaging method to suppress ambient light and detect the laser pulse arrival time. The system utilizes two orthogonal sinusoidal signals applied to the pixel as inputs, which are synchronized with a pulsed laser source. The detected signal phase indicates the arrival time. To evaluate the proposed system’s potential, we developed analytical and statistical models for assessing the phase error and precision of the arrival time under varying ambient light levels. The pixel simulation showed that the phase precision is less than 1% of the detection range when the ambient-to-signal ratio is 120. A proof-of-concept pixel array prototype was fabricated and characterized to validate the system’s performance. The pixel consumed, on average, 40 μW of power in operation with ambient light. The results demonstrate that the system can operate effectively under varying ambient light conditions and its potential for customization based on specific application requirements. This paper concludes by discussing the system’s performance relative to the existing direct time-of-flight technologies, identifying their strengths and limitations.

Current techniques of forest inventory rely on manual measurements and are slow and labor intensive. Recent developments in computer vision and depth sensing can produce accurate measurement data at significantly reduced time and labor costs. We developed the ForSense system to measure the diameters of trees at various points along the stem as well as stem straightness. Time use, mean absolute error (MAE), and root mean squared error (RMSE) metrics were used to compare the system against manual methods, and to compare the system against itself (reproducibility). Depth-derived diameter measurements of the stems at the heights of 0.3, 1.4, and 2.7 m achieved RMSE of 1.7, 1.5, and 2.7 cm, respectively. The ForSense system produced straightness measurement data that was highly correlated with straightness ratings by trained foresters. The ForSense system was also consistent, achieving sub-centimeter diameter difference with subsequent measures and less than 4% difference in straightness value between runs. This method of forest inventory, which is based on depth-image computer vision, is time efficient compared to manual methods and less computationally and technologically intensive compared to Structure-from-Motion (SFM) photogrammetry and ground-based LiDAR or terrestrial laser scanning (TLS). [ABSTRACT FROM AUTHOR]

Three-dimensional (3D) information is vital for providing detailed features of the physical world, which is used in numerous applications such as industrial inspection, automatic navigation and identity authentication. However, the implementations of 3D imagers always rely on bulky optics. Metasurfaces, as the next-generation optics, shows flexible modulation abilities and excellent performance combined with computer vision algorithm. Here, we demonstrate an active 3D positioning and imaging method with large field of view (FOV) by single fringe projection based on metasurface and solve the accurate and robust calibration problem with the depth uncertainty of 4 μm. With a compact metasurface projector, the demonstrated method can achieve submillimeter positioning accuracy under the FOV of 88°, offering robust and fast 3D reconstruction of the texture-less scene due to the modulation characteristic of the fringe. Such scheme may accelerate prosperous engineering applications with the continued growth of flat-optics manufacturing process by using metadevices.

3-D depth sensing is essential for many applications ranging from consumer electronics to robotics. Passive depth sensing techniques based on a double-helix (DH) point-spread-function (PSF) feature high depth estimation precision, minimal power consumption, and reduced system complexity compared to active sensing methods. Here, we propose and experimentally implemented a polarization-multiplexed DH metalens designed using an autonomous direct search algorithm, which utilizes two contra-rotating DH PSFs encoded in orthogonal polarization states to enable monocular depth perception. Using a reconstruction algorithm that we developed, concurrent depth calculation and scene reconstruction with minimum distortion and high resolution in all three dimensions were demonstrated.

This paper presents a depth-based hybrid method to generate safe flight corridors for a memoryless local navigation planner. It is first proposed to use raw depth images as inputs in the learning-based object-detection engine with no requirement for map fusion. We then employ an object-detection network to directly predict the base of polyhedral safe corridors in a new raw depth image. Furthermore, we apply a verification procedure to eliminate any false predictions so that the resulting collision-free corridors are guaranteed. More importantly, the proposed mechanism helps produce separate safe corridors with minimal overlap that are suitable to be used as space boundaries for path planning. The average intersection of union (IoU) of corridors obtained by the proposed algorithm is less than 2%. To evaluate the effectiveness of our method, we incorporated it into a memoryless planner with a straight-line path-planning algorithm. We then tested the entire system in both synthetic and real-world obstacle-dense environments. The obtained results with very high success rates demonstrate that the proposed approach is highly capable of producing safe corridors for memoryless local planning. [ABSTRACT FROM AUTHOR]

Object detection is an essential component of autonomous mobile robotic systems, enabling robots to understand and interact with the environment. Object detection and recognition have made significant progress using convolutional neural networks (CNNs). Widely used in autonomous mobile robot applications, CNNs can quickly identify complicated image patterns, such as objects in a logistic environment. Integration of environment perception algorithms and motion control algorithms is a topic subjected to significant research. On the one hand, this paper presents an object detector to better understand the robot environment and the newly acquired dataset. The model was optimized to run on the mobile platform already on the robot. On the other hand, the paper introduces a model-based predictive controller to guide an omnidirectional robot to a particular position in a logistic environment based on an object map obtained from a custom-trained CNN detector and LIDAR data. Object detection contributes to a safe, optimal, and efficient path for the omnidirectional mobile robot. In a practical scenario, we deploy a custom-trained and optimized CNN model to detect specific objects in the warehouse environment. Then we evaluate, through simulation, a predictive control approach based on the detected objects using CNNs. Results are obtained in object detection using a custom-trained CNN with an in-house acquired data set on a mobile platform and in the optimal control for the omnidirectional mobile robot. [ABSTRACT FROM AUTHOR]

We propose an omnidirectional measurement method without blind spots by using a convex mirror, which in principle does not cause chromatic aberration, and by using vertical disparity by installing cameras at the top and bottom of the image. In recent years, there has been significant research in the fields of autonomous cars and robots. In these fields, three-dimensional measurements of the surrounding environment have become indispensable. Depth sensing with cameras is one of the most important sensors for recognizing the surrounding environment. Previous studies have attempted to measure a wide range of areas using fisheye and full spherical panoramic cameras. However, these approaches have limitations such as blind spots and the need for multiple cameras to measure all directions. Therefore, this paper describes a stereo camera system that uses a device capable of taking an omnidirectional image with a single shot, enabling omnidirectional measurement with only two cameras. This achievement was challenging to attain with conventional stereo cameras. The results of experiments confirmed an improvement in accuracy of up to 37.4% compared to previous studies. In addition, the system succeeded in generating depth image that can recognize distances in all directions in a single frame, demonstrating the possibility of omnidirectional measurement with two cameras. [ABSTRACT FROM AUTHOR]

The purpose of this work is to prove the suitability of integrated single-photon avalanche diode (SPAD)-based indirect time-of-flight (iTOF) for sub-100 µm precision depth sensing using a correlation approach with GHz modulation frequencies. For this purpose, a prototype containing a single pixel consisting of an integrated SPAD, quenching circuit, and two independent correlator circuits was fabricated in a 0.35 µm CMOS process and characterized. It achieved a precision of 70 µm and a nonlinearity of less than 200 µm at a received signal power of less than 100 pW. Sub-mm precision was achieved with a signal power of less than 200 fW. These results and the simplicity of our correlation approach underline the great potential of SPAD-based iTOF for future depth sensing applications. [ABSTRACT FROM AUTHOR]

This work presents a novel framework for web-based environment-aware rendering and interaction in augmented reality based on WebXR and three.js. It aims at accelerating the development of device-agnostic Augmented Reality (AR) applications. The solution allows for a realistic rendering of 3D elements, handles geometry occlusion, casts shadows of virtual objects onto real surfaces, and provides physics interaction with real-world objects. Unlike most existing state-of-the-art systems that are built to run on a specific hardware configuration, the proposed solution targets the web environment and is designed to work on a vast range of devices and configurations. Our solution can use monocular camera setups with depth data estimated by deep neural networks or, when available, use higher-quality depth sensors (e.g., LIDAR, structured light) that provide a more accurate perception of the environment. To ensure consistency in the rendering of the virtual scene a physically based rendering pipeline is used, in which physically correct attributes are associated with each 3D object, which, combined with lighting information captured by the device, enables the rendering of AR content matching the environment illumination. All these concepts are integrated and optimized into a pipeline capable of providing a fluid user experience even on middle-range devices. The solution is distributed as an open-source library that can be integrated into existing and new web-based AR projects. The proposed framework was evaluated and compared in terms of performance and visual features with two state-of-the-art alternatives. [ABSTRACT FROM AUTHOR]

To obtain more consistent measurements through the course of a wheat growing season, we conceived and designed an autonomous robotic platform that performs collision avoidance while navigating in crop rows using spatial artificial intelligence (AI). The main constraint the agronomists have is to not run over the wheat while driving. Accordingly, we have trained a spatial deep learning model that helps navigate the robot autonomously in the field while avoiding collisions with the wheat. To train this model, we used publicly available databases of prelabeled images of wheat, along with the images of wheat that we have collected in the field. We used the MobileNet single shot detector (SSD) as our deep learning model to detect wheat in the field. To increase the frame rate for real-time robot response to field environments, we trained MobileNet SSD on the wheat images and used a new stereo camera, the Luxonis Depth AI Camera. Together, the newly trained model and camera could achieve a frame rate of 18–23 frames per second (fps)—fast enough for the robot to process its surroundings once every 2–3 inches of driving. Once we knew the robot accurately detects its surroundings, we addressed the autonomous navigation of the robot. The new stereo camera allows the robot to determine its distance from the trained objects. In this work, we also developed a navigation and collision avoidance algorithm that utilizes this distance information to help the robot see its surroundings and maneuver in the field, thereby precisely avoiding collisions with the wheat crop. Extensive experiments were conducted to evaluate the performance of our proposed method. We also compared the quantitative results obtained by our proposed MobileNet SSD model with those of other state-of-the-art object detection models, such as the YOLO V5 and Faster region-based convolutional neural network (R-CNN) models. The detailed comparative analysis reveals the effectiveness of our method in terms of both model precision and inference speed.

Recent advances in computer vision and deep learning have shown that the fusion of depth information can significantly enhance the performance of RGB-based damage detection and segmentation models. However, alongside the advantages, depth-sensing also presents many practical challenges. For instance, the depth sensors impose an additional payload burden on the robotic inspection platforms limiting the operation time and increasing the inspection cost. Additionally, some lidar-based depth sensors have poor outdoor performance due to sunlight contamination during the daytime. In this context, this study investigates the feasibility of abolishing depth-sensing at test time without compromising the segmentation performance. An autonomous damage segmentation framework is developed, based on recent advancements in vision-based multi-modal sensing such as modality hallucination (MH) and monocular depth estimation (MDE), which require depth data only during the model training. At the time of deployment, depth data becomes expendable as it can be simulated from the corresponding RGB frames. This makes it possible to reap the benefits of depth fusion without any depth perception per se. This study explored two different depth encoding techniques and three different fusion strategies in addition to a baseline RGB-based model. The proposed approach is validated on computer-generated RGB-D data of reinforced concrete buildings subjected to seismic damage. It was observed that the surrogate techniques can increase the segmentation IoU by up to 20.1% with a negligible increase in the computation cost. Overall, this study is believed to make a positive contribution to enhancing the resilience of critical civil infrastructure. [ABSTRACT FROM AUTHOR]

For drone vision and navigation, low-power 3-D depth sensing with robust operations against strong/weak light and various weather conditions is crucial. CMOS image sensor (CIS) and light detection and ranging (LiDAR) can provide high-fidelity imaging. However, CIS lacks depth sensing and has difficulty in low light conditions. LiDAR is expensive with issues of dealing with strong direct interference sources. Ultrasound imaging system (UIS), on the other hand, is robust in various weather and light conditions and is cost-effective. However, in air channel, it often suffers from long image reconstruction latency and low framerate. To address these issues, we present a UIS application-specific integrated circuit (ASIC) that adopts the one-shot transmitter (TX) and on-chip per-voxel receiver (RX) beamfocusing (PV-RXBF) image reconstruction scheme. The ASIC adopts the designs of fully differential charge-reuse high-voltage TX (FDCR-HVTX), digital back-end (DBE), and an on-chip power management unit (PMU). FDCR-HVTX generates 28 $\text{V}_{\mathrm {pp}}$ pulses and reduces the average power consumption by 25% by charge reuse (CR). The DBE achieves 7.76- $\mu \text{s}$ processing latency and 9.83M-FocalPoint/s throughput to effectively translate real-time 3-D image streaming at 24 frames/s. A prototype UIS, with an $8\times 8$ bulk piezo transducer array, is assembled with the proposed ASIC and a wireless data transmission module [field-programmable gate array (FPGA) + ESP32] on an entry-level consumer drone, and the real-time wireless 3-D image streaming at 24 frames/s with a range of 7 m is verified while the drone is flying. The ASIC implemented in 180-nm 1P6M Standard CMOS occupies 32.5 mm2 and consumes 142.3 mW. [ABSTRACT FROM AUTHOR]

To obtain more consistent measurements through the course of a wheat growing season, we conceived and designed an autonomous robotic platform that performs collision avoidance while navigating in crop rows using spatial artificial intelligence (AI). The main constraint the agronomists have is to not run over the wheat while driving. Accordingly, we have trained a spatial deep learning model that helps navigate the robot autonomously in the field while avoiding collisions with the wheat. To train this model, we used publicly available databases of prelabeled images of wheat, along with the images of wheat that we have collected in the field. We used the MobileNet single shot detector (SSD) as our deep learning model to detect wheat in the field. To increase the frame rate for real-time robot response to field environments, we trained MobileNet SSD on the wheat images and used a new stereo camera, the Luxonis Depth AI Camera. Together, the newly trained model and camera could achieve a frame rate of 18–23 frames per second (fps)—fast enough for the robot to process its surroundings once every 2–3 inches of driving. Once we knew the robot accurately detects its surroundings, we addressed the autonomous navigation of the robot. The new stereo camera allows the robot to determine its distance from the trained objects. In this work, we also developed a navigation and collision avoidance algorithm that utilizes this distance information to help the robot see its surroundings and maneuver in the field, thereby precisely avoiding collisions with the wheat crop. Extensive experiments were conducted to evaluate the performance of our proposed method. We also compared the quantitative results obtained by our proposed MobileNet SSD model with those of other state-of-the-art object detection models, such as the YOLO V5 and Faster region-based convolutional neural network (R-CNN) models. The detailed comparative analysis reveals the effectiveness of our method in terms of both model precision and inference speed. [ABSTRACT FROM AUTHOR]

Increasingly complex automated driving functions, specifically those associated with free space detection (FSD), are delegated to convolutional neural networks (CNNs). If the dataset used to train the network lacks diversity, modality, or sufficient quantities, the driver policy that controls the vehicle may induce safety risks. Although most autonomous ground vehicles (AGVs) perform well in structured surroundings, the need for human intervention significantly rises when presented with unstructured niche environments. To this end, we developed an AGV for seamless indoor and outdoor navigation to collect realistic multimodal data streams. We demonstrate one application of the AGV when applied to a self-evolving FSD framework that leverages online active machine-learning (ML) paradigms and sensor data fusion. In essence, the self-evolving AGV queries image data against a reliable data stream, ultrasound, before fusing the sensor data to improve robustness. We compare the proposed framework to one of the most prominent free space segmentation methods, DeepLabV3+. DeepLabV3+ is a state-of-the-art semantic segmentation model composed of a CNN and an autodecoder. In consonance with the results, the proposed framework outperforms DeepLabV3+. The performance of the proposed framework is attributed to its ability to self-learn free space. This combination of online and active ML removes the need for large datasets typically required by a CNN. Moreover, this technique provides case-specific free space classifications based on the information gathered from the scenario at hand. [ABSTRACT FROM AUTHOR]

This paper presents a depth-based hybrid method to generate safe flight corridors for a memoryless local navigation planner. It is first proposed to use raw depth images as inputs in the learning-based object-detection engine with no requirement for map fusion. We then employ an object-detection network to directly predict the base of polyhedral safe corridors in a new raw depth image. Furthermore, we apply a verification procedure to eliminate any false predictions so that the resulting collision-free corridors are guaranteed. More importantly, the proposed mechanism helps produce separate safe corridors with minimal overlap that are suitable to be used as space boundaries for path planning. The average intersection of union (IoU) of corridors obtained by the proposed algorithm is less than 2%. To evaluate the effectiveness of our method, we incorporated it into a memoryless planner with a straight-line path-planning algorithm. We then tested the entire system in both synthetic and real-world obstacle-dense environments. The obtained results with very high success rates demonstrate that the proposed approach is highly capable of producing safe corridors for memoryless local planning.

A 3D point cloud is typically constructed from depth measurements acquired by sensors at one or more viewpoints. The measurements suffer from both quantization and noise corruption. To improve quality, previous works denoise a point cloud a posteriori after projecting the imperfect depth data onto 3D space. Instead, we enhance depth measurements directly on the sensed images a priori, before synthesizing a 3D point cloud. By enhancing near the physical sensing process, we tailor our optimization to our depth formation model before subsequent processing steps that obscure measurement errors. Specifically, we model depth formation as a combined process of signal-dependent noise addition and non-uniform log-based quantization. The designed model is validated (with parameters fitted) using collected empirical data from a representative depth sensor. To enhance each pixel row in a depth image, we first encode intra-view similarities between available row pixels as edge weights via feature graph learning. We next establish inter-view similarities with another rectified depth image via viewpoint mapping and sparse linear interpolation. This leads to a maximum a posteriori (MAP) graph filtering objective that is convex and differentiable. We minimize the objective efficiently using accelerated gradient descent (AGD), where the optimal step size is approximated via Gershgorin circle theorem (GCT). Experiments show that our method significantly outperformed recent point cloud denoising schemes and state-of-the-art image denoising schemes in two established point cloud quality metrics. [ABSTRACT FROM AUTHOR]

This study introduces a custom-built RGB-D-based machine vision system designed to accurately estimate the mass and volume of whole kimchi cabbage (WC) and longitudinally cut kimchi cabbage (LCC). Given the pivotal role of kimchi cabbage (KC) in both Asian and Western diets, accurate post-harvest assessment of its mass and volume is critical for quality control, sorting, and pricing. Conventional manual measurements and visual estimations are laborious and inaccurate. Our research leveraged RGB-D data to refine machine learning models and enhance the extraction and analysis of 2D, 3D, and colorimetric features for a more reliable estimation approach. The results demonstrate that integrating 3D and colorimetric features markedly improves the estimation accuracy, with notable success in mass estimation for LCC (R² = 0.913, ratio of performance to deviation (RPD) = 3.38) and robust volume predictions for both cabbage types (R² > 0.90, RPD > 3). However, challenges such as potential over-exclusion of outer leaves in LCC and the need for more advanced WC mass estimation techniques have been identified. Future work will focus on refining the feature extraction methods and assessing various imaging environments to enhance the precision of mass and volume predictions across different forms of KC. • Novel RGB-D-based system accurately estimates mass and volume of kimchi cabbage. • Integration of 3D and colorimetric features enhances estimation accuracy. • Significant success in mass estimation for longitudinally cut cabbage (R² = 0.913). • Robust volume predictions achieved for both whole and cut cabbage (R² > 0.90). • Study addresses critical need for reliable post-harvest assessment methods. [ABSTRACT FROM AUTHOR]

Object detection is an essential component of autonomous mobile robotic systems, enabling robots to understand and interact with the environment. Object detection and recognition have made significant progress using convolutional neural networks (CNNs). Widely used in autonomous mobile robot applications, CNNs can quickly identify complicated image patterns, such as objects in a logistic environment. Integration of environment perception algorithms and motion control algorithms is a topic subjected to significant research. On the one hand, this paper presents an object detector to better understand the robot environment and the newly acquired dataset. The model was optimized to run on the mobile platform already on the robot. On the other hand, the paper introduces a model-based predictive controller to guide an omnidirectional robot to a particular position in a logistic environment based on an object map obtained from a custom-trained CNN detector and LIDAR data. Object detection contributes to a safe, optimal, and efficient path for the omnidirectional mobile robot. In a practical scenario, we deploy a custom-trained and optimized CNN model to detect specific objects in the warehouse environment. Then we evaluate, through simulation, a predictive control approach based on the detected objects using CNNs. Results are obtained in object detection using a custom-trained CNN with an in-house acquired data set on a mobile platform and in the optimal control for the omnidirectional mobile robot.

The purpose of this work is to prove the suitability of integrated single-photon avalanche diode (SPAD)-based indirect time-of-flight (iTOF) for sub-100 µm precision depth sensing using a correlation approach with GHz modulation frequencies. For this purpose, a prototype containing a single pixel consisting of an integrated SPAD, quenching circuit, and two independent correlator circuits was fabricated in a 0.35 µm CMOS process and characterized. It achieved a precision of 70 µm and a nonlinearity of less than 200 µm at a received signal power of less than 100 pW. Sub-mm precision was achieved with a signal power of less than 200 fW. These results and the simplicity of our correlation approach underline the great potential of SPAD-based iTOF for future depth sensing applications.

This work presents a novel framework for web-based environment-aware rendering and interaction in augmented reality based on WebXR and three.js. It aims at accelerating the development of device-agnostic Augmented Reality (AR) applications. The solution allows for a realistic rendering of 3D elements, handles geometry occlusion, casts shadows of virtual objects onto real surfaces, and provides physics interaction with real-world objects. Unlike most existing state-of-the-art systems that are built to run on a specific hardware configuration, the proposed solution targets the web environment and is designed to work on a vast range of devices and configurations. Our solution can use monocular camera setups with depth data estimated by deep neural networks or, when available, use higher-quality depth sensors (e.g., LIDAR, structured light) that provide a more accurate perception of the environment. To ensure consistency in the rendering of the virtual scene a physically based rendering pipeline is used, in which physically correct attributes are associated with each 3D object, which, combined with lighting information captured by the device, enables the rendering of AR content matching the environment illumination. All these concepts are integrated and optimized into a pipeline capable of providing a fluid user experience even on middle-range devices. The solution is distributed as an open-source library that can be integrated into existing and new web-based AR projects. The proposed framework was evaluated and compared in terms of performance and visual features with two state-of-the-art alternatives.

We propose an omnidirectional measurement method without blind spots by using a convex mirror, which in principle does not cause chromatic aberration, and by using vertical disparity by installing cameras at the top and bottom of the image. In recent years, there has been significant research in the fields of autonomous cars and robots. In these fields, three-dimensional measurements of the surrounding environment have become indispensable. Depth sensing with cameras is one of the most important sensors for recognizing the surrounding environment. Previous studies have attempted to measure a wide range of areas using fisheye and full spherical panoramic cameras. However, these approaches have limitations such as blind spots and the need for multiple cameras to measure all directions. Therefore, this paper describes a stereo camera system that uses a device capable of taking an omnidirectional image with a single shot, enabling omnidirectional measurement with only two cameras. This achievement was challenging to attain with conventional stereo cameras. The results of experiments confirmed an improvement in accuracy of up to 37.4% compared to previous studies. In addition, the system succeeded in generating depth image that can recognize distances in all directions in a single frame, demonstrating the possibility of omnidirectional measurement with two cameras.

This paper summarizes our results and findings made throughout the past years on the way towards a one millimeter thin, flexible, and scalable foil camera. The sensor part of the camera consists of multiple thin-film luminescent concentrator (LC) layers, each of which is sensitive to a different band of the light spectrum. A special optical micro-structure cut into the edges of the LC layers multiplexes the transported light signal into a variant of the Radon transform of the image focused on the LC surface. For increasing the camera's depth of field beyond the sensor surface, various thin-film imaging layers, such as optical Söller collimators realized by means of X-ray lithography on a PMMA wafer, have been investigated. The flexibility and scalability of our thin-film camera has the potential to lead to new human–computer interfaces that are unconstrained in shape and sensing-distance. Applications such as contact-less sensing, smart skin sensors for autonomous robots and industrial machines, and environment sensing for vehicles are imaginable. [ABSTRACT FROM AUTHOR]

Recognition of locomotion mode (LM) has been used to control the lower limb powered prosthetic and orthotic devices for navigating autonomously on a variety of terrains. In addition, the knowledge of environmental features (EFs) can be utilized to modulate the joint kinetics of prosthetic devices in a precise manner. In this article, a novel LM recognition strategy has been proposed along with EFs estimation utilizing a combination of laser distance sensors (LDSs) and inertial measurement units (IMUs). LM was classified from outputs of eight sensors, i.e., six LDSs and two IMUs, utilizing support vector machine (SVM) followed by a majority voting scheme. A cloud of points was generated in the sagittal plane, which was further used to determine the EFs leveraging curve fitting techniques. The efficiency of the proposed method was tested with six healthy subjects and for a variety of LMs, such as walking on the level-ground, ramp, and stairs. The overall accuracy of LM classification and EF estimation was found to be equal to 97.85% and 95.36%, respectively. The proposed study is an accurate, computationally inexpensive, and cost-effective solution and can be used efficiently to control lower limb prosthetic devices. The novelty of this study is the simplification of the LM classification and EF estimation methods compared to a camera-based technique to facilitate real-time control of powered prostheses. [ABSTRACT FROM AUTHOR]

In this study, an optical depth-sensing system whose focus distance was equal to its object distance was constructed by using a lens with a shortened depth of field. In addition to a mechanical control lens that captured two-dimensional images, a set of negative microlens array films used to shorten the depth of field was installed in this system. The developed system can scan a region and obtain the relationship curve between the focus distance and the motor step. Depth information is acquired by discerning the sharpness of the object’s outline and determining whether the captured object is in focus. The motor step is initialized when the object is in focus. The optical path of the developed system is simple, and its volume is minimal; thus, the developed system is suitable for being combined with cell phone lenses.

Smart dustbins and frameworks have been in conversation for a long time. Many innovations have been used to maintain cleanliness and hygiene in the society.Our smart system is the contactless trash collection using ESP32 interfaced with an ultrasonic sensor and servo motor. The proposed procedure for the organization of consumers is a powerful and effective cycle. This automation of waste likewise diminishes human exertion and subsequently the expense of the entire process. This framework can be executed at any spot effortlessly and inside a sensible measure of time. In this proposed method, the collection of wastes can be achieved by using sensors and actuators for supervising the lid of the dustbin, ultimately reducing human efforts. It is an easy to implement model which can be proposed anywhere to a certain extent. The global way of identifying and managing waste is becoming more efficient and smart. Also this invention motivates individuals to develop good hygiene habits. This can be implemented at any place and can be administered with the help of mobile applications by senior citizens and differently abled people to keep their surroundings clean. [ABSTRACT FROM AUTHOR]

Rectification is a critical process for speckle projection systems to ensure reliable image matching in producing a depth map. However, few studies have discussed it up to the present. In this article, we present a flexible but robust approach to address this issue. It mainly consists of two steps: calibration and rectification. The calibration step takes one or several pairs of images of the projected speckle pattern on a plane to estimate a unit vector indicating the direction of the line where emitter center lies on. The rectification step virtually rotates the camera based on this vector to obtain rectified images. Given camera intrinsic parameters, this approach only needs a minimum configuration consisting of two freely captured speckle images of a planar surface to complete system rectification, which makes it particularly suited for the case with no access to the internal projected speckle pattern. Elaborate experiments on real speckle images and synthetic images from different camera poses are conducted, demonstrating the effectiveness and robustness of the proposed approach. Furthermore, we also demonstrate that when the internal pattern is recovered from the rectified images, it reaches an accuracy comparable with the ones from the best state-of-the-art methods in performing a stereo matching, but with a much simpler operation to obtain. [ABSTRACT FROM AUTHOR]

Bridge inspections are a vital part of bridge maintenance and the main information source for Bridge Management Systems is used in decision-making regarding repairs. Without a doubt, both can benefit from the implementation of the Building Information Modelling philosophy. To fully harness the BIM potential in this area, we have to develop tools that will provide inspection accurate information easily and fast. In this paper, we present an example of how such a tool can utilise tablets coupled with the latest generation RGB-D cameras for data acquisition; how these data can be processed to extract the defect surface area and create a 3D representation, and finally embed this information into the BIM model. Additionally, the study of depth sensor accuracy is presented along with surface area accuracy tests and an exemplary inspection of a bridge pillar column.

Depth sensing with single-shot structured light is widely employed for depth measurement of dynamic scenes. For conventional single-shot structured light techniques, the projected pattern is coded with regards to color, shape, and intensity, aiming at assigning each pixel a unique label. However, the features of the projected pattern are extremely susceptible to the surface texture of the measured object, resulting in errors during performing decoding. In this study, a robust and simple binary pattern is proposed for high-accuracy depth sensing. Only two gray values are utilized in the designed pattern, which is insensitive to environmental noise. The pseudo two-dimensional De Bruijn sequence coding method is employed to reduce the size of the decoding window for robust matching. Short slashes are adopted as features of the pattern that greatly simplifies the feature extraction and pattern decoding. Compared with the Kinect and time-of-flight (TOF) depth cameras, our proposed method can acquire high-precision depth in quantitative and qualitative experiments. [ABSTRACT FROM AUTHOR]

In this study, an optical depth-sensing system whose focus distance was equal to its object distance was constructed by using a lens with a shortened depth of field. In addition to a mechanical control lens that captured two-dimensional images, a set of negative microlens array films used to shorten the depth of field was installed in this system. The developed system can scan a region and obtain the relationship curve between the focus distance and the motor step. Depth information is acquired by discerning the sharpness of the object's outline and determining whether the captured object is in focus. The motor step is initialized when the object is in focus. The optical path of the developed system is simple, and its volume is minimal; thus, the developed system is suitable for being combined with cell phone lenses. [ABSTRACT FROM AUTHOR]

Abstract Background Animal-attached sensors are increasingly used to provide insights on behaviour and physiology. However, such tags usually lack information on the structure of the surrounding environment from the perspective of a study animal and thus may be unable to identify potentially important drivers of behaviour. Recent advances in robotics and computer vision have led to the availability of integrated depth-sensing and motion-tracking mobile devices. These enable the construction of detailed 3D models of an environment within which motion can be tracked without reliance on GPS. The potential of such techniques has yet to be explored in the field of animal biotelemetry. This report trials an animal-attached structured light depth-sensing and visual–inertial odometry motion-tracking device in an outdoor environment (coniferous forest) using the domestic dog (Canis familiaris) as a compliant test species. Results A 3D model of the forest environment surrounding the subject animal was successfully constructed using point clouds. The forest floor was labelled using a progressive morphological filter. Trees trunks were modelled as cylinders and identified by random sample consensus. The predicted and actual presence of trees matched closely, with an object-level accuracy of 93.3%. Individual points were labelled as belonging to tree trunks with a precision, recall, and F$$_{\beta }$$ β score of 1.00, 0.88, and 0.93, respectively. In addition, ground-truth tree trunk radius measurements were not significantly different from random sample consensus model coefficient-derived values. A first-person view of the 3D model was created, illustrating the coupling of both animal movement and environment reconstruction. Conclusions Using data collected from an animal-borne device, the present study demonstrates how terrain and objects (in this case, tree trunks) surrounding a subject can be identified by model segmentation. The device pose (position and orientation) also enabled recreation of the animal’s movement path within the 3D model. Although some challenges such as device form factor, validation in a wider range of environments, and direct sunlight interference remain before routine field deployment can take place, animal-borne depth sensing and visual–inertial odometry have great potential as visual biologging techniques to provide new insights on how terrestrial animals interact with their environments.

This paper introduces a field-programmable gate array (FPGA)-based one-shot depth acquisition system, which is a kind of Kinect-like Depth sensing. The proposed implementation integrates the random binary pattern with an improved matching method, which is able to obtain dense and precise depth maps. Scene images captured by a camera are transferred to FPGA (Xilinx Virtex7 XC7VX690T2FFG1761C) through a CamerLink interface, and the acquired depth results are sent to host computer through a high-speed USB3.0 interface. An elaborate FPGA architecture is designed to realize efficient and accurate calculations. The proposed system outperforms the existing stereo systems, not only on the quality of depth acquisition, but also on the reconstruction speed for which our system achieves 90 fps at a resolution of 1280 × 960. Besides, Kinect-like depth sensing has an inherent advantage in deployments, much easier than other kinds of systems. Our proposed system has extensive applications not only in the field of vision processing, but also in the fields where block matching is needed.

Abstract The fringe pattern can realize high-resolution and dense depth sensing. However, the phase ambiguity is a challenge in the fringe pattern. In this paper, one period-code pattern is embedded to the fringe pattern by the frequency-division-multiplexing (FDM) framework. The nonparametric skew and De Bruijn sequence are utilized to determine the label of the period. For a more reliable phase unwrapping, three criteria are utilized to rectify the period numbers further. Quantitative and qualitative experiments show that the proposed method can achieve more reliable depth sensing compared with the counterparts. Even the measured scene contains discontinuous surfaces or sharp edges, the proposed algorithm can attain reliable depth.

Respiratory rate is a well-known to be a clinically important parameter with numerous clinical uses including the assessment of disease state and the prediction of deterioration. It is frequently monitored using simple spot checks where reporting is intermittent and often prone to error. We report here on an algorithm to determine respiratory rate continuously and robustly using a non-contact method based on depth sensing camera technology. The respiratory rate of 14 healthy volunteers was studied during an acute hypoxic challenge where blood oxygen saturation was reduced in steps to a target 70% oxygen saturation and which elicited a wide range of respiratory rates. Depth sensing data streams were acquired and processed to generate a respiratory rate (RRdepth). This was compared to a reference respiratory rate determined from a capnograph (RRcap). The bias and root mean squared difference (RMSD) accuracy between RRdepth and the reference RRcap was found to be 0.04 bpm and 0.66 bpm respectively. The least squares fit regression equation was determined to be: RRdepth = 0.99 × RRcap + 0.13 and the resulting Pearson correlation coefficient, R, was 0.99 (p < 0.001). These results were achieved with a 100% reporting uptime. In conclusion, excellent agreement was found between RRdepth and RRcap. Further work should include a larger cohort combined with a protocol to further test algorithmic performance in the face of motion and interference typical of that experienced in the clinical setting. [ABSTRACT FROM AUTHOR]

Objective: Hand hygiene is essential for preventing hospital-acquired infections but is difficult to accurately track. The gold-standard (human auditors) is insufficient for assessing true overall compliance. Computer vision technology has the ability to perform more accurate appraisals. Our primary objective was to evaluate if a computer vision algorithm could accurately observe hand hygiene dispenser use in images captured by depth sensors.Materials and Methods: Sixteen depth sensors were installed on one hospital unit. Images were collected continuously from March to August 2017. Utilizing a convolutional neural network, a machine learning algorithm was trained to detect hand hygiene dispenser use in the images. The algorithm's accuracy was then compared with simultaneous in-person observations of hand hygiene dispenser usage. Concordance rate between human observation and algorithm's assessment was calculated. Ground truth was established by blinded annotation of the entire image set. Sensitivity and specificity were calculated for both human and machine-level observation.Results: A concordance rate of 96.8% was observed between human and algorithm (kappa = 0.85). Concordance among the 3 independent auditors to establish ground truth was 95.4% (Fleiss's kappa = 0.87). Sensitivity and specificity of the machine learning algorithm were 92.1% and 98.3%, respectively. Human observations showed sensitivity and specificity of 85.2% and 99.4%, respectively.Conclusions: A computer vision algorithm was equivalent to human observation in detecting hand hygiene dispenser use. Computer vision monitoring has the potential to provide a more complete appraisal of hand hygiene activity in hospitals than the current gold-standard given its ability for continuous coverage of a unit in space and time. [ABSTRACT FROM AUTHOR]

Depth sensing devices enabled with an RGB camera, can be used to augment conventional images with depth information on a per-pixel basis. Currently available RGB-D sensors include the Asus Xtion Pro, Microsoft Kinect and Intel RealSense™. However, these sensors have certain limitations. Objects that are shiny, transparent or have an absorbing matte surface, create problems due to reflection. Also, there can be an interference in the IR pattern due to the use of multiple RGB-D cameras and the depth information is correctly interpreted only for short distances between the camera and the object. The proposed system, block matching stereo vision (BMSV) uses an RGB-D camera with rectified/non-rectified block matching and image pyramiding along with dynamic programming for human tracking and capture of accurate depth information from shiny/transparent objects. Here, the IR emitter generates a known IR pattern and the depth information is recovered by comparing the multiple views of the focused object. The depth map of the BMSV RGB-D camera and the resultant disparity map are fused. This fills any void regions that may have emerged due to interference or because of the reflective transparent surfaces and an enhanced dense stereo image is obtained. The proposed method is applied to a 3D realistic head model, a functional magnetic resonance image (fMRI) and the results are presented. Results showed an improvement in speed and accuracy of RGB-D sensors which in turn provided accurate depth information density irrespective of the object surface. [ABSTRACT FROM AUTHOR]

The paper presents a novel method for the classification of gait phases for power gait orthosis users based on machine learning. The classification uses depth images collected from a Time of Flight camera embedded in the crutches employed for the assisted gait. The machine learning algorithm foresees an initial phase of data collection and processing, identifying the 3D points belonging to the foot and those belonging to the floor. From these, a feature set is computed analyzing the values of percentiles of distances of the foot from the floor, and passed to a modified version of Random Forest classifier, called Sigma-z Random Forest. The classifier considers the uncertainties associated to each feature set and provides both the classification of the gait phase (stance or swing) and an associated confidence value. In this work, we propose the use of the confidence value to improve the reliability of the gait phase classification, by applying an optimized threshold to the confidence value obtained for each new frame. The algorithm has been tested on different subjects and environments. An average classification accuracy of 87.3% has been obtained (+6.3% with respect to the standard random forest classifier), with a minor loss of unclassifiable frames. Results highlight that unclassifiable samples are usually associated to transitions between stance and swing. [ABSTRACT FROM AUTHOR]

The novel depth-sensing system presented here revolutionizes structured light (SL) technology by employing metasurfaces and photonic crystal surface-emitting lasers (PCSELs) for efficient facial recognition in monocular depth-sensing. Unlike conventional dot projectors relying on diffractive optical elements (DOEs) and collimators, our system projects approximately 45,700 infrared dots from a compact 297-μm-dimention metasurface, drastically more spots (1.43 times) and smaller (233 times) than the DOE-based dot projector in an iPhone. With a measured field-of-view (FOV) of 158° and a 0.611° dot sampling angle, the system is lens-free and lightweight and boasts lower power consumption than vertical-cavity surface-emitting laser (VCSEL) arrays, resulting in a 5-10 times reduction in power. Utilizing a GaAs-based metasurface and a simplified optical architecture, this innovation not only addresses the drawbacks of traditional SL depth-sensing but also opens avenues for compact integration into wearable devices, offering remarkable advantages in size, power efficiency, and potential for widespread adoption.

Structured light techniques are widely used in the depth measurement. In this paper, we propose a colored speckle method for high-accuracy depth sensing. The projected pattern is a composite of the discrete phase information and the colored De Bruijn sequence information, which can prove the feasibility, in practice, and the robustness to the environment. A Gabor filter is used to estimate the wrapped phase. To alleviate the phase ambiguity of conventional measurement methods, the period of the phase is distributed in two channels of the colored pattern by the De Bruijn sequence decoding. A morphological closure is adopted to determine the De Bruijn code by the phase neighborhoods back and forth. The proposed method is suitable for a dynamic scene. The quantitative and qualitative experiments are conducted, in practice, to evaluate the performance of the proposed method. The results of the experiments prove that not only can our proposed method achieve high accuracy and dense depth map, but it is also suitable for color object and dynamic scenes indoors. [ABSTRACT FROM AUTHOR]