Search

Three-dimensional (3-D) stereo images can be generated via computer-based image processing of CORONA stereo pairs. To a certain extent, important terrain and surface feature data extracted from these stereo images can improve the survey of archaeological sites and the identification and mapping of major landscapes. In this study, we focused on the identification of the archaeological ruins of Liangzhu City. An optical stereo model (red/blue stereo image) of the Liangzhu site was created through computer-based mosaicking and processing of CORONA remote-sensing stereo pairs taken in the 1960s and 1970s. By importing the optical stereo model into mobile phones, tablet computers, and other mobile devices, the research team undertook real-time locating of ruins via human observation, on-site investigation, and image overlay during a field survey and identified several Liangzhu-period dams, some of which have been confirmed via archaeological field investigations. The research team later applied the same method to the identification of tombs at the site of the mausoleums of the six emperors of the Southern Song dynasty. The results further prove that this method is feasible and reliable and can be widely promoted and used for the identification of archaeological ruins. [ABSTRACT FROM AUTHOR]

Recently, deep learning‐based stereo matching has achieved great success. However, models trained on the source domain dataset encounter substantial performance degradation when directly tested on an unseen target domain dataset because of neglecting the generalization to out‐of‐distribution (OOD) stereo samples. This paper proposes a stereo sample generation‐based domain generalization network (SGDG‐Net) for stereo matching. Specifically, to expand the distribution span of training samples, OOD stereo samples are generated to assist training. To effectively generate OOD left samples, a style transfer‐based generation mechanism is proposed to transmit perturbations to the source left samples. In addition, to generate the OOD right samples, a disparity‐assisted generation strategy is proposed by using disparity map labels as auxiliary information. Experimental results demonstrate that the proposed SGDG‐Net produces remarkable results on four benchmark datasets. [ABSTRACT FROM AUTHOR]

Abstract Generalizable and accurate stereo depth estimation is vital for 3D reconstruction, especially in surgery. Supervised learning methods obtain best performance however, limited ground truth data for surgical scenes limits generalizability. Self‐supervised methods don't need ground truth, but suffer from scale ambiguity and incorrect disparity prediction due to inconsistency of photometric loss. This work proposes a two‐phase training procedure that is generalizable and retains the high performance of supervised methods. It entails: (1) performing self‐supervised representation learning of left and right views via masked image modelling (MIM) to learn generalizable semantic stereo features (2) utilizing the MIM pre‐trained model to learn robust depth representation via supervised learning for disparity estimation on synthetic data only. To improve stereo representations learnt via MIM, perceptual loss terms are introduced, which improve the model's stereo representations learnt by explicitly encouraging the learning of higher scene‐level features. Qualitative and quantitative performance evaluation on surgical and natural scenes shows that the approach achieves sub‐millimetre accuracy and lowest errors respectively, setting a new state‐of‐the‐art. Despite not training on surgical nor natural scene data for disparity estimation.

Abstract Depth estimation from monocular panoramic image is a crucial step in 3D reconstruction, which is a close relationship with virtual reality and metaverse technologies. In recent years, some methods, such as HRDFuse, BiFuse++, and UniFuse, have employed a two‐branch neural network leveraging two common projections: equirectangular and cubemap projections (CMPs). The equirectangular projection (ERP) provides a complete field of view but introduces distortion, while the CMP avoids distortion but introduces discontinuity at the boundary of the cube. In order to address the issue of distortion and discontinuity, the authors propose an efficient depth estimation fusion module to balance the feature mapping of the two projections. Moreover, for the ERP, the authors propose a novel inflated network architecture to extend the receptive field and effectively harness visual information. Extensive experiments show that the authors’ method predicts more clear boundaries and accurate depth results while outperforming mainstream panoramic depth estimation algorithms.

In recent years, vision systems have become essential in the development of advanced driver assistance systems or autonomous vehicles. Although deep learning methods have been the center of focus in recent years to develop fast and reliable obstacle detection solutions, they face difficulties in complex and unknown environments where objects of varying types and shapes are present. In this study, a novel non-AI approach is presented for finding the ground-line and detecting the obstacles in roads using v-disparity data. The main motivation behind the study is that the ground-line estimation errors cause greater deviations at the output. Hence, a novel ground plane is defined as a region in the v-disparity map by using random variables to minimize these errors. In this new approach, weighted least squares regression, outlier detection, and camera height approximation were utilized for determining the ground region with higher accuracy. KITTY-2 dataset was chosen to conduct validation and evaluation experiments of the proposed approach. The experiment results were presented in GitHub, and the performance comparison shows that the proposed approach provides at least 20% improvement over Hough transform, which is a widely used non-AI algorithm. The results were also compared with a recently published article data and the best outcome was obtained among them for the recall metric. [ABSTRACT FROM AUTHOR]

Depth estimation from monocular panoramic image is a crucial step in 3D reconstruction, which is a close relationship with virtual reality and metaverse technologies. In recent years, some methods, such as HRDFuse, BiFuse++, and UniFuse, have employed a two‐branch neural network leveraging two common projections: equirectangular and cubemap projections (CMPs). The equirectangular projection (ERP) provides a complete field of view but introduces distortion, while the CMP avoids distortion but introduces discontinuity at the boundary of the cube. In order to address the issue of distortion and discontinuity, the authors propose an efficient depth estimation fusion module to balance the feature mapping of the two projections. Moreover, for the ERP, the authors propose a novel inflated network architecture to extend the receptive field and effectively harness visual information. Extensive experiments show that the authors' method predicts more clear boundaries and accurate depth results while outperforming mainstream panoramic depth estimation algorithms. [ABSTRACT FROM AUTHOR]

Recently, online optimisation‐based scene flow estimation has attracted significant attention due to its strong domain adaptivity. Although online optimisation‐based methods have made significant advances, the performance is far from satisfactory as only flow priors are considered, neglecting scene priors that are crucial for the representations of dynamic scenes. To address this problem, the authors introduce a dual‐branch MLP‐based architecture to encode implicit scene representations from a source 3D point cloud, which can additionally synthesise a target 3D point cloud. Thus, the mapping function between the source and synthesised target 3D point clouds is established as an extra implicit regulariser to capture scene priors. Moreover, their model infers both flow and scene priors in a stronger bidirectional manner. It can effectively establish spatiotemporal constraints among the synthesised, source, and target 3D point clouds. Experiments on four challenging datasets, including KITTI scene flow, FlyingThings3D, Argoverse, and nuScenes, show that our method can achieve potential and comparable results, proving its effectiveness and generality. [ABSTRACT FROM AUTHOR]

Abstract Depth map super resolution has been paid much attention in 3D applications due to the limitation of depth sensors. Few textures in objects with clear contours along them is the most important characteristic of depth map. An efficient image representation should be directional, multiscale and anisotropic. From this we propose a novel multiscale and multidirection depth map super resolution framework with semantic inference to improve the quality of depth maps. In this framework, a multiscale and multidirection depth map contour fusion scheme captures and assembles intrinsic geometrical structures through a multiview non‐subsampled contourlet transform manner. This scheme not only isolates the discontinuities of contours but retains the smoothness along the contours. The semantic inference is also utilized to segment and label the depth map into objects/backgrounds‐level which are coplanar. Furthermore, a semantic‐aware label refinement strategy is introduced to correct the rarely inaccurate labels of the label map for upscaling the target pixel with pixels in the same object or background. Experimental results on benchmark depth map dataset demonstrate that the proposed multiscale and multidirection depth map super resolution framework with semantic inference has a significant improvement than the state‐of‐the‐art algorithms both visually and quantitatively.

Abstract In the current stereo matching field, the accuracy of the derived disparity map is highly dependent on the processing capability of ill regions. Fortunately, we find that the use of local information will eliminate the negative effects associated with ill regions. As a result of the above discovery, we propose the Concatenated Dilated Convolution (CDC) block and the Multi‐scale Feature Fusion module (MFF), which are capable of effectively extracting regional context by increasing the receptive field in parallel and channel‐wise ways. The CDC block can expand the receptive field by applying multiple dilated convolutions at different dilation rates in parallel to enhance the smoothness of the feature map. By constructing parallel CDC blocks in a multiple dilated manner, the MFF module can improve the smoothness of the feature map. In addition, to control the number of parameters in the MFF network, a high‐performance channel‐distribution algorithm is proposed, capable of adjusting the weights of each module and convolution in an adaptive manner while reducing the number of parameters. Extensive experiments have demonstrated that MFF and CDC can effectively improve the performance of ill areas and networks with a minimal number of parameters.

Abstract Cost aggregation is crucial to the accuracy of stereo matching. A reasonable cost aggregation algorithm should aggregate costs within homogeneous regions where pixels have the same or similar disparities. Otherwise, the estimated disparity map is prone to the well‐known edge‐fattening issue and the problem of losing fine structures. However, current state‐of‐the‐art convolutional neural networks (CNNs) mainly do cost aggregation with square‐kernel convolutional layers that learn to adjust their kernel elements to make the actual receptive fields of the aggregated costs adapt to homogeneous regions with various shapes. This is a mechanism that easily leads to the above issues due to the translation equivalence and content‐insensitivity properties of CNNs. To tackle these problems, a novel densely connected asymmetric convolution block (Dense‐ACB) based on asymmetric convolutions is proposed to explicitly construct receptive fields with various shapes, which effectively alleviates the issues caused by mismatching shapes of receptive fields and homogeneous regions. The proposed Dense‐ACB brings new insight to CNN‐based stereo matching networks. Based on the proposed cost aggregation method, an efficient and effective stereo matching network is built, which not only achieves competitive overall accuracy compared with state‐of‐the‐art models but also effectively alleviates the edge‐fattening problem and preserves fine structures.

Depth map super resolution has been paid much attention in 3D applications due to the limitation of depth sensors. Few textures in objects with clear contours along them is the most important characteristic of depth map. An efficient image representation should be directional, multiscale and anisotropic. From this we propose a novel multiscale and multidirection depth map super resolution framework with semantic inference to improve the quality of depth maps. In this framework, a multiscale and multidirection depth map contour fusion scheme captures and assembles intrinsic geometrical structures through a multiview non‐subsampled contourlet transform manner. This scheme not only isolates the discontinuities of contours but retains the smoothness along the contours. The semantic inference is also utilized to segment and label the depth map into objects/backgrounds‐level which are coplanar. Furthermore, a semantic‐aware label refinement strategy is introduced to correct the rarely inaccurate labels of the label map for upscaling the target pixel with pixels in the same object or background. Experimental results on benchmark depth map dataset demonstrate that the proposed multiscale and multidirection depth map super resolution framework with semantic inference has a significant improvement than the state‐of‐the‐art algorithms both visually and quantitatively. [ABSTRACT FROM AUTHOR]

This paper introduces a systematic approach to pothole detection, emphasizing its significance in road maintenance and safety. The proposed system is comprised of meticulously designed modules that collectively contribute to achieving accurate and effective results. Commencing with data collection through cameras capturing road areas of interest, the images undergo manual cropping and resizing for standardization. The core principle of the system revolves around image enhancement, involving grayscale conversion, application of blurring techniques, and adjustments to brightness and contrast. Automatic thresholding extracts essential information encoded within pixels, paving the way for precise edge detection refined through morphological operations. The focus then shifts to pothole detection, incorporating a stereo camera setup to calculate depth and disparity of road surfaces. Critical steps, including image rectification, correspondence matching, and disparity calculations, contribute to the accurate identification and delineation of potholes using depth thresholds. Subsequent modules employ K-Means clustering to segregate regions of interest from the image background, and post-processing steps, including filtering, masking, and refinement, fine-tune the images. Utilizing the HSV color space for grayscale image refinement and the connected component method to isolate white pixel objects further enhance the system's capability. The final step involves the addition of bounding boxes around identified potholes, streamlining their identification process. This comprehensive methodology ensures effective image processing, precise pothole detection, and contributes significantly to road safety and maintenance efforts. [ABSTRACT FROM AUTHOR]

This paper constructs an energy model based on local features used in stereo matching. The local features include the similarity between different image areas, the matching cost function pattern, the connection between neighbor pixels, and the occlusion geometric relationship. Based on these features, we define the weight of each data term and smoothing term in the energy function and then design an algorithm to solve the energy model and get disparity results. The significant improvements of this paper include as following. 1) We modify the structure of the energy function. First, we define the weight of the data term based on the reliability of its corresponding disparity result, which is obtained by cost function features and the occlusion geometric relationship. Then we define the weight of the smoothing term by analyzing the characteristic relation between neighbor super-pixels. We can also reduce the computational complexity by detecting and reducing some low-strength connections. 2) We proposed an algorithm based on pairwise Markov random field (MRF) (Taniai et al., IEEE Trans Pattern Anal Machine Intell 40(11): 2725–2739, 2017) and local greedy iteratively, which can be used to solve the energy model. 3) In post-optimation, we select some areas with severe occlusion and fewer matching clues for post-interpolation fitting to optimize the results. The experiment shows that the proposed method reduced the average percentage of bad pixels (in bad 3) to 6.06 on the Middlebury dataset and 1.42 on the KITTI dataset. Finally, we compare our results with those of MC-Cnn (Zbontar and LeCun 2015), CF-Net (Shen et al., 2021), Guided-Stereo (Poggi et al., 2019), Gwc-Net (Guo et al., 2019) and Patchmatch-Net(PM-Net) (Wang et al., 2021) to verify the improved speed and accuracy of our algorithm, especially at recognizing the depth of changing edges and small objects. This paper's relevant research can contribute to practical engineering practices such as assisted vision, intelligent driving, and robot grasping control. [ABSTRACT FROM AUTHOR]

In the current stereo matching field, the accuracy of the derived disparity map is highly dependent on the processing capability of ill regions. Fortunately, we find that the use of local information will eliminate the negative effects associated with ill regions. As a result of the above discovery, we propose the Concatenated Dilated Convolution (CDC) block and the Multi‐scale Feature Fusion module (MFF), which are capable of effectively extracting regional context by increasing the receptive field in parallel and channel‐wise ways. The CDC block can expand the receptive field by applying multiple dilated convolutions at different dilation rates in parallel to enhance the smoothness of the feature map. By constructing parallel CDC blocks in a multiple dilated manner, the MFF module can improve the smoothness of the feature map. In addition, to control the number of parameters in the MFF network, a high‐performance channel‐distribution algorithm is proposed, capable of adjusting the weights of each module and convolution in an adaptive manner while reducing the number of parameters. Extensive experiments have demonstrated that MFF and CDC can effectively improve the performance of ill areas and networks with a minimal number of parameters. [ABSTRACT FROM AUTHOR]

An accurate measurement method of crack fabric is proposed. Considering the density, geometry and spatial distribution in cracks, a novel method suggests the formula of high-order fabric tensor in the orthogonal space to describe cracks quantitatively. The crack is automatically identified by morphology and determined by stereo scans of image processing techniques. The fabric tensor is defined based on the idea of probability and statistics. The zero-order tensor describes the average effect of the density of rectangular cracks, the second-order tensor describes the two dominant directions of spatial cracks, and the higher-order tensor more accurately describes the anisotropy of cracks. With the increase of the tensor order, the more accurate direction and degree of the anisotropy can be described, and the morphological automatic identification of crack and its physical meaning of quantitative determination are clear. The plane description of the crack can be described by the invariants of the plane tensor. One invariant describes the degree of the anisotropy, and the other invariant describes its direction. The scan line is rotated and scanned on the rock plane image to obtain the accurate distribution of plane cracks, and the distribution law can be accurately described by the proposed high-order fabric tensor. In this paper, the rationality and effectiveness of the proposed measurement method are well verified by CT image analyses. Highlights: An accurate measurement method of crack fabric is proposed. The expression of crack fabric tensor is redefined by the normalized idea. The plane crack fabric is automatically identified by morphology, determined by stereo scans, and described by the its invariants equivalently. On the basis of stereology, the rationality and effectiveness of the proposed measurement method are well verified by SEM and CT image analyses. [ABSTRACT FROM AUTHOR]

Objective: Determination of species from animal hair is an effective method in veterinary forensic investigations, research, endangered species and prevention of poaching. Since the hairs are resistant to deterioration, they can be stored as evidence for many years. In addition, pig hairs are often used in making brushes. When these brushes are used in the food industry, it raises questions about halal food. This study aimed to identify these hairs by examining the hair structure of wild pigs living in the Balıkesir region and revealing their characteristics. Materials and Methods: The hairs of 3 wild boars obtained from the İvrindi (Balıkesir) region were used. After the hairs taken from different parts of the pigs were cleaned, stereomicroscopy and macroscopic examination were performed and routine procedures were applied for scanning electron microscopic imaging. Result: In stereomicroscopy and macroscopic examination, it was determined that the length and thickness of the hairs in different regions varied significantly. In the study, the hairs were generally bifurcated from the upper 1/3 part. In the scanning electron microscopic images, the hardened cuticle patterns on the hair shaft, which have a scaly appearance, were detected, and their measurements were made. Scanning electron microscopic images determined that there were very small bifurcations from the hair shaft. However, it was thought that these hairs could not be used for species separation, since these parts would break off in the hairs used as brushes. Significant images could not be obtained in cross-sections. Conclusion: It is thought that it will be used as a source for the identification of the hairs of wild boars in the Balıkesir region. [ABSTRACT FROM AUTHOR]

Cost aggregation is crucial to the accuracy of stereo matching. A reasonable cost aggregation algorithm should aggregate costs within homogeneous regions where pixels have the same or similar disparities. Otherwise, the estimated disparity map is prone to the well‐known edge‐fattening issue and the problem of losing fine structures. However, current state‐of‐the‐art convolutional neural networks (CNNs) mainly do cost aggregation with square‐kernel convolutional layers that learn to adjust their kernel elements to make the actual receptive fields of the aggregated costs adapt to homogeneous regions with various shapes. This is a mechanism that easily leads to the above issues due to the translation equivalence and content‐insensitivity properties of CNNs. To tackle these problems, a novel densely connected asymmetric convolution block (Dense‐ACB) based on asymmetric convolutions is proposed to explicitly construct receptive fields with various shapes, which effectively alleviates the issues caused by mismatching shapes of receptive fields and homogeneous regions. The proposed Dense‐ACB brings new insight to CNN‐based stereo matching networks. Based on the proposed cost aggregation method, an efficient and effective stereo matching network is built, which not only achieves competitive overall accuracy compared with state‐of‐the‐art models but also effectively alleviates the edge‐fattening problem and preserves fine structures. [ABSTRACT FROM AUTHOR]

In recent years, convolutional‐neural‐network based stereo matching methods have achieved significant gains compared to conventional methods in terms of both speed and accuracy. Current state‐of‐the‐art disparity estimation algorithms require many parameters and large amounts of computational resources and are not suited for applications on edge devices. In this paper, an end‐to‐end light‐weight network (LWNet) for fast stereo matching is proposed, which consists of an efficient backbone with multi‐scale feature fusion for feature extraction, a 3D U‐Net aggregation architecture for disparity computation, and color guidance in a 2D convolutional neural network (CNN) for disparity refinement. MobileNetV2 is adopted as an efficient backbone in feature extraction. The channel attention module is applied to improve the representational capacity of features and multi‐resolution information is adaptively incorporated into the cost volume via cross‐scale connections. Further, a left‐right consistency check and color guidance refinement are introduced and a robust disparity refinement network is designed with skip connections and dilated convolutions to capture global context information and improve disparity estimation accuracy with little computational cost and memory space. Extensive experiments on Scene Flow, KITTI 2015, and KITTI 2012 demonstrate that the proposed LWNet achieves competitive accuracy and speed when compared with state‐of‐the‐art stereo matching methods. [ABSTRACT FROM AUTHOR]

China's first optical stereo mapping satellite with a sub-meter resolution, GaoFen-7 (GF-7), launched in November 2019, shows significant potential for providing high-resolution topographic and geomorphic data for quantitative research on active tectonics. However, no studies have evaluated the capability of the GF-7-generated digital elevation model (DEM) for quantitatively studying active tectonics. This study aimed to validate the accuracy of the DEMs extracted from GF-7 stereo imagery, with or without ground control points (GCPs), and evaluated the potential of applying GF-7 DEMs to active tectonics. First, GF-7 stereo images were processed to obtain DEMs with a spatial resolution of 2 m, utilizing three different methods, including block adjustment without GCPs, block adjustment with the aid of Google Earth images and SRTM DEM, and block adjustment with GCPs derived from the Ice, Cloud, and land Elevation Satellite-2 (ICESat-2) data. These three generated DEMs were called GF-7 DEMMethod1, GF-7 DEMMethod2, and GF-7 DEMMethod3, respectively, and were verified by the airborne LiDAR data in the Hasishan section of the Haiyuan fault. Second, the capability of the GF-7 DEMs for identifying active faults, fault scarps, and horizontal offsets was evaluated. Finally, 8 vertical and 13 horizontal offsets were measured based on three different GF-7 DEMs, and airborne LiDAR data were used to verify the measurements' accuracies. The results indicated that the accuracy of GF-7 DEMMethod1 was the worst and that of GF-7 DEMMethod3 was superior to that of GF-7 DEMMethod2. The GF-7 DEMs could effectively identify the apparent fault scarps and horizontal offsets. The RMSE values of the vertical offsets measured based on GF-7 DEMMethod1, GF-7 DEMMethod2, and GF-7 DEMMethod3 were 0.55 m, 0.55 m, and 0.41 m, respectively. The horizontal offsets yielded RMSE values of 3.98 m, 2.52 m, and 1.37 m, respectively. These findings demonstrated that vertical and horizontal offsets could be accurately measured using the DEMs generated from GF-7 stereo images. Meanwhile, our study indicated that the GCPs derived from ICESat-2 data could be utilized to improve the accuracies of the GF-7 DEM, and the measurements of vertical and horizontal offsets. [ABSTRACT FROM AUTHOR]

This paper presents a method on how to compute the position of a robotic arm in the 2D and 3D spaces. This method is slightly different from the well-known methods, such as forward or inverse kinematics. The method presented in this paper is an optical method, which uses two video cameras in stereo vision configuration to locate and compute the next move of a robotic arm in space. The method recognizes the coordinates of the markers placed at the joints of the robotic arm using the two video cameras. The coordinate points of these markers are connected with straight lines. Around certain points, circles are drawn. From the tangent to the circles, a non-Cartesian (orthogonal) coordinate system is drawn, which is enough to compute the target position of the robotic arm. All of these drawings are overlaid on the live video feed. This paper also presents another method for calculating the stereo distance using the triangulation method. An alternative method is also presented when a non-Cartesian (orthogonal) 3D coordinate system is created, which is used to compute the target position of the robotic arm in the 3D space. Because the system is in a loop, it can make micro-adjustments of the robotic arm, in order to be exactly in the desired position. In this way, there is no need to make calibrations for the robotic arm. In an industrial system, there is no need to stop the production line, which can be a really big cost saver. [ABSTRACT FROM AUTHOR]

Abstract Stereo matching algorithms have been developed for many years but basically focus only on the implementation of existing datasets and are rarely applied to real scenarios, such as industrial robot scenarios. Traditional stereo matching algorithms have a high error rate, and deep learning algorithms are difficult to obtain good results in real scenarios because of their weak generalisation ability and difficult access to training data. In order to use stereo matching algorithms for industrial robot guidance, it is better to design a new traditional algorithm with low time complexity for the characteristics of industrial robot scenarios dominated by planar facets. This paper proposes a new matching method based on subrows of pixels, instead of individual pixels, in order to improve robustness of matching and reduce running time. First, the pixel strings from the same row of the left and right images are divided into several colour‐identical or colour‐gradient segments. Then, the colour and length of the two left and right pixel segment are used as clues to determine a matching relation and obtain the matching type. Then, all match types can be determined according to non‐crossing mapping. Each match type can reason backward to the corresponding spatial state of the stimulus source so that the disparity of pixels in pixel segments representing the spatial state can be calculated. This new matching method makes full use of the stimulus homology constraints and projective geometric constraints of row‐aligned images. The method can obtain good results in industrial robot scenarios and be applied for industrial robot guidance.

Non-contact vibration measurements are relevant for non-invasively characterizing the mechanical behavior of structures. This paper presents a novel methodology for full-field vibrational analysis at high frequencies using the three-dimensional digital image correlation technique combined with the projection of a speckle pattern. The method includes stereo calibration and image processing routines for accurate three-dimensional data acquisition. Quantitative analysis allows the extraction of several deformation parameters, such as the cross-correlation coefficients, shape and intensity, as well as the out-of-plane displacement fields and mode shapes. The potential of the methodology is demonstrated on an Unmanned Aerial Vehicle wing made of composite material, followed by experimental validation with reference accelerometers. The results obtained with the projected three-dimensional digital image correlation show a percentage of error below 5% compared with the measures of accelerometers, achieving, therefore, high sensitivity to detect the dynamic modes in structures made of composite material. [ABSTRACT FROM AUTHOR]

Abstract Stereo image pairs can effectively enhance the performance of super‐resolution (SR) since both intra‐view and cross‐view information can be used. However, exploiting cross‐view information accurately is extremely challenging. Most recent methods use the attention mechanism to get stereo correspondence. But these methods ignore the high‐frequency information since they only utilise first‐order statistics, which leads to reducing the discriminative ability of the network. To address this issue, in this work, a parallax‐based second‐order mixed attention stereo SR network (PSMASSRnet) is proposed to integrate the cross‐view information from a stereo image pair for SR. Specifically, a novel parallax‐based second‐order mixed attention module (PSMAM) is developed to combine second‐order channel features and spatial features to obtain more discriminative representations. Furthermore, a dense cross‐atrous spatial pyramid pooling (ASPP) module is also presented, which can effectively explore the local and the multi‐scale features with different dilation rates to extract more discriminative features with fewer parameters and less execution. The extensive experiments on the KITTI2012, KITTI2015 and Middlebury datasets have demonstrated the superiority of the proposed PSMASSRnet over the state‐of‐the‐art methods in the aspects of both the quantitative metrics and the visual quality.

Abstract Despite recent advances in 3‐D pose estimation of human hands, thanks to the advent of convolutional neural networks (CNNs) and depth cameras, this task is still far from being solved in uncontrolled setups. This is mainly due to the highly non‐linear dynamics of fingers and self‐occlusions, which make hand model training a challenging task. In this study, a novel hierarchical tree‐like structured CNN is exploited, in which branches are trained to become specialised in predefined subsets of hand joints called local poses. Further, local pose features, extracted from hierarchical CNN branches, are fused to learn higher order dependencies among joints in the final pose by end‐to‐end training. Lastly, the loss function used is also defined to incorporate appearance and physical constraints about doable hand motions and deformations. Finally, a non‐rigid data augmentation approach is introduced to increase the amount of training depth data. Experimental results suggest that feeding a tree‐shaped CNN, specialised in local poses, into a fusion network for modelling joints' correlations and dependencies, helps to increase the precision of final estimations, showing competitive results on NYU, MSRA, Hands17 and SyntheticHand datasets.

This paper introduces PlaneLoc2 - a novel indoor global localization system designed to harness the potential of stereo cameras. A need for robust global localization that does not produce incorrect results (false positives) is present in almost every life-long autonomy task. We show that planar segments extracted from stereo vision data by a neural network enable such robust localization. Planar segments are easier to discriminate than keypoint features and provide easy-to-use geometric constraints. We propose an architecture that exploits a single deep neural network (DNN) to detect planar segments, produce appearance descriptors, and estimate segment geometry. Moreover, we introduce a novel view-based segment map and a novel pose retrieval procedure that considers the uncertainty of features to efficiently use the geometric constraints provided by them. We also show that the new learned descriptor provides better discrimination than the hand-crafted one. Finally, we present experimental results that show that our solution outperforms other state-of-the-art global localization methods and does not produce incorrect agent poses. For both test scenes it recognizes at least 15% more poses than the second best method without incorrect recognitions.

[Display omitted] Discrete orthogonal moments (DOMs) are widely applied in the field of image processing. However, most existing DOMs are used for flat image analysis and are less suitable for color stereo image processing. In light of this limitation, this paper firstly presents a moments type called octonion Hahn moments (OHMs) for the compact analysis of color stereo images, and also for preserving the internal relations between all the color channels of the stereo image. OHMs are introduced in both Cartesian and polar coordinates. Secondly, we present a modified version of 2D Henon map that exhibits improved chaotic behavior over its original version. This version is introduced for potential deployment in the field of information security. Thirdly, we present an efficient color stereo image encryption scheme in the transform domain using the modified Henon's map and OHMs that are defined in Cartesian space. This scheme guarantees a perfect reconstruction of the stereo image. Finally, we present a local zero-watermarking scheme for color stereo image copyrighting based on the proposed version of Henon's map and OHMs that are defined in the polar coordinate space. This scheme is originally designed to be robust against image rotation attack. Compared to excellent existing encryption schemes, our scheme shows superior performance in terms of high security level and good robustness to various attacks (classical, statistical, noise addition, cropping, etc.). Moreover, the proposed zero-watermarking method shows stronger resistance to different types of attacks (geometric, common image processing, noise, desynchronization, etc.) in comparison to recent zero-watermarking methods. [ABSTRACT FROM AUTHOR]

Electronic holography is an ideal 3-D display technique to ameliorate the vergence-accommodation conflict, which is a possible cause of visual fatigue and discomfort from viewing conventional stereoscopic 3-D displays. Previous studies have measured accommodative and vergence responses to holographic images and to real objects, and the results showed that these responses are in good agreement. To demonstrate the effectiveness of electronic holography as a possible solution to the vergence-accommodation conflict caused by viewing conventional stereoscopic 3-D displays, we measured both accommodative and vergence responses to reconstructed images of holograms and super multiview holographic stereograms. We also measured responses to real objects located at various distances and conventional two-view stereo images for comparison. The results indicate that the accommodative responses to the electro-holographic display and super multiview holographic stereogram change with the vergence response in a similar manner, as is the case with real objects as compared to conventional two-view stereo display. This suggests that the electronic holographic display and super multiview holographic stereograms are promising candidates for resolving the vergence-accommodation conflict. [ABSTRACT FROM AUTHOR]

Virtual Reality (VR) is becoming ubiquitous with the rise of consumer displays and commercial VR platforms. Such displays require low latency and high quality rendering of synthetic imagery with reduced compute overheads. Recent advances in neural rendering showed promise of unlocking new possibilities in 3D computer graphics via image-based representations of virtual or physical environments. Specifically, the neural radiance fields (NeRF) demonstrated that photo-realistic quality and continuous view changes of 3D scenes can be achieved without loss of view-dependent effects. While NeRF can significantly benefit rendering for VR applications, it faces unique challenges posed by high field-of-view, high resolution, and stereoscopic/egocentric viewing, typically causing low quality and high latency of the rendered images. In VR, this not only harms the interaction experience but may also cause sickness. To tackle these problems toward six-degrees-of-freedom, egocentric, and stereo NeRF in VR, we present the first gaze-contingent 3D neural representation and view synthesis method. We incorporate the human psychophysics of visual- and stereo-acuity into an egocentric neural representation of 3D scenery. We then jointly optimize the latency/performance and visual quality while mutually bridging human perception and neural scene synthesis to achieve perceptually high-quality immersive interaction. We conducted both objective analysis and subjective studies to evaluate the effectiveness of our approach. We find that our method significantly reduces latency (up to 99% time reduction compared with NeRF) without loss of high-fidelity rendering (perceptually identical to full-resolution ground truth). The presented approach may serve as the first step toward future VR/AR systems that capture, teleport, and visualize remote environments in real-time. [ABSTRACT FROM AUTHOR]

Modeling 3D visual saliency has received great attention due to the development of emerging 3D display technologies. Traditional methods relying on low-level features may not be efficient in interpreting 3D visual content from high-level semantic perspective. Despite numerous efforts dedicated to this area, existing 3D visual saliency detection methods do not necessarily excel in exploring the stereoscopic image saliency driven by the intra-view and inter-view dependencies among left and right views. In this paper, we propose a visual saliency detection method for stereoscopic images grounded on adaptive viewpoint feature enhancement via binocular vision. More specifically, the correlation among left and right views is investigated through a delicately designed binocular stereoscopic saliency feature aggregation module, enabling the generation of more representative saliency features towards binocular vision. Subsequently, to further aggregate the saliency features in multiple scales, we design a progressive attention-based saliency feature pyramid extraction module to effectively integrate the features from top-level to down-level based on the network hierarchy mechanism. The saliency maps are ultimately produced for stereoscopic images by evaluating the obtained saliency features. In addition, we create a stereoscopic image saliency dataset (SIS-3D) that includes 1086 stereoscopic image pairs with various content and their corresponding human eye fixation annotations, aiming to further facilitate the research on visual saliency detection for stereoscopic images. Extensive experiments demonstrate that our proposed method improves CC by an average of 4.02% compared to representative counterparts on the newly built saliency dataset and another publicly available dataset. [ABSTRACT FROM AUTHOR]

This paper is devoted to the important problem of subsea infrastructure inspection using autonomous underwater vehicles (AUVs). The accomplishment of the inspection mission requires high accuracy of AUV coordination with respect to objects, which is not always provided by standard hydroacoustic equipment. That is why we propose an approach based on stereo image processing that can provide submeter accuracy of AUV coordination required under subsea conditions. In the framework of a unified approach based on the use of an a priori formed point model of an object and the application of a structural coherence criterion, we propose and analyze several methods for recognition of underwater objects and coordinate referencing of the AUV to these objects. As a geometric model of an object, we consider its representation based on characteristic points, which determines the spatial structure of the object. In addition to the standard method for object model construction from preliminary measurements, a new method is proposed, which is based on automated technique of model generation from images offline. The proposed object recognition methods use the structural coherence criterion to match 3D point clouds with object models. The 3D point clouds are formed from stereo images captured by a camera using visual navigation. Feature points are generated and matched using the SURF detector and Harris corner detector. To improve the reliability of object identification, we propose joint processing of source images with their vectorized forms. The use of line segments obtained by vectorization significantly increases the number of identified characteristic points when matching a 3D point cloud with a model. Based on the identified points of an object, the matrix of AUV coordinate referencing to this object is computed. The efficiency of the considered methods is estimated and compared. In the computational experiments with model scenes, a virtual geometric model of a distributed subsea production system is used. The experiments on real-world data are carried out under laboratory conditions with the use of a Karmin2 stereo camera (Nerian's 3D stereo camera, baseline 25 cm). [ABSTRACT FROM AUTHOR]

Abstract Image matching is a well‐studied problem in computer vision. Conventional image matching is solved using image feature matching algorithms, and later deep learning techniques are also applied to tackle the problem. Here, a slope‐modelling framework is proposed by adopting the image matching techniques. First, image pairs of a slope scene are captured and camera calibration as well as image rectification are performed. Then, PatchMatch Filter (PMF‐S) and PWC‐Net techniques are adapted to solve the matching of image pairs. In the proposed PatchMatch Filter‐Census (PMF‐Census), slanted‐plane modelling, image census transform and gradient difference are employed in matching cost formulation. Later, nine matching points are manually selected from an image pair. Matching point pairs are further used in fitting a transformation matrix to relate the matching between the image pair. Then, the transformation matrix is applied to obtain a ground truth matching image for algorithm evaluation. The challenges in this matching problem are that the slope is of a homogenous region and it has a slanted‐surface geometric structure. In this work, it is found out that the error rate of the proposed PMF‐Census is significantly lower as compared with the PWC‐Net method and is more suitable in this slope‐modelling task. In addition, to show the robustness of the proposed PMF‐Census against the original PMF‐S, further experiments on some image pairs from Middlebury Stereo 2006 dataset are conducted. It is demonstrated that the error percentage by the proposed PMF‐Census is reduced significantly especially in the low‐texture and photometric distorted region, in comparison to the original PMF‐S algorithm. This further verifies the suitability of the PMF‐Census in modelling the outdoor low‐texture slope scene.

Existing stereo matching methods achieve satisfactory average accuracy on a whole predicted disparity map under common global metrics, but ignore the fine-grained performance at region level, especially for far regions in the scene, which is more crucial in actual auto-driving scenarios. There are two factors accounting for this problem: 1) Depth resolution. Existing methods use disparity-based sampling to extract matching candidates uniformly according to the disparity range, but leads to sparser sampling density at far regions than that of close regions in terms of depth range, resulting in low depth resolution in far regions. 2) Feature discriminability. Limited image resolution and inferior feature extraction at far regions result in the obtained features with low discrimination, which influences the subsequent matching process between stereo images. To improve the estimation accuracy of far regions and thus achieve a balanced performance at region level, we design a novel two-stage Balanced Stereo Matching Network (BSMNet) to address the above problems. The coarse stage of BSMNet introduces a direct depth-based sampling strategy, which generates matching candidates according to scene depth instead of disparity, thus improving the depth resolution and obtaining initial depth map with more balanced accuracy. Then, a depth refinement stage is proposed to solve the problem of low feature discriminability and further optimizes the initial depth map obtained from the coarse stage. It selects matching candidates and computes their similarity scores from a carefully designed adaptive feature volume guided by a learnable scale map, thus making the final estimation more accurate. Different from the existing methods that construct stereo matching based on disparity prediction, our proposed pipeline is to directly optimize on depth information. Experiments show that our BSMNet can obtain an obvious performance improvement at far regions without discarding that at close regions, so as to largely outperform existing state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In contemporary society full of stereoscopic images, how to assess visual quality of 3D images has attracted an increasing attention in field of Stereoscopic Image Quality Assessment (SIQA). Compared with 2D-IQA, SIQA is more challenging because some complicated features of Human Visual System (HVS), such as binocular interaction and binocular fusion, must be considered. In this paper, considering both binocular interaction and fusion mechanisms of the HVS, a hierarchical no-reference stereoscopic image quality assessment network (StereoIF-Net) is proposed to simulate the whole quality perception of 3D visual signals in human cortex, including two key modules: BIM and BFM. In particular, Binocular Interaction Modules (BIMs) are constructed to simulate binocular interaction in V2-V5 visual cortex regions, in which a novel cross convolution is designed to explore the interaction details in each region. In the BIMs, different output channel numbers are designed to imitate various receptive fields in V2-V5. Furthermore, a Binocular Fusion Module (BFM) with automatic learned weights is proposed to model binocular fusion of the HVS in higher cortex layers. The verification experiments are conducted on the LIVE 3D, IVC and Waterloo-IVC SIQA databases and three indices including PLCC, SROCC and RMSE are employed to evaluate the assessment consistency between StereoIF-Net and the HVS. The proposed StereoIF-Net achieves almost the best results compared with advanced SIQA methods. Specifically, the metric values on LIVE 3D, IVC and WIVC-I are the best, and are the second-best on the WIVC-II. [ABSTRACT FROM AUTHOR]

Stereoscopic visual fatigue (SVF) due to prolonged immersion in the virtual environment can lead to negative user experience, thus hindering the development of virtual reality (VR) industry. Previous studies have focused on investigating the evaluation indicators associated with SVF, while few studies have been conducted to reveal the underlying neural mechanism, especially in VR applications. In this paper, a modified Go/NoGo paradigm was adopted to induce SVF in VR environment with Go trials for maintaining participants’ attention and NoGo trials for investigating the neural effects under SVF. Random dot stereograms (RDSs) with 11 disparities were presented to evoke the depth-related visual evoked potentials (DVEPs) during 64-channel EEG recordings. EEG datasets collected from 15 participants in NoGo trials were selected to conduct individual processing and group analysis, in which the characteristics of the DVEPs components for various fatigue degrees were compared and independent components were clustered to explore the original cortex areas related to SVF. Point-by-point permutation statistics revealed that DVEPs sample points from 230 ms to 280 ms (component P2) in most brain areas changed significantly when SVF increased. Additionally, independent component analysis (ICA) identified that component P2 which originated from posterior cingulate cortex and precuneus, was associated statistically with SVF. We believe that SVF is rather a conscious status concerning the changes of self-awareness or self-location awareness than the performance reduction of retinal image processing. Moreover, we suggest that indicators representing higher conscious state may be a better indicator for SVF evaluation in VR environments. [ABSTRACT FROM AUTHOR]

Stereoscopic omnidirectional content, as a novel visual media, has drawn wide attention in recent years due to its ability in providing strong immersive experience. Since Stereoscopic Omnidirectional Images (SOIs) involve the properties from panoramic and stereoscopic visual perception, it is very challenging to establish an efficient and effective visual quality evaluation model for SOIs. To better measure the user’s experience in virtual reality, we put forward a novel deep learning framework to assess the quality of SOIs in this paper. Firstly, the deformable convolutions instead of standard convolutions are adopted to ensure the invariant receptive fields of convolutional kernels on Equi-Rectangular Projection (ERP). Secondly, according to the stereoscopic property, we use binocular-difference information and a coarse-to-fine mechanism to construct the binocular feature extraction network. Thirdly, a three-channel network involving left-view, right-view and binocular-difference channels is presented to simulate the process of monocular and binocular interactions, in which independent quality labels are provided for each channel to reflect the individual effect of monocular and binocular visions on the whole visual quality. Finally, experimental results on two available benchmark databases demonstrate the superiority of the proposed metric over the state-of-the-art blind quality assessment models in predicting the quality of SOIs. Moreover, our model is efficient in computational cost as the feature extraction is directly applied on ERP images. [ABSTRACT FROM AUTHOR]

Stereo matching algorithms have been developed for many years but basically focus only on the implementation of existing datasets and are rarely applied to real scenarios, such as industrial robot scenarios. Traditional stereo matching algorithms have a high error rate, and deep learning algorithms are difficult to obtain good results in real scenarios because of their weak generalisation ability and difficult access to training data. In order to use stereo matching algorithms for industrial robot guidance, it is better to design a new traditional algorithm with low time complexity for the characteristics of industrial robot scenarios dominated by planar facets. This paper proposes a new matching method based on subrows of pixels, instead of individual pixels, in order to improve robustness of matching and reduce running time. First, the pixel strings from the same row of the left and right images are divided into several colour‐identical or colour‐gradient segments. Then, the colour and length of the two left and right pixel segment are used as clues to determine a matching relation and obtain the matching type. Then, all match types can be determined according to non‐crossing mapping. Each match type can reason backward to the corresponding spatial state of the stimulus source so that the disparity of pixels in pixel segments representing the spatial state can be calculated. This new matching method makes full use of the stimulus homology constraints and projective geometric constraints of row‐aligned images. The method can obtain good results in industrial robot scenarios and be applied for industrial robot guidance. [ABSTRACT FROM AUTHOR]

Abstract Depth is essential information for autonomous robotics applications that need environmental depth values. The depth could be acquired by finding the matching pixels between stereo image pairs. Depth information is an inference from a matching cost volume that is composed of the distances between the possible pixel points on the pre-aligned horizontal axis of stereo images. Most approaches use matching costs to identify matches between stereo images and obtain depth information. Recently, researchers have been using convolutional neural network-based solutions to handle this matching problem. In this paper, a novel method has been proposed for the refinement of matching costs by using recurrent neural networks. Our motivation is to enhance the depth values obtained from matching costs. For this purpose, to attain an enhanced disparity map by utilizing the sequential information of matching costs in the horizontal space, recurrent neural networks are used. Exploiting this sequential information, we aimed to determine the position of the correct matching point by using recurrent neural networks, as in the case of speech processing problems. We used existing stereo algorithms to obtain the initial matching costs and then improved the results by utilizing recurrent neural networks. The results are evaluated on the KITTI 2012 and KITTI 2015 datasets. The results show that the matching cost three-pixel error is decreased by an average of 14.5% in both datasets.

The visual depth perception is composed of monocular and binocular depth cues. Studies show that in absence of binocular depth cues the performance of visuomotor tasks like pointing to or grasping objects is limited. Thus, binocular depth cues are of great importance for motor control required in everyday life. However, binocular depth cues like retinal disparity (basis for stereopsis) might be influenced due to developmental disorders of the visual system. For example, amblyopia in which one eye's visual input is not processed leads to loss of stereopsis. The primary amblyopia treatment is occlusion of the healthy eye to force the amblyopic eye to train. However, improvements in stereopsis are poor. Therefore, binocular treatments arose that equilibrate both eyes' visual input to enable binocular vision. However, most approaches rely on divided stimuli which do not account for loss of stereopsis. We created a Virtual Reality (VR) with reduced monocular depth cues in which a stereoscopic task is shown to both eyes simultaneously, consisting of two balls jumping towards the user. One ball appears closer to the user which must be identified. To evaluate the task performance the reaction time is measured. We validated our approach with 18 participants with stereopsis under three contrast settings including one leading to monocular vision. The number of correct responses reduces from 90% under binocular vision to 52% under monocular vision corresponding to random guessing. Our results indicate that it is possible to disable monocular depth cues and create a dynamic stereoscopic task inside a VR. [ABSTRACT FROM AUTHOR]

Some research progress has been made on fractional-order continuous orthogonal moments (FrCOMs) in the past two years. Compared with integer-order continuous orthogonal moments (InCOMs), FrCOMs increase the number of affine invariants and effectively improve numerical stability. However, the existing types of FrCOMs are still very limited, of which all are planar image oriented. No report on stereoscopic images is available yet. To this end, in this paper, FrCOMs corresponding to various types of InCOMs are first deduced, and then, they are combined with trinion theory to construct trinion FrCOMs (TFrCOMs) applicable to stereoscopic images. Furthermore, the reconstruction performance and geometric invariance of TFrCOMs are analyzed theoretically and experimentally. Finally, an application in the stereoscopic image zero-watermarking algorithm is investigated to verify the superior performance of TFrCOMs. [ABSTRACT FROM AUTHOR]

Many stereoscopic image retargeting (SIR) methods have been developed for automatically and intelligently resizing stereoscopic images and we cannot always rely on time-consuming subjective user studies to validate the performance of different SIR methods. It is therefore required to design reliable objective metrics for SIR quality evaluation. This paper extends our previous 2D aspect ratio similarity (ARS) metric to a stereo 3D version termed as StereoARS where the key idea is to investigate into retargeting inconsistency between the original stereo correspondences. Our proposed StereoARS operates via two stages: monocular quality estimation and binocular inconsistency detection. In the first stage, monocular quality estimation is performed by applying a modified ARS measure on the left and right views separately to quantify the quality degradation within each monocular view. In the second stage, binocular inconsistency detection is performed in both pixel-level and grid-level to characterize the influence of binocular rivalry and stereo visual discomfort on SIR quality. In addition, we also measure to what extent the original pixel visibility relation is preserved after SIR as another binocular quality factor. Finally, these monocular and binocular quality estimates are fused to produce an overall SIR quality score. Extensive experiments have demonstrated that StereoARS achieves better alignment with human subjective ratings than the existing metrics by a large margin. [ABSTRACT FROM AUTHOR]

This paper presents a novel approach to provide immersive free navigation with 6 Degrees of Freedom in real-time for natural and virtual scenery, for both static and dynamic content. Stemming from the state-of-the-art in Depth Image-Based Rendering and the OpenGL pipeline, this new View Synthesis method achieves free navigation at up to 90 FPS and can take any number of input views with their corresponding depth maps as priors. Video content can be played thanks to GPU decompression, supporting free navigation with full parallax in real-time. To render a novel viewpoint, each selected input view is warped using the camera pose and associated depth map, using an implicit 3D representation. The warped views are then blended all together to generate the chosen virtual view. Various view blending approaches specifically designed to avoid visual artifacts are compared. Using as few as four input views appears to be an optimal trade-off between computation time and quality, allowing to synthesize high-quality stereoscopic views in real-time, offering a genuine immersive virtual reality experience. Additionally, the proposed approach provides high-quality rendering of a 3D scenery on holographic light field displays. Our results are comparable - objectively and subjectively - to the state of the art view synthesis tools NeRF and LLFF, while maintaining an overall lower complexity and real-time rendering.

Abstract The traditional way of passive sensor data association relies on the estimation of candidate target position, and hence suffers from severe performance degradation due to the estimation error and the so‐called ghosts. The proposed new cost function is based on epipolar geometry theory that uses fundamental matrix to measure the correlation between sensor measurements directly without estimation of candidate target position. This approach has two superiorities: (1) it declines the accuracy loss caused by a large number of trigonometric operations, and (2) it greatly immunes to the impact of ghosts. A comparison with three traditional cost functions via numerical experiments demonstrates that the proposed cost function significantly improves the accuracy of data association, especially when the number of targets is more and the sensor distribution is closer.

Stereo image pairs can effectively enhance the performance of super‐resolution (SR) since both intra‐view and cross‐view information can be used. However, exploiting cross‐view information accurately is extremely challenging. Most recent methods use the attention mechanism to get stereo correspondence. But these methods ignore the high‐frequency information since they only utilise first‐order statistics, which leads to reducing the discriminative ability of the network. To address this issue, in this work, a parallax‐based second‐order mixed attention stereo SR network (PSMASSRnet) is proposed to integrate the cross‐view information from a stereo image pair for SR. Specifically, a novel parallax‐based second‐order mixed attention module (PSMAM) is developed to combine second‐order channel features and spatial features to obtain more discriminative representations. Furthermore, a dense cross‐atrous spatial pyramid pooling (ASPP) module is also presented, which can effectively explore the local and the multi‐scale features with different dilation rates to extract more discriminative features with fewer parameters and less execution. The extensive experiments on the KITTI2012, KITTI2015 and Middlebury datasets have demonstrated the superiority of the proposed PSMASSRnet over the state‐of‐the‐art methods in the aspects of both the quantitative metrics and the visual quality. [ABSTRACT FROM AUTHOR]

Circumboreal forests are currently experiencing rapid climate warming which is altering their structure, productivity, and status as a carbon sink. Very high-resolution (VHR; < 2 m) stereo-derived digital surface models are available to monitor these forests, but a similar resolution digital terrain model (DTM) is required to extract information about tree height, which is often used to estimate carbon content. To the best of our knowledge, no openly available VHR DTM currently exists. To address this need, we developed approaches to extract DTMs by filtering VHR stereo point clouds (PCs) in sparse canopies of Alaska. Our evaluation consisted of two stereo processing methods with three PC search radii at six different tree canopy cover (TCC) intervals. We found that VHR DTMs were robust for estimating bare ground at TCC intervals less than 40% with vertical errors < 1.6 m using airborne small footprint light detection and ranging (LiDAR) as reference. [ABSTRACT FROM AUTHOR]

Providing realistic immersive experiences through virtual reality (VR) has become a major goal of video technology. Techniques to generate monocular VR images have been extended to the stereoscopic mode by adding the sense of depth. Research on the generation of stereoscopic VR images relies on either specialized equipment or general-purpose cameras. We propose an efficient algorithm to generate stereoscopic 360° VR images from pictures captured using affordable conventional cameras. To overcome the limited ability of conventional cameras to harmonize left and right VR images, we propose a model that represents the relation between left and right views using Euler angles. Experimental results show that the proposed algorithm provides better-harmonized images than other methods both qualitatively and quantitatively. [ABSTRACT FROM AUTHOR]

Stereoscopic image retargeting aims at converting stereoscopic images to the target resolution adaptively. Different from 2D image retargeting, stereoscopic image retargeting needs to preserve both the shape structure of salient objects and depth consistency of 3D scenes. In this paper, we present a stereoscopic image retargeting method based on deep convolutional neural network to obtain high-quality retargeted images with both object shape preservation and scene depth preservation. First, a cross-attention extraction mechanism is constructed to generate attention map, which contains the valuable attention features of the left and right images and the common attention features between them. Second, since the disparity map can provide accurate depth information of objects in 3D scenes, a disparity-assisted 3D significance map generation module is utilized to further preserve the valuable depth information of stereoscopic images. Finally, in order to predict the retargeted stereoscopic images accurately, an image consistency loss is developed to preserve the geometric structure of salient objects, and a disparity consistency loss is introduced to eliminate depth distortions. Experimental results demonstrate that the proposed deep convolutional neural network can provide favorable stereoscopic image retargeting results. [ABSTRACT FROM AUTHOR]

The census transform is a non-parametric local transform that is widely used in stereo matching. This transform encodes the structural information of a local patch into a binary code stream representing the relative intensity ordering of the pixels within the patch. Despite its high performance in stereo matching, the census transform often generates identical binary code streams for two different patches because it simply thresholds the pixels within the patch at the center pixel intensity. To overcome this problem, we introduce a quaternary census transform that encodes the local structural information into a quaternary code stream by employing both the relative intensity ordering and the minimum visibility threshold of the human eye known as the just-noticeable difference. Moreover, because the human eye activates different areas of the retina based on brightness, the patch size for the proposed quaternary census transform adaptively varies depending on the luminance of each pixel. Experimental results on well-known Middlebury stereo datasets prove that the proposed transform outperforms the other census transform-based methods in terms of the accuracy of stereo matching.

In this paper, a fast and precise stereo matching system, which is mainly composed of adaptive stereo matching, recursive refinement and depth upsizing subsystems, is proposed to estimate high resolution depth maps. The adaptive stereo matching subsystem, which adaptively uses color gradient, color intensity and texture census costs, can reduce the most of computation and achieve good primary depth maps. The recursive refinement subsystem consists of median filtering, left-right check, hole filling, and bilateral filtering functions. The refinement subsystem is performed recursively to reduce the most errors occurred in occlusion regions such that we can obtain high-precision depth maps. Finally, the depth upsizing subsystem by referring original high-resolution images performs depth value upscaling, texture-selected interpolation, and weighted depth voting processes to finally obtain the high-precision and high-resolution depth maps. To evaluate the proposed algorithms in three subsystems, the performances of the proposed stereo matching system are exhibited in step-by-step approaches. In comparison of the existing methods, the accuracy performance and computation time of the proposed stereo matching system are finally demonstrated. For future 3D ultra-high-resolution videos, the proposed stereo matching system with low computation will be a good solution to extract high-precision and high-resolution depth information. [ABSTRACT FROM AUTHOR]

Stereoscopic projection mapping (PM) allows a user to see a three-dimensional (3D) computer-generated (CG) object floating over physical surfaces of arbitrary shapes around us using projected imagery. However, the current stereoscopic PM technology only satisfies binocular cues and is not capable of providing correct focus cues, which causes a vergence-accommodation conflict (VAC). Therefore, we propose a multifocal approach to mitigate VAC in stereoscopic PM. Our primary technical contribution is to attach electrically focus-tunable lenses (ETLs) to active shutter glasses to control both vergence and accommodation. Specifically, we apply fast and periodical focal sweeps to the ETLs, which causes the “virtual image” (as an optical term) of a scene observed through the ETLs to move back and forth during each sweep period. A 3D CG object is projected from a synchronized high-speed projector only when the virtual image of the projected imagery is located at a desired distance. This provides an observer with the correct focus cues required. In this study, we solve three technical issues that are unique to stereoscopic PM: The 3D CG object is displayed on non-planar and even moving surfaces; the physical surfaces need to be shown without the focus modulation; the shutter glasses additionally need to be synchronized with the ETLs and the projector. We also develop a novel compensation technique to deal with the “lens breathing” artifact that varies the retinal size of the virtual image through focal length modulation. Further, using a proof-of-concept prototype, we demonstrate that our technique can present the virtual image of a target 3D CG object at the correct depth. Finally, we validate the advantage provided by our technique by comparing it with conventional stereoscopic PM using a user study on a depth-matching task. [ABSTRACT FROM AUTHOR]

Compared with traditional 2D images, stereoscopic omnidirectional images (SOIs) usually have more complex perceptual factors due to the particularities of imaging and display, making the objective quality assessment of SOIs challenging. In this paper, we construct a large and diverse subjective SOIs database named as NBU-SOID for further research demand. And then, we propose a viewport perception based blind SOIs quality assessment (VP-BSOIQA) method by considering the impacts of viewport, user behavior and stereoscopic perception on human visual system, which is mainly composed of binocular perception model (BPM) and omnidirectional perception model (OPM). In the BPM, a binocular combination perception map is generated by the dimension reduction of stereopair and the weighting of binocular energy to reflect the binocular masking effect. In the OPM, several viewports are first created to ensure the consistency of evaluation objects. Then, the intra-viewport and inter-viewport weighting factors are designed with the common influences of visual attention and peripheral vision sensitivity to aggregate the novel multi-orientation structural features extracted from all potential viewports. Experimental results on the NBU-SOID and SOLID databases demonstrate that BPM and OPM can be robustly combined with the existing 2D image quality assessment (IQA) methods, thus averagely achieving 10.2% and 12.2% performance gain in terms of SRCC, respectively. In addition, the proposed VP-BSOIQA method outperforms the state-of-the-art blind IQA methods in predicting the quality of SOIs. [ABSTRACT FROM AUTHOR]