Search

Current research practices have revealed that the neural mechanisms driving visual consciousness alternations in binocular rivalry come from the primary visual cortex, i.e., stimulus rivalry can be induced by monocular neurons. However, the competition mechanisms of the monocular neurons remain unclear. In this paper, we probe the dynamical characteristics of a monolayer binocular rivalry model (which contains four monocular neurons) with different types of stimuli, including fixed inputs, swap, flicker, swap and flicker, swap and blanks, respectively. Firstly, we study the dynamic effects of the traditional stimuli with fixed inputs (but with different grating conditions) on the monolayer rivalry model. Results show that Hopf bifurcations can induce three types of dynamical behaviors: winner-take-all (WTA), rivalry oscillation (RIV), and same activity (SAM), which are similar to other binocular rivalry models. Besides, the simulation results indicate that the competition mechanisms of the monolayer binocular rivalry model are more consistent with the experimental results compared with the hierarchical rivalry model proposed by Wilson. Secondly, the dynamical mechanisms of the monolayer rivalry model with four types of time-dependent stimuli are investigated. More complex dynamical behaviors including WTA-Mod, RIV-Mod, SAM-Mod induced by torus bifurcations and cycle skipping, multi-cycle skipping, chaos induced by period-doubling bifurcation, appear with different types of periodic stimuli. Finally, we analyze the perceptual alternation mechanisms based on the temporal characteristics of the monolayer rivalry model, and results are more consistent with the empirical findings compared with the hierarchical rivalry model. Our simulations and analysis provide a new opportunity for future experiments to investigate the neural mechanisms of binocular rivalry.

Estimating the depth of the scene from a monocular image is an essential step for image semantic understanding. Practically, some existing methods for this highly ill-posed issue are still in lack of robustness and efficiency. This paper proposes a novel end-to-end depth esti- mation model with skip connections from a pre- trained Xception model for dense feature extrac- tion, and three new modules are designed to im- prove the upsampling process. In addition, ELU activation and convolutions with smaller kernel size are added to improve the pixel-wise regres- sion process. The experimental results show that our model has fewer network parameters, a lower error rate than the most advanced networks and requires only half the training time. The evalu- ation is based on the NYU v2 dataset, and our proposed model can achieve clearer boundary de- tails with state-of-the-art effects and robustness.

Binocular summation in strabismic amblyopia is typically reported as being absent or greatly reduced in behavioral studies and is thought to be because of a preferential loss of excitatory interactions between the eyes. Here, we studied how excitatory and suppressive interactions contribute to binocular contrast interactions along the visual cortical hierarchy of humans with strabismic and anisometropic amblyopia in both sexes, using source-imaged steady-state visual evoked potentials (SSVEP) over a wide range of relative contrast between the two eyes. Dichoptic parallel grating stimuli modulated at unique temporal frequencies in each eye allowed us to quantify spectral response components associated with monocular inputs (self-terms) and the response components because of interaction of the inputs of the two eyes [intermodulation (IM) terms]. Although anisometropic amblyopes revealed a similar pattern of responses to normal-vision observers, strabismic amblyopes exhibited substantially reduced IM responses across cortical regions of interest (V1, V3a, hV4, hMT+ and lateral occipital cortex), indicating reduced interocular interactions in visual cortex. A contrast gain control model that simultaneously fits self- and IM-term responses within each cortical area revealed different patterns of binocular interactions between individuals with normal and disrupted binocularity. Our model fits show that in strabismic amblyopia, the excitatory contribution to binocular interactions is significantly reduced in both V1 and extra-striate cortex, whereas suppressive contributions remain intact. Our results provide robust electrophysiological evidence supporting the view that disruption of binocular interactions in strabismus or amblyopia is because of preferential loss of excitatory interactions between the eyes.SIGNIFICANCE STATEMENTWe studied how excitatory and suppressive interactions contribute to binocular contrast interactions along the visual cortical hierarchy of humans with normal and amblyopic vision, using source-imaged SSVEP and frequency-domain analysis of dichoptic stimuli over a wide range of relative contrast between the two eyes. A dichoptic contrast gain control model was used to characterize these interactions in amblyopia and provided a quantitative comparison to normal vision. Our model fits revealed different patterns of binocular interactions between normal and amblyopic vision. Strabismic amblyopia significantly reduced excitatory contributions to binocular interactions, whereas suppressive contributions remained intact. Our results provide robust evidence supporting the view that the preferential loss of excitatory interactions disrupts binocular interactions in strabismic amblyopia.

In the present study, we investigated the difference between monocular augmented reality (AR) and binocular AR in terms of perception and cognition by using a task that combines the flanker task with the oddball task. A right- or left-facing arrowhead was presented as a central stimulus at the central vision, and participants were instructed to press a key only when the direction in which the arrowhead faced was a target. In a small number of trials, arrowheads that were facing in the same or opposite direction (flanker stimuli) were presented beside the central stimulus binocularly or monocularly as an AR image. In the binocular condition, the flanker stimuli were presented to both eyes, and, in the monocular condition, only to the dominant eye. The results revealed that participants could respond faster in the binocular condition than in the monocular one; however, only when the flanker stimuli were in the opposite direction was the response faster in the monocular condition. Moreover, the results of event-related brain potentials (ERPs) showed that all stimuli were processed in both the monocular and the binocular conditions in the perceptual stage; however, the influence of the flanker stimuli was attenuated in the monocular condition in the cognitive stage. The influence of flanker stimuli might be more unstable in the monocular condition than in the binocular condition, but more precise examination should be conducted in a future study.

Dense depth estimation based on a single image is a basic problem in computer vision and has exciting applications in many robotic tasks. Modelling fully supervised methods requires the acquisition of accurate and large ground truth data sets, which is often complex and expensive. On the other hand, self-supervised learning has emerged as a promising alternative to monocular depth estimation as it does not require ground truth depth data. In this paper, we propose a novel self-supervised joint learning framework for depth estimation using consecutive frames from monocular and stereo videos. Our architecture leverages two new ideas for improvement: (1) triplet attention and (2) funnel activation (FReLU). By adding triplet attention to the deep and pose networks, this module captures the importance of features across dimensions in a tensor without any information bottlenecks, making the optimisation learning framework more reliable. FReLU is used at the non-linear activation layer to grasp the local context adaptively in images, rather than using more complex convolutions at the convolution layer. FReLU extracts the spatial structure of objects by the pixel-wise modeling capacity provided by the spatial condition, making the details of the complex image richer. The experimental results show that the proposed method is comparable with the state-of-the-art self-supervised monocular depth estimation method.

This paper presents an independent stereo-vision based positioning system for docking operations. The low-cost system consists of an object detector and different 3D reconstruction techniques. To address the challenge of robust detections in an unstructured and complex outdoor environment, a learning-based object detection model is proposed. The system employs a complementary modular approach that uses data-driven methods, utilizing data wherever required and traditional computer vision methods when the scope and complexity of the environment are reduced. Both, monocular and stereo-vision based methods are investigated for comparison. Furthermore, easily identifiable markers are utilized to obtain reference points, thus simplifying the localization task. A small unmanned surface vehicle (USV) with a LiDAR-based positioning system was exploited to verify that the proposed vision-based positioning system produces accurate measurements under various docking scenarios. Field experiments have proven that the developed system performs well and can supplement the traditional navigation system for safety-critical docking operations.

Recently, great progress has been made in the automatic detection and segmentation of planar regions from monocular images of indoor scenes. This has been achieved thanks to the development of convolutional neural network architectures for the task and the availability of large amounts of training data usually obtained with the help of active depth sensors. Unfortunately, it is much harder to obtain large image sets outdoors partly due to limited range of active sensors. Therefore, there is a need to develop techniques that transfer features learned from the indoor dataset to segmentation of outdoor images. We propose such an approach that does not require manual annotations on the outdoor datasets. Instead, we exploit a network trained on indoor images and an automatically reconstructed point cloud to estimate the training ground truth on the outdoor images in an energy minimization framework. We show that the resulting ground truth estimate is good enough to improve the network weights. Moreover, the process can be repeated multiple times to further improve plane detection and segmentation accuracy on monocular images of outdoor scenes.

Markerless motion capture and understanding of professional non-daily human movements is an important yet unsolved task, which suffers from complex motion patterns and severe self-occlusion, especially for the monocular setting. In this paper, we propose SportsCap—the first approach for simultaneously capturing 3D human motions and understanding fine-grained actions from monocular challenging sports video input. Our approach utilizes the semantic and temporally structured sub-motion prior in the embedding space for motion capture and understanding in a data-driven multi-task manner. To enable robust capture under complex motion patterns, we propose an effective motion embedding module to recover both the implicit motion embedding and explicit 3D motion details via a corresponding mapping function as well as a sub-motion classifier. Based on such hybrid motion information, we introduce a multi-stream spatial-temporal graph convolutional network to predict the fine-grained semantic action attributes, and adopt a semantic attribute mapping block to assemble various correlated action attributes into a high-level action label for the overall detailed understanding of the whole sequence, so as to enable various applications like action assessment or motion scoring. Comprehensive experiments on both public and our proposed datasets show that with a challenging monocular sports video input, our novel approach not only significantly improves the accuracy of 3D human motion capture, but also recovers accurate fine-grained semantic action attribute.

Single-task learning in artificial neural networks will be able to learn the model very well, and the benefits brought by transferring knowledge thus become limited. In this regard, when the number of tasks increases (e.g., semantic segmentation, panoptic segmentation, monocular depth estimation, and 3D point cloud), duplicate information may exist across tasks, and the improvement becomes less significant. Multi-task learning has emerged as a solution to knowledge-transfer issues and is an approach to scene understanding which involves multiple related tasks each with potentially limited training data. Multi-task learning improves generalization by leveraging the domain-specific information contained in the training data of related tasks. In urban management applications such as infrastructure development, traffic monitoring, smart 3D cities, and change detection, automated multi-task data analysis for scene understanding based on the semantic, instance, and panoptic annotation, as well as monocular depth estimation, is required to generate precise urban models. In this study, a common framework for the performance assessment of multi-task learning methods from fixed-wing UAV images for 2D/3D city modelling is presented.

Our binocular world is seamlessly assembled from two retinal images that remain segregated until the cerebral cortex. Despite the coherence of this input, there is often an imbalance between the strength of these connections in the brain. ‘Eye dominance’ provides a measure of the perceptual dominance of one eye over the other. Theoretical models suggest that eye dominance is related to reciprocal inhibition between monocular units in the primary visual cortex, the first location where the binocular input is combined. As the specific inhibitory interactions in the binocular visual system critically depend on the presence of visual input, we sought to test the role of inhibition by measuring the concentrations of inhibitory (GABA) neurotransmitters during monocular visual stimulation of the dominant and the non-dominant eye. GABA-levels were acquired in V1 using a combined functional magnetic resonance imaging (fMRI) and magnetic resonance spectroscopy (MRS) sequence on a 7-Tesla MRI scanner. Individuals with stronger eye dominance had a greater difference in GABAergic inhibition between the eyes. This relationship was present only when the visual system was actively processing sensory input and was not present at rest. We provide the first evidence that imbalances in GABA levels during ongoing sensory processing are related to eye dominance in the human visual cortex. This provides strong support to the view that intracortical inhibition underlies normal eye dominance.SIGNIFICANCE STATEMENTWhat we see is shaped by excitation and inhibition in our brain. We investigated how eye dominance, the perceptual preference of one eye’s input over the other, is related to levels of inhibitory neurotransmitter GABA during monocular visual stimulation. GABAergic inhibition is related to eye dominance, but only when the visual system is actively processing sensory input. This provides key support for the view that imbalances in visual competition that are observed in the normal visual system arise from an inability of GABA signalling to suppress the stronger sensory representation.

That binocular viewing confers an advantage over monocular viewing for detecting isolated low luminance or low contrast objects, has been known for well over a century; however, the processes involved in combining the images from the two eyes are still not fully understood. Importantly, in natural vision, objects are rarely isolated but appear in context. It is well known that nearby contours can either facilitate or suppress detection, depending on their distance from the target and the global configuration. Here we report that at close distances collinear (but not orthogonal) flanking contours suppress detection more under binocular compared to monocular viewing, thus completely abolishing the binocular advantage, both at threshold and suprathreshold levels. In contrast, more distant flankers facilitate both monocular and binocular detection, preserving a binocular advantage up to about four times the detection threshold. Our results for monocular and binocular viewing, for threshold contrast discrimination without nearby flankers, can be explained by a gain control model with uncertainty and internal multiplicative noise adding additional constraints on detection. However, in context with nearby flankers, both contrast detection threshold and suprathreshold contrast appearance matching require the addition of both target-to-target and flank-to-target interactions occurring before the site of binocular combination. To test an alternative model, in which the interactions occur after the site of binocular combination, we performed a dichoptic contrast matching experiment, with the target presented to one eye, and the flanks to the other eye. The two models make very different predictions for abutting flanks under dichoptic conditions. Interactions after the combination site predict that the perceived contrast of the flanked target will be strongly suppressed, while interactions before the site predict the perceived contrast will be more or less veridical. The data are consistent with the latter model, strongly suggesting that the interactions take place before the site of binocular combination.

In this paper, a new direct computational approach to dense 3D reconstruction in autonomous driving is proposed to simultaneously estimate the depth and the camera motion for the motion stereo problem. A traditional Structure from Motion framework is utilized to establish geometric constrains for our variational model. The architecture is mainly composed of the texture constancy constraint, one-order motion smoothness constraint, a second-order depth regularize constraint and a soft constraint. The texture constancy constraint can improve the robustness against illumination changes. One-order motion smoothness constraint can reduce the noise in estimation of dense correspondence. The depth regularize constraint is used to handle inherent ambiguities and guarantee a smooth or piecewise smooth surface, and the soft constraint can provide a dense correspondence as initial estimation of the camera matrix to improve the robustness future. Compared to the traditional dense Structure from Motion approaches and popular stereo approaches, our monocular depth estimation results are more accurate and more robust. Even in contrast to the popular depth from single image networks, our variational approach still has good performance in estimation of monocular depth and camera motion.

This paper proposes the object depth estimation in real-time, using only a monocular camera in an onboard computer with a low-cost GPU. Our algorithm estimates scene depth from a sparse feature-based visual odometry algorithm and detects/tracks objects’ bounding box by utilizing the existing object detection algorithm in parallel. Both algorithms share their results, i.e., feature, motion, and bounding boxes, to handle static and dynamic objects in the scene. We validate the scene depth accuracy of sparse features with KITTI and its ground-truth depth map made from LiDAR observations quantitatively, and the depth of detected object with the Hyundai driving datasets and satellite maps qualitatively. We compare the depth map of our algorithm with the result of (un-) supervised monocular depth estimation algorithms. The validation shows that our performance is comparable to that of monocular depth estimation algorithms which train depth indirectly (or directly) from stereo image pairs (or depth image), and better than that of algorithms trained with monocular images only, in terms of the error and the accuracy. Also, we confirm that our computational load is much lighter than the learning-based methods, while showing comparable performance.

Monocular depth estimation is a very challenging task in computer vision, with the goal to predict per-pixel depth from a single RGB image. Supervised learning methods require large amounts of depth measurement data, which are time-consuming and expensive to obtain. Self-supervised methods are showing great promise, exploiting geometry to provide supervision signals through image warping. Moreover, several works leverage on other visual tasks (e.g. stereo matching and semantic segmentation) to further advance self-supervised monocular depth estimation. In this paper, we propose a novel framework utilizing monocular depth completion as an auxiliary task to assist monocular depth estimation. In particular, a knowledge transfer strategy is employed to enable monocular depth estimation to benefit from the effective feature representations learned by monocular depth completion task. The correlation between monocular depth completion and monocular depth estimation could be fully and effectively utilized in this framework. Only unlabeled stereo images are used in the proposed framework, which achieves a self-supervised learning paradigm. Experimental results on publicly available dataset prove that the proposed approach achieves superior performance to state-of-the-art self-supervised methods and comparable performance with supervised methods.

Real-time detection of human activities has become very important in terms of surveillance and security of Bank-Automated Teller Machines (ATMs), public offices because of the day-to-day increase in criminal activities. The current way of monitoring such constrained environments is done through monocular CCTV cameras which capture only RGB video. The RGB+D sensor provides depth data of the scene in addition to RGB data. To address the problem of online detection of abnormal activities in Bank ATMs, we propose a supervised deep learning framework based on multi-stream CNNs and RGB+D sensor. From the online video stream of RGB+D data, motion templates are created from RGB and depth video segments and then trained on CNNs to detect a suspicious event in ongoing activity. Moreover, due to the unavailability of any dataset for analyzing human activities in ATMs, we also contributed a novel RGB+D dataset in this paper. The proposed deep learning-based framework is evaluated on qualitative and quantitative statistical evaluation parameters and detect suspicious event with the precision of 0.932 and accuracy of 94.2%. Detailed statistical analysis of results shows that the proposed framework can detect the suspicious event in a real-time online manner before the abnormal activity gets completed.

Monocular cameras coupled with inertial measurements generally give high performance visual inertial odometry. However, drift can be significant with long trajectories, especially when the environment is visually challenging. In this paper, we propose a system that leverages Ultra–WideBand (UWB) ranging with one static anchor placed in the environment to correct the accumulated error whenever the anchor is visible. We also use this setup for collaborative SLAM: different robots use mutual ranging (when available) and the common anchor to estimate the transformation between each other, facilitating map fusion. Our system consists of two modules: a double layer ranging, visual, and inertial odometry for single robots, and a transformation estimation module for collaborative SLAM. We test our system on public datasets by simulating UWB measurements as well as on real robots in different environments. Experiments validate our system and show our method can outperform pure visual-inertial odometry by more than 20%, and in visually challenging environments, our method works even when the visual-inertial pipeline has significant drift. Furthermore, we can compute the inter-robot transformation matrices for collaborative SLAM at almost no extra computation cost.

This letter presents a hybrid framework, both in front-end and back-end, for monocular simultaneous localization and mapping (SLAM), capable of utilizing the robustness of feature matching and the accuracy of direct alignment. In the front-end, the feature-based method is first used for coarse pose estimation that is subsequently taken by the direct alignment module as initialization for further refinement. In the back-end, a double window structure is constructed based on the maintained semi-dense map and the sparse feature map, of which the states are optimized via a multi-layer optimization scheme based on the reprojection constraints and the relative pose constraints. Our evaluation on several public datasets demonstrates that this hybrid design retains the superior resilience to scene variations of salient features, and achieves better tracking accuracy due to the integration of the direct modules, leading to a comparable performance with the state-of-the-arts.

The work is devoted to one of the actual problems of modern strabismology — the study of the ability to stereo perception in children with non-paralytic strabismus.Purpose: to study the capability to stereovision with alternating presentation of stereostimuli in children with functional scotoma in non-paralytic strabismus.Patients and methods. 113 children with functional scotoma (FS) in non-paralytic strabismus were observed. We used stereostimuli with different characteristics in the following regimes of presentation: 1) the regime of simple monocular alternating (alternate presentation of an image for the right eye and the left eye); 2) the regime having an “empty” interval (black background) between monocular phases; 3) the regime having a binocular phase (a binocular image containing details corresponding to the stimuli for the right eye and the left eye) between monocular phases.Results. It was found that in 23 (20,3 %) children, the ability to stereo perception was completely absent. All these children had stable total FS (monocular vision). In the remaining 90 children (with unstable or regional FS), the ability to stereo perception was shown with some stimuli in some modes of their alternating presentation. For stimuli with a central arrangement of linear parts, the stereo effect was possible when they were presented in an alternating mode with an “empty” interval lasting from 20 to 70 ms in combination with the duration of monocular phases from 30 to 90 ms. For stimuli with a peripheral arrangement of linear elements, 22.1 % of children were able to stereo perception not only in the “empty” interval mode, but also in the simple alternation mode. At the same time, the greatest number of children capable of stereo perception was detected when using the mode with an “empty” interval of 30–60 ms and a duration of monocular phases of 40–60 ms. With random-dot stimuli, none of the children in this group were capable of stereo perception.Conclusion. Our results suggest that even in patients with FS in non-paralytic strabismus, stereo perception is possible under the conditions of alternating presentation of stimuli with certain characteristics. In this case, the most likely appearance of a stereo effect with stimuli containing peripheral linear elements that create a stereo effect when presented in an alternating mode with an empty interval between monocular phases.

Athletes exhibit better dynamic visual acuity (DVA) compared with nonathletes, whereas action video game players (VGPs) perform more similarly to controls despite having similar static visual acuity and refractive errors. The differences in DVA between groups were not related to differences in static visual acuity, refractive error, or smooth pursuit gain.The purpose of the study was to examine whether athletes and VGPs have superior DVA than controls (nonathletes, nongamers).Forty-six participants (15 athletes, 11 VGPs, 20 controls) aged 21.7 years (standard deviation, 2.8 years) were recruited. Participants were emmetropic with equivalent monocular and binocular static visual acuity between groups. Dynamic visual acuity was assessed using predictable (horizontal) and unpredictable (random) motion targets at velocities of 5, 10, 20, and 30°/s. Smooth pursuit eye movements were assessed using a horizontal motion step-ramp stimulus at the same speeds. This study was pre-registered with the Center for Open Science (https://osf.io/eu7qc).At 30°/s, there were significant main effects of group (F = 4.762, P = .01) and motion type (F = 9.538, P = .004). Tukey post hoc analysis for groups indicated that athletes performed better than did the control group (t = -2.919, P.02). An omnibus (group × motion type × speed) repeated measures ANOVA showed a main effect of speed (F = 110.137, P.001) and a speed × motion-type interaction (F = 27.825, P.001). Dynamic visual acuity decreased as speed increased, and the slope of the change was greater for random than for horizontal motion. Smooth pursuit gains were not significantly different between groups (P.05).Athletes have superior dynamic visual acuity performance compared with controls at 30°/s. This between-group difference cannot be fully explained by differences in smooth pursuit eye movements and therefore may reflect other differences between the groups.

This letter is concerned with polarimetric dense map reconstruction based on a polarization camera with the help of relative depth information as a prior. In general, polarization imaging is able to reveal information about surface normal such as azimuth and zenith angles, which can support the development of solutions to the problem of dense reconstruction, especially in texture-poor regions. However, polarimetric shape cues are ambiguous due to two types of polarized reflection (specular/diffuse). Although methods have been proposed to address this issue, they either are offline and therefore not practical in robotics applications, or use incomplete polarimetric cues, leading to sub-optimal performance. In this letter, we propose an online reconstruction method that uses full polarimetric cues available from the polarization camera. With our online method, we can propagate sparse depth values both along and perpendicular to iso-depth contours. Through comprehensive experiments on challenging image sequences, we demonstrate that our method is able to significantly improve the accuracy of the depthmap as well as increase its density, specially in regions of poor texture.

This manuscript presents GazeBase, a large-scale longitudinal dataset containing 12,334 monocular eye-movement recordings captured from 322 college-aged participants. Participants completed a battery of seven tasks in two contiguous sessions during each round of recording, including a – (1) fixation task, (2) horizontal saccade task, (3) random oblique saccade task, (4) reading task, (5/6) free viewing of cinematic video task, and (7) gaze-driven gaming task. Nine rounds of recording were conducted over a 37 month period, with participants in each subsequent round recruited exclusively from prior rounds. All data was collected using an EyeLink 1000 eye tracker at a 1,000 Hz sampling rate, with a calibration and validation protocol performed before each task to ensure data quality. Due to its large number of participants and longitudinal nature, GazeBase is well suited for exploring research hypotheses in eye movement biometrics, along with other applications applying machine learning to eye movement signal analysis. Classification labels produced by the instrument’s real-time parser are provided for a subset of GazeBase, along with pupil area., Measurement(s) eye movement measurement Technology Type(s) eye tracking device Factor Type(s) round • participant Sample Characteristic - Organism Homo sapiens Machine-accessible metadata file describing the reported data: 10.6084/m9.figshare.14761866

In this letter, we propose a novel semantic direct monocular simultaneous localization and mapping (SLAM) system that fuses the semantic information obtained by an advanced deep neural network (DNN) into direct sparse odometry with loop closure(LDSO), with the purpose of improving the localization accuracy and building a dense semantic map of the urban environment. For localization, we apply a point reselection strategy based on coarse semantic plane (CSP) constraints to discard static points inconsistent with the nearby co-plane points of the same semantic class and dynamic points beyond the visible range. Moreover, a point group movement consistency (PGMC) check is utilized to decrease the impact of moving dynamic objects. For the dense semantic map, we model numerous small semantic planes from well-estimated points to measure the depth of each static pixel, rather than conduct stereo matching. Experimental results show that our method is more accurate than LDSO and comparable with ORB-SLAM in terms of localization. Moreover, it is capable of building a dense semantic map of the urban environment for better scene understanding.

The work is devoted to one of the actual problems of current ophthalmology — creating effective methods of studying stereovision.The purpose — comparative analysis of the capability of stereoperception under conditions of using different regimes of alternating presentation of stereo stimuli with different characteristics in children with strabismus and in children without ophthalmopathology.Patients and methods. 294 school children — 167 children of the control group (without ophthalmopathology) and 127 children with non-paralytic strabismus without functional scotoma (FSS) were observed. We used stereostimuli with different characteristics in the following regimes of presentation: 1) the regime of simple monocular alternating (alternate presentation of an image for the right and left eye); 2) the regime having an “empty” interval (black background) between monocular phases; 3) the regime having a binocular phase (a binocular image containing details corresponding to the stimuli for the right eye and the left eye) between monocular phases.Results. It was found that the majority of children with non-paralytic strabismus, who are incapable of stereoperception with the classic Fly-test and Lang-test, can perceive the stereoeffect with alternating presentation of stereostimuli within individual ranges of durations of monocular phases, a binocular phase and an “empty” interval. In children of the control group when switching from the simple alternation regime to the “empty” interval regime the maximal durations of monocular phases, which preserved the stereoeffect, decreased and when switching to the binocular phase regime they significantly increased. In children with strabismus linear images are simpler for stereoperception than random-dot images as well as in children of the control group (p < 0.001); stimuli creating the effect of the frontoparallel separation of details get perceived better than those creating the decline effect or the turning effect (p < 0.001); stimuli creating the effect of the vertical stripes decline get perceived better than those creating the effect of the horizontal stripes turning (p < 0.001). However, as opposed to the children of the control group, in children with strabismus the stereoeffect gets formed better under conditions of the peripheral localization of linear details than under conditions of the central one.Conclusion. Using computer programs with different regimes of alternating presentation of stereostimuli with certain characteristics allows to effectively evaluate individual capability of stereoperception which is necessary for the personalized approach to the selection of visual stimuli and stimuli presentation regimes in functional treatment of patients with non-paralytic strabismus.

In order to improve the accuracy of the monocular distance measurement of the vehicle in front under sunny, cloudy, rainy, snowy, and foggy weather, an improved pixel-mapping monocular distance measurement method is proposed. This method is based on eight-connected domains to detect the front vehicle, obtain the line pixels of the target vehicle in the image, and fit the image line pixels to the corresponding real longitudinal distance function, and combine the fitted function with the internal and external parameters of the camera. An improved pixel-mapping monocular ranging model is obtained. Set up a test environment under different weather to verify the feasibility of the algorithm. The results show that in the four environments, the detectable distances are within 70m, 60m, 30m, and 40m respectively; the error of the improved pixel-mapping monocular ranging method is reduced by 0.6% on average compared with before the improvement, up to 0.92% ; The improved algorithm ranging errors under the four weathers are 1.8513%, 2.6987%, 4.0137%, and 2.5795% respectively, which achieves the purpose of improving the accuracy of the monocular distance measurement of the vehicle in front under multiple weather conditions.

We present a new self-supervised monocular depth estimation method with multi-scale texture detail enhancement. Based on the observation that the image texture detail and the semantic information have essential significance on the depth estimation, we propose to provide them to the network to learn more sharpness and structural integrity of depth. Firstly, we generate the filtered images and detail images by multi-scale decomposition and use a deep neural network to automatically learn their weights to construct the texture detail enhanced image. Then, we consider the semantic features by putting deep features from the VGG-19 network into a self-attention network, guide the depth decoder network to focus on the integrity of objects in the scene. Finally, we propose a scale-invariant smooth loss to improve the structural integrity of the predicted depth. We evaluate our method on the KITTI 2015 and Make3D datasets and apply the predicted depth to novel view synthesis. The experimental results show that it has achieved satisfactory results compared with the existing methods.

We propose a monocular depth estimator SC-Depth, which requires only unlabelled videos for training and enables the scale-consistent prediction at inference time. Our contributions include: (i) we propose a geometry consistency loss, which penalizes the inconsistency of predicted depths between adjacent views; (ii) we propose a self-discovered mask to automatically localize moving objects that violate the underlying static scene assumption and cause noisy signals during training; (iii) we demonstrate the efficacy of each component with a detailed ablation study and show high-quality depth estimation results in both KITTI and NYUv2 datasets. Moreover, thanks to the capability of scale-consistent prediction, we show that our monocular-trained deep networks are readily integrated into the ORB-SLAM2 system for more robust and accurate tracking. The proposed hybrid Pseudo-RGBD SLAM shows compelling results in KITTI, and it generalizes well to the KAIST dataset without additional training. Finally, we provide several demos for qualitative evaluation., Comment: Accept to IJCV. The source code is available at https://github.com/JiawangBian/SC-SfMLearner-Release

This paper proposes a method for monocular underwater depth estimation, which is an open problem in robotics and computer vision. To this end, we leverage publicly available in-air RGB-D image pairs for underwater depth estimation in the spherical domain with an unsupervised approach. For this, the in-air images are style-transferred to the underwater style as the first step. Given those synthetic underwater images and their ground truth depth, we then train a network to estimate the depth. This way, our learning model is designed to obtain the depth up to scale, without the need of corresponding ground truth underwater depth data, which is typically not available. We test our approach on style-transferred in-air images as well as on our own real underwater dataset, for which we computed sparse ground truth depths data via stereopsis. This dataset is provided for download. Experiments with this data against a state-of-the-art in-air network as well as different artificial inputs show that the style transfer as well as the depth estimation exhibit promising performance.

Learning-based multi-task models have been widely used in various scene understanding tasks, and complement each other, i.e., they allow us to consider prior semantic information to better infer depth. We boost the unsupervised monocular depth estimation using semantic segmentation as an auxiliary task. To address the lack of cross-domain datasets and catastrophic forgetting problems encountered in multi-task training, we utilize existing methodology to obtain redundant segmentation maps to build our cross-domain dataset, which not only provides a new way to conduct multi-task training, but also helps us to evaluate results compared with those of other algorithms. In addition, in order to comprehensively use the extracted features of the two tasks in the early perception stage, we use a strategy of sharing weights in the network to fuse cross-domain features, and introduce a novel multi-task loss function to further smooth the depth values. Extensive experiments on KITTI and Cityscapes datasets show that our method has achieved state-of-the-art performance in the depth estimation task, as well improved semantic segmentation.

In this article, we propose a quality assessment model-based on the projection invariant feature and the visual saliency for Stereoscopic Omnidirectional Images (SOIs). Firstly, the projection invariant monocular and binocular features of SOI are derived from the Scale-Invariant Feature Transform (SIFT) points to tackle the inconsistency between the stretched projection formats and the viewports. Secondly, the visual saliency model, which combines the chrominance and contrast perceptual factors, is used to facilitate the prediction accuracy. Thirdly, according to the characteristics of the panoramic image, we generate the weight map and utilize it as a location prior, which can be adapted to different projection formats. Finally, the proposed SOI quality assessment model fuses the projection invariant features, visual saliency, and location prior. Experimental results on both the NingBo University SOI Database (NBU-SOID) and Stereoscopic OmnidirectionaL Image quality assessment Database (SOLID) demonstrate the proposed metric on equi-rectangular projection format outperforms the state-of-the-art schemes, the pearson linear correlation coefficient and spearman rank order correlation coefficient performance are 0.933 and 0.933 on SOLID, and 0.907 and 0.910 on NBU-SOID, respectively. Meanwhile, the proposed algorithm is extended to another five representative projection formats and achieves superior performance.

Post-surgical or traumatic corectopia is among the rare causes of monocular diplopia. A 26-years-old student presented to the Institute with a complaint of monocular double vision in the left eye. He had a penetrating ocular injury in the left eye and subsequently, undergone for multiple ocular surgeries. Following the final intraocular lens implantation, he experienced a monocular double vision in his left eye. Upon contact lens clinic presentation, visual acuities were 20/20 in the right and 20/320 in the left eye (improved to 20/25 with pinhole). Slit-lamp examination on the left eye revealed scarring in the superior nasal quadrant of the cornea, irregular mid-dilated pupil with exposed aphakic and pseudophakic portions. A range of different optical management options were implemented to eliminate monocular diplopia and to correct refractive error. Finally, a combination of prosthetic soft contact lens and spectacle correction was able to remove diplopia and provide binocular single vision.

Three-dimensional display and virtual reality technology have been applied in minimally invasive surgery to provide doctors with a more immersive surgical experience. One of the most popular systems based on this technology is the Da Vinci surgical robot system. The key to build the in vivo 3-D virtual reality model with a monocular endoscope is an accurate estimation of depth and motion. In this article, a fully unsupervised learning method for depth and motion estimation using the continuous monocular endoscopic video is proposed. After the detection of highlighted regions, EndoMotionNet and EndoDepthNet are designed to estimate ego-motion and depth, respectively. The timing information between consecutive frames is considered with a long short-term memory layer by EndoMotionNet to enhance the accuracy of ego-motion estimation. The estimated depth value of the previous frame is used to estimate the depth of the next frame by EndoDepthNet with a multimode fusion mechanism. The custom loss function is defined to improve the robustness and accuracy of the proposed unsupervised-learning-based method. Experiments with the public datasets verify that the proposed unsupervised-learning-based continuous depth and motion estimation method can effectively improve the accuracy of depth and motion estimation, especially after processing the continuous frame.

This article presents an innovative framework of real-time vision-based pose tracking for asteroid using the contour information of the asteroid image. At the first-time instant, the tracking is initialized by the distance-based template matching and contour-based pose optimization. Subsequently, at each time instant, with the prediction of the extended Kalman filter (EKF) as initial guess, the pose of the asteroid is obtained in real time by geometrically fitting the contour of the projected asteroid CAD model over the contour of the asteroid image with M-estimation. The variance of the pose is calculated based on 1-order approximation inference, which enables EKF to generate the final pose estimation and predict the pose at the next time instant. Sufficient experiments validate the accuracy and the efficiency of the proposed method.

Depth estimation from single monocular image attracts increasing attention in autonomous driving and computer vision. While most existing approaches regress depth values or classify depth labels based on features extracted from limited image area, the resulting depth maps are still perceptually unsatisfying. Neither local context nor low-level semantic information is sufficient to predict depth. Learning based approaches suffer from inherent defects of supervision signals. This paper addresses monocular depth estimation with a general information exchange convolutional neural network. We maintain a high-resolution prediction throughout the network. Meanwhile, both low-resolution features capturing long-range context and fine-grained features describing local context can be refined with information exchange path stage by stage. Mutual channel attention mechanism is applied to emphasize interdependent feature maps and improve the feature representation of specific semantics. The network is trained under the supervision of improved log-cosh and gradient constraints so that the abnormal predictions have less impacts and the estimation can be consistent in high order. The results of ablation studies verify the efficiency of every proposed components. Experiments on the popular indoor and street-view datasets show competitive results compared with the recent state-of-the-art approaches.

The visual system exploits multiple signals, including monocular and binocular cues, to determine the motion of objects through depth. In the laboratory, sensitivity to different three-dimensional (3D) motion cues varies across observers and is often weak for binocular cues. However, laboratory assessments may reflect factors beyond inherent perceptual sensitivity. For example, the appearance of weak binocular sensitivity may relate to extensive prior experience with two-dimensional (2D) displays in which binocular cues are not informative. Here we evaluated the impact of experience on motion-in-depth (MID) sensitivity in a virtual reality (VR) environment. We tested a large cohort of observers who reported having no prior VR experience and found that binocular cue sensitivity was substantially weaker than monocular cue sensitivity. As expected, sensitivity was greater when monocular and binocular cues were presented together than in isolation. Surprisingly, the addition of motion parallax signals appeared to cause observers to rely almost exclusively on monocular cues. As observers gained experience in the VR task, sensitivity to monocular and binocular cues increased. Notably, most observers were unable to distinguish the direction of MID based on binocular cues above chance level when tested early in the experiment, whereas most showed statistically significant sensitivity to binocular cues when tested late in the experiment. This result suggests that observers may discount binocular cues when they are first encountered in a VR environment. Laboratory assessments may thus underestimate the sensitivity of inexperienced observers to MID, especially for binocular cues.

Depth estimation is one of the basic and important tasks in 3D vision. Recently, many works have been done in self-supervised depth estimation based on geometric consistency between frames. However, these research works still have difficulties in ill-posed areas, such as occlusion areas and texture-less areas. This work proposed a novel self-supervised monocular depth estimation method based on occlusion mask and edge awareness to overcome these difficulties. The occlusion mask divides the image into two classes, making the training of the network more reasonable. The edge awareness loss function is designed based on the edge obtained by the traditional method, so that the method has strong robustness to various lighting conditions. Furthermore, we evaluated the proposed method on the KITTI datasets. The occlusion mask and edge awareness are both beneficial to find corresponding points in ill-posed areas.

In recent years, tracking of articulated objects at high speed with monocular cameras has been gaining attention. This study presents a novel method for high-frame-rate articulated object tracking with a monocular camera. The method is an extended version of our previous research on high-speed monocular rigid object tracking. In this study, to realize tracking of an articulated object, we integrate dual-quaternion kinematics with our previous fast pixel-wise-posteriors (fast-PWP3D) tracking framework, and propose an auto-regressive (AR) process to encode the dynamic propagation of the estimated state vectors. We give a full three-dimensional derivation of the mathematical formulation of our method and show that our method is capable of tracking an articulated object having a large number of degrees of freedom with only a monocular camera, and is robust against dynamic environmental changes (e.g., illumination/partial occlusion). Moreover, we show an efficient implementation strategy of our method. The results of real-time experiments show that we achieved nearly 350 Hz performance when tracking a four degrees-of-freedom (4-DOF) articulated object with a monocular camera.

Depth estimation is a classic task in computer vision, which is of great significance for many applications such as augmented reality, target tracking and autonomous driving. Traditional monocular depth estimation methods are based on depth cues for depth prediction with strict requirements, e.g. shape-from-focus/ defocus methods require low depth of field on the scenes and images. Recently, a large body of deep learning methods have been proposed and has shown great promise in handling the traditional ill-posed problem. This paper aims to review the state-of-the-art development in deep learning-based monocular depth estimation. We give an overview of published papers between 2014 and 2020 in terms of training manners and task types. We firstly summarize the deep learning models for monocular depth estimation. Secondly, we categorize various deep learning-based methods in monocular depth estimation. Thirdly, we introduce the publicly available dataset and the evaluation metrics. And we also analysis the properties of these methods and compare their performance. Finally, we highlight the challenges in order to inform the future research directions.

Recently, real-time obstacle detection by monocular vision exhibits a promising prospect in enhancing the safety of unmanned surface vehicles (USVs). Since the obstacles that may threaten USVs generally appear below the water edge, most existing methods first detect the horizon line and then search for obstacles below the estimated horizon line. However, these methods detect horizon line only using edge or line features, which are susceptible to interference edges from clouds, waves, and land, eventually resulting in poor obstacle detection. To avoid being affected by interference edges, in this paper, we propose a novel horizon line detection method based on semantic segmentation. The method assumes a Gaussian mixture model (GMM) with spatial smoothness constraints to fit the semantic structure of marine images and simultaneously generate a water segmentation mask. The horizon line is estimated from the water boundary points via straight line fitting. Further, inspired by human visual attention mechanisms, an efficient saliency detection method based on background prior and contrast prior is presented to detect obstacles below the estimated horizon line. To reduce false positives caused by sun glitter, waves and foam, the continuity of the adjacent frames is employed to filter the detected obstacles. An extensive evaluation was conducted on a large marine image dataset collected by our ‘Jinghai VIII’ USV. The experimental results show that the proposed method significantly outperformed the recent state-of-the-art marine obstacle method by 22.07% in terms of F-score while running over 24 fps on an NVIDIA GTX1080Ti GPU.