Search

Humans can perform object manipulations in VR in spite of missing haptic and acoustic information. Whether their movements under these artificial conditions do still rely on motor programs based on natural experience or are impoverished due to the restrictions imposed by VR is unclear. We investigated whether reach-to-place and reach-to-grasp movements in VR can still be adapted to the task and to the specific properties of the objects being handled, or whether they reflect a stereotypic, task- and object-independent motor program. We analyzed reach-to-grasp and reach-to-place movements from participants performing an unconstrained ”set-the-table” task involving a variety of different objects in virtual reality. These actions were compared based on their kinematic features. We encountered significant differences in peak speed and the duration of the deceleration phase which are modulated depending on the action and on the manipulated object. The flexibility of natural human sensorimotor control thus is at least partially transferred and exploited in impoverished VR conditions. We discuss possible explanations of this behavior and the implications for the design of object manipulations in VR.

Combining machine learning in neural networks with multimodal fusion strategies offers an interesting potential for classification tasks but the optimum fusion strategies for many applications have yet to be determined. Here we address this issue in the context of human activity recognition, making use of a state-of-the-art convolutional network architecture (Inception I3D) and a huge dataset (NTU RGB+D). As modalities we consider RGB video, optical flow, and skeleton data. We determine whether the fusion of different modalities can provide an advantage as compared to uni-modal approaches, and whether a more complex early fusion strategy can outperform the simpler late-fusion strategy by making use of statistical correlations between the different modalities. Our results show a clear performance improvement by multi-modal fusion and a substantial advantage of an early fusion strategy

Contact events during manipulation tasks can be distinguished in two categories: intended and unintended. We investigated the categorization of contact events during an object placing task executed in virtual reality based on kinematic features measured during the movement segment previous to the contact. The experimental setup enabled us to generate unintended contacts by triggering unexpected interruptions of the placing movement. Experimental results indicate that the kinematic features enable the distinction of intended and unintended contacts independent of substantial variations of movement properties (amplitude, duration, velocity), unless the unintended contact occurs toward the end of the planned movement.

Contact events are essential to the execution of manipulation tasks in virtual reality because they mark the different phases of an interaction, e.g. initial hand-object or tool-object contact. Generally, the nature of contact events can be distinguished in two categories: intended and unintended. Whereas intended contacts typically signalize the successful execution of a task, unintended contacts are the result of bad motor planning or execution, or are caused by accidental movements.Here we explore the use of kinematic features to distinguish intended and unintended contacts during an object placing task executed in virtual reality. A handheld object was placed over a horizontal surface with a vertical planar movement. The experimental setup enabled us to generate unintended contacts by triggering unexpected interruptions during the placing movement.Experimental results show significant differences between distributions of the kinematic features across intended and unintended contacts. These results indicate that the proposed kinematic features enable a robust distinction between intended and unintended contacts independent of substantial variations of movement properties (amplitude, duration, velocity).

Deep residual networks for action recognition based on skeleton data can avoid the degradation problem, and a 56-layer Res-Net has recently achieved good results. Since a much “shallower” 11-layer model (Res-TCN) with a temporal convolution network and a simplified residual unit achieved almost competitive performance, we investigate deep variants of Res-TCN and compare them to Res-Net architectures. Our results outperform the other approaches in this class of residual networks. Our investigation suggests that the resistance of deep residual networks to degradation is not only determined by the architecture but also by data and task properties.

The Saturnian moon Enceladus with its extensive water bodies underneath a thick ice sheet cover is a potential candidate for extraterrestrial life. Direct exploration of such extraterrestrial aquatic ecosystems requires advanced access and sampling technologies with a high level of autonomy. A new technological approach has been developed as part of the collaborative research project Enceladus Explorer (EnEx). The concept is based upon a minimally invasive melting probe called the IceMole. The force-regulated, heater-controlled IceMole is able to travel along a curved trajectory as well as upwards. Hence, it allows maneuvers which may be necessary for obstacle avoidance or target selection. Maneuverability, however, necessitates a sophisticated on-board navigation system capable of autonomous operations. The development of such a navigational system has been the focal part of the EnEx project. The original IceMole has been further developed to include relative positioning based on in-ice attitude determination, acoustic positioning, ultrasonic obstacle and target detection integrated through a high-level sensor fusion. This paper describes the EnEx technology and discusses implications for an actual extraterrestrial mission concept.

The human visual system exhibits substantially different properties between foveal and peripheral vision. Peripheral vision is special in that it has to compress data onto fewer units by reduced visual acuity and larger receptive fields, yielding greatly reduced performance on many tasks such as object recognition. However, here we show that the pooling operations implemented by peripheral vision provide exactly the invariance properties required by a self-localization task. We test the effect of different pooling sizes, as well as acuity reduction, on localization, object recognition, and scene categorization tasks. We find that peripheral pooling, but not reduced acuity, affects localization performance positively, whereas it is detrimental to object recognition performance.

The evaluation of printing machines poses the problem of how distortions like streaks caused by the machine can be detected and assessed automatically. Although luminance variations in prints can be measured quite precisely, the measured functions bear little relevance for the lightness of streaks and other distortions of prints as perceived by human observers. First, the measurements sometimes indicate changes of luminance in regions which are perceived as homogeneous by humans. Second, the measured strength of a distortion correlates often weakly with its perceived strength, which is influenced by a variety of factors, like the shape of a streak’s luminance profile and the distribution of luminance variations in its spatial surround. We have used a model of human perception, based on fundamental neurophysiological and psychophysical properties of the visual system, in order to predict the perceptual strength of streaks (i.e. the distortion as perceived by a human observer) from the measured phys...

The ability to quickly recognize the number of objects in our environment is a fundamental cognitive function. However, it is far from clear which computations and which actual neural processing mechanisms are used to provide us with such a skill. Here we try to provide a detailed and comprehensive analysis of this issue, which comprises both the basic mathematical foundations and the peculiarities imposed by the structure of the visual system and by the neural computations provided by the visual cortex. We suggest that numerosity should be considered as a mathematical invariant. Making use of concepts from mathematical topology--like connectedness, Betti numbers, and the Gauss-Bonnet theorem--we derive the basic computations suited for the computation of this invariant. We show that the computation of numerosity is possible in a neurophysiologically plausible fashion using only computational elements which are known to exist in the visual cortex. We further show that a fundamental feature of numerosity perception, its Weber property, arises naturally, assuming noise in the basic neural operations. The model is tested on an extended data set (made publicly available). It is hoped that our results can provide a general framework for future research on the invariance properties of the numerosity system.

A new experimental platform called “VirtuSphere” is helping scientists to understand how we perceive and process spatial information. Their findings will help scientists to develop the next generation of assistive devices and robotic systems.

We investigated audiovisual interactions in motion perception by behavioral experiments testing both, the influence of visual stimuli on auditory apparent motion and the influence of auditory stimuli on visual apparent motion perception. A set of loudspeakers with an LED mounted in the middle of each speaker cone was arranged in a semicircle. Apparent motion streams were presented for each modality alone in the unimodal conditions. In the bimodal conditions, stimuli of the second modality were added to fill the temporal and spatial gaps of the sampled trajectory of the reference stream. The participants' task was to observe the quasi-naturalistic stimulus sequences and to perform a standard classification. The addition of stimuli of the second modality indeed facilitated apparent motion perception. Bimodal presentation increased the upper temporal interval up to which the stimuli could be separated in time while still being perceived as continuous motion. We interpret these results as evidence for an ecologically advantageous audiovisual motion integration mechanism which operates beyond the constraints of strict spatiotemporal coincidence. Functional considerations suggest that this mechanism may represent an amodal stage suited for the processing of both unimodal and bimodal signals.

„VirtuSphere”: Eine neuartige Experimentierplattform hilft besser zu verstehen, wie raumliche Informationen vom Menschen wahrgenommen und verarbeitet werden. Davon konnen die Planung und Realisierung von Roboter- und Assistenzsystemen profitieren.

Many tasks involve the precise estimation of speed and position of moving objects, for instance to catch or avoid objects that cohabit in our environment. Many of these objects are characterised by signal representations in more than one modality, such as hearing and vision. The aim of this study was to investigate the extent to which the simultaneous presentation of auditory and visual signals enhances the estimation of motion speed and instantaneous position. Observers are asked to estimate the instant when a moving object arrives at a target spatial position by pressing a response button. This task requires observers to estimate the speed of the moving object and to calibrate the timing of their manual response such that it coincides with the true arrival time of the moving object. When both visual and auditory motion signals are available, the variability in estimating the arrival time of the moving object is significantly reduced compared to the variability in the unimodal conditions. This reduction in variability is consistent with optimal integration of the auditory and visual speed signals. The average bias in the estimated arrival times depends on the motion speed: for medium speeds (17deg/s) observers' subjective arrival times are earlier than the true arrival times; for high speeds (47deg/s) observers exhibit a (much smaller) bias in the other direction. This speed-dependency suggests that the bias is due to an error in estimating the motion speeds rather than an error in calibrating the timing of the motor response. Finally, in this temporal localization task, the bias and variability show similar patterns for motion defined by vision, audition or both.

We investigate a hybrid system for autonomous exploration and navigation, and implement it in a virtual mobile agent, which operates in virtual spatial environments. The system is based on several distinguishing properties. The representation is not map-like, but based on sensorimotor features, i.e. on combinations of sensory features and motor actions. The system has a hybrid architecture, which integrates a bottom-up processing of sensorimotor features with a top-down, knowledge-based reasoning strategy. This strategy selects the optimal motor action in each step according to the principle of maximum information gain. Two sensorimotor levels with different behavioural granularity are implemented, a macro-level, which controls the movements of the agent in space, and a micro-level, which controls its eye movements. At each level, the same type of hybrid architecture and the same principle of information gain are used for sensorimotor control. The localisation performance of the system is tested with large sets of virtual rooms containing different mixtures of unique and non-unique objects. The results demonstrate that the system efficiently performs those exploratory motor actions that yield a maximum amount of information about the current environment. Localisation is typically achieved within a few steps. Furthermore, the computational complexity of the underlying computations is limited, and the system is robust with respect to minor variations in the spatial environments.

It is often assumed that humans represent large-scale spatial environments as cognitive maps, but the exact features of these representations are still unclear. We investigate the structure of this representation with the impossible worlds paradigm by testing whether the information provided by virtual environments (VEs) with arbitrary violations of geometrical rules is rectified ("distorted") to become compatible with a map-like structure. The experiments were conducted in virtual reality using a natural locomotion interface. The subjects' task was to explore possible and impossible VEs carefully to achieve a "full understanding". After each trial, they had to "blindly" reproduce the path through the environment from memory in a VE with impoverished visual cues. We have found no evidence for angular or configurational distortions or alterations in the blind reproductions of impossible VEs. Blind reproduction indicates that impossible VEs do not require a transformation into a "possible" format to make them fit into the cognitive map. This suggests that the representation may not be similar to a map in a bounded sense of interpretation but requires more generalized concepts for its understanding.

The interaction of biological agents within the real world is based on their abilities and the affordances of the environment. By contrast, the classical view of perception considers only sensory features, as do most object recognition models. Only a few models make use of the information provided by the integration of sensory information as well as possible or executed actions. Neither the relations shaping such an integration nor the methods for using this integrated information in appropriate representations are yet entirely clear. We propose a probabilistic model integrating the two information sources in one system. The recognition process is equipped with an utility maximization principle to obtain optimal interactions with the environment

It is well known that the detection thresholds for stationary auditory and visual signals are lower if the signals are presented bimodally rather than unimodally, provided the signals coincide in time and space. Recent work on auditory-visual motion detection suggests that the facilitation seen for stationary signals is not seen for motion signals. We investigate the conditions under which motion perception also benefits from the integration of auditory and visual signals. We show that the integration of cross-modal local motion signals that are matched in position and speed is consistent with thresholds predicted by a neural summation model. If the signals are presented in different hemi-fields, move in different directions, or both, then behavioural thresholds are predicted by a probability-summation model. We conclude that cross-modal signals have to be co-localised and co-incident for effective motion integration. We also argue that facilitation is only seen if the signals contain all localisation cues that would be produced by physical objects.

Linear operations can only partially exploit the statistical redundancies of natural scenes, and nonlinear operations are ubiquitous in visual cortex. However, neither the detailed function of the nonlinearities nor the higher-order image statistics are yet fully understood. We suggest that these complicated issues can not be tackled by one single approach, but require a range of methods, and the understanding of the crosslinks between the results. We consider three basic approaches: (i) State space descriptions can theoretically provide complete information about statistical properties and nonlinear operations, but their practical usage is confined to very low-dimensional settings. We discuss the use of representation-related state-space coordinates (multivariate wavelet statistics) and of basic nonlinear coordinate transformations of the state space (e.g., a polar transform). (ii) Indirect methods, like unsupervised learning in multi-layer networks, provide complete optimization results, but no direct information on the statistical properties, and no simple model structures. (iii) Approximation by lower-order terms of power-series expansions is a classical strategy that has not yet received broad attention. On the statistical side, this approximation amounts to cumulant functions and higher-order spectra (polyspectra), on the processing side to Volterra Wiener systems. In this context we suggest that an important concept for the understanding of natural scene statistics, of nonlinear neurons, and of biological pattern recognition can be found in AND-like combinations of frequency components. We investigate how the different approaches can be related to each other, how they can contribute to the understanding of cortical nonlinearities such as complex cells, cortical gain control, end-stopping and other extraclassical receptive field properties, and how we can obtain a nonlinear perspective on overcomplete representations and invariant coding in visual cortex.

Linear filtering is a basic concept in neural models of early sensory information processing. In particular the visual system has been described to perform a wavelet-like multi-channel decomposition by a set of independent spatial-frequency selective filter mechanisms. Here we suggest that this principle of linear filtering deserves a critical re-evaluation. We propose that an optimal adaptation to natural scene statistics would require AND-like nonlinear interactions between the frequency-selective filter channels. We describe how this hypothesis can be tested by predicted violations of the principle of linearity that should be observable if cortical neurons would actually implement the proposed nonlinearities. We further explain why these effects might have been easily overlooked in earlier tests of the linearity of neurons in primary visual cortex.

We investigated the extent to which auditory and visual motion signals are combined when observers are asked to predict the location of a virtually moving target. In Condition 1, the unimodal and bimodal signals were noisy, but the target object was continuously visible and audible; in Condition 2, the virtually moving object was hidden (invisible and inaudible) for a short period prior to its arrival at the target location. Our main finding was that the facilitation due to simultaneous visual and auditory input is very different for the two conditions. When the target is continuously visible and audible (Condition 1), the bimodal performance is twice as good as the unimodal performances, thus suggesting a very effective integration mechanism. On the other hand, if the object is hidden for a short period (Condition 2) and the task therefore requires the extrapolation of motion speed over a temporal and spatial period, the facilitation due to both sensory inputs is almost absent, and the bimodal performance is limited by the visual performance.

The ability to localize ourselves in the outdoor world based on visual input even in absence of prior positional information is an important skill of our daily lives that comes naturally to us. However, the underlying mechanisms of this ability are poorly understood. Here, we show how simple texture statistics can be sufficient to provide a strong prior for the self-localization tasks. We find that statistics of common outdoor features such as tree density, foliage type or road structure provide a stronger cue for self-localization than the matching and recognition of less common landmarks such as lamp posts. We encourage the use of such common feature vectors as priors for self-localization systems and hypothesize that humans may use similar priors to assess the location from an unknown image.

The estimation of the cardinality of objects in a spatial environment requires a high degree of invariance. Numerous experiments showed the immense abstraction ability of the numerical cognition system in humans and other species. It eliminates almost all structures of the objects and determines the number of objects in a scene. Based on concepts and quantities like connectedness and Gaussian curvature, we provide a general solution to this problem and apply it to the numerosity estimation from visual stimuli.

Spatial interaction of biological agents with their environment is based on the cognitive processing of sensory as well as motor information. There are many models for sole sensory processing but only a few for integrating sensory and motor information into a unifying sensorimotor approach. Additionally, neither the relations shaping the integration are yet clear nor how the integrated information can be used in an underlying representation. Therefore, we propose a probabilistic model for integrated processing of sensory and motor information by combining bottom-up feature extraction and top-down action selection embedded in a Bayesian inference approach. The integration of sensory perceptions and motor information brings about two main advantages: (i) Their statistical dependencies can be exploited by representing the spatial relationships of the sensor information in the underlying joint probability distribution and (ii) a top-down process can compute the next most informative region according to an information gain strategy. We evaluated our system in two different object recognition tasks. We found that the integration of sensory and motor information significantly improves active object recognition, in particular when these movements have been chosen by an information gain strategy.

The geometric concept of curvature can help to deal with some important aspects of information processing in natural and artificial vision systems. The paper briefly reviews earlier results regarding the relationship between the notions of curvature, the processing of information, and the modelling of end-stopped neurons in the visual cortex. It then focuses on the difference between Gaussian curvature and the Riemann tensor and reveals the corresponding logical structures. Furthermore, it is shown how multidimensional deviations from flatness can be measured by two-dimensional curvature operators. In the context of image-sequence analysis, a new relationship between the Riemann tensor components and the computation of the optical flow is found.

We investigate the hypothesis that the main representation which underlies human navigation is not static and map-like, but rather is of an inherently sensorimotor nature, i.e. results from a combination of sensory features and motor actions. This is suggested by recent psychological and neurobiological results, and receives further support from an own study of human navigation in manipulated virtual reality environments. To investigate the presumed sensorimotor representation we design a hybrid architecture which integrates a bottom-up processing of sensorimotor features with a top-down reasoning that is based on the principle of maximum information gain. This architecture is implemented in an agent that operates in a VR environment and is able to use a minimum number of exploratory actions to orient itself within this environment.

Virtual reality systems provide a convenient means of studying human cognition and performance on a wide range of tasks for which real-world testing is cost prohibitive or difficult to control. For the results of such studies to be valid, it is important to ensure that aspects of the virtual experience do not alter a participant's behavior or performance on the experimental tasks. This can be particularly difficult when using novel locomotion interfaces that require training. Training procedures should not be completed until movement tasks can be performed at a high level of ability and they do not interfere with concurrent cognitive tasks. A study is described in which subjects were trained to locomote in the Virtusphere, an interface resembling a “human-sized hamster ball.” The effectiveness of training is discussed in terms of both movement abilities and performance on a concurrent cognitive task. Movement performance was tracked as subjects learned to travel through a virtual environment. Additionally, subjects simultaneously completed cognitive tasks in a dual-task selective-interference paradigm. Results showed very rapid improvement on movement measures, including distance traveled and the ratio of collisions to distance traveled, with performance improvement becoming gradual within a few minutes. However, results also highlight persistent problems with concurrent spatial memory tasks, indicating that the training is not really done when performance on the movement metrics levels off.

Human amblyopes display reduced contrast sensitivities, suffer from perceptual distortion, and their letter acuities are worse than is predicted from grating visibility. We sought the origin of these dysfunctions by measuring normal and amblyopic sensitivities to various forms of well-defined image distortion, namely band-limited phase quantization, phase quantization with additional amplitude modulation, and grey-scale modification. Our results prove the existence of an amblyopic quasi-blindness to image structure, that cannot be explained in terms of contrast detection. We discuss these findings within the computational scheme of image decomposition into local amplitude and local phase values. They are consistent with the assumption of amblyopic eyes being impaired in processing local phase but having the local amplitude (or “energy”, possibly at reduced gain) at their disposal. Phrased in physiological terms, we propose a scheme of complex-cells-only vision in amblyopia. We also provide a demonstration of how amblyopic eyes may see the test stimuli and natural images by generating local amplitude and phase representations at limited phase resolution. Copyright © 1996 Elsevier Science Ltd.

A new interband vector quantization of a human vision-based image representation is presented. The feature specific vector quantizer (FVQ) is suited for data compression beyond second-order decorrelation. The scheme is derived from statistical investigations of natural images and the processing principles of biological vision systems, the initial stage of the coding algorithm is a hierarchical, and orientation-selective, analytic bandpass decomposition, realized by even- and odd-symmetric filter pairs that are modeled after the simple cells of the visual cortex. The outputs of each even- and odd-symmetric filter pair are interpreted as real and imaginary parts of an analytic bandpass signal, which is transformed into a local amplitude and a local phase component according to the operation of cortical complex cells. Feature-specific multidimensional vector quantization is realized by combining the amplitude/phase samples of all orientation filters of one resolution layer. The resulting vectors are suited for a classification of the local image features with respect to their intrinsic dimensionality, and enable the exploitation of higher order statistical dependencies between the subbands. This final step is closely related to the operation of cortical hypercomplex or end-stopped cells. The codebook design is based on statistical as well as psychophysical and neurophysiological considerations, and avoids the common shortcomings of perceptually implausible mathematical error criteria. The resulting perceptual quality of compressed images is superior to that obtained with standard vector quantizers of comparable complexity.

We are interested in a bio-inspired representation of space that can be used for the design of autonomous systems and assistive devices. Experiments with subjects navigating in physically impossible environments indicate that the basic human representation of space cannot be map-like. Rather, it seems to be sensori motor in nature, combining sensory features with motor actions. We argue that such a sensori motor representation is important for the design of artificial systems that have to deal with spatial data. Here we demonstrate that this refers to both autonomous systems for spatial exploration and navigation and to systems for human assistance, in particular a navigation assistance and training system for people with Alzheimer's disease (AD). Disorientation and getting lost behavior are early signs of AD in most patients. The proposed system will focus on cognitive deficits associated with the disease by using and training skills that are still likely to be intact in the individual person. By means of guiding the attention to salient and autobiographically important landmarks, and practicing routes between them, the system will help to build up and train a more resilient sensori-motor representation. The user interface focuses on combining the visual scenes of the incorporated landmarks with a simplified map of the surrounding area.

In this paper we consider how the representation of images as surfaces, and their characterizations via surface differential forms, can be related to the concept of redundancy in the intensity signal. In contrast to common approaches, the basic surface types (planar, parabolic, elliptic/hyperbolic) are not seen as equal-priority classes, but as corresponding to different degrees of redundancy. This leads to a new approach to image representation and region labeling based upon generalized curvature measures. Furthermore, we employ different reconstruction algorithms to show that elliptic surface patches carry the significant information in natural images. Based upon deterministic and stochastic relaxation techniques, these algorithms allow one to reconstruct the original image from (i) "elliptic intensities" only and (ii) curvature measures which are zero for nonelliptic regions.

Determining one's position within the environment is a basic feature of spatial behavior and spatial cognition. This task is of inherently sensorimotor nature in that it results from a combination of sensory features and motor actions, where the latter comprise exploratory movements to different positions in the environment. Biological agents achieve this in a robust and effortless fashion, which prompted us to investigate a bio-inspired architecture to study the localization process of an artificial agent which operates in virtual spatial environments. The spatial representation in this architecture is based on sensorimotor features that comprise sensory sensory features as well as motor actions. It is hierarchically organized and its structure can be learned in an unsupervised fashion by an appropriate clustering rule. In addition, the architecture has a temporal belief update mechanism which explicitly utilizes the statistical correlations of actions and locations. The architecture is hybrid in integrating bottom-up processing of sensorimotor features with topdown reasoning which is able to select optimal motor actions based on the principle of maximum information gain. The architecture operates on two sensorimotor levels, a macro-level, which controls the movements of the agent in space, and on a micro-level, which controls its eye movements. As a result, the virtual mobile agent is able to localize itself within an environment using a minimum number of exploratory actions.

We present an approach for the direct detection of flow discontinuities which avoids explicit computation of a dense optic flow field. It is based on regarding the time varying image as a hypersurface in four-dimensional space and on using the Gaussian curvature properties of this hypersurface as a direct indicator for the presence of motion discontinuities. An easy to implement, nonlinear operator is suggested and possible extensions of the basic scheme are discussed.

Humans utilize region-based hierarchical representations in the context of navigation. We propose a computational model for representing region hierarchies and define criteria for automatically generating them. We devise a cognitively plausible online wayfinding algorithm exploiting the hierarchical decomposition given by regions. The algorithm allows an agent to derive plans with decreasing detail level along paths, enabling the agent to obtain the next action in logarithmic time and complete solutions in almost linear time. The resulting paths are reasonable approximations of optimal shortest paths.

We consider the coding properties of multilayer LNL (linear-nonlinear-linear) systems. Such systems consist of interleaved layers of linear transforms (or filter banks), nonlinear mappings, linear transforms, and so forth. They can be used as models of visual processing in higher cortical areas (V2, V4), and are also interesting with respect to image processing and coding. The linear filter operations in the different layers are optimized for the exploitation of the statistical redundancies of natural images. We explain why even simple nonlinear operations-like ON/OFF rectification-can convert higher-order statistical dependencies remaining between the linear filter coefficients of the first layer to a lower order. The resulting nonlinear coefficients can then be linearly recombined by the second-level filtering stage, using the same principles as in the first stage. The complete nonlinear scheme is invertible, i.e., information is preserved, if nonlinearities like ON/OFF rectification or gain control are employed. In order to obtain insights into the coding efficiency of these systems we investigate the feature selectivity of the resulting nonlinear output units and the use of LNL systems in image compression.

We investigate the hypothesis that the basic representation of space which underlies human navigation does not resemble an image-like map and is not restricted by the laws of Euclidean geometry. For this we developed a new experimental technique in which we use the properties of a virtual environment (VE) to directly influence the development of the representation. We compared the navigation performance of human observers under two conditions. Either the VE is consistent with the geometrical properties of physical space and could hence be represented in a map-like fashion, or it contains severe violations of Euclidean metric and planar topology, and would thus pose difficulties for the correct development of such a representation. Performance is not influenced by this difference, suggesting that a map-like representation is not the major basis of human navigation. Rather, the results are consistent with a representation which is similar to a non-planar graph augmented with path length information, or with a sensorimotor representation which combines sensory properties and motor actions. The latter may be seen as part of a revised view of perceptual processes due to recent results in psychology and neurobiology, which indicate that the traditional strict separation of sensory and motor systems is no longer tenable.

Humans are very efficient in the analysis, exploration and representation of their environment. Based on the neurobiological and cognitive principles of human information processing, we develop a system for the automatic identification and exploration of spatial configurations. The system sequentially selects "informative" regions (regions of interest), identifies the local structure, and uses this information for drawing efficient conclusions about the current scene. The selection process involves low-level, bottom-up processes for sensory feature extraction, and cognitive top-down processes for the generation of active motor commands that control the positioning of the sensors towards the most informative regions. Both processing levels have to deal with uncertain data, and have to take into account previous knowledge from statistical properties and learning. We suggest that this can be achieved in a hybrid architecture which integrates a nonlinear filtering stage modelled after the neural computations performed in the early stages of the visual system, and a cognitive reasoning strategy that operates in an adaptive fashion on a belief distribution.

The control of attention and of the associated saccadic fixations is to a large degree determined by the image content, that is, by salient local features in the image. The nature of this salient features and of the underlying strategy for their selection can be investigated within the context of natural scene statistics. For this, saccadic eye movements of human observers are recorded for a variety of natural and artificial test images, and the statistical properties of the fixated image regions are analyzed. The second-order statistics indicate that regions with higher spatial variance have a significantly higher probability to be fixated. Local contrast is therefore a salient feature. It was difficult to find additional differences in the local power spectra that can yield unequivocal information about a preference for specific local form properties. Differences that can be more easily interpreted as structural properties are derived from an investigation with higher-order statistics (bispectral density). The results indicate that nonredundant, intrinsically two-dimensional image features such as curved lines and edges, occlusions, isolated spots, and so on, play an important role in the saccadic selection process. Such features cannot be extracted by the classic spatial-frequency selective filter mechanisms but require nonlinear AND-like interactions between frequency components as possibly provided by end-stopping or hypercomplex properties related to the extraclassical receptive field.

Any mobile agent able to interact with moving objects or other mobile agents requires the ability to process motion shapes. The human visual system is an excellent, fast and proven machinery for dealing with such information. In order to obtain insight into the properties of this biological machine and to transfer it to artificial agents we analyze the limitations and capabilities of human perception of motion shapes. Here we present new empirical results on the classification, extrapolation and prediction of motion shape with varying degrees of complexity. In addition, results on the processing of multisensory spatio-temporal information will be presented. We make use of our earlier argument for the existence of a spatio-temporal memory in early vision and use the basic properties of this structure in the first layer of a neural network model. We discuss major architectural features of this network, which is based on Kohonens self-organizing maps. This network can be used as an interface to further representational stage on which motion vectors are implemented in a qualitative way. Both components of this hybrid model are constrained by the results gained in the psychophysical experiments.

Natural images contain considerable statistical redundancies beyond the level of second-order correlations. To identify the nature of these higher-order dependencies, we analyze the bispectra and trispectra of natural images. Our investigations reveal substantial statistical dependencies between those frequency components which are aligned to each other with respect to orientation. We argue that operators which are selective to local intrinsic dimensionality can optimally exploit such redundancies. We also show that the polyspectral structure we find for natural images helps to understand the hitherto unexplained superiority of orientation-selective filter decompositions over isotropic schemes like the Laplacian pyramid. However any essentially linear scheme can only partially exploit this higher-order redundancy. We therefore propose nonlinear i2D-selective operators which exhibit close resemblance to hypercomplex and end-stopped cells in the visual cortex. The function of these operators can be interpreted as a higher-order whitening of the input signal.

The standard model of visual processing is based on the selective properties of linear spatial filters which are tuned to different orientations and radial frequencies. This standard model is well suited for the description of a wide range of phenomena in vision but it is not clear, whether the whole range of basic properties of early vision is entirely within the models explanatory scope. Here we suggest that there exists a basic selective processing property in early vision which is definitely outside the explanatory scope of the standard model: the selectivity for intrinsically 2D signals. This property has already been observed in the classical experiments of Hubel and Wiesel, and has more recently been found in more complex form in the extra-classical receptive field properties of various visual neurons. We show here that this selectivity cannot be described within the framework of linear spatial filtering because of reasons which lie at the heart of the theory o f linear systems: the restriction of such systems to OR- combinations of their intrinsically 1D eigenfunctions. We present a general nonlinear framework for the modeling of i2D-selective systems which is based on AND-like combinations of frequency components, and which is closely related to the Wiener-Volterra representation of nonlinear systems. To our knowledge, i2D-selectivity is the only non- standard property for which such a theoretical framework yet exists. The framework enables the combination of the nonlinear i2D-selectivity with other basic selectivities of visual neurons, for examples with simple and complex-like properties, and makes it thus possible, to construct models for the variety of neurophysiological observations on the i2D-selective processing in visual neurons. As an insight of general interest for the recent discussion on second-order properties in early vision, the framework reveals the existence of extended equivalence classes in which nonlinear schemes can have very dissimilar structural properties, and lead nevertheless to identical input-output relations. Finally, there is a close relation between i2D-selectivity and the higher-order statistical redundancies in natural images.

Based on an information theoretical approach, we investigate feature selection processes in saccadic object and scene analysis. Saccadic eye movements of human observers are recorded for a variety of natural and artificial test images. These experimental data are used for a statistical evaluation of the fixated image regions. Analysis of second-order statistics indicates that regions with higher spatial variance have a higher probability to be fixated, but no significant differences beyond these variance effects could be found at the level of power spectra. By contrast, an investigation with higher-order statistics, as reflected in the bispectral density, yielded clear structural differences between the image regions selected by saccadic eye movements as opposed to regions selected by a random process. These results indicate that nonredundant, intrinsically two-dimensional image features like curved lines and edges, occlusions, isolated spots, etc. play an important role in the saccadic selection process which must be integrated with top-down knowledge to fully predict object and scene analysis by human observers.

In this paper we show that all images are topologically equivalent. Nevertheless, one can define useful pseudotopological properties that are related to what is usually referred to as topological perception. The computation of such properties involves low-level structures, which correspond to end-stopped and dot-responsive visual neurons. Our results contradict the common belief that the ability for perceiving topological properties must involve higher-order, cognitive processes.

Publisher Summary This chapter presents some arguments regarding the relevance of the concept of intrinsic dimensionality. It explains how intrinsic dimensionality is related to the statistical properties of natural images and to basic phenomena in visual neurophysiology and psychophysics. Related approaches in signal processing, with particular emphasis on the interpretation of images as surfaces are also discussed in. The chapter then presents the derivation of the conditions for the i2D selectivity of polynomial operators. It first illustrates that i2D operators cannot be linear and then considers the logical gating function of polynomial operators with respect to the processing of intrinsic dimensionality. The derivation of the necessary and sufficient conditions for quadratic i2D operators are also provided in this chapter. This chapter also presents examples of different types of i2D operators and discusses isotropic and orientation-selective versions of i2D operators, also exploring the relationship between intrinsic dimensionality and higher-order statistics.

The perception of an image by a human observer is usually modeled as a parallel process in which all parts of the image are treated more or less equivalently, but in reality the analysis of scenes is a highly selective procedure, in which only a small subset of image locations is processed by the precise and efficient neural machinery of foveal vision. To understand the principles behind this selection of the 'informative' regions of images we have developed a hybrid system, which consists of a combination of a knowledge-based reasoning system wit a low-level preprocessing by linear and nonlinear neural operators. This hybrid system is intended as a first step towards a compete model of the sensorimotor system of saccadic scene analysis. In the analysis of a scene, the system calculates in each step which eye movement has to be made to reach a maximum of information about the scene. The possible information gain is calculated by means of a parallel strategy which is suitable for adaptive reasoning. The output of the system is a fixation sequence, and finally, a hypothesis about the scene.

The classical approach in vision research - the derivation of basically linear filter models form experiments with simple artificial test stimuli - is currently undergoing a major revision. Instead of trying to keep the dirty environment out of our clean labs we put it now right into the focus of scientific exploration. The new approach has a close relation to basic engineering strategies for electronic image processing since its major concept is the exploration of the statistical redundancies of the environment by appropriate neural transformations. The standard engineering methods are not sufficient, however. Even a basic biological feature like orientation selectivity requires the consideration of higher-order statistics, like cumulants or polyspectra. Furthermore, there exists an abundance of nonlinear phenomena in biological vision, for example the phase-invariance of complex cells, cortical gain control, or end-stopping, which make it necessary to consider unconventional modeling approaches like differential geometry or Volterra-Wiener system. By use of such methods we cannot only gain a deeper understanding of the adaption of the visual system to the complex natural environment, but we can also make the biological system an inspiring source for the design of novel strategies in electronic image processing.

The processing and representation of motion information is addressed from an integrated perspective comprising low- level signal processing properties as well as higher-level cognitive aspects. For the low-level processing of motion information we argue that a fundamental requirement is the existence of a spatio-temporal memory. Its key feature, the provision of an orthogonal relation between external time and its internal representation, is achieved by a mapping of temporal structure into a locally distributed activity distribution accessible in parallel by higher-level processing stages. This leads to a reinterpretation of the classical concept of `iconic memory' and resolves inconsistencies on ultra-short-time processing and visual masking. The spatial-temporal memory is further investigated by experiments on the perception of spatio-temporal patterns. Results on the direction discrimination of motion paths provide evidence that information about direction and location are not processed and represented independent of each other. This suggests a unified representation on an early level, in the sense that motion information is internally available in form of a spatio-temporal compound. For the higher-level representation we have developed a formal framework for the qualitative description of courses of motion that may occur with moving objects.