Search

Abstract Interacting with objects in our environment requires determining their locations, often with respect to surrounding objects (i.e., allocentrically). According to the scene grammar framework, these usually small, local objects are movable within a scene and represent the lowest level of a scene’s hierarchy. How do higher hierarchical levels of scene grammar influence allocentric coding for memory-guided actions? Here, we focused on the effect of large, immovable objects (anchors) on the encoding of local object positions. In a virtual reality study, participants (n = 30) viewed one of four possible scenes (two kitchens or two bathrooms), with two anchors connected by a shelf, onto which were presented three local objects (congruent with one anchor) (Encoding). The scene was re-presented (Test) with 1) local objects missing and 2) one of the anchors shifted (Shift) or not (No shift). Participants, then, saw a floating local object (target), which they grabbed and placed back on the shelf in its remembered position (Response). Eye-tracking data revealed that both local objects and anchors were fixated, with preference for local objects. Additionally, anchors guided allocentric coding of local objects, despite being task-irrelevant. Overall, anchors implicitly influence spatial coding of local object locations for memory-guided actions within naturalistic (virtual) environments.

Interacting with objects in our environment requires determining their locations, often with respect to surrounding objects (i.e., allocentrically). According to the scene grammar framework, these usually small, local objects are movable within a scene and represent the lowest level of a scene's hierarchy. How do higher hierarchical levels of scene grammar influence allocentric coding for memory-guided actions? Here, we focused on the effect of large, immovable objects (anchors) on the encoding of local object positions. In a virtual reality study, participants (n = 30) viewed one of four possible scenes (two kitchens or two bathrooms), with two anchors connected by a shelf, onto which were presented three local objects (congruent with one anchor) (Encoding). The scene was re-presented (Test) with 1) local objects missing and 2) one of the anchors shifted (Shift) or not (No shift). Participants, then, saw a floating local object (target), which they grabbed and placed back on the shelf in its remembered position (Response). Eye-tracking data revealed that both local objects and anchors were fixated, with preference for local objects. Additionally, anchors guided allocentric coding of local objects, despite being task-irrelevant. Overall, anchors implicitly influence spatial coding of local object locations for memory-guided actions within naturalistic (virtual) environments. [ABSTRACT FROM AUTHOR]

Describing our visual environments is challenging because although an enormous amount of information is simultaneously available to the visual system, the language channel must impose a linear order on that information. Moreover, the production system is at least moderately incremental, meaning that it interleaves planning and speaking processes. Here, we address how the operations of these two cognitive systems are coordinated given their different characteristics. We propose the concept of a perceptual clause, defined as an interface representation that allows the visual and linguistic systems to exchange information. The perceptual clause serves as the input to the language formulator, which translates the representation into a linguistic sequence. Perceptual clauses capture speakers’ ability to describe visual scenes coherently while at the same time taking advantage of the incremental abilities of the language production system. [ABSTRACT FROM AUTHOR]

This work investigates the linearisation strategies used by speakers when describing real-world scenes to better understand production plans for multi-utterance sequences. In this study, 30 participants described real-world scenes aloud. To investigate which semantic features of scenes predict order of mention, we quantified three features (meaning, graspability, and interactability) using two techniques (whole-object ratings and feature map values). We found that object-level semantic features, namely those affordance-based, predicted order of mention in a scene description task. Our findings provide the first evidence for an object-related semantic feature that guides linguistic ordering decisions and offers theoretical support for the role of object semantics in scene viewing and description. [ABSTRACT FROM AUTHOR]

• Existing learning-based color constancy methods typically exhibit poor generalization between images taken by varying cameras. • Gray-based assumptions hold the natural property of camera-independent. • Gray-based assumptions are less accurate in different scenarios due to their fixity. • Scene semantics can contribute to inferring and modifying fixed gray-based assumptions. Current learning-based color constancy methods are typically employed to find camera-specific illuminant mappings. Consequently, these methods exhibit poor generalization to images captured by varying cameras. In this paper, we present a camera-independent learning method based on Scene Semantics, and we call it CISS. Inspired by the camera-independent property of gray-based methods, CISS does not directly estimate camera-specific illuminant by training model as most learning methods do. Instead, the model's output is transformed into camera-independent scene statistics related to gray-based assumptions to avoid being affected by camera variations. Based on these estimated scene statistics, illuminant can be calculated indirectly. To estimate scene statistics accurately, CISS designs illuminant-invariance scene semantics features as input to the model. Then, the model estimates scene statistics for each input image in terms of scene semantics with exemplar-based learning. Experiments show that, on several public datasets, CISS is able to outperform present methods for multi-cameras color constancy, and is flexible enough to be well generalized to the unseen camera without fine-tuning by additional images. [ABSTRACT FROM AUTHOR]

Abstract Scene perception and trajectory forecasting are two fundamental challenges that are crucial to a safe and reliable autonomous driving (AD) system. However, most proposed methods aim at addressing one of the two challenges mentioned above with a single model. To tackle this dilemma, this paper proposes spatio‐temporal semantics and interaction graph aggregation for multi‐agent perception and trajectory forecasting (ST‐SIGMA), an efficient end‐to‐end method to jointly and accurately perceive the AD environment and forecast the trajectories of the surrounding traffic agents within a unified framework. ST‐SIGMA adopts a trident encoder–decoder architecture to learn scene semantics and agent interaction information on bird’s‐eye view (BEV) maps simultaneously. Specifically, an iterative aggregation network is first employed as the scene semantic encoder (SSE) to learn diverse scene information. To preserve dynamic interactions of traffic agents, ST‐SIGMA further exploits a spatio‐temporal graph network as the graph interaction encoder. Meanwhile, a simple yet efficient feature fusion method to fuse semantic and interaction features into a unified feature space as the input to a novel hierarchical aggregation decoder for downstream prediction tasks is designed. Extensive experiments on the nuScenes data set have demonstrated that the proposed ST‐SIGMA achieves significant improvements compared to the state‐of‐the‐art (SOTA) methods in terms of scene perception and trajectory forecasting, respectively. Therefore, the proposed approach outperforms SOTA in terms of model generalisation and robustness and is therefore more feasible for deployment in real‐world AD scenarios.

Scene perception and trajectory forecasting are two fundamental challenges that are crucial to a safe and reliable autonomous driving (AD) system. However, most proposed methods aim at addressing one of the two challenges mentioned above with a single model. To tackle this dilemma, this paper proposes spatio‐temporal semantics and interaction graph aggregation for multi‐agent perception and trajectory forecasting (ST‐SIGMA), an efficient end‐to‐end method to jointly and accurately perceive the AD environment and forecast the trajectories of the surrounding traffic agents within a unified framework. ST‐SIGMA adopts a trident encoder–decoder architecture to learn scene semantics and agent interaction information on bird's‐eye view (BEV) maps simultaneously. Specifically, an iterative aggregation network is first employed as the scene semantic encoder (SSE) to learn diverse scene information. To preserve dynamic interactions of traffic agents, ST‐SIGMA further exploits a spatio‐temporal graph network as the graph interaction encoder. Meanwhile, a simple yet efficient feature fusion method to fuse semantic and interaction features into a unified feature space as the input to a novel hierarchical aggregation decoder for downstream prediction tasks is designed. Extensive experiments on the nuScenes data set have demonstrated that the proposed ST‐SIGMA achieves significant improvements compared to the state‐of‐the‐art (SOTA) methods in terms of scene perception and trajectory forecasting, respectively. Therefore, the proposed approach outperforms SOTA in terms of model generalisation and robustness and is therefore more feasible for deployment in real‐world AD scenarios. [ABSTRACT FROM AUTHOR]

It is a worldwide witnessed fact that traditional manual driving mechanism will be superseded by Autonomous Vehicles [AVs] in coming years. Autonomous vehicles are going to be most foreseen development in the automotive industry. That would require Decision Making System which will enable AVs to intuitively interpret the real-time situations around. Most importantly scene recognition on streets & extracting relevant semantics from the scene is challenging task. So, image classification & object detection techniques using Deep Convolutional Neural Networks [DCNN] are going to play vital role in every other methodology designed for scene semantics extraction. As per the extracted scene semantics DMS actuates the necessary devices which control the speed of vehicle & steering angel. So for that matter information extraction from road scene images covering all aspects to take intuitive decisions has huge concern with overall performance of the AV's. [ABSTRACT FROM AUTHOR]

Abstract A major problem in human cognition is to understand how newly acquired information and long-standing beliefs about the environment combine to make decisions and plan behaviors. Over-dependence on long-standing beliefs may be a significant source of suboptimal decision-making in unusual circumstances. While the contribution of long-standing beliefs about the environment to search in real-world scenes is well-studied, less is known about how new evidence informs search decisions, and it is unclear whether the two sources of information are used together optimally to guide search. The present study expanded on the literature on semantic guidance in visual search by modeling a Bayesian ideal observer’s use of long-standing semantic beliefs and recent experience in an active search task. The ability to adjust expectations to the task environment was simulated using the Bayesian ideal observer, and subjects’ performance was compared to ideal observers that depended on prior knowledge and recent experience to varying degrees. Target locations were either congruent with scene semantics, incongruent with what would be expected from scene semantics, or random. Half of the subjects were able to learn to search for the target in incongruent locations over repeated experimental sessions when it was optimal to do so. These results suggest that searchers can learn to prioritize recent experience over knowledge of scenes in a near-optimal fashion when it is beneficial to do so, as long as the evidence from recent experience was learnable.

In real-world vision, people prioritise the most informative scene regions via eye-movements. According to the cognitive guidance theory of visual attention, viewers allocate visual attention to those parts of the scene that are expected to be the most informative. The expected information of a scene region is coded in the semantic distribution of that scene. Meaning maps have been proposed to capture the spatial distribution of local scene semantics in order to test cognitive guidance theories of attention. Notwithstanding the success of meaning maps, the reason for their success has been contested. This has led to at least two possible explanations for the success of meaning maps in predicting visual attention. On the one hand, meaning maps might measure scene semantics. On the other hand, meaning maps might measure scene features, overlapping with, but distinct from, scene semantics. This study aims to disentangle these two sources of information by considering both conceptual information and non-semantic scene entropy simultaneously. We found that both semantic and non-semantic information is captured by meaning maps, but scene entropy accounted for more unique variance in the success of meaning maps than conceptual information. Additionally, some explained variance was unaccounted for by either source of information. Thus, although meaning maps may index some aspect of semantic information, their success seems to be better explained by non-semantic information. We conclude that meaning maps may not yet be a good tool to test cognitive guidance theories of attention in general, since they capture non-semantic aspects of local semantic density and only a small portion of conceptual information. Rather, we suggest that researchers should better define the exact aspect of cognitive guidance theories they wish to test and then use the tool that best captures that desired semantic information. As it stands, the semantic information contained in meaning maps seems too ambiguous to draw strong conclusions about how and when semantic information guides visual attention. [ABSTRACT FROM AUTHOR]

Real-world scenes contain low-level visual features (e.g.,edges, colors) and high-level semantic features (e.g., objectsand places). Traditional visual perception models assume thatintegration of low-level visual features and segmentation of thescene must occur before high-level semantics are perceived.This view implies that low-level visual features of a scenealone do not carry semantic information related to that scene.Here we present evidence that suggests otherwise. We showthat high-level semantics can be preserved in low-level visualfeatures, and that different high-level semantics can bepreserved in different types of low-level visual features.Specifically, the ‘disorder’ of a scene is preserved in edgefeatures better than color features, whereas the converse is truefor ‘naturalness.’ These findings suggest that semanticprocessing may start earlier than thought before, andintegration of low-level visual features and segmentation of thescene may occur after semantic processing has begun, or inparallel.

Most existing image captioning methods use only the visual information of the image to guide the generation of captions, lack the guidance of effective scene semantic information, and the current visual attention mechanism cannot adjust the focus intensity on the image. In this article, we first propose an improved visual attention model. At each timestep, we calculated the focus intensity coefficient of the attention mechanism through the context information of themodel, then automatically adjusted the focus intensity of the attention mechanism through the coefficient to extract more accurate visual information. In addition, we represented the scene semantic knowledge of the image through topic words related to the image scene, then added them to the language model. We used the attention mechanism to determine the visual information and scene semantic information that the model pays attention to at each timestep and combined them to enable the model to generate more accurate and scene-specific captions. Finally, we evaluated our model on Microsoft COCO (MSCOCO) and Flickr30k standard datasets. The experimental results show that our approach generates more accurate captions and outperforms many recent advanced models in various evaluation metrics. [ABSTRACT FROM AUTHOR]

A major problem in human cognition is to understand how newly acquired information and long-standing beliefs about the environment combine to make decisions and plan behaviors. Over-dependence on long-standing beliefs may be a significant source of suboptimal decision-making in unusual circumstances. While the contribution of long-standing beliefs about the environment to search in real-world scenes is well-studied, less is known about how new evidence informs search decisions, and it is unclear whether the two sources of information are used together optimally to guide search. The present study expanded on the literature on semantic guidance in visual search by modeling a Bayesian ideal observer's use of long-standing semantic beliefs and recent experience in an active search task. The ability to adjust expectations to the task environment was simulated using the Bayesian ideal observer, and subjects' performance was compared to ideal observers that depended on prior knowledge and recent experience to varying degrees. Target locations were either congruent with scene semantics, incongruent with what would be expected from scene semantics, or random. Half of the subjects were able to learn to search for the target in incongruent locations over repeated experimental sessions when it was optimal to do so. These results suggest that searchers can learn to prioritize recent experience over knowledge of scenes in a near-optimal fashion when it is beneficial to do so, as long as the evidence from recent experience was learnable. [ABSTRACT FROM AUTHOR]

In this paper, we define a new method for analyzing object-scene contextual relationships using computational linguistics: Linguistic Analysis of Scene Semantics, or LASS. LASS uses linguistic semantic similarity relationships between scene object and context labels embedded in a vector-space language model: Facebook Research's fastText. Importantly, the use of fastText permits semantic similarity score calculation between any set of strings and thus elements of any set of image data for which labels are available. Scene semantic similarity scores are then embedded in object segmentation mask locations in the image, creating a semantic similarity map. LASS can also be fully automated by generating context and object labels, as well as object segmentation masks, using deep learning. We compare semantic similarity maps between human- and neural network-generated annotations on a corpus of images taken from the LabelMe database. Semantic similarity maps produced by the fully automated LASS have a number of desirable properties, while maintaining a high degree of spatial and semantic similarity to them. Finally, we use LASS to evaluate the distribution of semantically consistent scene elements in space. Both show relatively uniform distributions of semantic relatedness to scene context, suggesting that contextually appropriate objects are likely to be found in all image regions. Taken together, these results suggest that LASS is accurate, automatic, flexible, and useful in a number of research contexts such as scene grammar and novelty detection. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Zhejiang University (Science Edition) is the property of Journal of Zhejiang University (Science Edition) Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Speaking (1989) inspired research on topics such as word selection, syntactic formulation, and dialogue, but an issue that remains understudied is linearisation: the question of how speakers organise a series of utterances into a coherent sequence, allowing both speaker and listener to keep track of what has been said and what will come next. In this paper we describe a new line of research investigating linearisation during scene description tasks, and we argue that, as Pim Levelt suggested in 1981 and in the 1989 book, the need to linearise arises from attentional constraints in the language system. Our work shows that attentional, visual, and linguistic processes are flexibly coordinated during scene descriptions, and that speakers not only respond to what the eye sees, but also to what the mind anticipates finding in the visual world. [ABSTRACT FROM AUTHOR]

With the increasing adoption of autonomous mobile robots in the construction industry, accurate localization and mapping in dynamic construction environments have become paramount. This is typically tackled via Simultaneous Localization and Mapping (SLAM) techniques. Primarily designed for static environments, traditional SLAM systems struggle to maintain robustness and accuracy in dynamic settings. To address this challenge, this study presents an enhanced visual SLAM system specifically tailored for dynamic construction environments. The proposed system, named vSLAM-Con, introduces an adaptive dynamic object segmentation method, utilizing an innovative AD-keyframes selection mechanism grounded on optical flow magnitude to diminish computational overhead while preserving competitive tracking accuracy. Additionally, a semantic-based feature update process is developed, leveraging scene understanding and continuous observation to augment the reliability of tracking features. This system's performance, evaluated on both an established public benchmark and a custom construction dataset, shows substantial improvements over the baseline and competitive results with the state-of-the-art algorithms. More importantly, it largely reduces the processing time compared to state-of-the-arts, demonstrating robust tracking performance even under highly dynamic conditions. The findings highlight the system's potential to contribute significantly to autonomous robotics in construction, offering more accurate navigation and interaction capabilities in complex, ever-changing environments. [ABSTRACT FROM AUTHOR]

We propose a novel framework for semantically enhancing a 3D indoor scene in agreement with a user-provided guide word. To do so, we make changes to furniture colors and place small objects in the scene. The relevance of specific furniture colors and small objects to each guide word is learned from a database of annotated images, taking into account both their frequency and specificity to that guide word. Enhancement suggestions are generated by optimizing a scoring function, which combines the relevance of both enhancement factors, i.e., furniture colors and small objects. During optimization, a submodular set function is adopted to ensure that a diverse set of enhancement suggestions is produced. Our experiments show that this framework can generate enhancement suggestions that are both compatible with the input guide word, and comparable to ones designed by humans. [ABSTRACT FROM AUTHOR]

People have an amazing ability to identify objects and scenes with only a glimpse. How automatic is this scene and object identification? Are scene and object semantics-let alone their semantic congruity-processed to a degree that modulates ongoing gaze behavior even if they are irrelevant to the task at hand? Objects that do not fit the semantics of the scene (e.g., a toothbrush in an office) are typically fixated longer and more often than objects that are congruent with the scene context. In this study, we overlaid a letter T onto photographs of indoor scenes and instructed participants to search for it. Some of these background images contained scene-incongruent objects. Despite their lack of relevance to the search, we found that participants spent more time in total looking at semantically incongruent compared to congruent objects in the same position of the scene. Subsequent tests of explicit and implicit memory showed that participants did not remember many of the inconsistent objects and no more of the consistent objects. We argue that when we view natural environments, scene and object relationships are processed obligatorily, such that irrelevant semantic mismatches between scene and object identity can modulate ongoing eye-movement behavior. [ABSTRACT FROM AUTHOR]

Many studies suggest a large capacity memory for briefly presented pictures of whole scenes. At the same time, visual working memory of scene elements is limited to only a few items. We examined the role of retroactive interference in limiting memory for visual details. Participants viewed a scene for five seconds and then, after a short delay containing either a blank screen or ten distracter scenes, answered questions about the location, color and identity of objects in the scene. We found that the influence of the distracters depended on whether they were from a similar semantic domain, such as kitchen or airport. Increasing the number of similar scenes reduced, and eventually eliminated, memory for scene details. Although scene memory was firmly established over the initial study period, this memory was fragile and susceptible to interference. This may help to explain the discrepancy in the literature between studies showing limited visual working memory and those showing a large capacity memory for scenes.

3D scene shape retrieval is a brand new but important research direction in content-based 3D shape retrieval. To promote this research area, two Shape Retrieval Contest (SHREC) tracks on 2D scene sketch-based and image-based 3D scene model retrieval have been organized by us in 2018 and 2019, respectively. In 2018, we built the first benchmark for each track which contains 2D and 3D scene data for ten (10) categories, while they share the same 3D scene target dataset. Four and five distinct 3D scene shape retrieval methods have competed with each other in these two contests, respectively. In 2019, to measure and compare the scalability performance of the participating and other promising Query-by-Sketch or Query-by-Image 3D scene shape retrieval methods, we built a much larger extended benchmark for each type of retrieval which has thirty (30) classes and organized two extended tracks. Again, two and three different 3D scene shape retrieval methods have contended in these two tracks, separately. To solicit state-of-the-art approaches, we perform a comprehensive comparison of all the above methods and an additional new retrieval methods by evaluating them on the two benchmarks. The benchmarks, evaluation results and tools are publicly available at our track websites (Yuan et al., 2019 [1]; Abdul-Rashid et al., 2019 [2]; Yuan et al., 2019 [3]; Abdul-Rashid et al., 2019 [4]), while code for the evaluated methods are also available: http://github.com/3DSceneRetrieval. • Build a basic and an extended sketch/image-based 3D scene retrieval benchmark. • Evaluate 14 top sketch/image-based 3D scene retrieval methods on the two benchmarks. • Solicit and identify the state-of-the-art methods and promising related techniques. • Conduct cross-benchmark and scalability performance evaluations. • Make evaluation results and source code of the evaluated methods publicly available. [ABSTRACT FROM AUTHOR]

Existing color constancy methods are all based on specific assumptions such as the spatial and spectral characteristics of images. As a consequence, no algorithm can be considered as universal. However, with the large variety of available methods, the question is how to select the method that performs best for a specific image. To achieve selection and combining of color constancy algorithms, in this paper natural image statistics are used to identify the most important characteristics of color images. Then, based on these image characteristics, the proper color constancy algorithm (or best combination of algorithms) is selected for a specific image. To capture the image characteristics, the Weibull parameterization (e.g., grain size and contrast) is used. It is shown that the Weibull parameterization is related to the image attributes to which the used color constancy methods are sensitive. An MoG-classifier is used to learn the correlation and weighting between the Weibull-parameters and the image attributes (number of edges, amount of texture, and SNR). The output of the classifier is the selection of the best performing color constancy method for a certain image. Experimental results show a large improvement over state-of-the-art single algorithms. On a data set consisting of more than 11,000 images, an increase in color constancy performance up to 20 percent (median angular error) can be obtained compared to the best-performing single algorithm. Further, it is shown that for certain scene categories, one specific color constancy algorithm can be used instead of the classifier considering several algorithms. [ABSTRACT FROM PUBLISHER]

Memorizing critical objects and their locations is an essential part of everyday life. In the present study, incidental encoding of objects in naturalistic scenes during search was compared to explicit memorization of those scenes. To investigate if prior knowledge of scene structure influences these two types of encoding differently, we used meaningless arrays of objects as well as objects in real-world, semantically meaningful images. Surprisingly, when participants were asked to recall scenes, their memory performance was markedly better for searched objects than for objects they had explicitly tried to memorize, even though participants in the search condition were not explicitly asked to memorize objects. This finding held true even when objects were observed for an equal amount of time in both conditions. Critically, the recall benefit for searched over memorized objects in scenes was eliminated when objects were presented on uniform, non-scene backgrounds rather than in a full scene context. Thus, scene semantics not only help us search for objects in naturalistic scenes, but appear to produce a representation that supports our memory for those objects beyond intentional memorization., (© 2014 ARVO.)