Search

Semantic segmentation is a crucial task in vision measurement systems that involves understanding and segmenting different objects and regions within an image. Over the years, numerous RGB-D semantic segmentation methods have been developed, leveraging the encoder-decoder architecture to achieve outstanding performance. However, existing methods have two main problems that constrain further performance improvement. Firstly, in the encoding stage, existing methods have a weak ability to fuse cross-modal information, and low-quality depth maps can easily lead to poor feature representation. Secondly, in the decoding stage, the upsampling of high-level semantic information may cause the loss of contextual information, and low-level features from the encoder may bring noises to the decoder through skip connections. To solve these issues, we propose a novel Encoding Fusion and Decoding Correction Network (EFDCNet) for RGB-D indoor semantic segmentation. First, in the encoding stage of EFDCNet, we focus on extracting valuable information from low-quality depth maps, and employ a channel-wise filter to select informative depth features. Additionally, we establish the global dependencies between RGB and depth features via the self-attention mechanism to enhance the cross-modal feature interactions, extracting discriminant and powerful features. Then, in the decoding stage of EFDCNet, we use the highest-level information as semantic guidance to compensate for the upsampling information and filter out noise from the low-level encoder features propagated through the skip connections to the decoder. Extensive experiments conducted on two widely-used RGB-D indoor semantic segmentation datasets demonstrate that the proposed EFDCNet surpasses the performance of relevant state-of-the-art methods. The code is available at https://github.com/ Mark9010/EFDCNet • EFDCNet achieves encoding fusion and decoding correction. • EFM efficiently integrates features in the encoder from different modalities. • DCM corrects features in the decoder with the semantic information. • EFDCNet gains great improvements in the public datasets. [ABSTRACT FROM AUTHOR]

• We propose a VFNet for semi-supervised saliency prediction in omnidirectional images (ODIs). • We propose a new form of perturbation, i.e., DropView, to fully use unlabeled ODIs. • We propose a VFAI module to fit the distribution of visual saliency in different viewports in ODIs. • Our VFNet achieves competitive performance as compared with 10 SOTA methods. Compared with the annotated data for the 2D image saliency prediction task, the annotated data for training omnidirectional image (or 360 ° image) saliency prediction models are not sufficient. Most existing fully-supervised saliency prediction methods for omnidirectional images (ODIs) adopt a scheme, first training the methods on a labeled large 2D image saliency prediction dataset and then fine-tuning the methods on the labeled tiny ODI saliency prediction dataset. However, this strategy is time-consuming and may not inadequately mine the visual features built in ODIs. To explore the visual attributes targeted at ODIs and address the shortage of labels on these ODIs, in this paper, we propose an end-to-end semi-supervised network, namely VFNet, which relies on viewport features and only utilizes ODIs as training data, for ODI saliency prediction. Concretely, we adopt consistency regularization as our semi-supervised learning framework. The predictions between main and auxiliary saliency inference networks in the VFNet enforce consistency. Aiming at ODIs, we introduce a new form of perturbation, i.e. , DropView, to improve the effectiveness of consistency regularization. By randomly dropping out different 360 ° cubemap viewport features before the auxiliary saliency inference network, the proposed DropView enhances the robustness of the final ODI saliency prediction. To adaptively interact with the equirectangular and different cubemap viewport features according to their contributions, we introduce a Viewport Feature Adaptive Integration (VFAI) module and deploy the VFAI module at different levels in the VFNet to raise the capacity of feature encoding of our VFNet. Compared with state-of-the-art fully-supervised methods, our VFNet with fewer labeled training data achieves competitive performance demonstrated by extensive experiments on two publicly available datasets. [ABSTRACT FROM AUTHOR]

Most of the deep convolutional neural networks (CNNs) based RGBD saliency models either regard the RGB and depth cues as the same status or trust the depth information excessively. However, they ignore that the low-quality depth map is an interference factor and the multi-level deep features that originated from RGB images contain abundant appearance information. Therefore, we propose a novel RGBD saliency model, where the attention-guided bottom-up and top-down modules are powerfully combined by using multi-level deep RGB features, to utilize the deep RGB and depth features in a sufficient and appropriate way. Concretely, a two-stream structure based bottom-up module is first constructed to dig and fuse the RGB and depth information, yielding the fused deep feature. Besides, the module embeds the depth cue based attention maps to guide the indication of salient objects. Then, considering the abundant appearance information, a top-down module is deployed to perform coarse-to-fine saliency inference, where the fused deep feature is progressively integrated with appearance information. Similarly, the attention map is also inserted into this module for locating salient objects. Extensive experiments are performed on five public RGBD datasets and the corresponding experimental results firmly demonstrate the effectiveness and superiority of our model when compared with the state-of-the-art RGBD saliency models. [ABSTRACT FROM AUTHOR]

Audio-visual saliency prediction (AVSP) is a task that aims to model human attention patterns in the perception of auditory and visual scenes. Given the challenges associated with perceiving and combining multi-modal saliency features from videos, this paper presents a multi-sensory framework for AVSP. This framework is designed to extract audio, motion and image saliency features and integrate them effectively, which can then serve as a general architecture for the AVSP task. To obtain multi-sensory information, we develop a three-stream encoder that extracts audio, motion and image saliency features. In particular, we utilize a pre-trained encoder with knowledge related to image saliency to extract saliency features for each frame. The image saliency features are then incorporated with motion features using a spatial attention module. For motion features, 3D convolutional neural networks (CNNs) like S3D are commonly used in AVSP models. However, these networks are unable to effectively capture the global motion relationship in videos. To tackle this problem, we incorporate Transformer- and MLP-based motion encoders into the AVSP models. To learn joint audio-visual representations, an audio-visual fusion block is exploited to enhance the correlation between audio and visual motion features under the supervision of a cosine similarity loss in a self-supervised manner. Finally, a multi-stage decoder integrates audio, motion and image saliency features to generate the final saliency map. We evaluate our methods on six audio-visual eye-tracking datasets. Experimental results demonstrate that our method achieves compelling performance compared to the state-of-the-art methods. The source code is available at https://github.com/oraclefina/MSPI. • A three-stream multisensory framework for audio-visual saliency prediction. • Image saliency model offers image saliency for feature fusion with dynamic saliency. • A self-supervised audio-visual fusion block fuses auditory and visual features. • Our method is examined with various motion encoders of 3D CNN, Transformer and MLP. [ABSTRACT FROM AUTHOR]