Search

This paper proposes a network, referred to as Multi-View Stereo TRansformer (MVSTR) for depth estimation from multi-view images. By modeling long-range dependencies and epipolar geometry, the proposed MVSTR is capable of extracting dense features with global context and 3D consistency, which are crucial for reliable matching in Multi-View Stereo (MVS). Specifically, to tackle the problem of the limited receptive field of existing CNN-based MVS methods, a global-context Transformer module is designed to establish intra-view long-range dependencies so that global contextual features of each view are obtained. In addition, to further enable features of each view to be 3D-consistent, a 3D-consistency Transformer module with an epipolar feature sampler is built, where epipolar geometry is modeled to effectively facilitate cross-view interaction. Experimental results show that the proposed MVSTR achieves the best overall performance on the DTU dataset and demonstrates strong generalization on the Tanks & Temples benchmark dataset.

In video coding, it is a challenge to deal with scenes with complex motions, such as rotation and zooming. Although affine motion compensation (AMC) is employed in Versatile Video Coding (VVC), it is still difficult to handle non-translational motions due to the adopted hand-craft block-based motion compensation. In this paper, we propose a deep affine motion compensation network (DAMC-Net) for inter prediction in video coding to effectively improve the prediction accuracy. To the best of our knowledge, our work is the first attempt to deal with the deformable motion compensation based on CNN in VVC. Specifically, a deformable motion-compensated prediction (DMCP) module is proposed to compensate the current encoding block through a learnable way to estimate accurate motion fields. Meanwhile, the spatial neighboring information and the temporal reference block as well as the initial motion field are fully exploited. By effectively fusing the multi-channel feature maps from DMCP, an attention-based fusion and reconstruction (AFR) module is designed to reconstruct the output block. The proposed DAMC-Net is integrated into VVC and the experimental results demonstrate that the proposed method considerably enhances the coding performance.

Pepper (Capsicum) are consumed worldwide as vegetables and food additives due to their pungent taste. Capsaicinoids are the bioactive compounds that confer the desired pungency to pepper fruits. Capsaicinoid biosynthesis was thought to occur exclusively in fruit placenta. Recently, biosynthesis in the pericarp of extremely pungent varieties was discovered, however, the mechanism of capsaicinoid biosynthesis regulation in the pericarp remains largely unknown. Here, the capsaicinoid contents of placenta and pericarp were analyzed. The results indicated that the Capsicum chinense pericarp accumulated a vast amount of capsaicinoids. Expression of the master regulator MYB31 and capsaicinoid biosynthesis genes (CBGs) were significantly upregulated in the pericarp in C. chinense accessions compared to accessions in other tested species. Moreover, in fruit of extremely-pungent 'Trinidad Moruga Scorpion' (C. chinense) and low-pungent '59' inbred line (C. annuum), the capsaicinoid accumulation patterns in the pericarp were consistent with expression levels of CBGs and MYB31. Silencing MYB31 in 'Trinidad Moruga Scorpion' pericarp leads to a significantly decreased CBGs transcription level and capsaicinoids content. Taken together, our results provide insights into the molecular mechanism arising from the expression of MYB31 in the pericarp that results in exceedingly hot peppers.

Visual quality of images captured by mobile devices is often inferior to that of images captured by a Digital Single Lens Reflex (DSLR) camera. This paper presents a novel generative adversarial network-based mobile image enhancement method, referred to as MIEGAN. It consists of a novel multi-module cascade generative network and a novel adaptive multi-scale discriminative network. The multi-module cascade generative network is built upon a two-stream encoder, a feature transformer, and a decoder. In the two-stream encoder, a luminance-regularizing stream is proposed to help the network focus on low-light areas. In the feature transformation module, two networks effectively capture both global and local information of an image. To further assist the generative network to generate the high visual quality images, a multi-scale discriminator is used instead of a regular single discriminator to distinguish whether an image is fake or real globally and locally. To balance the global and local discriminators, an adaptive weight allocation is proposed. In addition, a contrast loss is proposed, and a new mixed loss function is developed to improve the visual quality of the enhanced images. Extensive experiments on the popular DSLR photo enhancement dataset and MIT-FiveK dataset have verified the effectiveness of the proposed MIEGAN.

Magnetic resonance imaging (MRI) can provide multi-modality MR images by setting task-specific scan parameters, and has been widely used in various disease diagnosis and planned treatments. However, in practical clinical applications, it is often difficult to obtain multi-modality MR images simultaneously due to patient discomfort, and scanning costs, etc. Therefore, how to effectively utilize the existing modality images to synthesize missing modality image has become a hot research topic. In this paper, we propose a novel confidence-guided aggregation and cross-modality refinement network (CACR-Net) for multi-modality MR image synthesis, which effectively utilizes complementary and correlative information of multiple modalities to synthesize high-quality target-modality images. Specifically, to effectively utilize the complementary modality-specific characteristics, a confidence-guided aggregation module is proposed to adaptively aggregate the multiple target-modality images generated from multiple source-modality images by using the corresponding confidence maps. Based on the aggregated target-modality image, a cross-modality refinement module is presented to further refine the target-modality image by mining correlative information among the multiple source-modality images and aggregated target-modality image. By training the proposed CACR-Net in an end-to-end manner, high-quality and sharp target-modality MR images are effectively synthesized. Experimental results on the widely used benchmark demonstrate that the proposed method outperforms state-of-the-art methods.

In three-dimensional video system, the texture and depth videos are jointly encoded, and then the Depth Image Based Rendering (DIBR) is utilized to realize view synthesis. However, the compression distortion of texture and depth videos, as well as the disocclusion problem in DIBR degrade the visual quality of the synthesized view. To address this problem, a Two-stream Attention Network (TSAN)-based synthesized view quality enhancement method is proposed for 3D-High Efficiency Video Coding (3D-HEVC) in this paper. First, the shortcomings of the view synthesis technique and traditional convolutional neural networks are analyzed. Then, based on these analyses, a TSAN with two information extraction streams is proposed for enhancing the quality of the synthesized view, in which the global information extraction stream learns the contextual information, and the local information extraction stream extracts the texture information from the rendered image. Third, a Multi-Scale Residual Attention Block (MSRAB) is proposed, which can efficiently detect features in different scales, and adaptively refine features by considering interdependencies among spatial dimensions. Extensive experimental results show that the proposed synthesized view quality enhancement method achieves significantly better performance than the state-of-the-art methods.

Purpose: Pancreatic cancer is characterized by stromal desmoplasia and perineural invasion (PNI). We sought to explore the contribution of pancreatic stellate cells (PSC) activated by paracrine Sonic Hedgehog (SHH) in pancreatic cancer PNI and progression.Experimental Design: In this study, the expression dynamics of SHH were examined via immunohistochemistry, real-time PCR, and Western blot analysis in a cohort of carcinomatous and nonneoplastic pancreatic tissues and cells. A series of in vivo and in vitro assays was performed to elucidate the contribution of PSCs activated by paracrine SHH signaling in pancreatic cancer PNI and progression.Results: We show that SHH overexpression in tumor cells is involved in PNI in pancreatic cancer and is an important marker of biologic activity of pancreatic cancer. Moreover, the overexpression of SHH in tumor cells activates the hedgehog pathway in PSCs in the stroma instead of activating tumor cells. These activated PSCs are essential for the promotion of pancreatic cancer cell migration along nerve axons and nerve outgrowth to pancreatic cancer cell colonies in an in vitro three-dimensional model of nerve invasion in cancer. Furthermore, the coimplantation of PSCs activated by paracrine SHH induced tumor cell invasion of the trunk and nerve dysfunction along sciatic nerves and also promoted orthotropic xenograft tumor growth, metastasis, and PNI in in vivo models.Conclusions: These results establish that stromal PSCs activated by SHH paracrine signaling in pancreatic cancer cells secrete high levels of PNI-associated molecules to promote PNI in pancreatic cancer. Clin Cancer Res; 20(16); 4326–38. ©2014 AACR.

Hydropower dams are an important green renewable energy technology, but their effect on the gut microbes of fish in different habitats surrounding the dams is unclear. We collected the gut of seven fish species (n = 109 fish) both upstream and downstream of a dam in Xijiang River basin, China, and identified the microbes present by 16s rRNA pyrosequencing. A total of 9,071 OTUs were identified from 1,576,253 high-quality tags with 97% sequence similarity. Our results indicated that the gut microbial diversity of upstream fish was significantly higher than that of downstream fish, though the dominant microbial species were similar and mainly comprised Proteobacteria (mean 35.0%), Firmicutes (20.4%) and Actinobacteria (15.6%). The presence of the dam markedly altered the gut microbial composition in Squaliobarbus curriculusand Hypostomus plecostomus. Moreover, we found specificity in the composition of gut microorganisms in fishes of different diets and pelagic levels, whereas the omnivorous Pseudohemiculter dispar had a higher level of species richness and diversity of gut bacteria compared with the other species. The results of the functional analysis showed that the abundance of microorganisms related to energy metabolism (e.g., amino acid metabolism, carbohydrate metabolism, biosynthesis metabolism) was significantly higher in the gut of upstream fish than in downstream fish. Our results showed that the hydropower station affected downstream levels of chlorophyll-a, total nitrogen and total organic carbon. Canonical correspondence analysis showed that water temperature, Hg and chlorophyll-a significantly affected gut microbial composition. These results are important for assessing the impact of hydropower plant on fish gut microbes and their potential environmental risks.

Stereoscopic image retargeting aims at converting stereoscopic images to the target resolution adaptively. Different from 2D image retargeting, stereoscopic image retargeting needs to preserve both the shape structure of salient objects and depth consistency of 3D scenes. In this paper, we present a stereoscopic image retargeting method based on deep convolutional neural network to obtain high-quality retargeted images with both object shape preservation and scene depth preservation. First, a cross-attention extraction mechanism is constructed to generate attention map, which contains the valuable attention features of the left and right images and the common attention features between them. Second, since the disparity map can provide accurate depth information of objects in 3D scenes, a disparity-assisted 3D significance map generation module is utilized to further preserve the valuable depth information of stereoscopic images. Finally, in order to predict the retargeted stereoscopic images accurately, an image consistency loss is developed to preserve the geometric structure of salient objects, and a disparity consistency loss is introduced to eliminate depth distortions. Experimental results demonstrate that the proposed deep convolutional neural network can provide favorable stereoscopic image retargeting results.

Device-to-device (D2D) communication technology is widely acknowledged as an emerging candidate to alleviate the wireless traffic explosion problem for the next generation wireless local area network (WLAN) IEEE 802.11ax. In this paper, we integrate D2D communication into IEEE 802.11ax for maximizing the spectrum efficiency. Due to the spectrum scarcity, the number of available resource units (RUs) is typically less than D2D pairs and stations (STAs), which makes the management of spectrum resources more complex. To tackle this issue, we design an efficient resource management algorithm for uplink random access and resource allocation in D2D-enabled 802.11ax, which provides more channel access and resource reuse opportunities for STAs and D2D pairs. Specifically, we propose an enhanced back-off mechanism and derive the optimal contention window (CW) by a theoretical model, which improves the uplink random access efficiency. To tackle the complex interference problem in RU scheduling phase, we develop an efficient and low-complexity resource allocation algorithm based on the maximal independent set (MIS), which can schedule multiple D2D pairs to share the same RU with the specific STA. Simulation results demonstrate that the proposed algorithm significantly improves the network performance in terms of system throughput, collision rate, complete time, and channel utilization.

The recent years have witnessed significant advances in deep video action recognition. However, the performances of deep learning-based video action recognition methods are limited in the case of relatively small samples or no labeled samples. Therefore, trying to use unlabeled video data to generate clustering label is essential for small sample learning and zero sample learning. In this paper, we propose a novel deep video action clustering network, which aims to learn the similarity relationship among the unlabeled video samples, and generate the clustering label for each video sample. Specifically, the proposed method simultaneously learns the spatio-temporal features and subspace representations under a jointly optimized framework. It consists of a 3D U-Net self-representation generator, a video-clip reconstruction discriminator, and a confidence-based feedback mechanism. The 3D U-Net self-representation generator learns the spatio-temporal features of the video clips and produces subspace representation matrix. Then, the similarity graph is constructed based on this subspace representation matrix, and the clustering result is obtained. In the learning procedure, a confidence-based feedback mechanism is designed to feed the high-confidence labels of partial samples back to further guide the subspace structure learning, so that the optimal result can be obtained. During training, the video-clip reconstruction discriminator is introduced to evaluate the reconstructed video clips, which is beneficial for capturing the discriminative spatio-temporal features. Experimental results on a video benchmark dataset demonstrate the effectiveness of the proposed method.

Progoitrin (2-hydroxy-3-butenyl glucosinolate, PRO) is the main source of bitterness of Brassica plants. Research on the biosynthesis of PRO glucosinolate can aid the understanding of the nutritional value in Brassica plants. In this study, four ODD genes likely involved in PRO biosynthesis were cloned from Chinese kale. These four genes, designated as BocODD1–4, shared 75–82% similarities with the ODD sequence of Arabidopsis. The sequences of these four BocODDs were analyzed, and BocODD1 and BocODD2 were chosen for further study. The gene BocODD1,2 showed the highest expression levels in the roots, followed by the leaves, flowers, and stems, which is in accordance with the trend of the PRO content in the same tissues. Both the expression levels of BocODD1,2 and the content of PRO were significantly induced by high- and low-temperature treatments. The function of BocODDs involved in PRO biosynthesis was identified. Compared with the wild type, the content of PRO was increased twofold in the over-expressing BocODD1 or BocODD2 plants. Meanwhile, the content of PRO was decreased in the BocODD1 or BocODD2 RNAi lines more than twofold compared to the wildtype plants. These results suggested that BocODD1 and BocODD2 may play important roles in the biosynthesis of PRO glucosinolate in Chinese kale.

RGB-D salient object detection aims at identifying attractive objects in a scene by combining the color image and depth map. However, due to the differences between RGB-D image pairs, it is a key issue to utilize cross-modal data effectively. In this paper, we propose a novel RGB-D salient object detection method via cross-modal joint feature extraction and low-bound fusion loss. A two-stream framework is designed to generate the saliency maps for the RGB image and depth map. During the feature extraction, a cross-modal joint feature extraction module (CFM) is proposed to capture valuable joint features from the two streams. The CFM explores complementary information from the feature extraction and feeds the joint features to the aggregation stage of the network. Then, the fusion block (FB) is utilized to aggregate the multi-scale features of each stream and the joint features to generate the updated features. In addition, a low-bound fusion loss is designed to constrain the predictions of the two streams, to improve the lower bound of saliency values and generate a distinct saliency map. Experimental results on five datasets demonstrate that the proposed method achieves superior performances.

In this paper, we propose a two-stage weighting based perceptual quality assessment framework for asymmetrically distorted stereoscopic video (SV) sequences by temporal binocular rivalry. Firstly, a traditional 2D image quality assessment (IQA) method is employed to measure spatial distortion, and the temporal distortion is evaluated by the magnitude differences between motion vectors of distorted and reference video frames. Secondly, the structural strength (SS) computed by gradient map and the motion energy (ME) computed by frame difference map are used to estimate the intensity of visual stimulus in spatial and temporal domain respectively. Then, SS and ME are considered as the importance indexes to combine the quality scores of spatial and temporal distortion to estimate perceived distortion of single-view video sequences, which is denoted as the first-stage weighting. Finally, considering that the difference of intensity of visual stimulus between two eyes results in binocular rivalry, a novel temporal binocular rivalry inspired weighting method is designed to integrate the quality scores of left- and right-views for the final visual quality prediction of SV sequences, which is denoted as the second-stage weighting. Experimental results on Waterloo-IVC SV quality databases show that several specific examples of 2D-IQA methods within the proposed framework can obtain highly competitive performance over other existing ones.

Deep learning-based methods have achieved remarkable performance in single image super-resolution. However, these methods cannot be effectively applied in stereoscopic image super-resolution without considering the characteristics of stereoscopic images. In this article, an interaction module-based stereoscopic image super-resolution network (IMSSRnet) is proposed to effectively utilize the correlation information in stereoscopic images. The key insight of the network lies with how to explore the complementary information of one view to help the reconstruction of another view. Thus, an interaction module is designed to acquire the enhanced features by utilizing complementary information between different views. Specifically, the interaction module is composed of a series of interaction units with a residual structure. In addition, the single image features of left and right views are obtained by a spatial feature extraction module, which can be realized by any existing single image super-resolution models. In order to obtain high-quality stereoscopic images, a gradient loss is introduced to preserve the texture details in a view, and a disparity loss is developed to constrain the disparity relationship between different views. Experimental results demonstrate that the proposed method achieves a promising performance and outperforms the state-of-the-art methods.

Stereoscopic image retargeting aims to manipulate the stereoscopic images to fit various devices with different resolutions and prescribed aspect ratios. With the development of various types of three-dimensional (3D) displays, stereoscopic image retargeting becomes increasingly popular in the field of computer graphics. In this paper, we propose an unsupervised stereoscopic image retargeting network (USIR-Net) to address the problem of stereoscopic image retargeting without label information. By exploring the inter-view correlation and disparity relationship of stereoscopic images, two unsupervised losses are developed to guide the learning of stereoscopic image retargeting model. First, in view of the inter-view correlation, a view synthesis loss is proposed to guarantee the generation of high-quality stereoscopic images with accurate inter-view relationship. Second, by exploiting the consistency of stereoscopic images before and after the retargeting, a stereo cycle consistency loss, which consists of a content consistency term and a disparity consistency term, is developed to preserve the structure information and prevent binocular disparity inconsistency. Quantitative and qualitative experimental results demonstrate that the proposed method achieves superior performance compared with state-of-the-art methods.

Hyperspectral image (HSI) clustering is a challenging task due to the complex characteristics in HSI data, such as spatial-spectral structure, high-dimension, and large spectral variability. In this paper, we propose a novel deep spatial-spectral subspace clustering network (DS3C-Net), which explores spatial-spectral information via the multi-scale auto-encoder and collaborative constraint. Considering the structure correlations of HSI, the multi-scale auto-encoder is first designed to extract spatial-spectral features with different-scale pixel blocks which are selected as the inputs. Then, the collaborative constrained self-expressive layers are introduced between the encoder and decoder, to capture the self-expressive subspace structures. By designing a self-expressiveness similarity constraint, the proposed network is trained collaboratively, and the affinity matrices of the feature representation are learned in an end-to-end manner. Based on the affinity matrices, the spectral clustering algorithm is utilized to obtain the final HSI clustering result. Experimental results on three widely used hyperspectral image datasets demonstrate that the proposed method outperforms state-of-the-art methods.