Search

In this article, we propose a structure-aligned generative adversarial network framework to improve zero-shot learning (ZSL) by mitigating the semantic gap, domain shift, and hubness problem. The proposed framework contains two parts, i.e., a generative adversarial network with a softmax classifier part, and a structure-aligned part. In the first part, the generative adversarial network aims at generating pseudovisual features through the guiding generator and discriminator play the minimax two-player game together. At the same time, the softmax classifier is committed to increasing the interclass distance and reducing intraclass distance. Then, the harmful effect of domain shift and hubness problems can be mitigated. In another part, we introduce a structure-aligned module where the structural consistency between visual space and semantic space is learned. By aligning the structure between visual space and semantic space, the semantic gap between them can be bridged. The performance of classification is improved when the structure-aligned visual-semantic embedding space is transferred to the unseen classes. Our framework reformulates the ZSL as a standard fully supervised classification task using the pseudovisual features of unseen classes. Extensive experiments conducted on five benchmark data sets demonstrate that the proposed framework significantly outperforms state-of-the-art methods in both conventional and generalized settings.

Multimodal image-to-image translation based on generative adversarial networks (GANs) shows suboptimal performance in the visual domains with high internal variability, e.g., translation from multiple breeds of cats to multiple breeds of dogs. To alleviate this problem, we recast the training procedure as modeling distinct distributions which are observed sequentially, for example, when different classes are encountered over time. As a result, the discriminator may forget about the previous target distributions, known as catastrophic forgetting, leading to non-/slow convergence. Through experimental observation, we found that the discriminator does not always forget the previously learned distributions during training. Therefore, we propose a novel generator regulating GAN (GR-GAN). The proposed method encourages the discriminator to teach the generator more effectively when it remembers more of the previously learned distributions, while discouraging the discriminator to guide the generator when catastrophic forgetting happens on the discriminator. Both qualitative and quantitative results show that the proposed method is significantly superior to the state-of-the-art methods in handling the image data that are with high variability.

All things follow the cosmic equilibrium rule. As one of the important factors in evaluating images of aesthetic effects, the visual balance has not attracted much attention. In this study, a novel method is proposed to quantify the law of visual balance and automatically evaluate the aesthetic value of images. In the proposed method, the authors first analyse the colour composition of images using the K-means clustering method. Then, based on information aesthetics theory and perspective principles, a calculation method is proposed to find the visual centre of gravity of images. Finally, the aesthetic effect of the image based on visual balance is evaluated according to the positional relationship between the visual centre of gravity and the physical centre of the image. With extensive experimental results, the authors demonstrate qualitatively and quantitatively that the proposed evaluation method is basically consistent with the intuitive experience of most human beings. Moreover, experiments are also conducted on a large and diversified benchmark data set that are competitive with the current state of the art.

Generating a new Chinese font from a multitude of references is an easy task, while it is quite difficult to generate it from a few references. In this paper, we investigate the problem of arbitrary Chinese font generation from a single reference and propose a deep learning based model, named One-reference Chinese Font Generation Network (OCFGNet), to automatically generate any arbitrary Chinese font from a single reference. Based on the disentangled representation learning, we separate the representations of stylized Chinese characters into style and content representations. Then we design a neural network consisting of the style encoder, the content encoder and the joint decoder for the proposed model. The style encoder extracts the style features of style references and maps them onto a continuous Variational Auto-Encoder (VAE) latent variable space while the content encoder extracts the content features of content references and maps them to the content representations. Finally, the joint decoder concatenates both representations in layer-wise to generate the character which has the style of style reference and the content of content reference. In addition, based on Generative Adversarial Network (GAN) structure, we adopt a patch-level discriminator to distinguish whether the received character is real or fake. Besides the adversarial loss, we not only adopt L1-regularized per-pix loss, but also combine a novel loss term Structural SIMilarity (SSIM) together to further drive our model to generate clear and satisfactory results. The experimental results demonstrate that the proposed model can not only extract style and content features well, but also have good performance in the generation of Chinese fonts from a single reference.

Raman spectroscopy is a powerful tool for molecular chemical analysis and bioimaging, which shows an astonishing sensitivity when combined with a huge enhancement by the coherence and surface effects. Noble metal nanoparticles have been commonly used for the spontaneous surface-enhanced Raman scattering (SERS) and for the surface-enhanced coherent anti-Stokes Raman scattering (SECARS) spectroscopies, as they provide large enhancement factors via the electromagnetic and chemical mechanisms. Recently, two-dimensional (2D) semiconductors, such as monolayer molybdenum disulfide (MoS2), were used for potential SERS applications as cheaper substrates compared to noble metal nanoparticles. However, the coherent enhancement of SECARS on 2D materials has not been previously explored. Here we present the experimental SECARS measurements of pyridine–ethanol solutions containing 2D MoS2 nanocrystals with the giant chemical enhancement factor of 109 over coherent anti-Stokes Raman scattering (CARS), which is attribute...

The main challenge for most image–text tasks, such as zero-shot, is the way to measure the semantic similarity between visual and textual feature vectors. The common solution is to map the image feature vectors and text feature vectors into the Hilbert space and then rank the similarity by the inner product between feature vectors. In this paper, we learn the feature representation of images and their sentence descriptions by different deep neural networks to learn about the inner-modal correspondences between visual and language data. We then use a joint embedding structure based on angle calculation for measuring the semantic similarity between visual and textual features. In the proposed method, a constant factor b keeps the similarities of positive samples and negative samples at a certain distance. Since the proposed cosine similarity method involves both normalization and vectors computation, we also develop the learning algorithm on neural networks for expressing the semantic features of texts and images. We applied the angle-based method to the challenging Caltech-UCSD Birds and the Oxford-102 Flowers datasets. The experiments demonstrate good performances on both recognition and retrieval tasks.

The key of zero-shot learning is to use the visual-semantic embedding to transfer the knowledge from seen classes to unseen classes. In this paper, we propose to build the visual-semantic embedding by maximizing the mutual information between visual features and corresponding attributes. Then, the mutual information between visual and semantic features can be utilized to guide the knowledge transfer from seen domain to unseen domain. Since we are primarily interested in maximizing mutual information, we introduce the noise-contrastive estimation to calculate lower-bound value of mutual information. Through the noise-contrastive estimation, we reformulate zero-shot learning as a binary classification problem, i.e., classifying the matching visual-semantic pairs (positive samples) and mismatching visual-semantic pairs (negative/noise samples). Experiments conducted on five datasets demonstrate that the proposed mutual information estimators outperforms current state-of-the-art methods both in conventional and generalized zero-shot learning settings.

Lateral two-dimensional (2D) transitional metal dichalcogenide (TMD) heterostructures have recently attracted a wide attention as promising materials for optoelectronic nano-devices. Due to the nanometer-size width of the lateral heterojunction, the study of its photoemission properties is challenging and requires using subwavelength optical techniques. We investigated the photoresponse of a lateral 2D WSe2-MoSe2 heterostructure using tip-enhanced photoluminescence (TEPL) with sub-nanometer scale variable tip-sample distance. The controllable gold-coated plasmonic silver tip allows for both optical nano-imaging and plasmon-induced hot electron injection into the heterostructure, mediated by the depletion region between the MoSe2 and WSe2. This directionality contributes to the unequal enhancement of the TEPL signals from the MoSe2 and WSe2 sides of the heterojunction. By adjusting the tip-sample distance, we demonstrated the controllability of the hot electron injection rate. In particular, when the tip-sample distance reached the sub-nanometer van der Waals separation, the quantum plasmonic effects facilitated the hot electron injection. The properties of the directional hot electron injection in the quantum plasmonic regime make the lateral 2D MoSe2-WSe2 heterostructures promising for quantum nano-devices with tunable photoresponse.

Calligraphy is the cultural treasure of the Chinese nation for five millenniums, which has always been loved by the Chinese people. This paper collects a large number of characters of Chinese calligraphy and builds a Chinese characters calligraphy data set. By establishing three different Convolution Neural Network (CNN) models, the features of calligraphy handwriting are extracted. In addition, we use some techniques to improve the robustness and generalization ability of the CNN model, so that the model can adapt to more classification tasks. The experimental results prove that the proposed method can not only well identify the style of different calligraphers, but also have good performance in the classification of the font format of soft pen calligraphy and hard pen calligraphy.

The inverse mapping of GANs’ (Generative Adversarial Nets) generator has a great potential value. Hence, some works have been developed to construct the inverse function of generator by directly learning or adversarial learning. While the results are encouraging, the problem is highly challenging and the existing ways of training inverse models of GANs have many disadvantages, such as hard to train or poor performance. Due to these reasons, we propose a new approach based on using inverse generator (IG) model as encoder and pre-trained generator (G) as decoder of an AutoEncoder network to train the IG model. In the proposed model, the difference between the input and output, which are both the generated image of pre-trained GAN’s generator, of AutoEncoder is directly minimized. The optimizing method can overcome the difficulty in training and inverse model of an non one-to-one function. We also applied the inverse model of GANs’ generators to image searching and translation. The experimental results prove that the proposed approach works better than the traditional approaches in image searching.

Cyclooxygenase-2 (COX-2) plays a crucial role in the development and invasion of gastric cancer. COX-2 inhibitors have been shown to be chemopreventive against gastrointestinal cancers. In vitro studies have suggested that the mechanisms may be related to induction of apoptosis and inhibition of angiogenesis. COX-2 may also have impact on E-cadherin. In our study we investigate the effect of Celecoxib on expression of E-cadherin and serum soluble E-cadherin, as well as on apoptosis and angiogenesis in patients with gastric cancer. Fifty nine gastric cancer patients were randomly divided into two groups: Surgery group (n = 22), in which patients underwent surgical resection after diagnosis, and Celecoxib + Surgery group (n = 37), in which patients received oral Celecoxib 200 mg twice daily for 7 days before curative resection. Twenty healthy subjects (Healthy controls) were recruited as normal controls. After curative resection, COX-2, E-cadherin, VEGF, and MVD were detected by immunohistochemistry. Serum soluble E-cadherin was quantitatively measured using a commercially available enzyme-linked immunosorbent assay kit. Apoptosis was determined by TUNEL assay. Significantly decreased expression of COX-2, increased E-cadherin and apoptosis, decreased VEGF and MVD were observed in gastric cancer tissues from patients receiving Celecoxib compared to Surgery group. Compared to Healthy controls, the serum soluble E cadherin levels were higher in gastric cancer patients which were decreased by Celecoxib. This in vivo study demonstrated that Celecoxib induces apoptosis and inhibit angiogenesis of gastric cancer. Its impact on E-cadherin may suggest that this agent may suppress the invasion of advanced gastric cancer.