Search

In recent decades, the development of multimedia and computer vision has sparked significant interest among researchers in the field of automatic image annotation. However, much of the research has primarily focused on using a single graph for annotating images in semi-supervised learning. Conversely, numerous approaches have explored the integration of multi-view or image segmentation techniques to create multiple graph structures. Yet, relying solely on a single graph proves to be challenging, as it struggles to capture the complete manifold of structural information. Furthermore, the computational complexity of building multiple graph structures based on multi-view or image segmentation is substantial and time-consuming. To address these issues, we propose a novel method called "Central Attention with Multi-graphs for Image Annotation." Our approach emphasizes the critical role of the central image region in the annotation process. Remarkably, we demonstrate that impressive performance can be achieved by leveraging just two graph structures, composed of central and overall features, in semi-supervised learning. To validate the effectiveness of our proposed method, we conducted a series of experiments on benchmark datasets, including Corel5K, ESPGame, and IAPRTC12. These experiments provide empirical evidence of our method’s capabilities. [ABSTRACT FROM AUTHOR]

Due to their vast size, the growing number of digital images found in personal archives and on websites has become unmanageable, making it challenging to retrieve images from these large databases accurately. While these collections are popular due to their convenience, they often need to be equipped with proper indexing information, making it difficult for users to find what they need. One of the most significant challenges in computer vision and multimedia is image annotation, which involves labeling images with descriptive keywords. However, computers need to possess the capability to understand the essence of images in the same way that humans do, and people can only identify images based on their visual attributes rather than their deeper semantic meaning. Therefore, image annotation requires keywords to effectively communicate the contents of an image to a computer system. However, raw pixels in an image need to provide more information to generate semantic concepts, making image annotation a complex task. Unlike text annotation, where the dictionary linking words to semantics is well established, image annotation lacks a clear definition of “words” or “sentences” that can be associated with the meaning of the image, known as the semantic gap. To address this challenge, this study aimed to characterize image content meaningfully to make information retrieval easier. An improved automatic image annotation (AIA) system was proposed to bridge the semantic gap between low-level computer features and human interpretation of images by assigning one or multiple labels to images. The proposed AIA system can convert raw image pixels into semantic-level concepts, providing a clearer representation of the image content. The study combined the ResNet50 and slantlet transform with word2vec and principal component analysis with t-distributed stochastic neighbor embedding to balance precision and recall. This allowed the researchers to determine the optimal model for the proposed ResNet50-SLT AIA framework. A Word2vec model with ResNet50-SLT was used with principal component analysis and t-distributed stochastic neighbor embedding to improve IA prediction accuracy. The distributed representation approach involved encoding and storing information about image features. The proposed AIA system utilized seq2seq to generate sentences depending on feature vectors. The system was implemented on the most popular datasets (Flickr8k, Corel-5k, ESP-Game). The results showed that the newly developed AIA scheme overcame the computational time complexity associated with most existing image annotation models during the training phase for large datasets. The performance evaluation of the AIA scheme showed its excellent flexibility of annotation, improved accuracy, and reduced computational costs, thus outperforming the existing state-of-the-art methods. In conclusion, this AIA framework can provide immense benefits in accurately selecting and extracting image features and easily retrieving images from large databases. The extracted features can effectively be used to represent the image, thus accelerating the annotation process and minimizing the computational complexity.

Color similarity has been a key feature for content-based image retrieval by contemporary search engines, such as Google. In this study, we compare the visual content information of images, obtained through color histograms, with their corresponding hashtag sets in the case of Instagram posts. In previous studies, we had concluded that less than 25% of Instagram hashtags are related to the actual visual content of the image they accompany. Thus, the use of Instagram images' corresponding hashtags for automatic image annotation is questionable. In this study, we are answering this question through the computational comparison of images' low-level characteristics with the semantic and syntactic information of their corresponding hashtags. The main conclusion of our study on 26 different subjects (concepts) is that color histograms and filtered hashtag sets, although related, should be better seen as a complementary source for image retrieval and automatic image annotation.

Copyright of Journal of Ordnance Equipment Engineering is the property of Chongqing University of Technology and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The number of digital images uploaded in the virtual world is rapidly growing every day. Therefore, an automatic image annotation system that can retrieve information from these images seems to be in high demand. One of the challenges in this field is the imbalanced data sets and the difficulty of successfully learning tags from them. Even if a nearly balanced data set exists for image annotation, it is unlikely to find a single learner, which could learn all tags with the same accuracy. In this paper, we suggest a novel integration system that selects an elite group of models from all existing annotation models and then combines them to take the best advantage of each model's learning technique. The proposed system studies the data sets of selected models without the need for direct access to those data sets. As this algorithm is independent of the annotation models or data sets, it could be used to combine the currently available annotation models and those developed in future, along with their data sets and learning models. We believe the proposed approach has the potential of becoming an integrated ground for automatic image annotation models. [ABSTRACT FROM AUTHOR]

In this paper, based on in-depth analysis of remote multimedia images, the automatic annotation and classification of graphics are tested and analyzed by algorithms of deep learning. To reduce the time of remote multimedia image labeling and online classification, and improve efficiency, we study the use of deep learning methods to automate annotation and online classification of remote multimedia images. An image is re-labeling algorithm based on modeling the correlation of hidden feature dimensions is proposed to improve the effect of hidden feature models by modeling the correlation between hid feature dimensions. The algorithm constructs the correlation between each pair of dimensions in the hidden features through the outer product operation to form a two-dimensional interactive graph. The information in the interaction graph is refined layer by layer by using the ability of the convolutional neural network to model local features, and finally, a representation of the correlation of all dimensions in the hidden features is formed to realize the re-labeling of social images. The experimental results show that this method can utilize the hidden feature information more effectively and improve the image re-labeling results. The light-weight feature extraction network significantly reduces the number of model parameters at the expense of a small amount of detection accuracy, and the FPN pyramid structure enhances the feature characterization ability of the feature extraction network. The performance is close to that of the Yolo algorithm. [ABSTRACT FROM AUTHOR]

Automatic image annotation (AIA) has been adopted in different applications such as image retrieval and classification. Deep Learning is used in AIA to extract image features and then convert these features into text descriptions and labels. However, conventional AIA models that employ deep learning methods suffer from various shortcomings, such as poor annotation performance. This work proposes an AIA model based on convolutional neural networks (CNNs), generative adversarial networks (GANs), and transfer learning. GANs have attracted a lot of interest because of its ability to generate data without explicitly using probability density. Thus, it has proven its usefulness in image annotation and image augmentation. In this work, an Auxiliary classifier-GAN (ACGAN) has been used, where the discriminator predicts the class of an image rather than taking it as a given input; therefore, the stabilization of the training stage is ensured, and the generation of high-quality images is provided. Transfer learning is also used to enhance the performance of the classification. The proposed model outperforms the best state-of-the-art models in terms of MiAP, F-measure and error rate using ImageClef, ESPGame and IAPR-TC12 datasets.

Every day, websites and personal archives generate an increasing number of photographs. The extent of these archives is unfathomable. The ease of usage of these enormous digital image collections contributes to their popularity. However, not all of these databases provide appropriate indexing data. As a result, it’s tough to find information that the user is interested in. Thus, in order to find information about an image, it is necessary to classify its content in a meaningful way. Image annotation is one of the most difficult issues in computer vision and multimedia research. The objective is to convert an image into a single or numerous labels. This necessitates a grasp of the visual content of an image. The necessity for unambiguous information to build semantic-level concepts from raw image pixels is one of the challenges of image annotation. Unlike text annotation, where a dictionary links words to their meaning, raw picture pixels are insufficient to construct semantic-level notions directly. A simple syntax, on the other hand, is well specified for combining letters to form words and words to form sentences. The automatic feature extraction for automatic annotation was the emphasis of this paper. And they employed a deep learning convolutional neural network to build and improve image coding and annotation capabilities. Performance of the suggested technique on the Corel-5K, ESP-Game, and IAPRTC-12 datasets. Finally, experimental findings on three data sets were used to demonstrate the usefulness of this model for image annotation.

Automatic image annotation is a key technology in image understanding and pattern recognition, and is becoming increasingly important in order to annotate large-scale images. In the past decade, the nearest neighbor model-based AIA (Automatic image annotation) methods have been proved to be the most successful in all classical models. This model has four major challenges including semantic gap, label-imbalance, wider range labels, and weak-labeling. In this paper, we propose a novel annotation model based on three-pass KNN (k-Nearest Neighbor) to address the aforementioned challenges. The key idea is to identify appropriate neighbors at each pass KNN. In the first pass KNN, we identify the several most relevant categories based on label feature rather than visual feature as traditional models. In the second pass KNN, we determine the relevant images based on multi-modal (visual and textual label) embedding features. As the test image has not been annotated with any label, we propose a pre-annotation strategory before image annotation to improve the semantic level. In the third pass KNN, we capture relevant labels from semantically and visually similar images and propagate them to the given unlabeled image. In contrast with traditional nearest neighbor based methods, our method can inherently alleviate the problems of semantic gap, label-imbalance, and wider range labels. In addition, to alleviate the issue of weak-labeling, we propose label refinement for training images. Extensive experiments on three classical benchmark datasets and MS-COCO demonstrate that the proposed method significantly outperforms the state-of-the-art in terms of per-label and per-image metrics.

Automatic image annotation is an active field of research in which a set of annotations are automatically assigned to images based on their content. In literature, some works opted for handcrafted features and manual approaches of linking concepts to images, whereas some others involved convolutional neural networks (CNNs) as black boxes to solve the problem without external interference. In this work, we introduce a hybrid approach that combines the advantages of both CNN and the conventional concept-to-image assignment approaches. J-image segmentation (JSEG) is firstly used to segment the image into a set of homogeneous regions, then a CNN is employed to produce a rich feature descriptor per area, and then, vector of locally aggregated descriptors (VLAD) is applied to the extracted features to generate compact and unified descriptors. Thereafter, the not too deep clustering (N2D clustering) algorithm is performed to define local manifolds constituting the feature space, and finally, the semantic relatedness is calculated for both image–concept and concept–concept using KNN regression to better grasp the meaning of concepts and how they relate. Through a comprehensive experimental evaluation, our method has indicated a superiority over a wide range of recent related works by yielding F1 scores of 58.89% and 80.24% with the datasets Corel 5k and MSRC v2, respectively. Additionally, it demonstrated a relatively high capacity of learning more concepts with higher accuracy, which results in N+ of 212 and 22 with the datasets Corel 5k and MSRC v2, respectively. [ABSTRACT FROM AUTHOR]

Automatic image annotation plays a significant role in image understanding, retrieval, classification, and indexing. Today, it is becoming increasingly important in order to annotate large-scale social media images from content-sharing websites and social networks. These social images are usually annotated by user-provided low-quality tags. The topic model is considered as a promising method to describe these weak-labeling images by learning latent representations of training samples. The recent annotation methods based on topic models have two shortcomings. First, they are difficult to scale to a large-scale image dataset. Second, they can not be used to online image repository because of continuous addition of new images and new tags. In this paper, we propose a novel annotation method based on topic model, namely local learning-based probabilistic latent semantic analysis (LL-PLSA), to solve the above problems. The key idea is to train a weighted topic model for a given test image on its semantic neighborhood consisting of a fixed number of semantically and visually similar images. This method can scale to a large-scale image database, as training samples involved in modeling are a few nearest neighbors rather than the entire database. Moreover, this proposed topic model, online customized for the test image, naturally addresses the issue of continuous addition of new images and new tags in a database. Extensive experiments on three benchmark datasets demonstrate that the proposed method significantly outperforms the state-of-the-art especially in terms of overall metrics.

Nowadays, with the rapid growth of imaging and social network, huge volumes of image data are produced and shared on social media. Social image annotation has been an important and challenging task in the fields of computer vision and machine learning, which can facilitate large-scale image retrieval, indexing, and management. The four most challenges of social image annotation are semantic gap, tag refinement, label-imbalance, and annotation efficiency. To address these issues, we propose an efficient and effective annotation method based on the Mean of Positive Examples (MPE) corresponding to each label. First, we refine user-provided noisy tags by our proposed local smoothing process, and consider the refined tags as key features in contrast to the previous methods that consider them as side information, which significantly improves annotation performance. Second, we propose a weighted trans-media similarity measure method that fuses all modality information in identifying proper neighbors, which promotes the semantic level and eases image annotation. Third, our MPE model gives equal importance to all labels, thus, improving the annotation performance of infrequent labels without sacrificing that of frequent labels. Fourth, our MPE model can dramatically decrease space-time overheads, since the time cost of annotating an image is unaffected by the size of the training image dataset, but relying on the size of label vocabulary. Therefore, our proposed method can be applied to real-world large-scale online social image repositories. Extensive experiments on both benchmark datasets demonstrate the effectiveness and efficiency of our MPE model.

Automatic image annotation is a process in which computer systems automatically assign the textual tags related with visual content to a query image. In most cases, inappropriate tags generated by the users as well as the images without any tags among the challenges available in this field have a negative effect on the query's result. In this paper, a new method is presented for automatic image annotation with the aim at improving the obtained tags, as well as reducing the effect of unrelated tags. In the proposed method, first, the initial tags are determined by extracting the low-level features of the image and using neighbor voting method. Afterwards, each initial tag is assigned by a degree based on the neighbor image features of the query image. Finally, they will be ranked based on summing the degrees of each tag and the best tags will be selected by removing the unrelated tags. The experiments conducted on the proposed method using the NUS-WIDE dataset and the commonly used evaluation metrics demonstrate the effectiveness of the proposed system compared to the previous works.

In an image annotation model based on deep learning, the number of neurons in its output layer is proportional to the vocabulary of the annotation, i.e., the model structure changes with a change in the vocabulary, thereby reducing the accuracy of image annotation. To solve this problem, in this study a new annotation model combining the improved Wasserstein generative adversarial network (GAN) and word2vec was proposed. First, the tagged vocabulary was mapped to a fixed multidimensional word vector by word2vec. Second, a neural network model (GAN-IW) was constructed by using the generated confrontation network. It was observed that the number of neurons in the output layer was equal to the dimension of the multidimensional word vector and no longer relevant to the vocabulary. Finally, the model was tested for the Corel 5K and IAPRTC-12 image annotation datasets. Compared to the convolutional neural network regression method, the model accuracy, the recall rate, and the F1 value increased by 16%, 6%, and 9%, respectively, when the model was tested on the Corel 5K dataset. Compared to the two-pass K-nearest neighbor models, our model accuracy, recall rate, and F1 value were increased by 8%, 6%, and 4%, respectively, when the model was tested on the IAPRTC-12 dataset. The experimental results showed that the GAN-IW model can solve the problem of change in the number of output neurons with a change in the vocabulary and the number of labels annotated with each image is adaptive, making the results of model annotation more in line with the actual image annotation. [ABSTRACT FROM AUTHOR]

Increasing proliferation of images due to multimedia capabilities of hand-held devices has resulted in loss of source information resulting from inherent mobility. These images are cumbersome to search out once stored away from their original source because they drop their descriptive data. This work, developed a model to encapsulate descriptive metadata into the Exif section of image header for effective retrieval and mobility. The resulting metadata used for retrieval purposes was mobile, searchable and non-obstructive.

Nearest Neighbor method (KNN) is a typical method to solve the problem of automatic image annotation (AIA). However, traditional AIA methods based on KNN only consider the relationships among images and labels. In this paper, we propose an improved KNN image annotation method based on a tag semantic extension model (TSEM). Our approach uses the convolutional neural network (CNN) to extract image features and predicts image tags automatically via nearest features. Different from existing work, the proposed method considers correlations among images, correlations between images and labels and those among labels. Additionally, a label quantity prediction (LQP) model is proposed to predict the number of tags, which further improves the tag prediction accuracy. Comparison experiments were performed on three typical image datasets Corel5k, ESP game and laprtc12. Experimental results show that the average F1 of our model is 0.427, which outperforms the state-of-the-art KNN image annotation methods. [ABSTRACT FROM AUTHOR]

Automatic image annotation is an active field of research in which a set of annotations are automatically assigned to images based on their content. In literature, some works opted for handcrafted features and manual approaches of linking concepts to images, whereas some others involved convolutional neural networks (CNNs) as black boxes to solve the problem without external interference. In this work, we introduce a hybrid approach that combines the advantages of both CNN and the conventional concept-to-image assignment approaches. J-image segmentation (JSEG) is firstly used to segment the image into a set of homogeneous regions, then a CNN is employed to produce a rich feature descriptor per area, and then, vector of locally aggregated descriptors (VLAD) is applied to the extracted features to generate compact and unified descriptors. Thereafter, the not too deep clustering (N2D clustering) algorithm is performed to define local manifolds constituting the feature space, and finally, the semantic relatedness is calculated for both image–concept and concept–concept using KNN regression to better grasp the meaning of concepts and how they relate. Through a comprehensive experimental evaluation, our method has indicated a superiority over a wide range of recent related works by yielding F1 scores of 58.89% and 80.24% with the datasets Corel 5k and MSRC v2, respectively. Additionally, it demonstrated a relatively high capacity of learning more concepts with higher accuracy, which results in N+ of 212 and 22 with the datasets Corel 5k and MSRC v2, respectively.

Automatic image annotation aims to assign relevant keywords to images and has become a research focus. Although many techniques have been proposed to solve this problem in the last decade, giving promissing performance on standard datasets, we propose a novel automatic image annotation technique in this paper. Our method uses a label transfer mechanism to automatically recommend those promising tags to each image by using the category information of images. As image representation is one of the key technique in image annotation, we use sparse coding based spatial pyramid matching and deep convolutional neural networks to model image features. And metric learning technique is further used to combine these features to achieve more effective image representation in this paper. Experimental results illustrate that the proposed method get similar or better results than the state-of-the-art methods on three standard image datasets. [ABSTRACT FROM AUTHOR]

Automatic Image Annotation (AIA) remains as a challenge in computer vision with real-world applications, due to the semantic gap between high-level semantic concepts and low-level visual appearances. Contextual tags attached to visual images and context semantics among semantic concepts can provide further semantic information to bridge this gap. In order to effectively capture these semantic correlations, we present a novel approach called Multi-modal Multi-concept-based Deep Neural Network (M2-DNN) in this study, which models the correlations of visual images, contextual tags, and multi-concept semantics. Unlike traditional AIA methods, our M2-DNN approach takes into account not only single-concept context semantics, but also multi-concept context semantics with abstract scenes. In our model, a multi-concept such as { "plane" , "buildings" } is viewed as one holistic scene concept for concept learning. Specifically, we first construct a multi-modal Deep Neural Network (DNN) as a concept classifier for visual images and contextual tags, and then employ it to annotate unlabeled images. Second, real-world databases commonly include many difficult concepts that are hard to be recognized, such as concepts with similar appearances, concepts with abstract scenes, and rare concepts. To effectively recognize them, we utilize multi-concept semantics inference and multi-modal correlation learning to refine semantic annotations. Finally, we estimate the most relevant labels for each of unlabeled images through a new strategy of label decision. The results of our comprehensive experiments on two publicly available datasets have shown that our method performs favourably compared with several other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Automatic Image Annotation (AIA) aims to provide a semantic description for the content of image by assigning a set of textual labels. The recent approaches mainly focus on the improvement of single model and neglect the potential advantages of different models. In order to make full use of the advantages of different annotation models, Dual Model based on Multi-Label Selection Algorithm(DM-SA) is proposed in this research which combines a discriminative model with a nearest-neighbor-based model. The algorithm takes consideration of the advantages of each model, thus provides better annotation performance. A deep Convolutional Neural Network (CNN) is used to obtain visual representation of images first, then a discriminative model, CNN with Label Smoothing (CNN-LS), and a nearest-neighbor-based model, 2PKNN with Canonical Correlation Analysis (2PKNN-CCA) generate candidate label set respectively. Finally, a multi-label selection algorithm based on inverse document frequency is adopted to assign the final labels from two candidate label sets. Experimental results based on Corel5K and IAPRTC-12 datasets show that the proposed method can achieve state-of-the-art performance for average recall, 0.52 and 0.42 on Corel5K and IAPRTC-12 respectively. [ABSTRACT FROM AUTHOR]

This paper introduces an automatic image annotation framework based on multi-auxiliary information which aims at improving the annotation performance. We propose three novel ideas in the framework of annotation: 1) multi-information extraction: besides various visual features, tag co-occurrence, and user interest vector are added to enrich the multi-auxiliary information; 2) initial labeling: based on the traditional term frequency-inverse document frequency model-we utilize the visibility of words and extended tag set to enhance the result of initial labeling and propose a more efficient model, TF-IDF, visibility and extended tag set model; and 3) tag refinement: by considering multi-auxiliary information, including multi-visual content, tag co-occurrence, and user interest similarity, we propose the multi-information alllabels model for tag refinement. The tag refinement process is formalized as an optimization problem by adjusting confidence score set by the initial labeling model. Experimental results demonstrate that, compared with the state-of-the-art methods, our method achieves the best performance on MIR-Flickr data sets, outperforming the second best by 2%.

Image annotation is the task of assigning keywords or identifiers to images, holistically or in specific regions. These keywords serve as descriptors of high-level semantics to facilitate retrieval and organization of visual information. It plays an important role in content-based image understanding, as well as in areas such as object recognition in robotics, content-based image searching and knowledge extraction. Automatic image annotation is usually approached by means of supervised classification, where a set of previously annotated images is required to train a learning algorithm that later predicts the labels for new images. This paper proposes a novel ensemble classifier for the supervised image annotation task inspired in chain classifiers. In the proposed approach a chain of individual classifiers is build, where each classifier is trained by using a different modality. In addition, the input space of models in the chain is augmented with the output of the preceding model in the sequence. Each model in the chain deals with the same classification problem, making the proposed method an ensemble model build from multimodal data. To the best of our knowledge, chain classifiers have not been used in this particular setting. Experimental results in a challenging image collection show that the proposed method is able to obtain an f −value superior to 0.5, outperforming related work. [ABSTRACT FROM AUTHOR]

Automatic image annotation as a typical multi-label learning problem, has gained extensive attention in recent years owing to its application in image semantic understanding and relevant disciplines. Nevertheless, existing annotation methods share the same challenge that labels annotated on the training images are usually incomplete and unclean, while the need for adequate training data is costly and unrealistic. Being aware of this, we propose a dual low-rank regularized multi-label learning model under a graph regularized semi-supervised learning framework, which can effectively capture the label correlations in the learned feature space, and enforce the label matrix be self-recovered in label space as well. To be specific, the proposed approach firstly puts forward a label matrix refinement approach, by introducing a label coefficient matrix to build a linear self-recovery model. Then, graph Laplacian regularization is introduced to make use of a large number of unlabeled images by enforcing the local geometric structure on both labeled and unlabeled images. Lastly, we exploit dual trace norm regularization on both feature mapping matrix and self-recovery coefficient matrix to capture the correlations among different labels in both feature space and label space, and control the model complexity as well. Empirical studies on four real-world image datasets demonstrate the effectiveness and efficiency of the proposed framework. [ABSTRACT FROM AUTHOR]