Search

Multiple Instance Learning (MIL) is a learning paradigm, where training instances are arranged in sets, called bags, and only bag-level labels are available during training. This learning paradigm has been successfully applied in various real-world scenarios, including medical image analysis, object detection, image classification, drug activity prediction, and many others. This survey paper presents a comprehensive analysis of MIL, highlighting its significance, recent advancements, methodologies, applications, and evolving trends across diverse domains. The survey begins by explaining the core principles that form the basis of MIL and how it differs from traditional learning approaches. This sets the foundation for comprehending the distinct challenges and techniques of solving MIL problems. Next, we discuss how supervised learning algorithms are tailored to support MIL and combine this discussion with a review of seminal MIL algorithms as well as the latest innovations that incorporate neural networks, deep learning architectures, and attention techniques. This comprehensive analysis helps to understand the strengths, limitations, and adaptability of these methods across diverse data modalities, complexities, and applications. In summary, this survey paper provides an essential resource for researchers, practitioners, and enthusiasts seeking a comprehensive understanding of Multiple Instance Learning. It covers foundational concepts, traditional methods, recent advancements, and future directions. By providing a holistic view of MIL's dynamic landscape, this paper aims to inspire further innovation and exploration in this ever-evolving field. • A survey on the current state of the Multiple Instance Learning. • We provide applications of Multiple instance learning in various domains. • We discuss how existing supervised learning algorithms are modified to support MIL. • Publicly available datasets and open research challenges in MIL are discussed. [ABSTRACT FROM AUTHOR]

Roundwood sorting is still a manual process in many Swiss sawmills, requiring employees to visually inspect and categorize thousands of logs per day. The heavy workload can be both physically and mentally taxing and can lead to increased rates of human error. State-of-the-art automation systems like X-ray log scanners are expensive and difficult to integrate into existing process lines. This paper proposes a novel recommendation system that leverages recent advances in image classification to automate roundwood classification by quality and species. The system integrates a camera to capture cross-sectional images of logs and record numerical data, such as length, taper, and diameter. The analysis of the resulting dataset highlights the challenges of data imbalance and noise, which makes classification difficult and, in some cases, impossible. However, by using selected datasets with reduced noise, state-of-the-art Convolutional Neural Networks (CNNs) can extract quality and species features. Quality models learn from a manually selected and simplified dataset, featuring samples that experts can clearly classify based on the image's information. Species models are trained on a label-noise-reduced dataset, reflecting real-world complexity. The accuracy on the selected dataset for three quality classes is 80%. The species determination is less challenging and reaches 91% accuracy on a synchronized dataset for the main species spruce and fir. Overall, this paper highlights the potential of Machine Learning in augmenting the roundwood sorting processes and presents a novel system that can improve the efficiency and accuracy of the process. [Display omitted] • Automation of roundwood sorting: Replaces manual sorting with image-based AI. • Integrated camera system in roundwood sorting to collect labeled dataset. • Species prediction: 91% accuracy in spruce–fir distinction. • Quality prediction on complexity reduced dataset: 80% accuracy between three main quality levels. • Efficient, adaptable & scalable system which is easy to integrate into existing process lines. [ABSTRACT FROM AUTHOR]

Recent years have witnessed a growing interest in neural network-based medical image classification methods, which have demonstrated remarkable performance in this field. Typically, convolutional neural network (CNN) architectures have been commonly employed to extract local features. However, the transformer, a newly emerged architecture, has gained popularity due to its ability to explore the relevance of remote elements in an image through a self-attention mechanism. Despite this, it is crucial to establish not only local connectivity but also remote relationships between lesion features and capture the overall image structure to improve image classification accuracy. Therefore, to tackle the aforementioned issues, this paper proposes a network based on multilayer perceptrons (MLPs) that can learn the local features of medical images on the one hand and capture the overall feature information in both spatial and channel dimensions on the other hand, thus utilizing image features effectively. This paper has been extensively validated on COVID19-CT dataset and ISIC 2018 dataset, and the results show that the method in this paper is more competitive and has higher performance in medical image classification compared with existing methods. This shows that the use of MLP to capture image features and establish connections between lesions is expected to provide novel ideas for medical image classification tasks in the future. • SCNet learns the comprehensive and important information. • S-MLP divides feature maps into regions for MLP to enhance overall features. • C-MLP groups channels to reduce overfitting and improve generalization. [ABSTRACT FROM AUTHOR]

Transformer-based methods have achieved significant success in hyperspectral image (HSI) classification, which attribute to the strong capability of capturing the global dependencies from the input. However, the existing Transformer-based HSI classification methods are challenged in retrieving sufficient abound local information using the linear projection modules. Moreover, they do not fully use the hierarchical representations extracted from the original hyperspectral images. To overcome these challenges, this paper proposes a novel transformer model, called Residual Densely Transformer Network (RDTN) to comprehensively exploit the multi-hierarchical features and the local–global dependencies along the spatial–spectral dimensions from the hyperspectral images. Specially, the proposed RDTN is built with two modules: a Cross-Scale Convolution Attention (CSCA) module to extract abundant local spatial–spectral features using the multiscale convolution attention layers, and a Local Residual Transformer Block (LRTB) to respectively capture the abundant global dependencies along the spatial–spectral dimensions. Additionally, LRTB uses a residual connection operation to make full use of the hierarchical representations of all the transformer encoder layers. After acquiring dense global representations, we introduce a Global Residual Connection (GRC) to jointly fuse the local features obtained by CSCA, and then feed the fusion representations into the final avg-pooling layer and a classifier to predict the category. Finally, we conduct the extensive experiments based on four public benchmarks datasets, whose results are demonstrated that the proposed RDTN outperforms the state-of-the-art methods. The codes of this work are available at https://github.com/xiachangxue/DeepHyperX. [ABSTRACT FROM AUTHOR]

It is a crucial means for humans to perceive geomorphic features and landscape architectures by classifying ground objects in hyperspectral images (HSIs). Currently, the exponential development of neural networks has provided a powerful support for the accurate HSI classification. However, existing neural network-based methods usually rely solely on the data to drive the classification model, lacking attention to valuable land-cover distribution knowledge in HSIs. In view of this, to utilize hyperspectral data and distribution knowledge simultaneously, a data and knowledge-driven deep multiview fusion network based on diffusion model (DKDMN) is proposed in this paper. DKDMN extracts knowledge from unlabeled data in HSIs through a diffusion model-based knowledge learning framework (DMKLF), and combines raw hyperspectral data with the acquired knowledge through a designed deep multiview network architecture (DMNA) to mine complicated land-cover distribution information and reflect sample relationships. First, the proposed DMKLF utilizes the data distribution reconstructed by the diffusion model as a knowledge source for one view to enhance the network cross-sample awareness ability. On the other hand, the original HSI patches are considered a data source for another view, which co-drives DMNA with the unsupervised diffusion knowledge extracted by DMKLF to perform effective feature extraction. Second, taking into account the characteristics of each view and the feature similarity between these two views, a joint loss function specifically for DMNA is suggested to minimize the difference between the model predictions and the real labels. Finally, a multi-backbone integration classification framework (MBICF) is designed by deeply fusing three vision architectures to capture multi-scale spectral features and local–global features, thereby achieving pixel-wise classification effectively. Experimental results on four publicly available HSI datasets demonstrate that the proposed DKDMN achieves competitive classification accuracy compared with other state-of-the-art methods. For instance, the proposed DKDMN achieves an overall accuracy improvement of 1.62% and 2.18% on the Indian Pines and Salinas Valley datasets, respectively, compared to the multiple vision architecture-based hybrid network (MVAHN). The related code will be released at https://github.com/ZJier/DKDMN. • Designing a data and knowledge-driven HSI classification method. • Learning unsupervised spectral spatial distribution knowledge. • Combining data source and knowledge source by deep multiview network architecture. • Realizing HSI classification by multi-backbone integration classification framework. [ABSTRACT FROM AUTHOR]

Text recognition in scene images is still considered as a challenging task for the computer vision and pattern recognition community. For text images affected by multiple adverse factors, such as occlusion (due to obstacles) and poor quality (due to blur and low resolution), the performance of the state-of-the-art scene text recognition methods degrades. The key reason is that the existing encoder–decoder framework follows fixed left-to-right decoding order, which lacks sufficient contextual information. In this paper, we present a novel decoding order where good-quality characters can first be decoded followed by low-quality characters, which preserves the contextual information irrespective of the aforementioned difficult scenarios. Our method, named NDOrder, extracts visual features with a ViT encoder and then decodes with the Random Order Generation module (ROG) for learning to decode with random decoding orders and the Vision-Content-Position module (VCP) for exploiting the connections among visual information, content and position. In addition, a new dataset named OLQT (Occluded and Low-Quality Text) is created by manually collecting text images that suffer from occlusion or low-quality from several standard text recognition datasets. The dataset is now available at https://github.com/djzhong1/OLQT. Experiments on OLQT and public scene text recognition benchmarks show that the proposed method achieves state-of-the-art performance. • The NDOrder can recognize text images with occluded and low-quality characters. • The ROG module ensures the method work with a novel decoding order. • The VCP module constructs a robust connection among image, content and position. • We contribute to a new scene text dataset OLQT with 245 text instances. [ABSTRACT FROM AUTHOR]

• End-to-End learning using a deep hierarchical CNN to identify crack regions. • Precision through Kernel Dynamics that accurately characterise the crack pattern. • Parallel channels that enable learning crack pattern at multiple scales. Structural health monitoring plays a crucial role in assessing the condition of civil structures and ensuring their safety and maintenance. One important aspect of structural assessment is the detection and analysis of cracks, which can occur in various structures such as bridges, buildings, tunnels, dams, monuments, and roadways. Recent advancements in crack classification and segmentation techniques have utilized convolutional neural network (CNN) variations. In this paper, we propose an end-to-end learning approach utilizing a deep hierarchical CNN for crack region identification. The proposed architecture leverages high pass kernels in the early convolutional layers to accurately capture the crack patterns, complemented by learnable filters to facilitate adaptive learning. Furthermore, parallel channels within the network enable the learning of cracks at multiple scales, accommodating the varying severity and characteristics of cracks. The model is trained using data at different scales and levels, ensuring comprehensive learning from the lowest to the highest convolutional layers. To validate the effectiveness of our approach, we conducted experimental evaluations using end-to-end training on the Concrete Crack Images for Classification dataset, Asphalt Crack Dataset, and SDNET2018 dataset for crack classification. The results demonstrate the robustness and efficacy of the proposed methodology in accurately identifying crack regions across different datasets. [ABSTRACT FROM AUTHOR]

Rice is a crucial agricultural crop, yet it frequently suffers from various diseases, leading to decreased yields and, in severe cases, crop failure. Diseases significantly affect rice growth and yield, resulting in economic losses and food security challenges. The role of image recognition in identifying rice diseases is critical in agricultural production. It enables automated and efficient detection of rice diseases, which is essential for effective management, ensuring food security and sustainable agriculture. To address issues like background noise and edge blurring in rice disease image capture, as well as challenges in determining the optimal learning rate during the training of traditional rice disease recognition networks, a novel method based on PSOC-DRCNet is proposed for rice disease recognition.. First, tto solve the problem of background interference, Dual Mode Attention (DMA) is proposed to adaptively capture meaningful regions in rice disease images. Second, the Residual Adaptive Block(RAB) is proposed, which utilizes dimensional changes and channel attention to solve edge blur problems. Then, a Cross entropy and regularized mixed Loss function (CerLoss), is proposed to optimize the learning strategy of the model in the process of processing datasets and enhance the performance and generalization ability of the model to avoid overfitting problems. Ultimately, In response to the cumbersome problem of finding the optimal learning rate, we propose using Particle Swarm Optimization Chameleon (PSOC) to find the optimal learning rate and train the PSOC-DRCNet model on our custom dataset and compare it with other existing methods and the final average classification accuracy of PSOC-DRCNet is 93.88% with an F1 score of 0.940. We compare it with other existing methods. It is proved that the average classification accuracy of our model under hyper-parameter unification is 92.65% F1 score is 0.928. We validated the PSOC-DRCNet by conducting comparative analyses with other models and through generalization experiments and module effectiveness tests. Additionally, the practicality of PSOC-DRCNet was confirmed through its application in real-world scenarios. The methods proposed in this paper successfully enable the identification of various diseases in rice leaves, offering a practical solution for incorporating deep learning into the agricultural production process. Furthermore, these findings serve as a valuable reference for disease identification in other crops. [ABSTRACT FROM AUTHOR]

Current sonar image recognition methods excel in closed-set and balanced scenarios, but real underwater data often follow an open-set and long-tailed distribution, leading to misclassifications, especially among tail classes. Although open-set long-tail recognition (OLTR) tasks have received attention in natural images in recent years, there has been a lack of systematic research in sonar images. To address this gap, we present the first comprehensive study and analysis of open-set long-tail recognition in sonar images (Sonar-OLTR). In this paper, we establish a Sonar-OLTR benchmark by introducing the Nankai Sonar Image Dataset (NKSID), a new collection of 2617 real-world forward-looking sonar images. We investigate the challenges posed by long-tail distributions in existing open-set recognition (OSR) evaluation metrics for sonar images and propose two improved evaluation metrics. Using this benchmark, we conduct a thorough examination of state-of-the-art OSR, long-tail recognition, OLTR, and out-of-distribution detection algorithms. Additionally, we propose a straightforward yet effective integrated Sonar-OLTR approach as a new baseline. This method introduces a Push the right Logit Up and the wrong logit Down (PLUD) loss to increase feature space margins between known and unknown classes, as well as head and tail classes within known classes. Extensive experimental evaluation based on the benchmark demonstrates the performance and speed advantages of PLUD, providing insights for future Sonar-OLTR research. The code and dataset are publicly available at https://github.com/Jorwnpay/Sonar-OLTR. • Enhancing underwater robot usability by tackling open-set & long-tail challenges. • Building the first benchmark: new dataset, improved metrics, and experimental plan. • Our method improves model's ability via feature synthesis and specialized loss function. • Our method outperforms state-of-the-art on three different sonar image datasets. [ABSTRACT FROM AUTHOR]

Visual question answering (VQA) is a critical multimodal task in which an agent must answer questions according to the visual cue. Unfortunately, language bias is a common problem in VQA, which refers to the situation where the model generates answers solely based on the surface-level correlations between the question-answer pairs in the training set, without fully understanding the visual content. To reduce the language bias, Several recent approaches increase the image-dependency by introducing auxiliary tasks. However, these auxiliary tasks balance the data by adding extra manual image annotations or simply constructing counterfactual samples, without fully exploring the intrinsic information of the samples themselves. In this paper, we tackle the language bias problem by proposing an adaptive self-supervised counterfactual learning (ASCL) method to enhance the model's understanding of images. We propose a new adaptive feature selection module to mine the intrinsic information of the samples. This module can adaptively divides the image into question-relevant visual positive objects and question-irrelevant visual negative objects based on the given question. The question-relevant visual positive objects are used directly to generate the predicted answer, in order to reduce the influence of visual distracting information on the model's understanding of the image and ensure the actual cause of the answer. The question-irrelevant visual negative objects are treated as counterfactual samples to guide model training and prevent the model from being driven by language bias. To avoid incorrect classification of images on the classification edge during training, we propose an adaptive contrastive loss learning method that automatically adjusts the measurement distance to increase the distance between images on the classification edge. Our method has been extensively evaluated on the VQA-CP dataset, demonstrating its effectiveness and yielding improved results. Specifically, by leveraging the LMH model as a foundation, we achieve state-of-the-art performance on both the VQA CPv1 and VQA CPv2 datasets. Notably, our method significantly enhances the accuracy of the baseline, with improvements of 10.36% on the VQA CPv2 dataset and 9.38% on the VQA CPv1 dataset. The source code is publicly available at: https://github.com/shuxy0120/ASCL. • A feature selection method mines the intrinsic information of samples. • A novel learnable adaptive method for feature selection. • A counterfactual learning method uses causal inference to mitigate language bias. • An adaptive contrastive loss learning method to automatically adjust the distance. [ABSTRACT FROM AUTHOR]

Representing image sets as subspaces on Grassmann manifold and leveraging the Riemannian geometry of this space has proven to be highly effective in various visual recognition tasks. However, the computational cost associated with operating on Riemannian manifolds, especially of high-dimensional ones, poses a significant limitation to the applicability of existing techniques. Moreover, the image sets with labels are scarce and valuable in real-world scenarios, where the manual annotation is time-consuming and costly. To tackle these issues, in this paper, we propose an unsupervised Grassmannian dimensionality reduction method for image set classification and clustering, which simultaneously performs local structure learning and dimensionality reduction under the subspace view. Specifically, instead of relying on the pre-defined similarity matrix that obtained from raw data and may contain much noise, we adaptively learn the local structures from the projected low-dimensional Grassmann manifold. The refined structures then provide more discriminative power to learn the accurate manifold-to-manifold mapping. To better preserve the global structure of the data, this mapping is constrained to satisfy the orthogonal condition. In addition, our model can be compatible with F-norm OLPP for Grassmann manifold when the squared F-norm is transformed into projection metric. A re-weighted iterative scheme is carefully developed to optimize the problem. Experimental results on several benchmark datasets for image set classification and clustering tasks clearly demonstrate that our model consistently outperforms baselines, yielding significant improvements. • We propose a novel unsupervised DR method to work on non-linear Grassmann manifold. • We jointly learn the mapping and local structure to alleviate the effect of noise. • We further measure the distance for Grassmann manifold in terms of F-norm. • The experimental results demonstrate the effectiveness of the proposed model. [ABSTRACT FROM AUTHOR]

• To design an automatic multi-disease detection in CXR images by ensemble model. • To adopt ODeepLabv3 to perform segmentation process for multi-disease detection. • To adopt a new ensemble model named Optimized Ensemble Transfer Learning (OETL). • To design a Mutation Rate-based Lion Algorithm (MR-LA) to obtain high accuracy. • To optimize the epochs in ensemble classifier with the help of MR-LA algorithm. Chest radiography is a comparatively low-cost, most probably applied in a medical process that brings crucial information to discover diagnostic conclusions. Chest X-rays (CXR) are used in the prognosis of chest diseases that includes asthma, lung cancer, pneumonia, and COVID-19. Artificial intelligence detects the multi-disease automatically to enhance its proficiency and performance by solving image detection complications with various machine learning and deep learning approaches. Convolutional Neural Network (CNN) has been designed for the advancement of computerized recognition systems. For the classification of medical images, texture, shape, size, and tissue constitution are essential features for detecting diseases. Hence, huge input features are merged with deep CNN validated to increase the effectiveness of CXR analysis. In addition, multi-scale features are visualized in CNN for the detection of variable sizes of thoracic diseases. This paper has implemented a novel multi-disease diagnosis model using chest X-ray images through deep learning approaches, and thus, it helps to minimize the computational cost. The standard CXR images are gathered from the standard datasets. After, the pre-processing is performed for cleaning and contrast the images. Further, the image segmentation is carried out using Optimized DeepLabv3 (ODeepLabv3), where the optimization of parameters in DeepLabv3 is done by a new Mutation Rate-based Lion Algorithm (MR-LA). Then, multi-disease classification is carried out through Optimized Ensemble Transfer Learning (OETL), where the OETL is carried out through VGG16, ResNet, ImageNet, MobileNet, GoogleNet, Inception, and Xception. Here, the parameter optimization of all models is done by the same MR-LA. The proposed model improved the effectiveness and performance in expressions of sensitivity, precision, and specificity factors and get higher classification and detection accuracy. The outcome of the recommended method over other models is identified by the relative analysis conducted. [ABSTRACT FROM AUTHOR]

The skin, one of the most crucial organs of the human body, serves as a barrier between the body and the external environment. Early detection of skin diseases is imperative to reduce mortality rates, as some untreated conditions may progress to skin cancer. Segmentation and classification of lesions represent pivotal and interrelated endeavors within the realm of skin disease diagnosis. For this purpose, this paper presents a comprehensive diagnostic framework for segmentation and classification of skin lesions, which integrates a Legendre multiwavelet transform-based fusion XNet (LW-XNet) with an improved soft attention dense connection convolutional network (ISA-DenseNet). LW-XNet combines the strengths of XNet in fusing different frequency components of images, along with the strong feature representation capability of LW bases with various regularities for overall contextual information and detailed information of dermoscopy images. Furthermore, its encoder devises a LWT channel concatenate (LCC) block to subdivide the image into eight wavelet coefficient feature images and perform concatenated processing on them, enabling it to better differentiate and comprehend the intricate features within dermoscopy images. Finally, ISA-DenseNet is utilized for multi-class classification of the segmented images. Experimental results demonstrate the superiority of the proposed framework over existing segmentation and classification methods. [ABSTRACT FROM AUTHOR]

Fundus images can be obtained non-invasively and be adopted to monitor the follow-up on various fundus diseases, such as high myopia. Therefore, the use of fundus images for the early screening of eye diseases has principal clinical significance. However, AI-based medical research continues to face two main challenges, lack of prior-knowledge guidance and complex fundus information. In this paper, we propose a pediatric high myopia classification model, the Attention-based Patch Residual Shrinkage network (APRSnet), which facilitates early clinical diagnosis by evaluating the importance of different image features. A collection of 2492 high-resolution fundus images of children, including 768 images of high myopia and 1724 images of non-high myopia, is fed to APRSnet as the training dataset. To better understand how different features impact the classification, APRSnet is tested on multiple feature-enhanced fusion datasets. The result shows that, by removing irrelevant information in fundus images, APRSnet achieves an accuracy of 0.959 and an F1 score of 0.946 and outperforms all classic image classification networks we compared with. We also testify that, in fundus images, gradient information benefits severity classification the most since it helps with model convergence more than luminance and texture information. • An attention-based deep network is proposed for pediatric high myopia diagnosis. • We combine the strengths of both knowledge-guided and data-driven methods. • Extensive experiments are conducted to prove the effectiveness of our method. [ABSTRACT FROM AUTHOR]

In the traditional manufacturing industry, the safe operation of storage and retrieval (S/R) machines is vital for efficient automated warehouse management. Recently, deep learning-based monocular vision approaches have emerged as a promising solution for collision avoidance in automated storage and retrieval systems (AS/RS). However, the scarcity of specific datasets and the need for high-performance, lightweight neural network models pose challenges in the AS/RS domain. To address these challenges, this paper introduces a novel workflow for implementing a monocular anti-collision system for S/R machines. A semi-supervised annotation method leveraging a pretrained clustering model is proposed to expedite image collection and classification, resulting in a tailored dataset. Furthermore, the NightMix technique is introduced, combining Grayscale Enhancement and Floating Grid Mix method to enhance dataset diversity and capacity. Notably, a residual attention awareness module is integrated into MobileNet V3 to enhance its feature processing capabilities. Finally, the proposed approaches are validated through extensive experiments involving various design choices. The experimental results demonstrate that our methods are not only computationally efficient but also highly competitive in terms of learning performance. • Construction of an SSSM dataset using pretrained models. • NightMix technique is proposed for the data augmentation. • Proposal of a residual attention awareness module. [ABSTRACT FROM AUTHOR]

Real-world image classification data usually exhibits a challenging long-tailed distribution, attributed to the inherent difficulty in data collection. Existing ensemble approaches predominantly prioritize the empirical diversification of the ensemble model, sidelining its critical aspects of representation ability. A noticeable gap also exists in theoretical analysis elucidating the intricate relationship between ensemble model diversity and its generalization efficacy. This paper introduces a loosely coupled ensemble framework, Multiple Contrastive Experts (MCE) tailored for long-tailed image classification, aiming to bolster image representation while ensuring diversity within the proposed MCE. Leveraging skill-diverse classification losses, the experts in the ensemble model are able to specialize in different kinds of classes in the long-tailed distribution. An adapted supervised contrastive learning (SCL) loss is introduced to guide training for each feature learning branch to enhance the representation ability of MCE. Through the effective integration of the different losses from all experts, each expert model can be optimized coordinately. Moreover, the relationship between the generalization ability of MCE and its diversity is theoretically revealed against a single classification model. At last, extensive experiments were made over five widely used long-tailed image classification datasets. The results show that our proposed MCE is very competitive against the state-of-the-art methods, while maintaining a relatively lower computational cost. Besides, MCE has also exhibited superior performance with an increasing number of experts. [ABSTRACT FROM AUTHOR]

Remote sensing (RS) imagery provides invaluable insights into characterizing the Earth's land surface within the scope of Earth observation (EO). Technological advances in capture instrumentation, coupled with the rise in the number of EO missions aimed at data acquisition, have significantly increased the volume of accessible RS data. This abundance of information has alleviated the challenge of insufficient training samples, a common issue in the application of machine learning (ML) techniques. In this context, crowd-sourced data play a crucial role in gathering diverse information from multiple sources, resulting in heterogeneous datasets that enable applications to harness a more comprehensive spatial coverage of the surface. However, the sensitive nature of RS data requires ensuring the privacy of the complete collection. Consequently, federated learning (FL) emerges as a privacy-preserving solution, allowing collaborators to combine such information from decentralized private data collections to build efficient global models. This paper explores the convergence between the FL and RS domains, specifically in developing data classifiers. To this aim, an extensive set of experiments is conducted to analyze the properties and performance of novel FL methodologies. The main emphasis is on evaluating the influence of such heterogeneous and disjoint data among collaborating clients. Moreover, scalability is evaluated for a growing number of clients, and resilience is assessed against Byzantine attacks. Finally, the work concludes with future directions and serves as the opening of a new research avenue for developing efficient RS applications under the FL paradigm. The source code is publicly available at https://github.com/hpc-unex/FLmeetsRS. • This study reviews multiple federated learning methodologies for remote sensing • The experimentation results highlight privacy, scalability and classification. • The work sets the future directions of federated learning in remote sensing. • This decentralized approach opens a new avenue for remote sensing applications. [ABSTRACT FROM AUTHOR]

As random noises contained in the high-frequency data could interfere with the feature learning of deep networks, low-pass filtering or wavelet transform have been integrated with deep networks to exclude the high-frequency component of input image. However, useful image details like contour and texture are also lost in such a process. In this paper, we propose D ual-branch i nteractive C ross-frequency a ttention N etwork (DiCaN) to separately process low-frequency and high-frequency components of input image, such that useful information is extracted from high-frequency data and included in deep learning. Our DiCaN first decomposes input image into low-frequency and high-frequency components using wavelet decomposition, and then applies two parallel residual-style branches to extract features from the two components. We further design an interactive cross-frequency attention mechanism, to highlight the useful information in the high-frequency data and interactively fuse them with the features in low-frequency branch. The features learned by our framework are then applied for both image classification and object detection and tested using ImageNet-1K and COCO datasets. The results suggest that DiCaN achieves better classification performance than different ResNet variants. Both one-stage and two-stage detectors with our DiCaN backbone also achieve better detection performance than that with ResNet backbone. The code of DiCaN will be released. • Learn deep features from images high-frequency component. • Design interactive cross-frequency attention to fuse different frequency features. • Propose a general dual-branch deep network for better deep feature. [ABSTRACT FROM AUTHOR]

The accurate classification of Typhoon Intensity (TI) based on cloud patterns in satellite images is crucial for effective disaster warning and management. However, this task is particularly challenging due to the striking visual similarities among different sub-classes of typhoons, coupled with a long-tailed distribution of class occurrences. Existing deep learning methods often struggle with biased classification stemming from imbalanced datasets and face difficulty discerning subtle differences between categories. This paper proposes a novel solution to these challenges through a hybrid framework that integrates contrastive learning and classifier learning. Contrastive learning is employed to increase the separation of similar sub-classes within the feature space, while classifier learning aims to train a discriminative and unbiased classifier. The proposed approach is evaluated extensively on the publicly available DeepTI dataset, demonstrating enhanced performance for both prominent and less frequent classes. The model achieves the 70.88% accuracy for seven categories typhoon intensity classification. The code implementation is at https://github.com/chen-huanxin/contrastive-hybrid-network. • The proposed network combines contrastive learning with classifier learning. • The proposed network can learn the relation between samples. • Contrastive learning provides the network with more supervision. • The proposed network can learn an unbiased classifier from imbalanced dataset. [ABSTRACT FROM AUTHOR]

Multivariate time series classification is a significant research topic in the realm of data mining, which encompasses a wide array of practical applications in domains such as healthcare, energy systems, and traffic. The complex temporal and spatial dependencies inherent in multivariate time series pose challenges to classification tasks. Previous studies have usually focused on the time-domain information of multivariate time series. However, achieving accurate classification using only the time-domain information may be difficult. To address this challenge, a time and frequency-domain feature fusion network (TF-Net) for multivariate time series classification is proposed in this paper. Our model contains two modules, the time-domain module and the frequency-domain module. The time-domain module is used to capture the time-domain features of multivariate time series. It is constructed using CNNs and GCNs, enabling the capture of both temporal and spatial dependencies within the time-domain. The frequency-domain module is used to capture the frequency-domain features of multivariate time series data. In this module, we treat the frequency-domain features as images and innovatively transform the multivariate time series classification task into an image classification task. Our method is able to classify multivariate time series from both the time-domain and frequency-domain, which provides a new perspective for multivariate time series analysis. We conduct extensive experiments on the UAE archive, and the experimental results show that our model achieves the best performance compared to the ten state-of-the-art methods. • We proposed a deep model that captures the spatiotemporal dependence of time series. • We transform the time series classification task into an image classification task. • Our method can classify time series from both time and frequency-domain perspectives. [ABSTRACT FROM AUTHOR]

The approaches based on vision transformers (ViTs) are advancing the field of medical artificial intelligence (AI) and cancer diagnosis. Recently, many researchers have developed artificial intelligence methods for cancer diagnosis based on ViTs. In this paper, 98 pertinent articles since 2020 were carefully chosen from digital databases, including Google scholar, Elsevier, and Springer Link, to review the research progress of artificial intelligence methods for cancer imaging based on ViT. Method : The basic structure of ViT is introduced, and corresponding modules such as patch embedding, positional embedding, transformer encoder, multi-head self-attention (MSA), layer normalization (LN), and residual connections, multilayer perceptron (MLP) are elaborated; a comprehensive review of improved ViT models in the medical field is presented. The application of ViT technology in cancer analysis based on medical images was reviewed. Results : ViT has achieved great success in cancer diagnosis based on medical images, showing its advantages in image classification, image reconstruction, image detection, image segmentation, image registration, image fusion, and other tasks. In these task studies, the most common task is cancer image classification and segmentation. There is still a lot of room for improvement in the aspects of multi-task learning, multi-modal learning, model generality, generalization ability, and explainability, and it also faces the mutual restriction of model scale and performance. Conclusion : The ViT training model for cancer diagnosis can potentially improve. The ViT model of self-supervised learning and semi-supervised learning mechanism is promising research. The lightweight attention module design, ViTs based on mobile networks, and the development of 3DViT will promote cancer diagnosis based on medical images to be more accurate and efficient. [ABSTRACT FROM AUTHOR]

Facial phenotypes are extensively studied in medical and biological research, serving as critical markers that potentially indicate underlying genetic traits or medical conditions. With the recent advancements in big data, algorithms, and hardware, deep facial diagnosis, which employs deep learning techniques to systematically examine facial phenotypes and identify signs of certain diseases or medical conditions, has attracted significant attention and research, gradually emerging as a promising tool in precision medicine. Primarily limited by the scarcity of data for training facial diagnosis models, the accuracy of facial diagnosis for various conditions remains low up to now. In the past decade, RGB-D cameras, measuring depth information along with standard RGB capabilities, have proven superior in processing spatial details with more stability and accuracy. Motivated by the facts mentioned above, in this paper, we propose a Simulated Multimodal Framework, which effectively improves the computer-aided facial diagnosis performance of state-of-the-art models in experiments under different conditions. The underlying principle is to leverage the simulated depth by generative models to improve the performance of RGB image recognition. Furthermore, as a rapid and non-invasive tool for disease screening and detection, our proposal demonstrated an average accuracy improvement of over 20% compared to practicing physicians in the study. • Deep facial diagnosis offers rapid, non-invasive disease detection and screening. • Simulated Multimodal Framework effectively leverages estimated face depth. • Fine-grained bilinear classification boosts facial diagnosis with limited data. • Our proposal markedly elevates facial diagnosis performance in RGB mode. • AI surpasses practicing physicians with over 20% accuracy boost in the study. [ABSTRACT FROM AUTHOR]

In recent years, the use of deep learning methods in railway defect detection has expanded rapidly due to the technology's ability to improve accuracy and efficiency in fault diagnosis. However, modeling deep learning applications requires a large amount of data, and the labeling process for such datasets demands significant manpower. To address this gap this paper introduces a semi-supervised learning approach using a student–teacher model with YOLOv4 for automatically labeling images, aiming to reduce the manual effort required for image classification. We also present a novel dataset generated from images provided by the TCDD (The State Railways of the Republic of Turkey), which includes five distinct railway defects. Results of the experiments on the same test dataset show that the proposed student–teacher model not only improves YOLOv4's detection performance according to several decision metrics but also extends the training set with high-confidence pseudo-labeled images. • A novel dataset is generated for five common railway defects. • Student–teacher model effectively generates accurate labels for railway defects. • Automated image labeling process via student–teacher model minimizes manual effort. • Significant increase is achieved in mAP and F1-score in defect detection. [ABSTRACT FROM AUTHOR]

Bringing artificial intelligence on board Earth observation satellites unlocks unprecedented possibilities to extract actionable items from various image modalities at the global scale in real time. This is of paramount importance nowadays, as downlinking large amounts of imagery is not only prohibitively expensive but also time-consuming. However, building deep learning solutions that could be deployed on board an edge device is challenging due to the limited manually-annotated satellite datasets and hardware constraints of an edge device. This paper addresses these challenges through harnessing a blend of data-centric and model-centric approaches to build a well-generalizing yet efficient and resource-frugal deep learning model for multi-class satellite image classification in the few-shot learning settings. This integrated strategy is formulated to enhance classification performance while accommodating the unique demands of an image analysis chain on board OPS-SAT, a nanosatellite operated by the European Space Agency. The experiments performed over a real-world dataset of OPS-SAT images delves into the interactions between data- and model-centric techniques, underscores the significance of synthesizing artificial training data and emphasizes the value of ensemble learning. However, they also caution against negative transfer in domain adaptation. This study sheds light on effective model training strategies and highlights the multifaceted challenges inherent in deep learning for practical Earth observation, contributing insights to the field of satellite image classification within the constraints of nanosatellite operations. To ensure reproducibility of our study, the implementation is available at https://github.com/ShendoxParadox/Few-shot-satellite-image-classification-OPS-SAT. • We propose data- and model-centric approach to few-shot image classification. • We establish a framework for building deep learning models from limited samples. • We investigate our approaches over the benchmark images captured by OPS- SAT. • We ensure full reproducibility of our study. • The proposed techniques offer high classification accuracy and resource frugality. [ABSTRACT FROM AUTHOR]

Fine-grained image recognition (FGIR) is a challenging task as it requires the recognition of sub-categories with subtle differences. Recently, the swin transformer has shown impressive performance in various fields. Our research has shown that swin transformer applied directly to FGIR is also highly effective compared to many other approaches and can be further enhanced with adaptive improvements. In this paper, we propose a novel swin transformer based architecture, named SwinFG, which enhances FGIR by leveraging shifted window based self-attention to locate discriminative regions. The self-attention computation fuses image patches together based on attention weights, enabling the subsequent influence of each patch to be tracked and its contribution to the extracted feature to be determined. This forms the basis for locating discriminative regions. To this end, we propose a series of transformations that integrate the attention weights of local windows in each block into attention maps, which can be recursively multiplied to track changes in the attention weights. As the discriminative regions are not entirely occupied by the foreground object, the background information is also expressed in the extracted feature inevitably. To address this, we propose conducting contrastive learning on features obtained from both the discriminative and background regions of a single image to enlarge their distance and further eliminate any potential influence from the background. We demonstrate the state-of-the-art performance of our model on four popular fine-grained benchmarks. (The code is available at https://anonymous.4open.science/r/swinFG-1DCE). • We enhance the swin transformer for FGIR, providing alternatives to CNN and ViT. • Our method integrates attention weights into maps for precise region tracking. • We mitigate background interference through fore-background contrastive learning. [ABSTRACT FROM AUTHOR]

Feature extraction is one of the fundamental problems in pattern recognition research. For image recognition, extracting effective image features is the key to accomplish the recognition task. In this paper, a partial differential equations-based multi-rank two-dimensional canonical correlation analysis (PDEs-MR2DC2A) is proposed for multi-view feature extraction and pattern classification. Unlike most of the previous researches on multi-view algorithms that work directly on the original 2D representation, in our approach, we first utilize the evolution process of PDEs to extract the feature matrix of per-view. In addition, we employ multi-rank left and right projecting matrices to maximize the correlation. The computational complexity of PDEs-MR2DC2A is also analyzed. To evaluate the effectiveness of the proposed algorithm, we conducted a series of performance comparisons with some existing methods on several popular datasets. The experimental results showed that our proposed algorithm performed very well on these datasets and outperformed the existing related methods on some metrics. • A novel multi-rank multi-view algorithm based on PDEs is proposed. • This approach can extract rotation and translation invariant features. • Numerous experimental results demonstrate the effectiveness of our proposed method. [ABSTRACT FROM AUTHOR]

In 2016, Google proposed Federated Learning (FL) as a novel paradigm to train Machine Learning (ML) models across the participants of a federation while preserving data privacy. Since its birth, Centralized FL (CFL) has been the most used approach, where a central entity aggregates participants' models to create a global one. However, CFL presents limitations such as communication bottlenecks, single point of failure, and reliance on a central server. Decentralized Federated Learning (DFL) addresses these issues by enabling decentralized model aggregation and minimizing dependency on a central entity. Despite these advances, current platforms training DFL models struggle with key issues such as managing heterogeneous federation network topologies, adapting the FL process to virtualized or physical deployments, and using a limited number of metrics to evaluate different federation scenarios for efficient implementation. To overcome these challenges, this paper presents Fedstellar, a novel platform designed to train FL models in a decentralized, semi-decentralized, and centralized fashion across diverse federations of physical or virtualized devices. Fedstellar allows users to create federations by customizing parameters like the number and type of devices training FL models, the network topology connecting them, the machine and deep learning algorithms, or the datasets of each participant, among others. Additionally, it offers real-time monitoring of model and network performance. The Fedstellar implementation encompasses a web application with an interactive graphical interface, a controller for deploying federations of nodes using physical or virtual devices, and a core deployed on each device, which provides the logic needed to train, aggregate, and communicate in the network. The effectiveness of the platform has been demonstrated in two scenarios: a physical deployment involving single-board devices such as Raspberry Pis for detecting cyberattacks and a virtualized deployment comparing various FL approaches in a controlled environment using MNIST and CIFAR-10 datasets. In both scenarios, Fedstellar demonstrated consistent performance and adaptability, achieving F 1 s c o r e s of 91%, 98%, and 91.2% using DFL for detecting cyberattacks and classifying MNIST and CIFAR-10, respectively, reducing training time by 32% compared to centralized approaches. • Open-source platform for training FL models in a decentralized fashion. • Support for diverse FL architectures, network topologies, datasets, and ML/DL models. • User-friendly setup and efficient monitoring of FL processes. • Successful deployment across diverse physical and virtualized devices. • Promising performance in tasks like cyberattack detection and image classification. [ABSTRACT FROM AUTHOR]

This paper presents a novel resume parser designed to effectively reorganize the textual content of any resume into its original section structure. Our work addresses two practical challenges overlooked by the existing literature: (i) ensuring the correct reading order of text retrieved from resume files and (ii) extracting individually all sections, as well as work experience and education subsections. By taking into account the observation that most resumes adhere to basic document templates, we reframe the reading order problem as a template identification task. Our experiments suggest that even a widely-used small model like EfficientNet-B0 can accurately identify common templates. Additionally, we propose a sequence tagging approach that simultaneously identifies all resume sections and some subsections. We implement and compare two solutions based on the well-known CRF and BERT models. Our evaluation provides strong evidence that the CRF can serve as a practical alternative to BERT, depending on hardware and budget constraints. They yield comparable results in terms of identifying resume sections, while BERT displays a substantial advantage when identifying education and work experience subsections. An interesting direction for future work is to expand our approach to ensure the correct ordering of a large family of templates. • A parser to retrieve the resume structure from a given pdf document. • A simple approach to ensure the correct resume reading order. • Two segmentation models to extract the sections and subsections from a resume. • An anonymized dataset for the resume template identification problem. [ABSTRACT FROM AUTHOR]

Understanding the attitudes of users about different topics through their huge amount of comments and opinions on social networks is an emerging hot issue. Analyzing the sentiments of these posts and comments provides useful information utilized for many applications. Multimodal sentiments often contain more information than single-modal sentiments, including text, image, audio and video, which leads to better performance of multimodal sentiment analysis (MSA) compared to single-modal sentiment analysis (SSA). In this paper, a capsule network-based deep ensemble transfer learning approach, called DSY-ETL-MSA, is proposed for MSA on images and texts, the results of which are fused using Yager theory. Capsule networks and ensemble learning methods improve classification performance, and the deep transfer learning approach reduces the training time. A hybrid deep architecture is used for automatic feature extraction in the proposed method. For analyzing the sentiment of image modality, a pre-trained VGG16 model, fine-tuned on the datasets, is used to extract high-level features for image classification. A capsule convolutional neural network (CNN) is also separately used for extracting image features and classifying them. In the text modality, the pre-trained GloVe model is exploited to embed words and 2 separate classifiers are employed for text classifications. The Yager fusion rules are finally used for early and late fusions of the results of the classifiers. The results of the text and image classifiers are combined separately as the early fusion and the final results of them are fused at the decision level as the late fusion. The proposed model is empirically evaluated on the MVSA and T4SA datasets. The significant performance improvement compared to a variety of former methods is shown in the experimental results. The final accuracies obtained by the proposed method on the MVSA and T4SA datasets are 0.9866 and 0.9996, respectively. [ABSTRACT FROM AUTHOR]

Different cultural backgrounds give birth to distinct styles of cultural heritage. In order to explore the relationship between the cultural relics' materials and the specific cultural attributes, a new method for the recognition of cultural relics' image materials based on computer vision and attention mechanism is proposed. Moreover, this paper analyzes the relationship between Chinese traditional cultural relics and Chinese dynasties as an example. The methodology of this work consists first of using Python to collect and manually screen cultural relic images of common material types. Then, the nine datasets with different materials are uploaded to the EfficientNet-B0 network with the attention mechanism for iterative training. The best weight model is stored and put to the test. Finally, the improved EfficientNet-B0 network is applied to recognize the cultural relics image datasets of each dynasty, and the relationship between the materials and the cultural attributes of each dynasty is analyzed. As for the outcomes, the experimental results show that the EfficientNet-B0 model, with the attention mechanism, can effectively enhance the extraction of image material information, and the accuracy of the recognized cultural relics image materials reaches up to 88.15%, with an average precision of 83.3%. The comparison experiment on the material dataset shows that the proposed method has excellent ability in the recognition of material image compared with other common image recognition methods. [ABSTRACT FROM AUTHOR]

Image resizing is frequently used as a preprocessing step in many computer vision tasks, especially in medical applications. While tuning of the resizing method is usually omitted in the studies, there are many problems in which the exact influence of resampling on image textures and gradients is significant. The paper presents an in-depth analysis of image reconstruction's impact on two inherent tasks in medical image analysis: segmentation and classification. The proposed study is conducted on the renal diagnosis dataset in which the kidney is segmented, and three renal tumours are classified. A novel image reconstruction method is introduced, namely Sampling Kantorovich Algorithm (SKA). It is compared to six other popular techniques widely used in image processing. Based on the qualitative and quantitative analyses, we proved that choice of image reconstruction method impacts the system's overall performance. SKA turns out to be the best performing method in the classification setup. It boosts performance to 75% of the weighted F1-score by approximately 3 percentage points (pp) compared to the best baseline solution. In kidney segmentation, the SKA improves efficiency by over 2pp compared to other resizing methods. The results presented in this paper may apply to a wide range of medical image processing problems. • Image resolution enhancement is beneficial for computer-aided diagnosis systems. • The Sampling Kantorovich Algorithm (SKA) improves the quality of CT scans. • SKA works well in neural network-based systems for classification and segmentation. • The proposed method boosts the performance of renal image recognition by 2-3pp. • Both downstream tasks equally benefit from the additional processing step. [ABSTRACT FROM AUTHOR]

Many routing algorithms in capsule networks (CapsNets) suffer seriously from a large amount of parameter training during iterations. To address this issue, a non-iterative CapsNet with interdependent agreement routing (IAR-CapsNet) is proposed in this paper, which relieves the parameter redundancy by reducing the pose dimensionality as well as obtains desired performance by introducing descriptive subcomponents. A dual attention mechanism is developed to pay special attention to equivariant pose information and invariant descriptor information. To be specific, the key information weights are assigned to different components separately through a multi-branch attention mechanism to fulfill the information transfer between the capsule layers. Then, a reciprocal agreement routing without iterations is designed to enhance such information transfer between layers. Finally, the experiments show that the proposed IAR-CapsNet achieves the best classification accuracy with only a 1.36% error rate on the smallNORB dataset and the best robustness performance in robustness experiments. • A novel capsule network with non-iterative and interdependent agreement routing. • A dual-attention mechanism is developed to respectively concentrate to pose and descriptor information. • Capsule with equivariant pose and invariant descriptor. • Theoretical analysis for the necessity of the squashing function. [ABSTRACT FROM AUTHOR]

In this paper, we introduce the Spatial Bias to learn global knowledge without self-attention in convolutional neural networks. Owing to the limited receptive field, conventional convolutional neural networks suffer from learning long-range dependencies. Non-local neural networks have struggled to learn global knowledge, but unavoidably have too heavy a network design due to the self-attention operation. Therefore, we propose a fast and lightweight Spatial Bias that efficiently encodes global knowledge without self-attention on convolutional neural networks. Spatial Bias is stacked on the feature map and convolved together to adjust the spatial structure of the convolutional features. Because we only use the convolution operation in this process, ours is lighter and faster than traditional methods based on the heavy self-attention operation. Therefore, we learn the global knowledge on the convolution layer directly with very few additional resources. Our method is very fast and lightweight due to the attention-free non-local method while improving the performance of neural networks considerably. Compared to non-local neural networks, the Spatial Bias use about × 10 times fewer parameters while achieving comparable performance with 1.6 ∼ 3.3 times more throughput on a very little budget. Furthermore, the Spatial Bias can be used with conventional non-local neural networks to further improve the performance of the backbone model. We show that the Spatial Bias achieves competitive performance that improves the classification accuracy by + 0. 79 % and + 1. 5 % on ImageNet-1K and CIFAR-100 datasets. Additionally, we validate our method on the MS-COCO and ADE20K datasets for downstream tasks involving object detection and semantic segmentation. • We introduce Spatial Bias, a lightweight non-local method in neural networks. • The proposed method learns long-range dependencies with a convolution layer. • The proposed Spatial Bias is fairly quick and show competitive performance. • The proposed Spatial Bias can be used with the existing non-local methods. [ABSTRACT FROM AUTHOR]

Artificial intelligence (AI) technology has significant potential in Earth sciences, particularly in mineral identification for industrial exploration, geological mapping, and archaeological research. However, traditional methods are time-consuming, expensive, and complex. And existing mineral identification methods based on mineral photos face several critical challenges, including lack of consideration for natural image features captured in real environments, limitations of single-label classification which does not align with multi-mineral occurrences in nature, and growing computational complexity as the number of identifiable mineral labels increases. Therefore, this paper proposes an efficient mineral identification model based on multi-label image classification, focusing on natural environmental features. First, realistic feature datasets are created by simulating mineral photos in real environments. Then, the model uses the query-label (Query2Label) framework, with MaxViT-T (Multi-Axis Vision Transformer-Tiny) as the feature extraction network and the asymmetric loss function. Knowledge distillation is employed to improve identification accuracy while reducing computational complexity. The proposed model achieves an impressive average identification accuracy of 84.74% on a dataset of 495,756 mineral photos, surpassing existing models like ResNet-101, ML-GCN (Multi-Label Graph Convolutional Network), and SRN (Spatial Regularization Net). It maintains a lower parameter count and computational complexity. In the end, ablation experiments demonstrate the effectiveness of each optimization scheme. • Selecting image noise and color shift methods for realistic mineral data simulation. • Proposing an enhanced two-stage image multi-Label identification model framework. • Applying knowledge distillation for multi-label classification subtask decomposition. • Being with low computational complexity and high mineral identification accuracy. [ABSTRACT FROM AUTHOR]

In this paper, we present a novel dimensionality reduction term (i.e. TripletPCA and ContrastivePCA) designed to enhance the robustness of pair-based loss functions in metric learning against image corruptions. Notably, our approach achieves this without the need for any data preprocessing steps. Our method can be seamlessly integrated into existing models, such as a block with Plug & Play framework. To the best of our knowledge, our TripletPCA and ContrastivePCA represent the first attempts to incorporate dimensionality reduction directly into pair-based metric learning losses for end-to-end metric learning. By projecting image features into low-dimensional vectors, our proposed loss functions effectively retain the essential components of images while mitigating the impact of corrupted features. Consequently, our metric learning loss function accurately computes feature distances through the projection. Experimental results demonstrate that our dimensionality reduction term can be easily incorporated into various types of existing deep neural networks. This integration leads to a substantial improvement in performance on standard benchmark datasets for corrupted image classification tasks. On several corruption datasets, we achieve an average performance improvement of 10.55% compared to existing baseline methods. Our code is available at https://github.com/Juryun/TDRL. [ABSTRACT FROM AUTHOR]

Deep learning-based no-reference image quality assessment (NRIQA) methods have demonstrated advanced performance. In this paper, a deep learning-based NRIQA method with strong error-aware and content-aware capabilities is proposed, which consists of Error-Aware Reconstruction Network (EARNet) module, Content Feature Extraction Network (CFENet) module, and Subjective Quality Regression Network (SQRNet) module. We first build a database to pre-train EARNet to obtain the ability to extract error features. For content features, CFENet pre-trained on large-scale image classification tasks is adopted to extract. The pre-trained EARNet and CFENet are serially connected with SQRNet so that the features received by SQRNet are both error-aware and content-aware. Extensive experimental results show that the proposed method achieves state-of-the-art performance on many well-known IQA databases. The robustness of the proposed method is verified on the large-scale Waterloo Exploration Database (WED), and its superiority is demonstrated by the group maximum differentiated (gMAD) competition game. Furthermore, we also verify that the proposed EARNet is highly extensible, which can further improve the performance of the existing deep learning-based NRIQA method. • A blind image quality assessment method based on deep learning is proposed. • The proposed model has both error and content awareness. • The proposed method has advanced performance and high scalability. • Creatively designed a joint loss to guide the model in generating error maps. [ABSTRACT FROM AUTHOR]

The performance of diagnostic Computer-Aided Design (CAD) systems for retinal diseases depends on the quality of the retinal images being screened. Thus, many studies have been developed to evaluate and assess the quality of such retinal images. However, most of them did not investigate the relationship between the accuracy of the developed models and the quality of the visualization of interpretability methods for distinguishing between gradable and non-gradable retinal images. Consequently, this paper presents a novel framework called "FGR-Net" to automatically assess and interpret underlying fundus image quality by merging an autoencoder network with a classifier network. The FGR-Net model also provides an interpretable quality assessment through visualizations. In particular, FGR-Net uses a deep autoencoder to reconstruct the input image in order to extract the visual characteristics of the input fundus images based on self-supervised learning. The extracted features by the autoencoder are then fed into a deep classifier network to distinguish between gradable and ungradable fundus images. FGR-Net is evaluated with different interpretability methods, which indicates that the autoencoder is a key factor in forcing the classifier to focus on the relevant structures of the fundus images, such as the fovea, optic disk, and prominent blood vessels. Additionally, the interpretability methods can provide visual feedback for ophthalmologists to understand how our model evaluates the quality of fundus images. The experimental results showed the superiority of FGR-Net over the state-of-the-art quality assessment methods, with an accuracy of > 89 % and an F1-score of > 87 %. The code is publicly available at https://github.com/saifalkh/FGR-Net. • FGR-NET is a deep learning model to assess fundus image gradeability. • FGR-NET consists of two parallel deep networks: autoencoder and classifier. • Using self-supervised learning, autoencoder is to extract the images' key features. • A classifier is to distinguish between gradable and ungradable fundus images. • Interpretability is performed to understand how FGR-NET evaluates the gradeability. [ABSTRACT FROM AUTHOR]

Spiking neural networks (SNNs) have garnered significant attention due to their ultra-high-speed and ultra-low-power operation, rendering them suitable for a range of energy-efficient applications. However, their performance in image generation is relatively mediocre, resulting in a lack of an effective generative replay mechanism to handle sequential probabilistic representations of multiple datasets. Consequently, this limitation often leads to catastrophic forgetting. This paper introduces a novel SNN lifelong learning framework namely Dynamic Lifelong learning with Spiking Generative Networks (DL-SGN), aimed at providing better generation ability while mitigating the problem of catastrophic forgetting in a continual learning scenario. DL-SGN comprises three key components: dynamic experts, a student, and an assistant. The dynamic expert is implemented as a dynamically expanding mixture model, with a proposed network expansion mechanism Dynamic Knowledge Adversarial Fusion (DKAF) facilitating the automatic handling of an increasing number of tasks. the student module adopts an SNN-based Variational Autoencoder (VAE) and classifier, leveraging accumulated knowledge from each expert. To enhance the student's ability to generalize to images, we introduce a discriminator as an assistant module trained using adversarial training. We validate the effectiveness of the proposed by conducting experiments on image generation and classification in a lifelong learning environment, and the results substantiate the effectiveness of DL-SGN. • The DL-SGN framework is proposed to addresses catastrophic forgetting in SNNs. • Our model is based on multiple dynamic experts and adversarial training. • Develop a dynamic knowledge adversarial fusion to expand expert module's capacity. • Employ an assistant module to enhance the student module's generalization ability. [ABSTRACT FROM AUTHOR]

Use of social media for communication, sharing or expressing views, broadcasting news, threatening and blackmailing has become an integral part of society. One such activity is understanding multi-cultural wedding images uploaded on social media. This paper presents a novel method based on the combination of U-Net, Convolutional Neural Network and Random Forest for classification of multicultural wedding images. In the case of wedding images, bride and bridegroom draw the attention of the viewers. This observation led to propose a U-Net for segmenting the region of bride and bridegroom in a novel way. Similarly, it is noted that the costumes of bride and bridegroom are vital information for differentiating different cultures. This cue motivated us to extract features using CNN for classification. Since the extracted features using CNN are capable of discriminating images of different classes, we propose a simple and effective Random-Forest for Multicultural Wedding Image Classification. The efficiency of the proposed model is demonstrated by testing it on our own dataset of six multi-cultural wedding classes and standard dataset of wedding and non-wedding images classes. Experimental results on both the datasets show that the proposed model outperforms the state-of-the-art models in terms of average classification rate. [ABSTRACT FROM AUTHOR]

Domain adaptation provides the possibility of utilizing the knowledge gained from an auxiliary domain to accomplish the task in another related domain. In this paper, a common feature representation of these domains is learned through a new proposed idea to reduce inter-domain differences more precisely which results in higher accuracy for unsupervised domain adaptation. To decrease these divergences, the proposed method finds a subspace that reduces data distribution discrepancy between domains. To achieve this goal, minimization of the divergence of the class conditional probability distributions is considered. By considering class conditional distributions, more discriminative information of data is preserved compared to applying marginal distributions. Also, an explicit parametric distribution of the source and target domains is considered to reduce the discrepancy between the two domains data which results in higher accuracy compared to the other relevant domain adaptation methods. Experimental studies on benchmark image classification tasks confirmed our assumptions and indicated the significant improvement given by the proposed method compared to the other state-of-the-art methods. • Aligning the feature distribution of the source and target domains explicitly. • Introducing a new loss function to impose the common distribution on the features. • Reducing the class conditional probability distribution distance between the domains. [ABSTRACT FROM AUTHOR]

Deep learning architectures have demonstrated outstanding results in image classification in the last few years. However, applying sophisticated neural network architectures in small datasets remains challenging. In this context, transfer learning is a promising approach for dealing with this scenario. Generally, the available pre-trained architectures adopt a standard fixed input, which usually implies resizing and cropping the input images in the preprocessing phase, causing information loss. Besides, images present visual features in different scales in real-world scenarios, and most common approaches do not consider this fact. In this paper, we propose an approach that applies transfer learning for dealing with small datasets and leverages visual features extracted by pre-trained models from different scales. We based our approach on graph convolutional networks (GCN) that take graphs representing the images in different scales as input and whose nodes are characterized by features extracted by pre-trained models from regular image patches of different scales. Since GCN can deal with graphs with different numbers of nodes, our approach can deal naturally with images of heterogeneous sizes without discarding relevant information. We evaluated our approach in two datasets: a set of geological images and a publicly available dataset, both presenting characteristics that challenge traditional approaches. We tested our approach by adopting three different pre-trained models as feature extractors: two efficient pre-trained CNN models (DenseNet and ResNeXt) and one Vision Transformer (CLIP). We compared our approach with two conventional approaches for dealing with image classification. The experiments show that our approach achieves better results than the conventional approaches for this task. • We propose multiscale patch-based graphs to represent image information. • Transfer learning is adopted for dealing with small datasets. • The approach can handle images with heterogeneous sizes. • The approach represents visual information at multiple scales. • The approach is base on convolutional graph neural networks. [ABSTRACT FROM AUTHOR]

This paper aims to address the challenges posed by complex scenery image classification. Most of the existing deep learning networks are trained and evaluated using ImageNet. However, when these models are applied to scenery images, dramatic performance degradation is observed due to the change in data characteristics. To challenge the prevailing practices in network design, we investigate the impact of altering data on the performance of deep networks. Specifically, we introduce a novel data-oriented network design to emphasize the importance of considering the unique characteristics of the data. Our proposed approach is a Deep-Narrow Network, which incorporates a Dilated Pooling module built upon the ResNet architecture. Compared to ResNet, our approach achieves a significant reduction of floating-point operations by 51.5% and in the number of parameters by 54.5%. Remarkably, despite the reduction in computational complexity and model size, our design exhibits a 0.4% increase in overall accuracy. This approach offers an efficient and effective means of scaling the network according to the data characteristics while maintaining highly competitive performance. • We analyze the effect of depth and width in CNNs from the perspective of image characteristics. • We demonstrate that deeper networks are more suitable for complex scene recognition. • We propose a Deep-Narrow Network with Dilated Pooling for improved scene recognition. [ABSTRACT FROM AUTHOR]

In recent years, pruning methods have been proposed to reduce the size of image classification models based on CNNs and to shorten their inference times. However, most of them are based on setting the less important parameters of the model to zero, but not on reshaping the network. Therefore, in this paper we propose a pruning methodology for sequential CNNs that modifies the shape of the network (both convolutional and fully connected layers) and was applied to four sequential networks (a custom, AlexNet, VGG11 and VGG16) for the "Original and counterfeit Colombian peso banknotes" dataset. The pruned models were evaluated quantitatively and qualitatively. First, in terms of accuracy against FLOPs and parameter reduction. Second, using HiResCAM to explain the patterns on which the model is based for decision making. Quantitative results show a reduction in parameters and FLOPs of about 75% for a reduction in accuracy of up to 0.5% across all models in this study. With larger reductions of about 95%, the AlexNet-pruned and VGG16-pruned models significantly reduced their accuracy (38.1% and 21.9%, respectively), while the custom-pruned model and the VGG11-pruned model reduced their accuracy by only 0.3% and 0.9%, respectively. Furthermore, using HiResCAM, it was observed that the custom-pruned model and the VGG11-pruned model better preserved the original model activations, even at high pruning percentages, and achieved lower accuracy reductions than their competitors. With the proposed methodology, any sequential model can be pruned and reshaped while largely retaining the accuracy of the unpruned model, or even improving it for low pruning percentages. [ABSTRACT FROM AUTHOR]

Emotion recognition based on facial expressions has low accuracy and doubtful reliability because of the existence of fake expressions. In this paper, a method is proposed to recognize fake emotions based on multivisual information generated from single information sources, including facial expressions, eye states and physiological signals captured from video. An algorithm based on a graph neural network is used to extract spatial and spectral domain features from facial images for facial expression recognition. A model-based method is used for decomposing RGB signals into heart rates. A deep model trained by a labeled dataset that we created is used to segment the eye region. After obtaining the signals extracted from video, different fusion strategies are applied to evaluate emotion recognition performance based on multiple signals. In the experiment, the CK+, TFEID, JAFFE, RAF, PURE, and ESLD datasets are used to measure the accuracy of facial expression recognition, heart rate detection and eye region segmentation. The results show that multimodality is effective in improving accuracy and that eye state can be considered a cue for trusted emotion recognition. Compared with the method based on the eye state and noncontact physiological signal, the accuracy of multimodality can be improved by 26.19% and 9.52%, respectively. [ABSTRACT FROM AUTHOR]

• An Image-to-text model NeoDescriber for Neoclassical building description. • Four architectural image databases were built to train visual modules. • Higher BERTScore than BLIP-2 for architectural style description. Architectural style description is important for the renewal and reuse of historical monuments. However, the identification of Neoclassical buildings is more difficult than that of other European architectural styles owing to the revival of classicism and its numerous variants. This paper proposes the image-to-text model NeoDescriber for the automatic identification and description of Neoclassical buildings based on the combination of dedicated visual classification and detection modules organized by expert knowledge in architecture. NeoDescriber takes building façade images as input and follows a coarse-to-fine description logic to generate structured annotations covering the whole building and key building elements. We built several building image datasets composed of more than 300 historical monuments to train and validate different visual modules. The results showed the acceptance performance of both classification and detection modules, with average precisions ranging from 73.6% to 95.2%. The generated text descriptions were then compared with results from BLIP-2 by qualitative and quantitative analyses. With human descriptions as a reference, the results showed that NeoDescriber's generated descriptions were more precise and detailed, having a higher BERTScore for Neoclassical monuments. Thus, NeoDescriber could successfully recognize and describe Neoclassical buildings. [ABSTRACT FROM AUTHOR]

Images contain rich information and can induce various emotions in the audience. Image emotion classification aims to identify the emotion categories that images can evoke. It is widely used in mental health assessment, human–computer interaction, etc. There are two main problems in existing image emotion classification methods: (1) Most of them only focus on a single emotion label; (2) The global structural relationship among semantic objects in the image is ignored. Therefore, this paper proposes an Image Emotion classification method based on Multi-Graph Multi-Label learning (IE-MGML). In contrast to the existing approaches, the image is transformed into a graph-based representation by extracting the emotional features of semantic objects and calculating the similarity between the features. The local (semantic objects) features and global structure (relationship among semantic objects) features of the image are fused by the relationship between nodes. Furthermore, the graph representation of an image from the perspective of multiple emotional features is pooled and modeled as a graph bag containing multiple graphs (i.e., multi-graph). In multi-graph learning, the graph kernel directly evaluates a graph-label dependency score to avoid the loss of structural information caused by graph-instance degradation. The bag(image)-label dependency score is obtained by aggregating the graph-label dependency score from different perspectives through the aggregation function. The problem of error accumulation in the learning process is handled by proposing a threshold-based ranking loss objective function. Moreover, the non-convex optimization problem is addressed using a subgradient descent algorithm to deal with the required high-dimensional space computation. Experimental results on three general image emotion datasets show that the proposed method outperforms the state-of-the-art methods. • Model the image emotion classification problem as a multi-graph learning problem. • Propose a multi-graph multi-label learning algorithm based on global structure. • Build three image emotion datasets in the form of multi-graph multi-label. • Solve the problem of perceptual subjectivity by multi-label classification. • Propose a threshold-based loss function to solve the error accumulation problem. [ABSTRACT FROM AUTHOR]

In image set classification, affine hull based models have achieved good performance. However, these models only focus on holistic images and ignore the local information of images. To incorporate local information, this paper develops a cooperative operator and proposes a cooperative linear regression model (CLRM) for image set classification. Specifically, CLRM first proposes a block partitioning strategy to obtain local information. Image sets and local information are both modeled as affine hulls. Then, a linear regression model embedded with cooperative operator is proposed to incorporate different information and compute the distance between the different affine hulls. Finally, a combined distance measure is proposed to incorporate different information for classification. Extensive experiments on different recognition tasks such as face recognition on image/video based databases and object classification show that the proposed cooperative operator can improve the performance of the model and the proposed CLRM achieves competitive performance compared with the state-of-the-art classifiers. • Proposing a block partitioning strategy to extract local information. • Modeling local information as affine hull for image set classification. • Developing a cooperative operator to integrate different information as a whole. • Proposing a linear regression model embedded with the cooperative operator. • Proposing an efficient optimization algorithm to solve the model. [ABSTRACT FROM AUTHOR]

Content tags of a given image are the basic mechanism used in social networking services. One of the most commonly used websites like Instagram, Facebook, or Twitter is the hashtag. In this paper, we propose a hybrid solution for automatic image description extraction. A network architecture consists of a module for removing main objects from the image for additional classification of other elements on the background via U-NET (neural model designed for segmentation). Obtained images are classified by multi-branch convolutional networks. The results of such a classification are used to describe the probability of belonging to particular classes. Then, the probability of belonging to each class is used to find related linguistic values through the Skip-Gram module. The main advantage of the solution is the possibility of extending the network with additional branches to increase the number of main classes of objects. The proposal was analyzed through the object's classification, the use of learning transfer, and image description on two datasets, i.e. Animals-10 and HARRISON16. Experimental results show that the proposal is a very flexible solution for extending the model to other classes without the need to overtrain the entire architecture and reach higher classification accuracy than other models for social networking services. • A model for describing images through linguistic words that can be used as hashtags on social networking sites. • Extending neural model by adding new branches to classify images without the need to train the entire model. • Flexibility of extending the model with further linguistic classes. • Automatic image analysis by separate classification of main objects and background. [ABSTRACT FROM AUTHOR]

Content tags of a given image are the basic mechanism used in social networking services. One of the most commonly used websites like Instagram, Facebook, or Twitter is the hashtag. In this paper, we propose a hybrid solution for automatic image description extraction. A network architecture consists of a module for removing main objects from the image for additional classification of other elements on the background via U-NET (neural model designed for segmentation). Obtained images are classified by multi-branch convolutional networks. The results of such a classification are used to describe the probability of belonging to particular classes. Then, the probability of belonging to each class is used to find related linguistic values through the Skip-Gram module. The main advantage of the solution is the possibility of extending the network with additional branches to increase the number of main classes of objects. The proposal was analyzed through the object's classification, the use of learning transfer, and image description on two datasets, i.e. Animals-10 and HARRISON16. Experimental results show that the proposal is a very flexible solution for extending the model to other classes without the need to overtrain the entire architecture and reach higher classification accuracy than other models for social networking services. • A model for describing images through linguistic words that can be used as hashtags on social networking sites. • Extending neural model by adding new branches to classify images without the need to train the entire model. • Flexibility of extending the model with further linguistic classes. • Automatic image analysis by separate classification of main objects and background. [ABSTRACT FROM AUTHOR]

Classification methods that are based on hyperspectral images (HSIs) are playing an increasingly significant role in the processes of target detection, environmental management, and mineral mapping as a result of the fast development of hyperspectral remote sensing technology. Improving classification performance is still a significant problem, however, as a result of the high dimensionality and redundancy of hyperspectral image sets (HSIs), as well as the presence of class imbalance in hyperspectral datasets. In the past few years, convolutional neural networks (CNNs) and graph convolutional networks (GCNs) have achieved good results in HSI classification, but CNNs struggle to achieve good accuracy in low samples, while GCNs have a huge computational cost. To resolve these issues, this paper proposes a Multi-Feature Fusion of 3D-CNN and Graph Attention Network MFFCG. The algorithm consists of two elements: the 3D-CNN, which produces good classification for 3D HSI cube data, and GAT-based encoder and decoder modules that help in improving the classification accuracy of the 3D-CNN. Finally, the multiple features are merged with the help of two neural network models. We further develop two optimized GAT models named GAT1 and GAT2, which are used with different layers of 3D-CNN. Algorithms after merging with 3D-CNN are named MFFCG-1 and MFFCG-2, which produce better classification results then other developed methods. Experiments on three public HSI datasets show that the proposed methods perform better than other state-of-the-art methods using the limited training samples and in low classification time. [ABSTRACT FROM AUTHOR]