Search

This paper introduces Knowledge Distillation of Vision Transformer (KDViT), a novel approach for medical image classification. The Vision Transformer architecture incorporates a self-attention mechanism to autonomously learn image structure. The input medical image is segmented into patches and transformed into low-dimensional linear embeddings. Position information is integrated into each patch, and a learnable classification token is appended for classification, thereby preserving spatial relationships within the image. The output vectors are then fed into a Transformer encoder to extract both local and global features, leveraging the inherent attention mechanism for robust feature extraction across diverse medical imaging scenarios. Furthermore, knowledge distillation is employed to enhance performance by transferring insights from a large teacher model to a small student model. This approach reduces the computational requirements of the larger model and improves overall effectiveness. Integrating knowledge distillation with two Vision Transformer models not only showcases the novelty of the proposed solution for medical image classification but also enhances model interpretability, reduces computational complexity, and improves generalization capabilities. The proposed KDViT model achieved high accuracy rates of 98.39%, 88.57%, and 99.15% on the SARS-CoV-2-CT, COVID-CT, and iCTCF datasets respectively, surpassing the performance of other state-of-the-art methods.

In natural language processing (NLP), a Question Answering System (QAS) refers to a system or model that is designed to understand and respond to user queries in natural language. As we navigate through the recent advancements in QAS, it can be observed that there is a paradigm shift of the methods used from traditional machine learning and deep learning approaches towards transformer-based language models. While significant progress has been made, the utilization of these models for historical QAS and the development of QAS for Malay language remain largely unexplored. This research aims to bridge the gaps, focusing on developing a Multilingual QAS for history of Malaysia by utilizing a transformer-based language model. The system development process encompasses various stages, including data collection, knowledge representation, data loading and pre-processing, document indexing and storing, and the establishment of a querying pipeline with the retriever and reader. A dataset with a collection of 100 articles, including web blogs related to the history of Malaysia, has been constructed, serving as the knowledge base for the proposed QAS. A significant aspect of this research is the use of the translated dataset in English instead of the raw dataset in Malay. This decision was made to leverage the effectiveness of well-established retriever and reader models that were trained on English data. Moreover, an evaluation dataset comprising 100 question-answer pairs has been created to evaluate the performance of the models. A comparative analysis of six different transformer-based language models, namely DeBERTaV3, BERT, ALBERT, ELECTRA, MiniLM, and RoBERTa, has been conducted, where the effectiveness of the models was examined through a series of experiments to determine the best reader model for the proposed QAS. The experimental results reveal that the proposed QAS achieved the best performance when employing RoBERTa as the reader model. Finally, the proposed QAS was deployed on Discord and equipped with multilingual support through the incorporation of language detection and translation modules, enabling it to handle queries in both Malay and English. Doi: 10.28991/ESJ-2024-08-02-019 Full Text: PDF

In Natural Language Processing, creating training data for question answering (QA) systems typically requires significant effort and expertise. This challenge is amplified in few-shot scenarios where only a limited number of training samples are available. This paper proposes a novel pretraining framework to enhance few-shot question answering (FSQA) capabilities. It begins with the selection of the Discrete Reasoning Over the Content of Paragraphs (DROP) dataset, designed for English reading comprehension tasks involving various reasoning types. Data preprocessing converts question-answer pairs into a predefined template, consisting of a concatenated sequence of the question, a mask token with a prefix, and the context, forming the input sequence, while the target sequence includes the question and answer. The Question-Answer Replacement and Removal (QARR) technique augments the dataset by integrating the answer into the question and selectively removing words. Various templates for question-answer pairs are introduced. Models like BART, T5, and LED are then used to evaluate the framework’s performance, undergoing further pretraining on the augmented dataset with their respective architectures and optimization objectives. The study also investigates the impact of different templates on model performance in few-shot QA tasks. Evaluated on three datasets in few-shot scenarios, the QARR-T5 method outperforms state-of-the-art FSQA techniques, achieving the highest F1 scores of 81.7% in 16-shot and 32-shot, 82.7% in 64-shot, and 84.5% in 128-shot on the SQuAD dataset. This demonstrates the framework’s effectiveness in improving models’ generalization and performance on new datasets with limited samples, advancing few-shot QA.

Individual deep learning models showcase impressive performance; however, the capacity of a single model might fall short in capturing the full spectrum of intricate patterns present in the input data. Thus, relying solely on a single model may hamper the attainment of optimal results and broader generalization. In light of this, the paper presents an ensemble method that leverages the strengths of multiple Convolutional Neural Networks (CNNs) and Transformer models to elevate gait recognition performance. Additionally, a novel gait representation named windowed Gait Energy Image (GEI) is introduced, obtained by averaging gait frames irrespective of gait cycles. Firstly, the windowed GEI is input to the Convolutional Neural Networks and Transformer models to learn significant gait features. Each model is followed by a Multilayer Perceptron (MLP) to encode the relationship between the extracted features and corresponding class labels. Subsequently, the extracted gait features from each model are flattened and concatenated into a cohesive feature representation before passing through another MLP for subject classification. The performance of the proposed method was assessed on three datasets: OU-ISIR dataset D, CASIA-B, and OU-LP dataset. Experimental results demonstrated remarkable improvements compared to existing methods across all three datasets. The proposed method achieved accuracy rates of 100% on OU-ISIR D, 99.93% on CASIA-B, and 99.94% on OU-LP, showcasing the superior performance of the Ensemble CNN-ViT model using feature-level fusion compared to state-of-the-art methods.

The few-shot learning paradigm aims to generalize to unseen tasks with limited samples. However, a focus solely on class-level discrimination may fall short of achieving robust generalization, especially when neglecting instance diversity and discriminability. This study introduces a metric-based few-shot approach, named Self-supervised Feature Fusion with Transformer (SFT), which integrates self-supervised learning with a transformer. SFT addresses the limitations of previous approaches by employing two distinct self-supervised tasks in separate models during pre-training, thus enhancing both instance diversity and discriminability in the feature space. The training process unfolds in two stages: pre-training and transfer learning. In pre-training, each model undergoes training with specific self-supervised tasks to harness the benefits of enhanced feature space. In the subsequent transfer learning stage, model weights are frozen, acting as feature extractors. The features from both models are amalgamated using a feature fusion technique and are transformed into task-specific features by a transformer, boosting discrimination on unseen tasks. The combined features enable the model to learn a well-generalized representation, effectively tackling the challenges posed by few-shot tasks. The proposed SFT method achieves state-of-the-art results on three benchmark datasets in few-shot image classification.

Facial Expression Recognition (FER) is a machine learning problem that deals with recognizing human facial expressions. While existing work has achieved performance improvements in recent years, FER in the wild and under challenging conditions remains a challenge. In this paper, a lightweight patch and attention network based on MobileNetV1, referred to as PAtt-Lite, is proposed to improve FER performance under challenging conditions. A truncated ImageNet-pre-trained MobileNetV1 is utilized as the backbone feature extractor of the proposed method. In place of the truncated layers is a patch extraction block that is proposed for extracting significant local facial features to enhance the representation from MobileNetV1, especially under challenging conditions. An attention classifier is also proposed to improve the learning of these patched feature maps from the extremely lightweight feature extractor. The experimental results on public benchmark databases proved the effectiveness of the proposed method. PAtt-Lite achieved state-of-the-art results on CK+, RAF-DB, FER2013, FERPlus, and the challenging conditions subsets for RAF-DB and FERPlus.

Sentiment analysis, a pivotal task in natural language processing, aims to discern opinions and emotions expressed in text. However, existing methods for sentiment analysis face various challenges such as data scarcity, complex language patterns, and long-range dependencies. In this paper, we propose MPNet-GRUs, a hybrid deep learning model that integrates three key components: MPNet, BiGRU, and GRU. MPNet, a transformer-based pre-trained language model, enhances language understanding through masked and permuted pre-training. BiGRU and GRU, recurrent neural networks, capture long-term dependencies bidirectionally and unidirectionally. By combining the strengths of these models, MPNet-GRUs aims to provide a more effective and efficient solution for sentiment analysis. Evaluation on three benchmark datasets reveals the superior performance of MPNet-GRUs: 94.71% for IMDb, 86.27% for Twitter US Airline Sentiment, and 88.17% for Sentiment140, demonstrating its potential to advance sentiment analysis.

Knowledge Graph Question Answering (KGQA) plays a crucial role in extracting valuable insights from interconnected information. Existing methods, while commendable, face challenges such as contextual ambiguity and limited adaptability to diverse knowledge domains. This paper introduces TransKGQA, a novel approach addressing these challenges. Leveraging Sentence Transformers, TransKGQA enhances contextual understanding, making it adaptable to various knowledge domains. The model employs question-answer pair augmentation for robustness and introduces a threshold mechanism for reliable answer retrieval. TransKGQA overcomes limitations in existing works by offering a versatile solution for diverse question types. Experimental results, notably with the sentence-transformers/all-MiniLM-L12-v2 model, showcase remarkable performance with an F1 score of 78%. This work advances KGQA systems, contributing to knowledge graph construction, enhanced question answering, and automated Cypher query execution.

Multimodal Question Answering (MMQA) has emerged as a challenging frontier at the intersection of natural language processing (NLP) and computer vision, demanding the integration of diverse modalities for effective comprehension and response. While pre-trained language models (PLMs) exhibit impressive performance across a range of NLP tasks, the investigation of text-based approaches to address MMQA represents a compelling and promising avenue for further research and advancement in the field. Although recent research has delved into text-based approaches for MMQA, the attained results have been unsatisfactory, which could be attributed to potential information loss during the knowledge transformation processes. In response, a novel three-stage framework named UniRaG is proposed for tackling MMQA, which encompasses unified knowledge representation, context retrieval, and answer generation. At the initial stage, advanced techniques are employed for unified knowledge representation, including LLaVA for image captioning and table linearization for tabular data, facilitating seamless integration of visual and tabular information into textual representation. For context retrieval, a cross-encoder trained on sequence classification is utilized to predict relevance scores for question-document pairs, and a top-k retrieval strategy is employed to retrieve the documents with the highest relevance scores as the contexts for answer generation. Finally, the answer generation stage is facilitated by a text-to-text PLM, Flan-T5-Base, which follows the encoder-decoder architecture with attention mechanisms. During this stage, uniform prefix conditioning is applied to the input text for enhanced adaptability and generalizability. Moreover, contextual diversity training is introduced to improve model robustness by including distractor documents as negative contexts during training. Experimental results on the MultimodalQA dataset demonstrate the superior performance of UniRaG, surpassing the existing state-of-the-art methods across all scenarios with 67.4% EM and 71.3% F1. Overall, UniRaG showcases robustness and reliability in MMQA, heralding significant advancements in multimodal comprehension and question answering research.

Few-shot image classification remains a persistent challenge due to the intrinsic difficulty faced by visual recognition models in achieving generalization with limited training data. Existing methods primarily focus on exploiting marginal distributions and overlook the disparity between the product of marginals and the joint characteristic functions. This can lead to less robust feature representations. In this paper, we introduce DBDC-SSL, a method that aims to improve few-shot visual recognition models by learning a feature extractor that produces image representations that are more robust. To improve the robustness of the model, we integrate DeepBDC (DBDC) during the training process to learn better feature embeddings by effectively computing the disparity between product of the marginals and joint characteristic functions of the features. To reduce overfitting and improve the generalization of the model, we utilize an auxiliary rotation loss for self-supervised learning (SSL) in the training of the feature extractor. The auxiliary rotation loss is derived from a pretext task, where input images undergo rotation by predefined angles, and the model classifies the rotation angle based on the features it generates. Experimental results demonstrate that DBDC-SSL is able to outperform current state-of-the-art methods on 4 common few-shot image classification benchmark, which are miniImageNet, tieredImageNet, CUB and CIFAR-FS. For 5-way 1-shot and 5-way 5-shot tasks respectively, the proposed DBDC-SSL achieved the accuracy of 68.64±0.43 and 86.02±0.28 on miniImageNet, 73.88±0.48 and 89.03±0.29 on tieredImageNet, 84.67±0.39 and 94.76±0.16 on CUB, and 75.60±0.44 and 88.49±0.31 on CIFAR-FS.

Few-shot learning continues to pose a challenge as it is inherently difficult for visual recognition models to generalize with limited labeled examples. When the training data is limited, the process of training and fine-tuning the model will be unstable and inefficient due to overfitting. In this paper, we introduce NegCosIC: Negative Cosine Similarity-Invariance-Covariance Regularization, a method that aims to improve the mean accuracy from the perspective of stabilizing the fine-tuning process and regularizing variance. NegCosIC incorporates a negative simple cosine similarity loss to stabilize the parameters of the feature extractor during fine-tuning. In addition, NegCosIC integrates invariance loss and covariance loss to regularize the embeddings in order to reduce overfitting. Experimental results demonstrate that NegCosIC is able to bring substantial improvements over the current state-of-the-art methods. An in-depth worse case analysis is also conducted and shows that NegCosIC is able to outperform state-of-the-art methods on worst case accuracy. The proposed NegCosIC achieved 2.15% and 2.13% higher accuracy on miniImageNet 1-shot and 5-shot tasks, 3.22% and 2.67% higher accuracy on CUB 1-shot and 5-shot tasks, and 2.13% and 7.74% higher accuracy on CIFAR-FS 1-shot and 5-shot tasks in terms of worst-case accuracies.

Vision Transformers, known for their innovative architectural design and modeling capabilities, have gained significant attention in computer vision. This paper presents a dual-path approach that leverages the strengths of the Multi-Axis Vision Transformer (MaxViT) and the Improved Multiscale Vision Transformer (MViTv2). It starts by encoding speech signals into Constant-Q Transform (CQT) spectrograms and Mel Spectrograms with Short-Time Fourier Transform (Mel-STFT). The CQT spectrogram is then fed into the MaxViT model, while the Mel-STFT is input to the MViTv2 model to extract informative features from the spectrograms. These features are integrated and passed into a Multilayer Perceptron (MLP) model for final classification. This hybrid model is named the “MaxViT and MViTv2 Fusion Network with Multilayer Perceptron (MaxMViT-MLP).” The MaxMViT-MLP model achieves remarkable results with an accuracy of 95.28% on the Emo-DB, 89.12% on the RAVDESS dataset, and 68.39% on the IEMOCAP dataset, substantiating the advantages of integrating multiple audio feature representations and Vision Transformers in speech emotion recognition.

Sign language recognition presents significant challenges due to the intricate nature of hand gestures and the necessity to capture fine-grained details. In response to these challenges, a novel approach is proposed-Lightweight Attentive VGG16 with Random Forest (LAVRF) model. LAVRF introduces a refined adaptation of the VGG16 model integrated with attention modules, complemented by a Random Forest classifier. By streamlining the VGG16 architecture, the Lightweight Attentive VGG16 effectively manages complexity while incorporating attention mechanisms that dynamically concentrate on pertinent regions within input images, resulting in enhanced representation learning. Leveraging the Random Forest classifier provides notable benefits, including proficient handling of high-dimensional feature representations, reduction of variance and overfitting concerns, and resilience against noisy and incomplete data. Additionally, the model performance is further optimized through hyperparameter optimization, utilizing the Optuna in conjunction with hill climbing, which efficiently explores the hyperparameter space to discover optimal configurations. The proposed LAVRF model demonstrates outstanding accuracy on three datasets, achieving remarkable results of 99.98%, 99.90%, and 100% on the American Sign Language, American Sign Language with Digits, and NUS Hand Posture datasets, respectively.

Speech emotion recognition aims to automatically identify and classify emotions from speech signals. It plays a crucial role in various applications such as human-computer interaction, affective computing, and social robotics. Over the years, researchers have proposed different approaches for speech emotion recognition, leveraging various classifiers and features. However, despite the advancements, existing methods in speech emotion recognition still have certain limitations. Some approaches rely on handcrafted features that may not capture the full complexity of emotional information present in speech signals, while others may suffer from a lack of robustness and generalization when applied to different datasets. To address these challenges, this paper proposes a speech emotion recognition method that combines Mel spectrogram with Short-Term Fourier Transform (Mel-STFT) and the Improved Multiscale Vision Transformers (MViTv2). The Mel-STFT spectrograms capture both the frequency and temporal information of speech signals, providing a more comprehensive representation of the emotional content. The MViTv2 classifier introduces multi-scale visual modeling with different stages and pooling attention mechanisms. MViTv2 incorporates relative positional embeddings and a residual pooling connection to effectively model the interactions between tokens in the space-time structure, preserve essential information, and improve the efficiency of the model. Experimental results demonstrate that the proposed method generalizes well on different datasets, achieving an accuracy of 91.51% on the Emo-DB dataset, 81.75% on the RAVDESS dataset, and 64.03% on the IEMOCAP dataset.

Text-to-image synthesis is a fascinating area of research that aims to generate images based on textual descriptions. The main goal of this field is to generate images that match the given textual description in terms of both semantic consistency and image realism. While text-to-image synthesis has shown remarkable progress in recent years, it still faces several challenges, mainly related to the level of image realism and semantic consistency. To address these challenges, various approaches have been proposed, which mainly rely on Generative Adversarial Networks (GANs) for optimal performance. This paper provides a review of the existing text-to-image synthesis approaches, which are categorized into four groups: image realism, multiple scene, semantic enhancement, and style transfer. In addition to discussing the existing approaches, this paper also reviews the widely used datasets for text-to-image synthesis, including Oxford-102, CUB-200-2011, and COCO. The evaluation metrics used in this field are also discussed, including Inception Score, Fréchet Inception Distance, Structural Similarity Index, R-precision, Visual-Semantic Similarity, and Semantic Object Accuracy. The paper also offers a compilation of the performance of existing works in the field.

Speech emotion recognition presents a significant challenge within the field of affective computing, requiring the analysis and detection of emotions conveyed through speech signals. However, existing approaches often rely on traditional signal processing techniques and handcrafted features, which may not effectively capture the nuanced aspects of emotional expression. In this paper, an approach named “SCQT-MaxViT” is proposed for speech emotion recognition, combining signal processing, computer vision, and deep learning techniques. The method utilizes the Constant-Q Transform (CQT) to convert speech waveforms into spectrograms, providing high-frequency resolution and enabling the model to capture intricate emotional details. Additionally, the Multi-axis Vision Transformer (MaxViT) is employed for further representation learning and classification of the CQT spectrograms. MaxViT incorporates a multi-axis self-attention mechanism, facilitating both local and global interactions within the network and enhancing the ability of the model to learn meaningful features. Furthermore, the dataset is augmented using random time masking techniques to enhance the generalization capabilities. Achieving accuracies of 88.68% on the Emo-DB dataset, 77.54% on the RAVDESS dataset, and 62.49% on the IEMOCAP dataset, the proposed SCQT-MaxViT method exhibits promising performance in capturing and recognizing emotions in speech signals.

The task of Text-to-Image synthesis is a difficult challenge, especially when dealing with low-data regimes, where the number of training samples is limited. In order to address this challenge, the Self-Supervision Text-to-Image Generative Adversarial Networks (SS-TiGAN) has been proposed. The method employs a bi-level architecture, which allows for the use of self-supervision to increase the number of training samples by generating rotation variants. This, in turn, maximizes the diversity of the model representation and enables the exploration of high-level object information for more detailed image construction. In addition to the use of self-supervision, SS-TiGAN also investigates various techniques to address the stability issues that arise in Generative Adversarial Networks. By implementing these techniques, the proposed SS-TiGAN has achieved a new state-of-the-art performance on two benchmark datasets, Oxford-102 and CUB. These results demonstrate the effectiveness of the SS-TiGAN method in synthesizing high-quality, realistic images from text descriptions under low-data regimes.

Hand gesture recognition (HGR) is a rapidly evolving field with the potential to revolutionize human–computer interactions by enabling machines to interpret and understand human gestures for intuitive communication and control. However, HGR faces challenges such as the high similarity of hand gestures, real-time performance, and model generalization. To address these challenges, this paper proposes the stacking of distilled vision transformers, referred to as SDViT, for hand gesture recognition. An initially pretrained vision transformer (ViT) featuring a self-attention mechanism is introduced to effectively capture intricate connections among image patches, thereby enhancing its capability to handle the challenge of high similarity between hand gestures. Subsequently, knowledge distillation is proposed to compress the ViT model and improve model generalization. Multiple distilled ViTs are then stacked to achieve higher predictive performance and reduce overfitting. The proposed SDViT model achieves a promising performance on three benchmark datasets for hand gesture recognition: the American Sign Language (ASL) dataset, the ASL with digits dataset, and the National University of Singapore (NUS) hand gesture dataset. The accuracies achieved on these datasets are 100.00%, 99.60%, and 100.00%, respectively.

Recent successes in deep learning have inspired researchers to apply deep neural networks to Acoustic Event Classification (AEC). While deep learning methods can train effective AEC models, they are susceptible to overfitting due to the models’ high complexity. In this paper, we introduce EnViTSA, an innovative approach that tackles key challenges in AEC. EnViTSA combines an ensemble of Vision Transformers with SpecAugment, a novel data augmentation technique, to significantly enhance AEC performance. Raw acoustic signals are transformed into Log Mel-spectrograms using Short-Time Fourier Transform, resulting in a fixed-size spectrogram representation. To address data scarcity and overfitting issues, we employ SpecAugment to generate additional training samples through time masking and frequency masking. The core of EnViTSA resides in its ensemble of pre-trained Vision Transformers, harnessing the unique strengths of the Vision Transformer architecture. This ensemble approach not only reduces inductive biases but also effectively mitigates overfitting. In this study, we evaluate the EnViTSA method on three benchmark datasets: ESC-10, ESC-50, and UrbanSound8K. The experimental results underscore the efficacy of our approach, achieving impressive accuracy scores of 93.50%, 85.85%, and 83.20% on ESC-10, ESC-50, and UrbanSound8K, respectively. EnViTSA represents a substantial advancement in AEC, demonstrating the potential of Vision Transformers and SpecAugment in the acoustic domain.

Due to the rapid development of technology, social media has become more and more common in human daily life. Social media is a platform for people to express their feelings, feedback, and opinions. To understand the sentiment context of the text, sentiment analysis plays the role to determine whether the sentiment of the text is positive, negative, neutral or any other personal feeling. Sentiment analysis is prominent from the perspective of business or politics where it highly impacts the strategic decision making. The challenges of sentiment analysis are attributable to the lexical diversity, imbalanced dataset and long-distance dependencies of the texts. In view of this, a data augmentation technique with GloVe word embedding is leveraged to synthesize more lexically diverse samples by similar word vector replacements. The data augmentation also focuses on the oversampling of the minority classes to mitigate the imbalanced dataset problems. Apart from that, the existing sentiment analysis mostly leverages sequence models to encode the long-distance dependencies. Nevertheless, the sequence models require a longer execution time as the processing is done sequentially. On the other hand, the Transformer models require less computation time with parallelized processing. To that end, this paper proposes a hybrid deep learning method that combines the strengths of sequence model and Transformer model while suppressing the limitations of sequence model. Specifically, the proposed model integrates Robustly optimized BERT approach and Long Short-Term Memory for sentiment analysis. The Robustly optimized BERT approach maps the words into a compact meaningful word embedding space while the Long Short-Term Memory model captures the long-distance contextual semantics effectively. The experimental results demonstrate that the proposed hybrid model outshines the state-of-the-art methods by achieving F1-scores of 93%, 91%, and 90% on IMDb dataset, Twitter US Airline Sentiment dataset, and Sentiment140 dataset, respectively.

The rapid development of mobile technologies has made social media a vital platform for people to express their feelings and opinions. Understanding the public opinions can be beneficial for business and political entities in making strategic decisions. In light of this, sentiment analysis plays an important role to understand the polarity of the public opinions. This paper presents an ensemble hybrid deep learning model for sentiment analysis. The proposed ensemble model comprises three hybrid deep learning models which are the combination of Robustly optimized Bidirectional Encoder Representations from Transformers approach (RoBERTa), Long Short-Term Memory (LSTM), Bidirectional Long Short-Term Memory (BiLSTM) and Gated Recurrent Unit (GRU). In the hybrid deep learning model, RoBERTa is responsible for projecting the textual input sequence into a representative embedding space. Thereafter, the LSTM, BiLSTM and GRU capture the long-range dependencies in the embedding given the class. The predictions by the hybrid deep learning model are then amalgamated by averaging ensemble and majority voting, further improving the overall performance in sentiment analysis. In addition to that, the data augmentation with GloVe pre-trained word embedding has also been applied to alleviate the imbalanced dataset problems. The experimental results show that the proposed ensemble hybrid deep learning model outshines the state-of-the-art methods with the accuracy of 94.9%, 91.77%, and 89.81% on IMDb, Twitter US Airline Sentiment dataset and Sentiment140 dataset, respectively.