Search

Integrating Large Language Models (LLMs) with Knowledge Graphs (KGs) enhances the interpretability and performance of AI systems. This research comprehensively analyzes this integration, classifying approaches into three fundamental paradigms: KG-augmented LLMs, LLM-augmented KGs, and synergized frameworks. The evaluation examines each paradigm's methodology, strengths, drawbacks, and practical applications in real-life scenarios. The findings highlight the substantial impact of these integrations in fundamentally improving real-time data analysis, efficient decision-making, and promoting innovation across various domains. In this paper, we also describe essential evaluation metrics and benchmarks for assessing the performance of these integrations, addressing challenges like scalability and computational overhead, and providing potential solutions. This comprehensive analysis underscores the profound impact of these integrations on improving real-time data analysis, enhancing decision-making efficiency, and fostering innovation across various domains. [ABSTRACT FROM AUTHOR]

Word embeddings are a fundamental tool in natural language processing. Currently, word embedding methods are evaluated on the basis of empirical performance on benchmark data sets, and there is a lack of rigorous understanding of their theoretical properties. This paper studies word embeddings from a statistical theoretical perspective, which is essential for formal inference and uncertainty quantification. We propose a copula-based statistical model for text data and show that under this model, the now-classical Word2Vec method can be interpreted as a statistical estimation method for estimating the theoretical pointwise mutual information (PMI). We further illustrate the utility of this statistical model by using it to develop a missing value-based estimator as a statistically tractable and interpretable alternative to the Word2Vec approach. The estimation error of this estimator is comparable to Word2Vec and improves upon the truncation-based method proposed by Levy and Goldberg (Adv. Neural Inf. Process. Syst., 27, 2177–2185 2014). The resulting estimator also is comparable to Word2Vec in a benchmark sentiment analysis task on the IMDb Movie Reviews data set and a part-of-speech tagging task on the OntoNotes data set. [ABSTRACT FROM AUTHOR]

As the core of ontology integration, the task of ontology mapping is to find the semantic relationship between ontologies. Nevertheless, most existing ontology mapping methods only rely ontext information to calculate entity similarity, thereby disregarding semantic nuances and necessitating substantial manual intervention. Therefore, this paper introduces an ontology mapping method. Based on the traditional ontology mapping method, the process employs a deep learning model to mine the semantic information of entity concepts, entity properties and ontology structure to obtain the embedding vector. We use the similarity mechanism to calculate the similarity between different embedding vectors, and combine the similarity values obtained from multiple strategy entities into a similarity matrix. The similarity matrix serves as input to the support vector machine (SVM), and the ontology mapping problem is finally transformed into a binary classification problem. However, since the number of mapped pairs is much larger than the number of non-mapped pairs, the number of positive samples in the data set is much smaller than the number of negative samples. Therefore, based on the traditional SVM, the paper adopts cost-sensitive strategy to deal with the class imbalance problem. In comparative evaluations against contemporary ontology mapping techniques, our method demonstrates a noteworthy 5.0% enhancement in recall and a 3.0% improvement in F1 score when tested on both public benchmark datasets and domain-specific datasets. [ABSTRACT FROM AUTHOR]

Self-supervised learning enables the training of large neural models without the need for large, labeled datasets. It has been generating breakthroughs in several fields, including computer vision, natural language processing, biology, and speech. In particular, the state-of-the-art in several speech processing applications, such as automatic speech recognition or speaker identification, are models where the latent representation is learned using self-supervised approaches. Several configurations exist in self-supervised learning for speech, including contrastive, predictive, and multilingual approaches. There is, however, a crucial limitation in the majority of existing approaches: their high computational costs. These costs limit the deployment of models, the size of the training dataset, and the number of research groups that can afford research with large self-supervised models. Likewise, we should consider the environmental costs that high energy consumption implies. Efforts in this direction comprise optimization of existing models, neural architecture efficiency, improvements in finetuning for speech processing tasks, and data efficiency. But despite current efforts, more work could be done to address high computational costs in self-supervised representation learning. [ABSTRACT FROM AUTHOR]

Questions have a critical role in learning and teaching. People ask questions to obtain information and express interest in ideas. The Bristol scientific centre "We The Curious" launched "Project What If" in 2017 to inspire residents of Bristol to record their questions and pursue their curiosities. Researching these questions may help the museum better understand the curiosity of its audiences and create exhibitions or educational content that are more relevant to their interests and lives. The project managed to collect more than 10,000 questions on various topics, and more questions are being collected on a daily basis. With this large amount of data collected, it is time-consuming to process and analyse all the questions by humans. This research aims to apply artificial intelligence (AI) techniques and models in analysing these questions gathered by We The Curious. Meanwhile, in AI, there is a lack of tools that focus on processing and analysing the questions. Thus, we introduce a deep neural network called QBERT to process the questions for three tasks: question taxonomy, equivalent question detection, and question answering. Then we apply QBERT to provide an analysis of the questions collected by We The Curious, as well as comprehend Bristolians' curiosity. Then using QBERT, we categorise the We The Curious questions into 90 themes and 5,930 communities. Moreover, 436 questions are answered by one-sentence answers extracted from Wikipedia. [ABSTRACT FROM AUTHOR]

With increasing use of artificial intelligence (AI) in high-stakes contexts, a race for “trustworthy AI” is under way. However, Dorsch and Deroy (Philosophy & Technology 37, 62, 2024) recently argued that regardless of its feasibility, morally trustworthy AI is unnecessary: We should merely rely on rather than trust AI, and carefully calibrate our reliance using the reliability scores which are often available. This short commentary on Dorsch and Deroy engages with the claim that morally trustworthy AI is unnecessary and argues that since there are important limits to how good calibration based on reliability scores can be, some residual roles for trustworthy AI (if feasible) are still possible. [ABSTRACT FROM AUTHOR]

Topic classification is a challenging task in order to comprehend the subject matter or theme of the Indian mythology. It will enhance the performance of NLP-based systems, such as recommendation and semantic search engines, when dealing with texts containing mythology. This research focuses on developing transformer based models for automated topic classification of Indian mythological documents, which addresses the challenges of organizing and analyzing this rich and diverse corpus. We introduce PouranicTopic, a new annotated dataset containing over 200k verses from 7 major Hindu texts with canto, topic, and sentence labels. Additional datasets Similarity-based and Log-likelihood-based are created using sentence clustering techniques. The BERT, RoBERTa, and DistilBERT models are evaluated for canto and topic classification on these datasets. Clustering greatly improves the results on the Similarity-based dataset, but Log-likelihood-based dataset remains challenging. [ABSTRACT FROM AUTHOR]

With current success of large-scale pre-trained models (PTMs), how efficiently adapting PTMs to downstream tasks has attracted tremendous attention, especially for PTMs with billions of parameters. Previous work focuses on designing parameter-efficient tuning paradigms but needs to save and compute the gradient of the whole computational graph. In this paper, we propose Y -Tuning, an efficient yet effective paradigm to adapt frozen large-scale PTMs to specific downstream tasks. Y -Tuning learns dense representations for labels Y defined in a given task and aligns them to fixed feature representation. Without computing the gradients of text encoder at training phrase, Y -Tuning is not only parameter-efficient but also training-efficient. Experimental results show that for DeBERTaXXL with 1.6 billion parameters, Y -Tuning achieves performance more than 96% of full fine-tuning on GLUE Benchmark with only 2% tunable parameters and much fewer training costs. [ABSTRACT FROM AUTHOR]

Protein language models (PLMs) play a dominant role in protein representation learning. Most existing PLMs regard proteins as sequences of 20 natural amino acids. The problem with this representation method is that it simply divides the protein sequence into sequences of individual amino acids, ignoring the fact that certain residues often occur together. Therefore, it is inappropriate to view amino acids as isolated tokens. Instead, the PLMs should recognize the frequently occurring combinations of amino acids as a single token. In this study, we use the byte-pair-encoding algorithm and unigram to construct advanced residue vocabularies for protein sequence tokenization, and we have shown that PLMs pre-trained using these advanced vocabularies exhibit superior performance on downstream tasks when compared to those trained with simple vocabularies. Furthermore, we introduce PETA, a comprehensive benchmark for systematically evaluating PLMs. We find that vocabularies comprising 50 and 200 elements achieve optimal performance. Our code, model weights, and datasets are available at https://github.com/ginnm/ProteinPretraining. Scientific contribution: This study introduces advanced protein sequence tokenization analysis, leveraging the byte-pair-encoding algorithm and unigram. By recognizing frequently occurring combinations of amino acids as single tokens, our proposed method enhances the performance of PLMs on downstream tasks. Additionally, we present PETA, a new comprehensive benchmark for the systematic evaluation of PLMs, demonstrating that vocabularies of 50 and 200 elements offer optimal performance. [ABSTRACT FROM AUTHOR]

Cross-modal retrieval, the process of retrieving relevant data from one modality in response to a query in another, has become increasingly important with the growing amount of multimodal data. This paper proposes using a pretrained model CLIP as the backbone of a cross-modal retrieval system and explores various methods to enhance its performance. The proposed approach reduces the output feature dimension to 384, reducing model parameters, storage capacity, and retrieval time by 62.5%. By conducting cross-training on the training dataset, the model not only enhances its intermodal invariance but also achieves multimodal retrieval. The residual connections and an increased dropout ratio of 30% increase average retrieval performance. Additionally, we propose the utilization of class proxies as missing data to accomplish training in an incomplete (imbalanced) dataset. The proposed approach is evaluated on four benchmark datasets: Wikipedia, NUS-WIDE, Pascal-Sentence, and XmediaNet, achieving 3.4%, 1.9%, 2.3%, and 5.8% retrieval performance improvement, respectively. The results demonstrate the effectiveness of the proposed approach in significantly improving the performance of cross-modal retrieval systems, outperforming state-of-the-art methods on benchmark datasets while reducing the number of model parameters, retrieval time, and database storage space. [ABSTRACT FROM AUTHOR]

BERT is a representative pre-trained language model that has drawn extensive attention for significant improvements in downstream Natural Language Processing (NLP) tasks. The complex architecture and massive parameters bring BERT competitive performance but also result in slow speed at model inference time. To speed up BERT inference, FastBERT realizes adaptive inference with an acceptable drop in accuracy based on knowledge distillation and the early-exit technique. However, many factors may limit the performance of FastBERT, such as the teacher classifier that is not knowledgeable enough, the batch size shrinkage and the redundant computation of student classifiers. To overcome these limitations, we propose a new BERT inference method with GPU-Efficient Exit Prediction (GEEP). GEEP leverages the shared exit loss to simplify the training process of FastBERT from two steps into only one step and makes the teacher classifier more knowledgeable by feeding diverse Transformer outputs to the teacher classifier. In addition, the exit layer prediction technique is proposed to utilize a GPU hash table to handle the token-level exit layer distribution and to sort test samples by predicted exit layers. In this way, GEEP can avoid batch size shrinkage and redundant computation of student classifiers. Experimental results on twelve public English and Chinese NLP datasets prove the effectiveness of the proposed approach. The source codes of GEEP will be released to the public upon paper acceptance. [ABSTRACT FROM AUTHOR]

Natural Language Inference (NLI) is considered a representative task to test natural language understanding (NLU). In this work, we propose an extensible framework to collectively yet categorically test diverse Logical reasoning capabilities required for NLI (and, by extension, NLU). Motivated by behavioral testing, we create a semi-synthetic large test bench (363 templates, 363k examples) and an associated framework that offers the following utilities: (1) individually test and analyze reasoning capabilities along 17 reasoning dimensions (including pragmatic reasoning); (2) design experiments to study cross-capability information content (leave one out or bring one in); and (3) the synthetic nature enables us to control for artifacts and biases. We extend a publicly available framework of automated test case instantiation from free-form natural language templates (CheckList) and a well-defined taxonomy of capabilities to cover a wide range of increasingly harder test cases while varying the complexity of natural language. Through our analysis of state-of-the-art NLI systems, we observe that our benchmark is indeed hard (and non-trivial even with training on additional resources). Some capabilities stand out as harder. Further, fine-grained analysis and fine-tuning experiments reveal more insights about these capabilities and the models – supporting and extending previous observations; thus showing the utility of the proposed testbench. [ABSTRACT FROM AUTHOR]

Since OpenAI opened access to ChatGPT, large language models (LLMs) become an increasingly popular topic attracting researchers' attention from abundant domains. However, public researchers meet some problems when developing LLMs given that most of the LLMs are produced by industries and the training details are typically unrevealed. Since datasets are an important setup of LLMs, this paper does a holistic survey on the training datasets used in both the pre-train and fine-tune processes. The paper first summarizes 16 pre-train datasets and 16 fine-tune datasets used in the state-of-the-art LLMs. Secondly, based on the properties of the pre-train and fine-tune processes, it comments on pre-train datasets from quality, quantity, and relation with models, and comments on fine-tune datasets from quality, quantity, and concerns. This study then critically figures out the problems and research trends that exist in current LLM datasets. The study helps public researchers train and investigate LLMs by visual cases and provides useful comments to the research community regarding data development. To the best of our knowledge, this paper is the first to summarize and discuss datasets used in both autoregressive and chat LLMs. The survey offers insights and suggestions to researchers and LLM developers as they build their models, and contributes to the LLM study by pointing out the existing problems of LLM studies from the perspective of data. [ABSTRACT FROM AUTHOR]

In this paper, we introduce s2s-ft, a method for adapting pretrained bidirectional Transformer encoders, such as BERT and RoBERTa, to sequence-to-sequence tasks like abstractive summarization and question generation. By employing a unified modeling approach and well-designed self-attention masks, s2s-ft leverages the generative capabilities of pretrained Transformer encoders without the need for an additional decoder. We conduct extensive experiments comparing three fine-tuning algorithms (causal fine-tuning, masked fine-tuning, and pseudo-masked fine-tuning) and various pretrained models for initialization. Results demonstrate that s2s-ft achieves strong performance across different tasks and languages. Additionally, the method is successfully extended to multilingual pretrained models, such as XLM-RoBERTa, and evaluated on multilingual generation tasks. Our work highlights the importance of reducing the discrepancy between masked language model pretraining and sequence-to-sequence fine-tuning and showcases the effectiveness and expansibility of the s2s-ft method. [ABSTRACT FROM AUTHOR]

Efficiently treating cardiac patients before the onset of a heart attack relies on the precise prediction of heart disease. Identifying and detecting the risk factors for heart disease such as diabetes mellitus, Coronary Artery Disease (CAD), hyperlipidemia, hypertension, smoking, familial CAD history, obesity, and medications is critical for developing effective preventative and management measures. Although Electronic Health Records (EHRs) have emerged as valuable resources for identifying these risk factors, their unstructured format poses challenges for cardiologists in retrieving relevant information. This research proposed employing transfer learning techniques to automatically extract heart disease risk factors from EHRs. Leveraging transfer learning, a deep learning technique has demonstrated a significant performance in various clinical natural language processing (NLP) applications, particularly in heart disease risk prediction. This study explored the application of transformer-based language models, specifically utilizing pre-trained architectures like BERT (Bidirectional Encoder Representations from Transformers), RoBERTa, BioClinicalBERT, XLNet, and BioBERT for heart disease detection and extraction of related risk factors from clinical notes, using the i2b2 dataset. These transformer models are pre-trained on an extensive corpus of medical literature and clinical records to gain a deep understanding of contextualized language representations. Adapted models are then fine-tuned using annotated datasets specific to heart disease, such as the i2b2 dataset, enabling them to learn patterns and relationships within the domain. These models have demonstrated superior performance in extracting semantic information from EHRs, automating high-performance heart disease risk factor identification, and performing downstream NLP tasks within the clinical domain. This study proposed fine-tuned five widely used transformer-based models, namely BERT, RoBERTa, BioClinicalBERT, XLNet, and BioBERT, using the 2014 i2b2 clinical NLP challenge dataset. The fine-tuned models surpass conventional approaches in predicting the presence of heart disease risk factors with impressive accuracy. The RoBERTa model has achieved the highest performance, with micro F1-scores of 94.27%, while the BERT, BioClinicalBERT, XLNet, and BioBERT models have provided competitive performances with micro F1-scores of 93.73%, 94.03%, 93.97%, and 93.99%, respectively. Finally, a simple ensemble of the five transformer-based models has been proposed, which outperformed the most existing methods in heart disease risk fan, achieving a micro F1-Score of 94.26%. This study demonstrated the efficacy of transfer learning using transformer-based models in enhancing risk prediction and facilitating early intervention for heart disease prevention. [ABSTRACT FROM AUTHOR]

Transformer-based Pre-Trained Language Models currently dominate the field of Natural Language Inference (NLI). We first survey existing NLI datasets, and we systematize them according to the different kinds of logical inferences that are being distinguished. This shows two gaps in the current dataset landscape, which we propose to address with one dataset that has been developed in argumentative writing research as well as a new one building on syllogistic logic. Throughout, we also explore the promises of ChatGPT. Our results show that our new datasets do pose a challenge to existing methods and models, including ChatGPT, and that tackling this challenge via fine-tuning yields only partly satisfactory results. [ABSTRACT FROM AUTHOR]

This paper provides a comprehensive review of recent challenges and results in the field of generative AI with application to mobile telecommunications networks. The objective is to classify the literature using an approach that encompasses the type of generative AI technology employed, the functional purpose, and the specific component of the mobile network that each solution targets. Moreover, performance requirements for generative AI applications are considered. Thereafter, state-of-the-art generative AI algorithms and an examination of their use cases across various industry verticals are presented. The discussion extends to the current level of AI integration in telecom standardization bodies, such as the 3rd Generation Partnership Project (3GPP). Finally, the open research challenges that the generative AI technology aims to address are thoroughly investigated. [ABSTRACT FROM AUTHOR]

The prediction of molecular properties is a crucial aspect in drug discovery that can save a lot of money and time during the drug design process. The use of machine learning methods to predict molecular properties has become increasingly popular in recent years. Despite advancements in the field, several challenges remain that need to be addressed, like finding an optimal pre-training procedure to improve performance on small datasets, which are common in drug discovery. In our paper, we tackle these problems by introducing Relative Molecule Self-Attention Transformer for molecular representation learning. It is a novel architecture that uses relative self-attention and 3D molecular representation to capture the interactions between atoms and bonds that enrich the backbone model with domain-specific inductive biases. Furthermore, our two-step pretraining procedure allows us to tune only a few hyperparameter values to achieve good performance comparable with state-of-the-art models on a wide selection of downstream tasks. Scientific contribution: A novel graph transformer architecture for molecular property prediction is introduced. The task-agnostic methodology for pre-training this model is presented, improving target task performance with minimal hyperparameter tuning. A rigorous exploration of the design space for the self-attention layer is conducted to identify the optimal architecture. [ABSTRACT FROM AUTHOR]

Big data analytics has become a significant trend for many businesses as a result of the daily acquisition of enormous volumes of data. This information has been gathered because it may be important for dealing with complicated problems including fraud, marketing, healthcare, and cyber security. Analytics are used by big businesses like Facebook and Google to analyse and make decisions about their massive volumes of collected data. Such analyses and decisions impact both the present and the future of technology. The inherent non-linear properties in huge data may be captured by deep learning (DL) algorithms using automated feature extraction techniques. In order to estimate renewable energy, energy consumption, demand, and supply, among other things, over the short, medium, and long term, a complete and in-depth investigation of generative, hybrid, and discriminative DL models is being conducted. This article examines the benefits and drawbacks of DL that depends on a variety of deep neural networks, including recurrent neural networks, multilayer neural networks, auto encoders and long short-term memory. [ABSTRACT FROM AUTHOR]

The task of resolving repeated objects in natural languages is known as coreference resolution, and it is an important part of modern natural language processing. It is classified into two categories depending on the resolved objects, namely entity coreference resolution and event coreference resolution. Predicting coreference connections and identifying mentions/triggers are the major challenges in coreference resolution, because these implicit relationships are particularly difficult in natural language understanding in downstream tasks. Coreference resolution techniques have experienced considerable advances in recent years, encouraging us to review this task in the following aspects: current employed evaluation metrics, datasets, and methods. We investigate 10 widely used metrics, 18 datasets and 4 main technical trends in this survey. We believe that this work is a comprehensive roadmap for understanding the past and the future of coreference resolution. [ABSTRACT FROM AUTHOR]

Estimating the difficulty of a question in video QAs is one of the important reasoning steps to answer the question. However, no previous question difficulty estimators consider the association between multiple modalities though video QA is intrinsically a multi-modal task involving both text and video. To solve this problem, this paper proposes a novel question difficulty estimator using an enhanced directional modality attention transformer (DiMAT++). The proposed estimator adopts a CNN backbone network and a transformer to express a video modality and RoBERTa to express a text modality. However, these modalities are insufficient to classify the difficulty level of a question correctly, since they affect each other during performing video QAs. Therefore, in the proposed estimator, DiMAT++ captures directional associations from text modality to video modality and vice versa. DiMAT, the previous version of DiMAT++, does not represent the sequential information for each modality though it is designed to express the directional associations. Thus, DiMAT++ revises it to accept the sequential representations of each modality as its input. The effectiveness of the proposed estimator is verified with two benchmark video QA data sets. The experimental results indicate that the proposed estimator outperforms three baselines of heterogeneous attention mechanism (HAM), multi-modal fusion transformer (MMFT), and DiMAT, which proves that DiMAT++ is effective in improving the performance of video question difficulty estimation. [ABSTRACT FROM AUTHOR]

Neural network models enjoy success on language tasks related to Web documents, including news and Wikipedia articles. However, the characteristics of scientific publications pose specific challenges that have yet to be satisfactorily addressed: the discourse structure of scientific documents crucial in scholarly document processing (SDP) tasks, the interconnected nature of scientific documents, and their multimodal nature. We survey modern neural network learning methods that tackle these challenges: those that can model discourse structure and their interconnectivity and use their multimodal nature. We also highlight efforts to collect large-scale datasets and tools developed to enable effective deep learning deployment for SDP. We conclude with a discussion on upcoming trends and recommend future directions for pursuing neural natural language processing approaches for SDP. [ABSTRACT FROM AUTHOR]

The effectiveness of natural language processing models relies on various factors, including the architecture, number of parameters, data used during training, and the tasks they were trained on. Recent studies indicate that models pre-trained on large corpora and fine-tuned on task-specific datasets, covering multiple tasks, can generate remarkable results across various benchmarks. We propose a new approach based on a straightforward hypothesis: improving model performance on a target task by considering other artificial tasks defined on the same training dataset. By doing so, the model can gain further insights into the training dataset and attain a greater understanding, improving efficiency on the target task. This approach differs from others that consider multiple pre-existing tasks on different datasets. We validate this hypothesis by focusing on the problem of answering yes/no questions and introducing a multi-task model that outputs a span of the reference text, serving as evidence for answering the question. The task of span extraction is an artificial one, designed to benefit the performance of the model answering yes/no questions. We acquire weak supervision for these spans, by using a pre-trained extractive question answering model, dispensing the need for costly human annotation. Our experiments, using modern transformer-based language models, demonstrate that this method outperforms the standard approach of training models to answer yes/no questions. Although the primary objective was to enhance the performance of the model in answering yes/no questions, it was discovered that span texts are a significant source of information. These spans, derived from the question reference texts, provided valuable insights for the users to better comprehend the answers to the questions. The model's improved accuracy in answering yes/no questions, coupled with the supplementary information provided by the span texts, led to a more comprehensive and informative user experience. [ABSTRACT FROM AUTHOR]

Reading comprehension involves the process of reading and understanding textual information in order to answer questions related to it. It finds practical applications in various domains such as domain-specific FAQs, search engines, and dialog systems. Resource-rich languages like English, Japanese, Chinese, and most European languages benefit from the availability of numerous datasets and resources, enabling the development of machine reading comprehension (MRC) systems. However, building MRC systems for low-resource languages (LRL) with limited datasets, such as Vietnamese, Urdu, Bengali, and Hindi, poses significant challenges. To address this issue, this study utilizes quantitative analysis to conduct a systematic literature review (SLR) with the aim of comprehending the recent global shift in MRC research from high-resource languages (HRL) to low-resource languages. Notably, existing literature reviews on MRC lack comprehensive studies that compare techniques specifically designed for rich and low-resource languages. Hence, this study provides a comprehensive overview of the MRC research landscape in low-resource languages, offering valuable insights and a list of suggestions to enhance LRL–MRC research. [ABSTRACT FROM AUTHOR]

Deep learning (DL) has recently changed the development of intelligent systems and is widely adopted in many real-life applications. Despite their various benefits and potentials, there is a high demand for DL processing in different computationally limited and energy-constrained devices. It is natural to study game-changing technologies such as Binary Neural Networks (BNN) to increase DL capabilities. Recently remarkable progress has been made in BNN since they can be implemented and embedded on tiny restricted devices and save a significant amount of storage, computation cost, and energy consumption. However, nearly all BNN acts trade with extra memory, computation cost, and higher performance. This article provides a complete overview of recent developments in BNN. This article focuses exclusively on 1-bit activations and weights 1-bit convolution networks, contrary to previous surveys in which low-bit works are mixed in. It conducted a complete investigation of BNN's development—from their predecessors to the latest BNN algorithms/techniques, presenting a broad design pipeline and discussing each module's variants. Along the way, it examines BNN (a) purpose: their early successes and challenges; (b) BNN optimization: selected representative works that contain essential optimization techniques; (c) deployment: open-source frameworks for BNN modeling and development; (d) terminal: efficient computing architectures and devices for BNN and (e) applications: diverse applications with BNN. Moreover, this paper discusses potential directions and future research opportunities in each section. [ABSTRACT FROM AUTHOR]

Word representation learning is a fundamental technique in cognitive computation that plays a crucial role in enabling machines to understand and process human language. By representing words as vectors in a high-dimensional space, computers can perform complex natural language processing tasks such as sentiment analysis. However, most word representation learning models are trained in open-domain corpora, which results in suboptimal performance in domain-specific tasks. To address this problem, we propose a unified learning framework that leverages external hybrid sentiment knowledge to enhance the sentiment information of word distributed representations. Specifically, we automatically acquire domain- and target-dependent sentiment knowledge from multiple sources. To mitigate knowledge noise, we introduce knowledge expectation and knowledge context weights to filter the acquired knowledge items. Finally, we integrate the filtered sentiment knowledge into the word distributed representations via a learning framework to enrich their semantic information. Extensive experiments are conducted to verify the effectiveness of enhancing sentiment information in word representations for different sentiment analysis tasks. The experimental results show that the proposed models significantly outperform state-of-the-art baselines. Our work demonstrates the advantages of sentiment-enhanced word representations in sentiment analysis tasks and provides insights into the acquisition and fusion of sentiment knowledge from different domains for generating word representations with richer semantics. [ABSTRACT FROM AUTHOR]

To reliably and accurately classify complicated "big" datasets, machine learning models must be continually improved. This research proposes straightforward yet competitive neural networks for text classification, even though graph neural networks (GNN) have reignited interest in graph-based text classification models. Convolutional neural networks (CNN), artificial neural networks (ANN), and their refined "fine-tuned" models (denoted as FT-CNN and FT-ANN) are the names given to our proposed models. The models presented in this paper demonstrate that our simple models like (CNN, ANN, FT-CNN, and FT-ANN) can perform better than more complex GNN ones such as (SGC, SSGC, and TextGCN) and are comparable to others (i.e., HyperGAT and Bert). The process of fine-tuning is also highly recommended because it improves the performance and reliability of models. The performance of our suggested models on five benchmark datasets (namely, Reuters (R8), R52, 20NewsGroup, Ohsumed, and Mr) is vividly illustrated. According to the experimental findings, on the majority of the target datasets, these models—especially those that have been fine-tuned—perform surprisingly better than SOTA approaches, including GNN-based models. [ABSTRACT FROM AUTHOR]

Adversarial training has been empirically shown to be more prone to overfitting than standard training. The exact underlying reasons are still not fully understood. In this paper, we identify one cause of overfitting related to current practices of generating adversarial examples from misclassified samples. We show that, following current practice, adversarial examples from misclassified samples results in harder-to-classify samples than the original ones. This leads to a complex adjustment of the decision boundary during training and hence overfitting. To mitigate this issue, we propose A3T, an accuracy aware AT method that generate adversarial example differently for misclassified and correctly classified samples. We show that our approach achieves better generalization while maintaining comparable robustness to state-of-the-art AT methods on a wide range of computer vision, natural language processing, and tabular tasks. [ABSTRACT FROM AUTHOR]

This paper explores the application of deep learning in automating the scoring of open-ended candidate responses to pre-hire employment selection assessments. Using job applicant text data from pre-employment virtual assessment center exercises, three algorithmic approaches were compared: a traditional bag of words (BoW), long short-term memory (LSTM) models, and robustly optimized bidirectional encoder representations from transformers approach (RoBERTa). Measurement and assessment best practices were leveraged in the development of the candidate assessment items and human labels (subject matter experts' (SME) ratings on job-relevant competencies), producing a rich set of data to train the algorithms on. The trained models were used to score the candidate textual responses on the given competencies, and the level of agreement with expert human raters was assessed. Using data from three companies hiring for three different occupations and across seven competencies, three algorithmic approaches to automatically score text were evaluated, showcasing correlations between SME and algorithmically scored competencies on holdout samples that were very strong (avg r = 0.84 for the best performing method, RoBERTa) and nearly identical to the inter-rater reliability achieved by multiple expert raters following consensus (avg r = 0.85). Criterion-related validity, subgroup differences, and decision accuracy are investigated for each algorithmic approach. Lastly, the impact of smaller sample sizes to train the algorithms is explored. [ABSTRACT FROM AUTHOR]

This paper introduces a semantics-aware approach to natural language inference which allows neural network models to perform better on natural language inference benchmarks. We propose to incorporate explicit lexical and concept-level semantics from knowledge bases to improve inference accuracy. We conduct an extensive evaluation of four models using different sentence encoders, including continuous bag-of-words, convolutional neural network, recurrent neural network, and the transformer model. Experimental results demonstrate that semantics-aware neural models give better accuracy than those without semantics information. On average of the three strong models, our semantic-aware approach improves natural language inference in different languages. [ABSTRACT FROM AUTHOR]

The question answering system is frequently applied in the area of natural language processing (NLP) because of the wide variety of applications. It consists of answering questions using natural language. The problem is, in general, solved by employing a dataset that consists of an input text, a query, and the text segment or span from the input text that provides the question's answer. The ability to make human-level predictions from data has improved significantly thanks to deep learning models, particularly the Transformer architecture, which has been state-of-the-art in text-based models in recent years. This paper reviews studies related to the use of transformer models in the implementation of question-answering (QA) systems. The paper's first focus is on the attention and transformer models. A brief description of the architectures is presented by classifying them into models based on encoders, decoders, and on both Encoder-Decoder. Following that, we examine the most recent research trends in textual QA datasets by highlighting the architecture of QA systems and categorizing them according to various criteria. We survey also a significant set of evaluation metrics that have been developed in order to evaluate the models' performance. Finally, we highlight solutions built to simplify the implementation of Transformer models. [ABSTRACT FROM AUTHOR]

In this work we present the Chatbot Interaction with Artificial Intelligence (CI-AI) framework as an approach to the training of a transformer based chatbot-like architecture for task classification with a focus on natural human interaction with a machine as opposed to interfaces, code, or formal commands. The intelligent system augments human-sourced data via artificial paraphrasing in order to generate a large set of training data for further classical, attention, and language transformation-based learning approaches for Natural Language Processing (NLP). Human beings are asked to paraphrase commands and questions for task identification for further execution of algorithms as skills. The commands and questions are split into training and validation sets. A total of 483 responses were recorded. Secondly, the training set is paraphrased by the T5 model in order to augment it with further data. Seven state-of-the-art transformer-based text classification algorithms (BERT, DistilBERT, RoBERTa, DistilRoBERTa, XLM, XLM-RoBERTa, and XLNet) are benchmarked for both sets after fine-tuning on the training data for two epochs. We find that all models are improved when training data is augmented by the T5 model, with an average increase of classification accuracy by 4.01%. The best result was the RoBERTa model trained on T5 augmented data which achieved 98.96% classification accuracy. Finally, we found that an ensemble of the five best-performing transformer models via Logistic Regression of output label predictions led to an accuracy of 99.59% on the dataset of human responses. A highly-performing model allows the intelligent system to interpret human commands at the social-interaction level through a chatbot-like interface (e.g. "Robot, can we have a conversation?") and allows for better accessibility to AI by non-technical users. [ABSTRACT FROM AUTHOR]

There is a deluge of AI-assisted decision-making systems, where our data serve as proxy to our actions, suggested by AI. The closer we investigate our data (raw input, or their learned representations, or the suggested actions), we begin to discover "bugs". Outside of their test, controlled environments, AI systems may encounter situations investigated primarily by those in other disciplines, but experts in those fields are typically excluded from the design process and are only invited to attest to the ethical features of the resulting system or to comment on demonstrations of intelligence and aspects of craftmanship after the fact. This communicative impasse must be overcome. Our idea is that philosophical and engineering considerations interact and can be fruitfully combined in the AI design process from the very beginning. We embody this idea in the role of a philosopher engineer. We discuss the role of philosopher engineers in the three main design stages of an AI system: deployment management (what is the system's intended use, in what environment?); objective setting (what should the system be trained to do, and how?); and training (what model should be used, and why?). We then exemplify the need for philosopher engineers with an illustrative example, investigating how the future decisions of an AI-based hiring system can be fairer than those contained in the biased input data on which it is trained; and we briefly sketch the kind of interdisciplinary education that we envision will help to bring about better AI. [ABSTRACT FROM AUTHOR]

Aspect-based sentiment classification aims to predict the sentiment polarity on given aspect terms in a sentence. Recent works have incorporated syntactic information by developing graph neural networks (GNNs) over dependency trees to better establish connections between aspect items and their related context. However, the advancement is restricted because the dependency tree derived by the external parser is not entirely accurate, especially for high complexity and arbitrary expression datasets. To address this constraint, we propose a dynamic multichannel fusion mechanism based on the Graph AttenTion network and BERT (DMF-GAT-BERT), which regards the complementarity of semantic and syntactic information captured by GAT and BERT, respectively. Specifically, to alleviate the damage of incorrect dependency tree information to the model, we propose a two-layer dynamic fusion mechanism to adaptively adjust the fusion weight of semantic and syntax-related information channels. In addition, to capture accurate syntactic features, we propose an attentive layer ensemble (ALE) to integrate the contextual features learned by GAT in different layers. We conducted experiments on four datasets with different complexity, the Laptop, Restaurant, Twitter, and MAMS datasets, and achieved 80.38%, 86.10%, 76.22%, and 83.86% accuracy, respectively, outperforming robust baseline approaches. [ABSTRACT FROM AUTHOR]

In scientific literature and industry, semantic and context-aware Natural Language Processing-based solutions have been gaining importance in recent years. The possibilities and performance shown by these models when dealing with complex Human Language Understanding tasks are unquestionable, from conversational agents to the fight against disinformation in social networks. In addition, considerable attention is also being paid to developing multilingual models to tackle the language bottleneck. An increase in size has accompanied the growing need to provide more complex models implementing all these features without being conservative in the number of dimensions required. This paper aims to provide a comprehensive account of the impact of a wide variety of dimensional reduction techniques on the performance of different state-of-the-art multilingual siamese transformers, including unsupervised dimensional reduction techniques such as linear and nonlinear feature extraction, feature selection, and manifold techniques. In order to evaluate the effects of these techniques, we considered the multilingual extended version of Semantic Textual Similarity Benchmark (mSTSb) and two different baseline approaches, one using the embeddings from the pre-trained version of five models and another using their fine-tuned STS version. The results evidence that it is possible to achieve an average reduction of 91.58 % ± 2.59 % in the number of dimensions of embeddings from pre-trained models requiring a fitting time 96.68 % ± 0.68 % faster than the fine-tuning process. Besides, we achieve 54.65 % ± 32.20 % dimensionality reduction in embeddings from fine-tuned models. The results of this study will significantly contribute to the understanding of how different tuning approaches affect performance on semantic-aware tasks and how dimensional reduction techniques deal with the high-dimensional embeddings computed for the STS task and their potential for other highly demanding NLP tasks. [ABSTRACT FROM AUTHOR]

Recently, pre-trained language models (PLMs) have become core components in a wide range of natural language processing applications. However, PLMs like BERT and RoBERTa are typically trained with a large amount of unlabeled text corpora which requires extremely high computational cost. Curriculum learning (CL) is a learning strategy for training a model from easy samples to hard ones that has potential to alleviate this problem. Nevertheless, how to determine the difficulty measure of training samples for PLMs and an effective training scheduler are still open questions. In this study, we focus on the length of input text as the difficulty measure and propose a new CL approach called length-based CL. We analyze the effectiveness of the length-based difficulty measure in terms of convergence speed and GLUE scores using a limited amount of corpus. By combining maximum available batch size with the length-based difficulty measure, we show that our length-based CL model can achieve 1.5 times faster convergence speed in pre-training and better performances on downstream tasks. Furthermore, we expand the corpus to evaluate various pacing functions (training schedulers) for the length-based CL with respect to the computational time and generalization performance. Through experiments with a larger corpus, we find that our proposed Square scheduler achieved less computational time in pre-training and obtained the best generalization performance on downstream tasks. [ABSTRACT FROM AUTHOR]

Crowdsourcing holds great potential: macro-task crowdsourcing can, for example, contribute to work addressing climate change. Macro-task crowdsourcing aims to use the wisdom of a crowd to tackle non-trivial tasks such as wicked problems. However, macro-task crowdsourcing is labor-intensive and complex to facilitate, which limits its efficiency, effectiveness, and use. Technological advancements in artificial intelligence (AI) might overcome these limits by supporting the facilitation of crowdsourcing. However, AI's potential for macro-task crowdsourcing facilitation needs to be better understood for this to happen. Here, we turn to affordance theory to develop this understanding. Affordances help us describe action possibilities that characterize the relationship between the facilitator and AI, within macro-task crowdsourcing. We follow a two-stage, bottom-up approach: The initial development stage is based on a structured analysis of academic literature. The subsequent validation & refinement stage includes two observed macro-task crowdsourcing initiatives and six expert interviews. From our analysis, we derive seven AI affordances that support 17 facilitation activities in macro-task crowdsourcing. We also identify specific manifestations that illustrate the affordances. Our findings increase the scholarly understanding of macro-task crowdsourcing and advance the discourse on facilitation. Further, they help practitioners identify potential ways to integrate AI into crowdsourcing facilitation. These results could improve the efficiency of facilitation activities and the effectiveness of macro-task crowdsourcing. [ABSTRACT FROM AUTHOR]

Practical ability manifested through robust and reliable task performance, as well as information relevance and well-structured representation, are key factors indicative of understanding in the philosophical literature. We explore these factors in the context of deep learning, identifying prominent patterns in how the results of these algorithms represent information. While the estimation applications of modern neural networks do not qualify as the mental activity of persons, we argue that coupling analyses from philosophical accounts with the empirical and theoretical basis for identifying these factors in deep learning representations provides a framework for discussing and critically evaluating potential machine understanding given the continually improving task performance enabled by such algorithms. [ABSTRACT FROM AUTHOR]

Pollen identification is a sub-discipline of Palynology, which has broad applications in several fields such as allergy control, paleoclimate reconstruction, criminal investigation, and petroleum exploration. Among these, pollen allergy is a common and frequent disease worldwide. Accurate and rapid identification of pollen species under the electron microscope help medical staff in pollen forecast and interrupt the natural course of pollen allergy. The current pollen species identification needs to rely on professional researchers to identify pollen particles in pictures manually, and this time-consuming and laborious way cannot meet the requirements of pollen forecasting. Recently, the self-attention based Transformer has attracted considerable attention in vision tasks, such as image classification. However, pure self-attention lacks local operations on pixels and requires large-scale dataset pretraining to achieve comparable performance to convolutional neural networks (CNN). In this study, we propose a new Vision Transformer pipeline for image classification. First, we design a FeatureMap-to-Token (F2T) module to perform token embedding on the input image. A global self-attention operation is performed on the basis of tokens with local information, and the hierarchical design of CNN is applied to the Vision Transformer, combining local and global strengths in multiscale spaces. Second, we use a distillation strategy to learn the feature representation in the output space of the teacher network to further learn the inductive bias in the CNN to improve the recognition accuracy. Experiments demonstrate that the proposed model achieves CNN-equivalent performance under the same conditions after being trained from scratch on the electron-microscopic pollen dataset. It also requires less model parameters and training time. Code for the model is available at https://github.com/dkbshuai/PyTorch-Our-S. [ABSTRACT FROM AUTHOR]

In humans, Attention is a core property of all perceptual and cognitive operations. Given our limited ability to process competing sources, attention mechanisms select, modulate, and focus on the information most relevant to behavior. For decades, concepts and functions of attention have been studied in philosophy, psychology, neuroscience, and computing. For the last 6 years, this property has been widely explored in deep neural networks. Currently, the state-of-the-art in Deep Learning is represented by neural attention models in several application domains. This survey provides a comprehensive overview and analysis of developments in neural attention models. We systematically reviewed hundreds of architectures in the area, identifying and discussing those in which attention has shown a significant impact. We also developed and made public an automated methodology to facilitate the development of reviews in the area. By critically analyzing 650 works, we describe the primary uses of attention in convolutional, recurrent networks, and generative models, identifying common subgroups of uses and applications. Furthermore, we describe the impact of attention in different application domains and their impact on neural networks' interpretability. Finally, we list possible trends and opportunities for further research, hoping that this review will provide a succinct overview of the main attentional models in the area and guide researchers in developing future approaches that will drive further improvements. [ABSTRACT FROM AUTHOR]

The Answer Sentence Selection (AS2) task is defined as the task of ranking the candidate answers for each question based on a matching score. The matching score is the probability of being a correct answer for a given question. Detecting the question class and matching it with the named entities of the answer sentence to narrow down the search space was used in primary question answering systems. We used this idea in the state-of-the-art text matching models namely, Transformer-based language models. In this paper, we proposed two different architectures: Ent-match and Ent-add, while using two different question classifiers: Convolutional Neural Network-based (CNN-based) and rule-based. The proposed models outperform the state-of-the-art AS2 model, namely TANDA and RoBERTa-base on both TREC-QA and Wiki-QA datasets. Using Wiki-QA, the Ent-add (CNN-based) model outperforms the TANDA model by 2.1% and 1.9% improvement over Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR) metrics, respectively. Over the TREC-QA dataset the Ent-match (CNN-based) model outperformed the TANDA model with 1.5% and 1.4% improvement over MAP and MRR, respectively. [ABSTRACT FROM AUTHOR]

Data-driven algorithms are studied and deployed in diverse domains to support critical decisions, directly impacting people's well-being. As a result, a growing community of researchers has been investigating the equity of existing algorithms and proposing novel ones, advancing the understanding of risks and opportunities of automated decision-making for historically disadvantaged populations. Progress in fair machine learning and equitable algorithm design hinges on data, which can be appropriately used only if adequately documented. Unfortunately, the algorithmic fairness community, as a whole, suffers from a collective data documentation debt caused by a lack of information on specific resources (opacity) and scatteredness of available information (sparsity). In this work, we target this data documentation debt by surveying over two hundred datasets employed in algorithmic fairness research, and producing standardized and searchable documentation for each of them. Moreover we rigorously identify the three most popular fairness datasets, namely Adult, COMPAS, and German Credit, for which we compile in-depth documentation. This unifying documentation effort supports multiple contributions. Firstly, we summarize the merits and limitations of Adult, COMPAS, and German Credit, adding to and unifying recent scholarship, calling into question their suitability as general-purpose fairness benchmarks. Secondly, we document hundreds of available alternatives, annotating their domain and supported fairness tasks, along with additional properties of interest for fairness practitioners and researchers, including their format, cardinality, and the sensitive attributes they encode. We summarize this information, zooming in on the tasks, domains, and roles of these resources. Finally, we analyze these datasets from the perspective of five important data curation topics: anonymization, consent, inclusivity, labeling of sensitive attributes, and transparency. We discuss different approaches and levels of attention to these topics, making them tangible, and distill them into a set of best practices for the curation of novel resources. [ABSTRACT FROM AUTHOR]

Deep-learning based natural language processing (NLP) models are proven vulnerable to adversarial attacks. However, there is currently insufficient research that studies attacks to neural machine translations (NMTs) and examines the robustness of deep-learning based NMTs. In this paper, we aim to fill this critical research gap. When generating word-level adversarial examples in NLP attacks, there is a conventional trade-off in existing methods between the attacking performance and the amount of perturbations. Although some literature has studied such a trade-off and successfully generated adversarial examples with a reasonable amount of perturbations, it is still challenging to generate highly successful translation attacks while concealing the changes to the texts. To this end, we propose a novel Hybrid Attentive Attack method to locate language-specific and sequence-focused words, and make semantic-aware substitutions to attack NMTs. We evaluate the effectiveness of our attack strategy by attacking three high-performing translation models. The experimental results show that our method achieves the highest attacking performance compared with other existing attacking strategies. [ABSTRACT FROM AUTHOR]

Propaganda is a rhetorical technique designed to serve a specific topic, which is often used purposefully in news article to achieve our intended purpose because of its specific psychological effect. Therefore, it is significant to be clear where and what propaganda techniques are used in the news for people to understand its theme efficiently during our daily lives. Recently, some relevant researches are proposed for propaganda detection but unsatisfactorily. As a result, detection of propaganda techniques in news articles is badly in need of research. In this paper, we are going to introduce our systems for detection of propaganda techniques in news articles, which is split into two tasks, Span Identification and Technique Classification. For these two tasks, we design a system based on the popular pretrained BERT model, respectively. Furthermore, we adopt the over-sampling and EDA strategies, propose a sentence-level feature concatenating method in our systems. Experiments on the dataset of about 550 news articles offered by SEMEVAL show that our systems perform state-of-the-art. [ABSTRACT FROM AUTHOR]