Search

The dual-channel assumption of the cognitive theory of multimedia learning suggests that importing a large amount of information through a single (visual or audio) channel overloads that channel, causing partial loss of information, while importing it simultaneously through multiple channels relieves the burden on them and leads to the registration of a larger amount of information. In light of such knowledge, this study investigates the possibility of reinforcing visual stimuli with audio for supporting e-learners in memorization tasks. Specifically, we consider three kinds of learning material and two kinds of audio stimuli and partially reinforce each kind of material with each kind of stimuli in an arbitrary way. In a series of experiments, we determine the particular type of audio, which offers the highest improvement for each kind of material. Our work stands out as being the first study investigating the differences in memory performance in relation to different combinations of learning content and stimulus. Our key findings from the experiments are: (i) E-learning is more effective in refreshing memory rather than studying from scratch, (ii) Non-informative audio is more suited to verbal content, whereas informative audio is better for numerical content, (iii) Constant audio triggering degrades learning performance and thus audio triggering should be handled with care. Based on these findings, we develop an ANN-based estimator to determine the proper moment for triggering audio (i.e. when memory performance is estimated to be declining) and carry out follow-up experiments for testing the integrated framework. Our contributions involve (i) determination of the most effective audio for each content type, (ii) estimation of memory deterioration based on learners' interaction logs, and (iii) the proposal of improvement of memory registration through auditory reinforcement. We believe that such findings constitute encouraging evidence the memory registration of e-learners can be enhanced with content-aware audio incorporation.

International audience; We propose a novel Mean-Shift method for data clustering, called Robust Mean-Shift (RMS). A new update equation for point iterates is proposed, mixing the ones of the standard Mean-Shift (MS) and the Blurring Mean-Shift (BMS). Despite its simplicity, the proposed method has not been studied so far. RMS can be set up in both a kernel-based and a nearest-neighbor (NN)-based fashion. Since the update rule of RMS is closer to BMS, the convergence of point iterates is conjectured based on the Chen's BMS convergence theorem. Experimental results on synthetic and real datasets show that RMS in several cases outperforms MS and BMS in the clustering task. In addition, RMS exhibits larger attraction basins than MS and BMS for identical parametrization; consequently, its kernel variant requires a lower aperture of the kernel function, and its NN variant a lower number of nearest neighbors compared to MS or BMS, to achieve optimal clustering results. In addition, the NN version of RMS does not need to specify a convergence threshold to stop the iterations, contrarily to the NN-BMS algorithm.

Refereed/Peer-reviewed Investigation of value based Reinforcement Learning algorithms exhibited a resurgence into mainstream research in 2015 following demonstration of super-human performance when applied to Atari 2600 games. Since then, significant media attention and hype have accompanied this area, and the field of Artificial Intelligence generally, spread across distinct categories. This review paper is focused exclusively on the progression of value based Reinforcement Learning in the last five years. We aim to distill the incremental improvements to stability and performance in this period, highlighting the minimal changes to the base algorithm over this time. This holds true with all but the one exception of the Recurrent Experience Replay in Distributed Reinforcement Learning algorithm, representing a fundamental shift and increase in agent performance through an advanced memory representation. We suggest a new focus area for value based Reinforcement Learning research.

In the field of WiFi sensing, as an alternative sensing source of the channel state information (CSI) matrix, the use of a beamforming feedback matrix (BFM)that is a right singular matrix of the CSI matrix has attracted significant interest owing to its wide availability regarding the underlying WiFi systems. In the IEEE 802.11ac/ax standard, the station (STA) transmits a BFM to an access point (AP), which uses the BFM for precoded multiple-input and multiple-output communications. In addition, in the same way, the AP transmits a BFM to the STA, and the STA uses the received BFM. Regarding BFM-based sensing, extensive real-world experiments were conducted as part of this study, and two key insights were reported: Firstly, this report identified a potential issue related to accuracy in existing uni-directional BFM-based sensing frameworks that leverage only BFMs transmitted for the AP or STA. Such uni-directionality introduces accuracy concerns when there is a sensing capability gap between the uni-directional BFMs for the AP and STA. Thus, this report experimentally evaluates the sensing ability disparity between the uni-directional BFMs, and shows that the BFMs transmitted for an AP achieve higher sensing accuracy compared to the BFMs transmitted from the STA when the sensing target values are estimated depending on the angle of departure of the AP. Secondly, to complement the sensing gap, this paper proposes a bi-directional sensing framework, which simultaneously leverages the BFMs transmitted from the AP and STA. The experimental evaluations reveal that bi-directional sensing achieves higher accuracy than uni-directional sensing in terms of the human localization task., Comment: 10 pages, 7 figures

Counting the number of people in a crowd has gained attention in the last decade. Due to its benefit to many applications such as crowd behavior analysis, crowd management, and video surveillance systems, etc. Counting crowded scenes, like stadiums, represents a challenging task due to the inherent occlusions and density of the crowd inside and outside the stadiums. Finding a pattern to control thousands of people and counting them is a challenging task. With the introduction of Convolutional Neural Networks (CNN), enables performing this task with acceptable performance. The accuracy of a CNN-based method is related to the size of data used for training. The availability of the dataset is sparse. In particular, there is no dataset in the literature that can be used for training applications for crowd scene. This paper proposes two main contributions including a new dataset for crowd counting, and a CNN-based method for counting the number of people and generating the crowd density maps. The proposed dataset for Football Supporters Crowd (FSC-Set) is composed of 6000 annotated images (manually) of different types of scenes that contain thousands of people gathering in or around the stadiums. FSC-Set contains more than 1.5 Million individuals. The collected images are captured under varying Fields of Views (FOV), illuminations, resolutions, and scales. The proposed dataset can also be utilized for other applications, such as individual's localization and face detection as well as team recognition from supporter images. Further, we propose a CNN-based method named FSCNet for crowd counting exploiting context-aware attention, spatial-wise attention, and channel-wise attention modules. The proposed method is evaluated on our established FSC-Set and other existing datasets then compared to state-of-the-art methods. The obtained results show satisfactory performances on all the datasets. The dataset is made publicly available and can be requested using the following link: https://sites.google.com/view/fscrowd-dataset/ 2013 IEEE. Scopus

We propose a novel method that can learn easy-to-interpret latent representations in real-world image datasets using a VAE-based model by splitting an image into several disjoint regions. Our method performs object-wise disentanglement by exploiting image segmentation and alpha compositing. With remarkable results obtained by unsupervised disentanglement methods for toy datasets, recent studies have tackled challenging disentanglement for real-world image datasets. However, these methods involve deviations from the standard VAE architecture, which has favorable disentanglement properties. Thus, for disentanglement in images of real-world image datasets with preservation of the VAE backbone, we designed an encoder and a decoder that embed an image into disjoint sets of latent variables corresponding to objects. The encoder includes a pre-trained image segmentation network, which allows our model to focus only on representation learning while adopting image segmentation as an inductive bias. Evaluations using real-world image datasets, CelebA and Stanford Cars, showed that our method achieves improved disentanglement and transferability.

Even if measuring the outcome of binary classifications is a pivotal task in machine learning and statistics, no consensus has been reached yet about which statistical rate to employ to this end. In the last century, the computer science and statistics communities have introduced several scores summing up the correctness of the predictions with respect to the ground truth values. Among these scores, the Matthews correlation coefficient (MCC) was shown to have several advantages over confusion entropy, accuracy, F1 score, balanced accuracy, bookmaker informedness, markedness, and diagnostic odds ratio: MCC, in fact, produces a high score only if the majority of the predicted negative data instances and the majority of the positive data instances are correct, and therefore it results being very trustworthy on imbalanced datasets. In this study, we compare MCC with two other popular scores: Cohen’s Kappa, a metric that originated in social sciences, and the Brier score, a strictly proper scoring function which emerged in weather forecasting studies. After explaining the mathematical properties and the relationships between MCC and each of these two rates, we report some use cases where these scores generate different values, which lead to discordant outcomes, where MCC provides a more truthful and informative result. We highlight the reasons why it is more advisable to use MCC rather that Cohen’s Kappa and the Brier score to evaluate binary classifications.

Distress image retrieval for infrastructure maintenance via self-trained deep metric learning using experts’ knowledge is proposed in this paper. Since engineers take multiple images of a single distress part for inspection of road structures, it is necessary to construct a similar distress image retrieval method considering the input of multiple images to support determination of the level of deterioration. Thus, the construction of an image retrieval method while selecting an effective input from multiple images is described in this paper. The proposed method performs deep metric learning by using a small number of effective images labeled by experts’ knowledge with information about their effectiveness and a large number of unlabeled images via a self-training approach. Specifically, an end-to-end learning approach that performs retraining of the model by assigning pseudo-labels to these unlabeled images according to the output confidence of the model is achieved. Thus, the proposed method can select an effective image from multiple images that are input at the retrieval as a query image. This is the main contribution of this paper. As a result, the proposed method realizes highly accurate retrieval of similar distress images considering the actual situation of inspection in which multiple images of a distress part are input.

Describing orally what we are seeing is a simple task we do in our daily life. However, in the natural language processing field, this simple task needs to be bridged by a textual modality that helps the system to generalize various objects in the image and various pronunciations in speech utterances. In this study, we propose an end-to-end Image2Speech system that does not need any textual information in its training. We use a vector-quantized variational autoencoder (VQ-VAE) model to learn the discrete representation of a speech caption in an unsupervised manner, where discrete labels are used by an image-captioning model. This self-supervised speech representation enables the Image2Speech model to be trained with the minimum amount of paired image-speech data while still maintaining the quality of the speech caption. Our experimental results with a multi-speaker natural speech dataset demonstrate our proposed text-free Image2Speech system’s performance close to the one with textual information. Furthermore, our approach also successfully outperforms the most recent existing frameworks with phoneme-based and grounding-based Image2Speech systems.

In this paper, we designed a registration framework that can be used to develop augmented reality environments, where all the real (including the users) and virtual elements are co-localized and registered in a common reference frame. The software is provided together with this paper, to contribute to the research community. The developed framework allows us to perform a quantitative assessment of interaction and egocentric perception in Augmented Reality (AR) environments. We assess perception and interaction in the peripersonal space through a 3D blind reaching task in a simple scenario and an interaction task in a kitchen scenario using both video (VST) and optical see-through (OST) head-worn technologies. Moreover, we carry out the same 3D blind reaching task in real condition (without head-mounted display and reaching real targets). This provides a baseline performance with which to compare the two augmented reality technologies. The blind reaching task results show an underestimation of distances with OST devices and smaller estimation errors in frontal spatial positions when the depth does not change. This happens with both OST and VST devices, compared with the real-world baseline. Such errors are compensated in the interaction kitchen scenario task. Thanks to the egocentric viewing geometry and the specific required task, which constrain the position perception on a table, both VST and OST have comparable and effective performance. Thus, our results show that such technologies have issues, though they can be effectively used in specific real tasks. This does not allow us to choose between VST and OST devices. Still, it provides a baseline and a registration framework for further studies and emphasizes the specificity of perception in interactive AR.

Heavy equipment operation is a responsible and difficult task causing mental workload on a human operator and exposing the operator to a range of harmful factors. Human factors and ergonomics in heavy equipment design have traditionally been focused on anthropometry and questionnaires. More advanced techniques involving biosignal measurements were not applied to heavy equipment, mainly due to the diversity of real working conditions that were hard to reproduce in a laboratory environment and that prevented ambulatory studies. Recent advances in wearable biosensors and real-time simulators produce the capability of using biosignals for improving the ergonomics of heavy equipment operation. The present paper reviews the use of biosignals in human factors and the ergonomics of heavy machines by focusing on stress detection for the last ten years. The aim of the paper is analyzing the previous implemented algorithms to find a set of biosignals and methods of stress identification that could be suitable for identifying stress in heavy equipment operators both in laboratory and ambulatory studies. The conclusion emphasizes successful stress identification methods and a combination of the algorithms from different studies that facilitate the use of heavy equipment operator’s applications. Also, feasible methods and directions for future research are considered.

Opinion mining has gained increasing importance to draw insights from social media content to support decision making. Despite the explosive growth of efforts on linguistic analysis to detect and track people’s opinions, more specifically when dealing with aspect-level opinion mining, the results are still away from generalization to real-world applications. Nowadays, emojis are getting excessively popular in social media communication as a complementary way to quickly express opinions and ideas in a visual manner. Two emoji-related issues are highlighted: ambiguity and misinterpretation of emojis’ sentiment and tendency of persons to adopt emojis more in positive cases. This paper aims at investigating to what extent the usage of emojis can contribute to the automated detection of sentiment polarity of text messages with focus on Twitter posts in the Arabic language, a widely spoken language but has complex morphology and limited reliable resources for sentiment analysis. For this purpose, after an extensive review of the state-of-the-art of emojis-related work, a dataset is composed and several feature extraction methods are applied for both text and emojis modalities. Moreover, various early and late fusion techniques are proposed to combine both modalities at different levels including feature, score, decision and hybrid. The experimental results revealed that emojis features can significantly improve the classification results, especially when integrated with text at the score level.

In this study, we focus on the federated learning (FL) based tactical edge network platform to cooperatively operate computation-hungry tasks as efficiently as possible. Based on the incentive of FL model training, each individual device makes offloading decisions for their tactical tasks in resource-constrained network environments. According to the ideas of two different bargaining solutions -weighted average solution and constant elasticity substitution solution - edge server and IoT devices work together in a coordinated manner to approximate a well-balanced system performance. Therefore, we can reach an agreement while exploring the mutual benefits to leverage a reciprocal consensus between different viewpoints. The main novelty of our approach is to investigate the dual-interactive bargaining process based on the interdependent relationship between IoT devices and the tactical edge server. To the best of our knowledge, this is the first work that jointly considers different bargaining solutions to handle tactical edge-assisted task offloading services. Based on the numerical simulation, it is demonstrated that the proposed approach can increase the system throughput, device payoff and device fairness up to 10%, 15% and 20%, respectively, in comparison with existing protocols.

Text to face generation is a sub-domain of text to image synthesis. It has a huge impact on new research areas along with the wide range of applications in the public safety domain. Due to the lack of dataset, the research work focused on the text to face generation is very limited. Most of the work for text to face generation until now is based on the partially trained generative adversarial networks, in which the pre-trained text encoder has been used to extract the semantic features of the input sentence. Later, these semantic features have been utilized to train the image decoder. In this research work, we propose a fully trained generative adversarial network to generate realistic and natural images. The proposed work trained the text encoder as well as the image decoder at the same time to generate more accurate and efficient results. In addition to the proposed methodology, another contribution is to generate the dataset by the amalgamation of LFW, CelebA and locally prepared dataset. The dataset has also been labeled according to our defined classes. Through performing different kinds of experiments, it has been proved that our proposed fully trained GAN outperformed by generating good quality images by the input sentence. Moreover, the visual results have also strengthened our experiments by generating the face images according to the given query.

Domain adaptation (DA) is a technology that transfers knowledge from the source domain to the target domain. General domain adaptation assume that the source and the target domain have the same label space. However, in practical application tasks, the label of the target domain is often only a subset of the source label. For this situation, partial domain adaptation is usually used as an effective solution to transfer knowledge from a large number of labeled data sets to unlabeled micro data sets. In this article, a weighted partial domain adaptation method is proposed to solve the Acoustic Scene Classification (ASC) problem. Our method establish a connection between source and target domains to do the partial domain adaptation. Experiments are carried out on TUT and ESC-50 datasets which show that our method achieves state-of-the-art results. What is more, we apply the algorithms to an optical fiber perimeter security system to complete early warning by identifying intrusion signals.

Recently Neural Architecture Search has drawn interest from researchers because of its ability to learn neural network architectures from data automatically. The differentiable methods are widely used because they can obtain better architectures with less computational resources. However, the method is with a mismatch between training and inference. In training process, we minimize the loss on the expected output of an ensemble of models, whereas we infer a single model at test time. In this paper we present a topology-sensitive approach to neural architecture search. Unlike previous work, we do not resort to the search strategy that works in a different scope of architecture topologies with inference. Instead, the topology of a neural network is explicitly modeled in the training/search process. We experiment with our method on the PTB and WikiText language modeling tasks. On the PTB dataset, it outperforms several strong baselines. Also, the newly-found architecture shows good transferability to named entity recognition and machine translation tasks. In particular, experimental results show our method outperforms the strong baseline by a BLEU score of 0.6 on IWSLT15 English-Vietnamese task and a BLEU score of 0.9 on WMT14 English-German task.

With the development of integrated technology and equipment systems, decision-making support has been increasingly applied in equipment research and development, especially in air combat. The development of airborne equipment can significantly improve the confrontation ability of manned fighters. Contribution evaluations restrict decision making in equipment research and development and are important in beyond-visual-range (BVR) air combat, which involves dynamic and uncertain enemy positions. This paper proposes a contribution evaluation model for BVR air combat based on autonomous maneuver decision making; this approach mainly includes a situation evaluation model, a maneuver decision model, and a one-to-one BVR air combat evaluation model. However, such a model includes a high-dimensional state and action space, which require many computations, especially if equipment changes occur. Then, an autonomous maneuver decision algorithm based on the influence diagram method is proposed; this algorithm uses a rolling time domain concept to improve combat fidelity and obtain useful equipment contribution evaluation results. Finally, one-to-one BVR air combat is simulated based on different aircraft models and airborne equipment. The simulation results show that the proposed maneuver decision model and countermeasure evaluation model can help experts quantify the contributions of equipment and provide adequate decision-making support to improve equipment systems.

Emotion-cause pair extraction (ECPE), which aims to extract emotions and the corresponding causes in documents, has a wide range of applications in network public opinion analysis. Current two-stage methods first extract emotion and cause clauses, and then pair them. However, there are two problems in these methods: 1) the unidirectional enhancement between emotion and cause extraction fails to make full use of the correlation between them; 2) the errors from the first stage directly degrade the performance of the second stage. To address these problems, we firstly propose a mutually auxiliary multitask model to promote the extraction of emotion and cause clauses by adding two auxiliary tasks which are identical to the original tasks. The proposed model uses the predicted results generated by the two auxiliary tasks as extra features of each other’s main tasks, so as to establish the bidirectional correlation between emotion and cause extraction. Secondly, to reduce the influence of error propagation on the second stage, we design a self-distillation method for pairwise tasks to train the proposed model, which further improve the accuracy of emotion and cause extraction. Experimental results on the ECPE benchmark dataset show that the proposed model has achieved good performance on emotion-cause pair extraction, outperforming the baseline models by 1.92% in F1 score.

Machine reading comprehension is a very challenging task, which aims to determine the answer span based on the given context and question. The newly developed pre-training language model has achieved a series of successes in various natural language understanding tasks with its powerful contextual representation ability. However, these pre-training language models generally lack the downstream processing structure for specific tasks, which limits further performance improvement. In order to solve this problem and deepen the model’s understanding of the question and context, this paper proposes S&I Reader. On the basis of the pre-training model, skimming, intensive reading, and gated mechanism modules are added to simulate the behavior of humans reading text and filtering information. Based on the idea of granular computing, a multi-granularity module for computing context granularity and sequence granularity is added to the model to simulate the behavior of human beings to understand the text from words to sentences, from parts to the whole. Compared with the previous machine reading comprehension model, our model structure is novel. The skimming module and multi-granularity module proposed in this paper are used to solve the problem that the previous model ignores the key information of the text and cannot understand the text with multi granularity. Experiments show that the model proposed in this paper is effective for both Chinese and English datasets. It can better understand the question and context and give a more accurate answer. The performance has made new progress on the basis of the baseline model.

Human motion similarity is practiced in many fields, including action recognition, anomaly detection, and human performance evaluation. While many computer vision tasks have benefited from deep learning, measuring motion similarity has attracted less attention, particularly due to the lack of large datasets. To address this problem, we introduce two datasets: a synthetic motion dataset for model training and a dataset containing human annotations of real-world video clip pairs for motion similarity evaluation. Furthermore, in order to compute the motion similarity from these datasets, we propose a deep learning model that produces motion embeddings suitable for measuring the similarity between different motions of each human body part. The network is trained with the proposed motion variation loss to robustly distinguish even subtly different motions. The proposed approach outperforms the other baselines considered in terms of correlations between motion similarity predictions and human annotations while being suitable for real-time action analysis. Both datasets and codes are released to the public.

The topic model has been widely applied to various research domains such as information retrieval, data mining, and so on. It can discover topics of texts in an unsupervised way. In the early years, most researches mainly focused on long texts. With the emergence of the Internet, the number of short texts is growing rapidly. Most existing schemes to solve the sparsity problems of short texts, are mainly based on data aggregation or model improvements. Among them, the Biterm Topic Model is one of the most representative models. It proposed a new way to model topics based on document-level word pairs and has shown creativity and effectiveness. However, this strategy ignores those semantically similar and rarely co-occurrent word pairs. What’s more, most researches ignore the multi-sense phenomenon in natural languages. In this paper, we utilize multi-sense word vectors to extract similar word pairs from the whole corpus by considering multiple senses. Based on this idea, we introduce a novel short-text topic model, which disambiguates multiple senses of words and generates more reasonable global biterms. Experimental results on two open-source English datasets have shown superiority to state-of-the-art topic models.

This paper presents a first study on solution representation learning for inducing greater alignment and hence positive transfers between distinct multi-objective optimization tasks that bear discrepancies in their original search spaces. We first establish a novel probabilistic model-based multi-objective transfer evolutionary optimization (TrEO) framework with solution representation learning, capable of activating positive transfers while simultaneously curbing the threat of negative transfers. In particular, well-aligned solution representations are learned via spatial transformations to handle mismatches in search space dimensionalities between distinct multi-objective problems, as well as to increase the overlap between their optimized search distributions. We then showcase different algorithmic instantiations and case studies of the proposed framework in applications spanning continuous as well as discrete optimization; illustrative examples include multi-objective engineering design and route planning of unmanned aerial vehicles. The experimental results show that our framework helps induce positive transfers by unveiling useful but hidden inter-task relationships, thus bringing about faster search convergence to solutions of high quality in multi-objective TrEO.

With the recent increase in requirement of both natural-language and visual information, the demand for research on seamless multi-modal processing for effective retrieval of these types of information has increased. However, because of the unstructured nature of images, it is difficult to retrieve images that accurately represent the input text. In this study, we utilized an augmented version of a multi-generator generative adversarial network that uses BERT embeddings and attention maps as input to enable grounded vocabulary for visual representations. We compared the performance of our proposed model with those of other state-of-the-art text input-based image retrieval methods on the MSCOCO and Flikr30K datasets, and the results showed the potential of our proposed method. Even with limited vocabulary, our proposed model was comparable to other state-of-the-art performances on R@10 or even exceed them in R@1. Moreover, we revealed the unique properties of our method by demonstrating how it could perform successfully even when using more descriptive text or short sentences as input.

Automation of Radio Access Network (RAN) operation is a fundamental feature to manage sustainable and efficient Beyond Fifth-generation wireless (5G) networks, in the context of the Next Generation Self-Organizing Network (NG-SON) vision. Machine Learning (ML) is already identified as the key ingredient of this vision, with new standardized and open architectures, like Open-RAN (O-RAN), taking momentum. In this paper, we propose models based on single-task and Multi-Task Learning (MTL) paradigms to address two RAN use cases, handover management and initial Modulation and Coding Scheme (MCS) selection. Traditional handover schemes have the drawback of taking into account the quality of the signals from the serving, and the target cell, before the handover. Also, initial MCS at the start of the session and after a handover usually is handled conservatively. The proposed ML solutions allow to address these drawbacks by 1) considering the expected Quality of Experience (QoE) resulting from the decision of a target cell to handover, as the driving principle of the handover decision and 2) using the experience extracted from network data to make smarter initial MCS allocations. In this line, we implement a realistic cellular simulation scenario by incorporating coverage holes to build an extensive database to train and test the proposed models. The results show that the ML-based models outperform the 3rd Generation Partnership Project (3GPP) standardized handover and initial MCS selection approaches by improving the QoE of users resulting from a handover and the throughput obtained upon establishing a new connection with a network. Besides that, using the obtained results, this paper extensively discusses the merits of leveraging the MTL model to address different, but related multiple RAN functions because it allows reusing a common learning architecture for multiple RAN use cases, which provides significant implementation advantages., This work was supported in part by under Grant PID2020-113832RB-C22 (ORIGIN)/AEI/10.13039/501100011033, and in part by Huawei Technologies, Sweden AB.

The last half-decade has seen a surge in deep learning research on irregular domains and efforts to extend convolutional neural networks (CNNs) to work on irregularly structured data. The graph has emerged as a particularly useful geometrical object in deep learning, able to represent a variety of irregular domains well. Graphs can represent various complex systems, from molecular structure, to computer and social and traffic networks. Consequent on the extension of CNNs to graphs, a great amount of research has been published that improves the inferential power and computational efficiency of graph-based convolutional neural networks (GCNNs). The research is incipient, however, and our understanding is relatively rudimentary. The majority of GCNNs are designed to operate with certain properties. In this survey we review of the state of graph representation learning from the perspective of deep learning. We consider challenges in graph deep learning that have been neglected in the majority of work, largely because of the numerous theoretical difficulties they present. We identify four major challenges in graph deep learning: dynamic and evolving graphs, learning with edge signals and information, graph estimation, and the generalization of graph models. For each problem we discuss the theoretical and practical issues, survey the relevant research, while highlighting the limitations of the state of the art. Advances on these challenges would permit GCNNs to be applied to wider range of domains, in situations where graph models have previously been limited owing to the obstructions to applying a model owing to the domains’ natures.

Unmanned aerial vehicles (UAVs) operate with large degrees of autonomy when executing tasks from a ground-based controller. In this work, we design an (immersive) interface for drone management. Our interface supports operators in mission execution of one or several (automated) UAVs within a fleet. This allows operators to assign tasks to UAVs, while safe operations are ensured through in-flight conflict detection and prevention. Our interface fulfills both functional (e.g., reliability) and non-functional requirements (e.g., perceived cognitive load during use). A key design challenge is the latency in information exchanges, which introduces information asymmetries between the humans and UAVs. We then validate the effectiveness of our interface design across several experiments. We find that our design reduces the perceived cognitive load and improves task performance. Altogether, this work has implications for designing interfaces in human-machine collaboration, so humans can effectively control, interact, or collaborate with automated machines, such as UAVs., IEEE Access, 9, ISSN:2169-3536

With the tremendous growth in the number of electronic documents, it is becoming challenging to manage the volume of information. Much research has focused on automatically summarizing the information available in the documents. Multi-Document Summarization (MDS) is one approach that aims to extract the information from the available documents in such a concise way that none of the important points are missed from the summary while avoiding the redundancy of information at the same time. This study presents an extensive survey of extractive MDS over the last decade to show the progress of research in this field. We present different techniques of extractive MDS and compare their strengths and weaknesses. Research work is presented by category and evaluated to help the reader understand the work in this field and to guide them in defining their own research directions. Benchmark datasets and standard evaluation techniques are also presented. This study concludes that most of the extractive MDS techniques are successful in developing salient and information-rich summaries of the documents provided.

Pommerman is a popular reinforcement learning environment because it imposes several challenges such as sparse and deceptive rewards and delayed action effects. In this paper, we propose an efficient reinforcement learning approach that uses a more efficient Monte Carlo tree search combined with action pruning and flexible imitation learning to accelerate the search performance, allowing the agent to avoid meaningless explorations and find some high-level strategies. Under the Pommerman benchmark, we evaluate the agent driven by the proposed approach against the heuristic and pure reinforcement learning baselines, and the results show that our method can yield a relatively high-level agent performance during combat, which demonstrates the efficiency of our method in this specific domain and its potential ability.

Mobile crowdsensing is a promising paradigm to leverage the power of people to collect large-scale spatially distributed data. This concept has been intensely studied to efficiently and securely complete sensing tasks at lower cost. The development of a unified platform designed to provide various types of sensing applications is among the major approaches to economical crowdsourcing. However, existing previous frameworks were not optimized for shared use among multiple organizers because they were largely vertically integrated systems. Security and user trust and confidence is also a significant issue a crowdsensing frameworks, given the potential security concerns. Therefore, in this study, we propose a network-side task allocation (NeSTA) framework to address the existing problems in mobile crowdsensing. The proposed framework enables the horizontal integration of sensing applications, in which mobile networks mediate communication among organizers and participants, significantly reducing the installation cost of individual applications. Privacy preservation is achieved by task distribution and allocation procedures, where the participants were obscured by organizers. The validity of the proposed NeSTA was confirmed by simulations with an analytical model using an open dataset. The results show that the proposed method exhibited computational efficiency over two orders of magnitude greater than the conventional approach. This advantage originates from the reduction of problem size by dividing the original problem into subproblems.

In this work we study the behavior of groups of autonomous vehicles, which are the part of the Internet of Vehicles systems. One of the challenging modes of operation of such systems is the case when the observability of each vehicle is limited and the global/local communication is unstable, e.g. in the crowded parking lots. In such scenarios the vehicles have to rely on the local observations and exhibit cooperative behavior to ensure safe and efficient trips. This type of problems can be abstracted to the so-called multi-agent pathfinding when a group of agents, confined to a graph, have to find collision-free paths to their goals (ideally, minimizing an objective function e.g. travel time). Widely used algorithms for solving this problem rely on the assumption that a central controller exists for which the full state of the environment (i.e. the agents current positions, their targets, configuration of the static obstacles etc.) is known and they cannot be straightforwardly be adapted to the partially-observable setups. To this end, we suggest a novel approach which is based on the decomposition of the problem into the two sub-tasks: reaching the goal and avoiding the collisions. To accomplish each of this task we utilize reinforcement learning methods such as Deep Monte Carlo Tree Search, Q-mixing networks, and policy gradients methods to design the policies that map the agents’ observations to actions. Next, we introduce the policy-mixing mechanism to end up with a single hybrid policy that allows each agent to exhibit both types of behavior – the individual one (reaching the goal) and the cooperative one (avoiding the collisions with other agents). We conduct an extensive empirical evaluation that shows that the suggested hybrid-policy outperforms standalone stat-of-the-art reinforcement learning methods for this kind of problems by a notable margin.

Visual scene understanding is the core task in making any crucial decision in any computer vision system. Although popular computer vision datasets like Cityscapes, MS-COCO, PASCAL provide good benchmarks for several tasks (e.g. image classification, segmentation, object detection), these datasets are hardly suitable for post disaster damage assessments. On the other hand, existing natural disaster datasets include mainly satellite imagery which have low spatial resolution and a high revisit period. Therefore, they do not have a scope to provide quick and efficient damage assessment tasks. Unmanned Aerial Vehicle(UAV) can effortlessly access difficult places during any disaster and collect high resolution imagery that is required for aforementioned tasks of computer vision. To address these issues we present a high resolution UAV imagery, FloodNet, captured after the hurricane Harvey. This dataset demonstrates the post flooded damages of the affected areas. The images are labeled pixel-wise for semantic segmentation task and questions are produced for the task of visual question answering. FloodNet poses several challenges including detection of flooded roads and buildings and distinguishing between natural water and flooded water. With the advancement of deep learning algorithms, we can analyze the impact of any disaster which can make a precise understanding of the affected areas. In this paper, we compare and contrast the performances of baseline methods for image classification, semantic segmentation, and visual question answering on our dataset., 11 pages

Assessing comprehension difficulties requires the ability to assess cognitive load. Changes in cognitive load induced by comprehension difficulties could be detected with an adequate time resolution using different biofeedback measures (e.g., changes in the pupil diameter). However, identifying the Spatio-temporal sources of content comprehension difficulties (i.e., when, and where exactly the difficulty occurs in content regions) with a fine granularity is a big challenge that has not been explicitly addressed in the state-of-the-art. This paper proposes and evaluates an innovative approach named Intelligent BiofeedbackAugmented Content Comprehension (TellBack) to explicitly address this challenge. The goal is to autonomously identify regions of digital content that cause user's comprehension difficulty, opening the possibility to provide real-time comprehension support to users. TellBack is based on assessing the cognitive load associated with content comprehension through non-intrusive cheap biofeedback devices that acquire measures such as pupil response or Heart Rate Variability (HRV). To identify when exactly the difficulty in comprehension occurs, physiological manifestations of the Autonomic Nervous System (ANS) such as the pupil diameter variability and the modulation of HRV are exploited, whereas the fine spatial resolution (i.e., the region of content where the user is looking at) is provided by eye-tracking. The evaluation results of this approach show an accuracy of 83.00% ± 0.75 in classifying regions of content as difficult or not difficult using Support Vector Machine (SVM), and precision, recall, and micro F1-score of 0.89, 0.79, and 0.83, respectively. Results obtained with 4 other classifiers, namely Random Forest, k-nearest neighbor, Decision Tree, and Gaussian Naive Bayes, showed a slightly lower precision. TellBack outperforms the state-of-the-art in precision & recall by 23% and 17% respectively.

Visual relationship detection aims to reason over relationships among salient objects in images, which has drawn increasing attention over the past few years. Inspired by human reasoning mechanisms, it is believed that external visual commonsense knowledge is beneficial for reasoning visual relationships of objects in images, which is however rarely considered in existing methods. In this paper, we propose a novel approach named Relational Visual-Linguistic Bidirectional Encoder Representations from Transformers (RVL-BERT), which performs relational reasoning with both visual and language commonsense knowledge learned via self-supervised pre-training with multimodal representations. RVL-BERT also uses an effective spatial module and a novel mask attention module to explicitly capture spatial information among the objects. Moreover, our model decouples object detection from visual relationship recognition by taking in object names directly, enabling it to be used on top of any object detection system. We show through quantitative and qualitative experiments that, with the transferred knowledge and novel modules, RVL-BERT achieves competitive results on two challenging visual relationship detection datasets. The source code is available at https://github.com/coldmanck/RVL-BERT., Comment: Published in IEEE Access

Multi-task evolutionary optimization is a new method proposed in recent years to solve optimization problems. Compared with traditional single-task optimization, the multi-task optimization mechanism based on evolutionary algorithms promotes the evolution of population by sharing the potential similarity and complementarity between different problems, so as to improve the performance and efficiency of solving problems. However, with the evolution of the population, the ability of one task to learn from other tasks may decrease, and the efficiency of knowledge transfer will also decrease. To address this issue and improve the quality of knowledge transfer among the tasks, this article proposes a new genetic transmission strategy and mutation strategy of multi task optimization algorithm, namely DOMLMFEA. In particular, in the genetic process, if two parent individuals are specific to different tasks, one parent individual is mapped to the vicinity of the other parent individual through task space mapping, and high-quality offspring individuals are generated through crossover. In addition, a dynamic opposite mutual learning mutation strategy based on wavelet basis function is introduced to generate promising solutions. It can effectively explore and develop in the unified search space and the subspace of each task, and improve the local search ability of the algorithm and the diversity of solutions. In order to evaluate the performance of DOMLMFEA, this paper tests on classical multi-task optimization problems, and compared with several other state-of-the-art MFEA variants. Numerical and simulation results show the effectiveness of the proposed DOMLMFEA.

Humans can categorize an object in three ways. For example, a car can be categorized as vehicles (superordinate), ground vehicles (basic) or car (subordinate). Different semantic levels of categorization is referred to these three categorization modes. There are different speed and accuracy for a similar object in the classification of these levels. However, much research has been done in this context, the trend of these levels is still questionable as to the accuracy and reaction time and the reason for this difference. In this paper, we examine the order of these levels and the reason for their differences. Here we show the superordinate advantage is declared and after this level the base level and the subordinate level are expressed respectively. To this end, first, we design an experiment to examine the semantic levels in the human eye system. The result of this experiment is the superordinate advantage. Actually, at the superordinate level at the same time, the reaction time was lower than other levels, and the efficiency was higher than other levels. In addition, a computational model is introduced that has time and can classify semantic levels. The model is trained for ten categories in this study. These ten categories are considered as subordinate level and five levels of basic and two levels of superordinate are expressed. We found that, at higher levels, such as superordinate, more neurons participate in the clustering task, so outcome is faster and more accurate results. Moreover, the proposed model has been tested for inverted input images to verify the model.

Cloud computing is a widely adopted platform for executing tasks of different application types that belong to the end users. In the cloud, application task is prone to failure for several reasons, such as software bug or exception, virtual or physical infrastructure failure. Cloud service providers are responsible for managing availability of scheduled computing tasks in order to provide high level QoS for their customers. Protecting task against failure is a challenging and not a trivial mission due to dynamic, heterogeneous and large distributed structure of the cloud environment. The existing works in the literature focus on task failure prediction and neglect the remedy (post) actions. In this work, we first study and analyze three publicly available large cluster datasets from Google, Alibaba, and Trinity, to characterize task failure in cloud computing platform. We then propose a failure-aware task scheduling framework that can predict the termination status for a set of given tasks during the runtime, and take the appropriate remedy actions. The framework uses deep learning methods named Artificial and Convolutional Neural Network, ANN and CNN, for different prediction purposes. In addition, we formalize the actions selection problem as Integer Linear Programming (ILP) model and propose a heuristic optimization solution that aims to minimize the failure probability of tasks and their resources usage. The results show ANN and CNN can achieve prediction accuracy of up to 94% and 92%, respectively using Google dataset. Moreover, the framework can protect up to 40% of tasks that are predicted as failed using Alibaba dataset by taking the appropriate remedy actions, and hence save many of cluster’s resources such as CPU and RAM.

Neural headline generation models have recently shown great results since neural network methods have been applied to text summarization. In this paper, we focus on news headline generation. We propose a news headline generation model based on a generative pre-training model. In our model, we propose a rich features input module. The headline generation model we propose only contains a decoder incorporating the pointer mechanism and the n-gram language features, while other generation models use the encoder-decoder architecture. Experiments on news datasets show that our model achieves comparable results in the field of news headline generation.

Brain–computer interface (BCI) technology has the potential to positively contribute to the educational learning environment, which faces many challenges and shortcomings. Cognitive and affective BCIs can offer a deep understanding of brain mechanisms, which may improve learning strategies and increase brain-based skills. They can offer a better empirical foundation for teaching–learning methodologies, including adjusting learning content based on brain workload, measuring student interest of a topic, or even helping students focus on specific tasks. The latest findings from emerging BCI technology, neuroscience, cognitive sciences, and psychology could be used in learning and teaching strategies to improve student abilities in education. This study investigates and analyzes the research on BCI patterns and its implementation for enhancing cognitive capabilities of students. The results showed that there is insufficient literature on BCI that addresses students with disabilities in the learning process. Further, our analysis revealed a bias toward the significance of cognitive process factors compared with other influential factors, such as the learning environment and emotions that influence learning. Finally, we concluded that BCI technology could improve students’ learning and cognitive skills—when consistently associated with the different pedagogical teaching–learning strategies—for better academic achievement.

Virtual reality (VR) has recently emerged as a platform that can be employed in the context of e-health applications. Even though the majority of VR-based applications focus on visual stimuli as the main content, audio also plays a very important role. If someone has an issue with auditory processing, the comprehension of auditory information is compromised. This condition reflects negatively on one’s quality of life, given the impact of audio perception on one’s ability to communicate effectively or to differentiate and ignore “noise”. This work aims to design a VR application that can be used to: (a) optimize multimodal VR experiences based on user trials (b) extract user data continuously throughout the experiment (c) evaluate a user’s auditory processing ability. To accomplish this goal, participants are required to localise a sound source in space in the presence of multiple listening conditions with simple and complex sound stimuli configurations. Data collected from users consists of physiological and objective metrics. The results of this study highlight the relationship between user behaviour (head movement, fixation points) and performance in the sound localisation task. This information can be used to design future applications with the purpose of training one’s auditory localisation ability. In addition, the evaluation compared the impact of using two interaction methods to perform this task: using a pointer or eye gaze to indicate the location of the target source. The findings from this study show statistically significant differences in terms of physiological response when subjects are exposed to different interaction methods, with greater immersion and performance for the pointer group.

Learning action policy for autonomous agents in a decentralized multi-agent environment has remained an interesting but difficult research problem. We propose to model this problem in a contextual bandit setting with delayed reward signals, in particular an individual short-term reward signal and a shared long-term reward signal. Our algorithm utilizes an approach with reward oracles to directly model these delayed reward signals and also relies on a learning scheme benefiting from the sampling guidance of an expert-designed policy. This algorithm is expected to apply to a wide range of problems, including those with constraints on accessing state transitions and those with implicit reward information. A demonstration, deployed by deep learning regressors, shows the effectiveness of the proposed algorithm in learning offensive action policy in the RoboCup Soccer 2D Simulation (RCSS) environment against a well-known adversary benchmark team compared to a baseline policy.

Affection is regarded as a new solution to coordinate task allocation problems in the multi-robot system. In this paper, we proposed an interpretable and computational affection model based on willingness and a new personality model named CASE to measure the heterogeneity of the robot. Emotion contagion model is used to calculate affective interaction among robots, and we use willingness to quantify the impact of affective factors on task allocation and as an important factor for task allocation. Task allocation algorithm is designed for affective robots, and simulate the application of the affection model at task allocation by a typical example. It displays the characteristics and novelty of affection in task allocation and to verify the correctness, effectiveness, and novelty of affective robot task allocation algorithm. The result of the simulation experiment shows that affective robots in multi-robot cooperation systems obtain a better solution than rational robots and express the state of the art of task allocation for effective robots.

In this paper, we address the problem of solving rearranging tasks using a robot. Rearranging tasks are challenging because they include many problems to solve at the same time, such as determining how to pick the items as well as planning how and where to place them. Solving a rearranging task usually consists of finding a set of pick-and-place instructions with a symbolic planner to perform the task. However, if the symbolic planner does not consider the robot’s capability to execute the instructions, it will likely generate many infeasible instructions, which wastes time in multiple trials and failures. Therefore, we propose to combine symbolic and motion planning to confirm a sequence of instructions before its execution by the robot. To achieve this combination, we use a Motion Feasibility Checker (MFC), which selects a set of feasible poses for the robot from a feasibility database. The MFC verifies that the instructions of the symbolic planning are valid and searches for a pick-and-place pair of poses to execute the instructions. We use a Monte Carlo Tree Search (MCTS) as the symbolic planner, and we combine it with the MFC when creating or expanding the states in the tree. After the MCTS finds a set of instructions for the rearranging task, we execute those instructions with the robot. As these instructions were previously validated, the robot is able to execute them. The proposed method was tested in a simulation environment that reproduces the scenario of rearranging products on a shelf of a convenience store. The experiment results show that the proposed method outperforms the conventional method in various initial states of increasing levels of difficulty.

Rapid expansion and the novel phenomenon of deep learning have manifested a variety of proposals and concerns in the area of video description, particularly in the recent past. Automatic event localization and textual alternatives generation for the complex and diverse visual data supplied in a video can be articulated as video description, bridging the two leading realms of computer vision and natural language processing. Several sequence-to-sequence algorithms are being proposed by splitting the task into two segments, namely encoding, i.e., getting and learning the insights of the visual representations, and decoding, i.e., transforming the learned representations to a sequence of words, one at a time. Implemented deep learning approaches have gained a lot of recognition for the reason of their superior computing capabilities and tremendous performance. However, the accomplishment of these algorithms strongly depends on the nature, diversity, and amount of data they are trained, validated and tested on. Techniques applied on insufficient and inadequate train/test data cannot deliver promising conclusions, consequently making it complicated to evaluate the quality of generated results. This survey focuses explicitly on the benchmark datasets, and evaluation metrics developed and deployed for video description tasks and their capabilities and limitations. Finally, we concluded with the need for essential enhancements and encouraging research directions on the topic.

Critical reading plays an important role in science learning, and previous studies have endeavored to objectively and precisely capture readers’ cognitive processing in reading scientific texts. Since many factors affect readers’ initiation and comprehension of scientific texts, studying the interactions of these factors was technically challenging for earlier studies. Recently, the use of artificial intelligent techniques for analyzing physiological signals has gained significant research attention, but exploitation of the educational data for proactive instructional use is still limited. This study proposed and evaluated an automatic approach incorporating the K-means++ clustering method for eye movement analytics. In this study, 64 undergraduate and graduate students read a multi-page popular science text while their eye movements were recorded. The results of the cluster analysis identified three patterns of reading behavior that were consistent and comparable to those of previous studies using self-reported measures and post-analysis analytics. Findings of the study support the potential and validity of a bottom-up, data-driven approach that can directly examine and analyze reading behaviors without interruption, and the contribution of the study to research and practice is outlined.

The paths generated by sampling-based path planning are generally not smooth and often generate multiple unnecessary robot posture changes in the task space. To mitigate such issues with a planned path from sampling-based path planners, shortcut-based path shortening algorithms are commonly adopted in the field of robot manipulator path planning as a post-processing step. In this paper, we analyze shortcut-based algorithms and propose a new approach based on the idea of parallelism for faster path shortening so that it can be more applicable in environments where a path has to be generated as quickly as possible to avoid collisions with other moving objects around the manipulator. Through performance comparisons in simulations, it is shown that the proposed approach can obtain a well-shortened as well as much smooth path compared to the original path faster than conventional shortcut-based algorithms and an optimization-based approach developed for collision-free path generation.