Search

Steganography and steganalysis are two interrelated aspects of the field of information security. Steganography seeks to conceal communications, whereas steganalysis aims to discover or, if possible, recover the data they contain. These two areas have garnered significant interest, especially among law enforcement agencies. Cybercriminals and even terrorists often employ steganography to avoid detection while in possession of incriminating evidence, even when that evidence is encrypted, since cryptography is prohibited or restricted in many countries. Therefore, a deep understanding of cutting-edge techniques for uncovering concealed information is essential in exposing illegal activities. Over the last few years, a number of strong and reliable steganography and steganalysis techniques have been introduced in the literature. This review paper provides a comprehensive overview of deep learning-based steganalysis techniques used to detect hidden information within digital media. The paper covers all types of cover in steganalysis, including image, audio, and video, and discusses the most commonly used deep learning techniques. In addition, the paper explores the use of more advanced deep learning techniques, such as deep transfer learning (DTL) and deep reinforcement learning (DRL), to enhance the performance of steganalysis systems. The paper provides a systematic review of recent research in the field, including data sets and evaluation metrics used in recent studies. It also presents a detailed analysis of DTL-based steganalysis approaches and their performance on different data sets. The review concludes with a discussion on the current state of deep learning-based steganalysis, challenges, and future research directions. • Detect covert messages through neural nets, enhancing steganalysis accuracy. • Steganalysis can benefit from auto-encoders, DNN, CNN, RNN, LSTM, GNN, DBN, and DRN. • Reinforcement learning, transfer learning, and adversarial-based methods are effective to steganalysis. • Transform domain, attention layer, and noise-based features empower steganalysis schemes. • Unsupervised learning extracts nuanced patterns from multimedia for detection. [ABSTRACT FROM AUTHOR]

Most existing Deep Reinforcement Learning (DRL) algorithms solely apply to discrete action or continuous action spaces. However, the agent often has both continuous and discrete action space, named hybrid action space. This paper proposes an action-decoupled algorithm for hybrid action space. Specifically, the hybrid action is decoupled, and then the original agent in the hybrid action space is abstracted into two agents. Each agent contains only discrete or continuous action space. The discrete and continuous actions are independent of each other to be executed simultaneously. We use the Soft Actor-Critic (SAC) algorithm as the optimization method and name our proposed algorithm Action Decoupled SAC (AD-SAC). We handle multi-agent problems using a framework of Centralized Training Decentralized Execution (CTDE) and then reduce the concatenation of partial agent observations to avoid the interference of redundant observations. We design a hybrid action space environment for Unmanned Aerial Vehicles (UAVs) path planning and gimbal scanning using AirSim. The results show that our algorithm has better convergence and robustness than the discretization, relaxation, and the Parametrized Deep Q-Networks Learning (P-DQN) algorithms. Finally, we carried out a Hardware in the Loop (HITL) simulation experiment based on Pixhawk to verify the feasibility of our algorithm. [ABSTRACT FROM AUTHOR]

Studies, with the increasing concern for mental health, have shown that interventions along with social support can reduce stress and depression. However, counselling centers do not have enough resources to provide counselling and social support to all the participants in their interest. This paper helps social support organizations (e.g., university counselling centers) sequentially select the participants for interventions. Meanwhile, Deep Reinforcement Learning (DRL) has shown significant success in learning an efficient policy for sequential decision-making problems in both fully observable environments and partially observable environments with small action space. In this paper, we consider emotion propagation from other neighbours of the influencees, initial uncertainties of mental states and influence in the student network. We propose a new architecture called DRLPSO (Deep Reinforcement Learning with Particle Swarm Optimization) to enhance learning performance in a partially observable environment with a large state and action space. DRLPSO consists of two stages: the Discrete Particle Swarm Optimization (DPSO) and Deep Q-learning integrated with Long Short-Term Memory (DQ-LSTM). In the first stage, we apply DPSO by initializing n particles that converge to multiple optimal actions for each belief state. In the second stage, the action with the best Q-value from the DPSO action set is executed to obtain belief and observation (history of action). We evaluated the proposed method empirically with the simulated student networks with mental state propagation compared to the state-of-the-art algorithms. The experimental results demonstrate that DRLPSO outperforms the state-of-the-art DRL methods by an average of 32%. [ABSTRACT FROM AUTHOR]

For NP-hard combinatorial optimization problems, it is usually challenging to find high-quality solutions in polynomial time. Designing either an exact algorithm or an approximate algorithm for these problems often requires significantly specialized knowledge. Recently, deep learning methods have provided new directions to solve such problems. In this paper, an end-to-end deep reinforcement learning framework is proposed to solve this type of combinatorial optimization problems. This framework can be applied to different problems with only slight changes of input, masks, and decoder context vectors. The proposed framework aims to improve the models in literacy in terms of the neural network model and the training algorithm. The solution quality of TSP and the CVRP up to 100 nodes are significantly improved via our framework. Compared with the best results of the state-of-the-art methods, the average optimality gap is reduced from 4.53% to 3.67% for TSP with 100 nodes and from 7.34% to 6.68% for CVRP with 100 nodes when using the greedy decoding strategy. Besides, the proposed framework can be used to solve a multi-depot CVRP case without any structural modification. Furthermore, our framework uses about 1/3 ∼ 3/4 training samples compared with other existing learning methods while achieving better results. The results performed on randomly generated instances, and the benchmark instances from TSPLIB and CVRPLIB confirm that our framework has a linear running time on the problem size (number of nodes) during training and testing phases and has a good generalization performance from random instance training to real-world instance testing. [ABSTRACT FROM AUTHOR]

With the rapidly growing fifth generation (5G) wireless data traffic, the cellular network has gradually become an important mode for the Industrial Internet of Things (IIoT). To give full play to the advantages of cellular network, it is desirable to design the optimal allocation strategy under the condition of limited network resources. In this paper, we consider the problem of allocating downlink power to mobile IIoT devices based on multi-agent deep reinforcement learning (MADRL). To maximize the total system capacity, each base station (BS)-user device (UE) is modeled as a RL agent to learn optimal allocation policy. A centralized training and distributed execution learning framework is proposed, the influence of parameter transmission delay on allocation policy is considered. The designed state space and reward function can adapt to large-scale networks on account of their expandability. Simulation results show the effectiveness and superiority of the proposed method in terms of the system sum rate. [ABSTRACT FROM AUTHOR]

Deep Q-Networks algorithm (DQN) was the first reinforcement learning algorithm using deep neural network to successfully surpass human level performance in a number of Atari learning environments. However, divergent and unstable behaviour have been long standing issues in DQNs. The unstable behaviour is often characterised by overestimation in the Q -values, commonly referred to as the overestimation bias. To address the overestimation bias and the divergent behaviour, a number of heuristic extensions have been proposed. Notably, multi-step updates have been shown to drastically reduce unstable behaviour while improving agent's training performance. However, agents are often highly sensitive to the selection of the multi-step update horizon (n), and our empirical experiments show that a poorly chosen static value for n can in many cases lead to worse performance than single-step DQN. Inspired by the success of n -step DQN and the effects that multi-step updates have on overestimation bias, this paper proposes a new algorithm that we call 'Elastic Step DQN' (ES-DQN) to alleviate overestimation bias in DQNs. ES-DQN dynamically varies the step size horizon in multi-step updates based on the similarity between states visited. Our empirical evaluation shows that ES-DQN out-performs n -step with fixed n updates, Double DQN and Average DQN in several OpenAI Gym environments while at the same time alleviating the overestimation bias. [ABSTRACT FROM AUTHOR]

In recent years, deep reinforcement learning (DRL) has been widely applied in various fields, and its vulnerability to adversarial samples, which is inherited from deep learning, has also received considerable attention for training robust DRL agents. However, most of existing attack approaches aim to perturb the observations of DRL agents, and poisoning attacks against DRL through perturbing the reward from the environment have been studied less. In this paper, we propose a novel reward poisoning adversarial attack method that employs the exploration strategy of reinforcement learning to help effectively attack a target. The adversary first builds a deep neural network to attain the exploration value of each state–action pair, i.e., the E -value, which is utilized to determine the appropriate time to attack the target and the sign of adversarial perturbation to generate adversary samples to the reward. The experimental results demonstrate that the reward poisoning attack algorithm based on E-values proposed in this paper enables an adversary to improve the efficiency of attacks with fewer adversarial samples. • Exploration strategies are applied to attacks against deep reinforcement learning. • A reward poisoning attack algorithm based on E-values (RPAAE) method is proposed. • The experimental results prove the effectiveness of the RPAAE method. [ABSTRACT FROM AUTHOR]

In the work setting of deep learning, most of the neural networks employed for visual object detection in recent years are based on bounding box regression. The performance of active detectors through multi-step decision-making is limited by the rough model design. However, from the perspective of cognitive science, the recognition in the human visual system is a decision process from coarse to fine. Based on the theory of "see the forest first, then the trees", this paper proposes a dynamic coarse-to-fine gaze strategy for active object detection, named AHDet, which takes the key points as the realization carrier of the coarse-to-fine concept. The detection process is divided into two steps, AIM and HIT. In the step of AIM, the positioning and prior bounding boxes for objects are given by detecting the center points, referring to the first glance. In the step of HIT, bounding boxes are dynamically adjusted to obtain compact bounding boxes with the help of the corner points, referring to the careful observation. With the design of the two-step coarse-to-fine gaze process, AHDet outperforms traditional approaches. A series of experiments performed on MS-COCO and PASCAL VOC dataset demonstrate the advantages of AHDet. [ABSTRACT FROM AUTHOR]

The linear active disturbance rejection control (LADRC) has been applied in many control practices and achieved satisfactory results. However, the controller with fixed parameters cannot achieve the optimal control performance for the controlled plant. In order to optimize the control effort of LADRC with online parameter adjustment, a deep reinforcement learning algorithm, the deep deterministic policy gradient (DDPG), is applied to enhance the LADRC in this paper for the ship course control and to obtain the optimized parameters of LADRC in different conditions. Specifically, the proposed strategy adopts the deep neural network to adjust the control parameters according to the measured states. This makes the proposed method applicable to the nonlinear control systems with high dimensional continuous states and actions. In addition, the stability of the closed-loop system is analyzed based on the Lyapunov method. In simulations, comparisons with Q-learning based LADRC and conventional LADRC controllers are also presented. Simulation results are provided to demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Artificial intelligence and all its supporting tools, e.g. machine and deep learning in computational intelligence-based systems, are rebuilding our society (economy, education, life-style, etc.) and promising a new era for the social welfare state. In this paper we summarize recent advances in data science and artificial intelligence within the interplay between natural and artificial computation. A review of recent works published in the latter field and the state the art are summarized in a comprehensive and self-contained way to provide a baseline framework for the international community in artificial intelligence. Moreover, this paper aims to provide a complete analysis and some relevant discussions of the current trends and insights within several theoretical and application fields covered in the essay, from theoretical models in artificial intelligence and machine learning to the most prospective applications in robotics, neuroscience, brain computer interfaces, medicine and society, in general. [ABSTRACT FROM AUTHOR]

Manual annotation of video segmentation datasets requires an immense amount of human effort, thus, reduction of human annotation costs is an active topic of research. While many papers deal with the propagation of masks through frames of a video, only a few results attempt to optimize annotation task selection. In this paper we present a deep learning based solution to the latter problem and train it using Reinforcement Learning. Our approach utilizes a modified version of the Dueling Deep Q-Network sharing weight parameters across the temporal axis of the video. This technique enables the trained agent to select annotation tasks from the whole video. We evaluate our annotation task selection method by means of a hierarchical supervoxel segmentation based mask propagation algorithm. [ABSTRACT FROM AUTHOR]

The Cognitive Packet Network (CPN) bases its routing decisions and flow control on the Random Neural Network (RNN) Reinforcement Learning algorithm; this paper proposes the addition of a Deep Learning (DL) Cluster management structure to the CPN for Quality of Service metrics (Delay Loss and Bandwidth), Cyber Security keys (User, Packet and Node) and Management decisions (QoS, Cyber and CEO). The RNN already models how neurons transmit information using positive and negative impulsive signals whereas the proposed additional Deep Learning structure emulates the way the brain learns and takes decisions; this paper presents a brain model as the combination of both learning algorithms, RNN and DL. The proposed model has been simulated under different network sizes and scenarios and it has been validated against the CPN itself without DL clusters. The simulation results are promising; the presented CPN with DL clusters as a mechanism to transmit, learn and make packet routing decisions is a step closer to emulate the way the brain transmits information, learns the environment and takes decisions. [ABSTRACT FROM AUTHOR]

[Display omitted] • A context helps the agent to learn knowledge from previous tasks. • A simple multi-task objective meta-training procedure is proposed to run faster. • A quadratic penalty technique is used to preserve the historical experience. • A predictive model is applied to further accelerate the adaptation phase. In this paper, we present a context-based meta-reinforcement learning approach to tackle the challenging data-inefficiency problem of Hyperparameter Optimization (HPO). Specifically, we design an agent which sequentially selects hyperparameters to maximize the expected accuracy of the machine learning algorithm on the validation set. First, we design a context variable that learns the latent embedding of prior experience, and the agent can solve the new tasks efficiently conditioned on it. Second, we employ a multi-task objective method that aims to maximize the average reward across all the meta-training tasks to meta-train the agent. Third, in the adaptation phase, we introduce a quadratic penalty technique to achieve better performance of the agent. Finally, to further improve the efficiency in the adaptation phase, we use a predictive model to evaluate the accuracy of machine learning algorithm instead of training it. We evaluate our approach on 18 real-world datasets and the results demonstrate that our approach outperforms other state-of-the-art optimization methods in terms of test set accuracy and runtime performance. [ABSTRACT FROM AUTHOR]

The increasing complexity of power systems makes existing deep reinforcement learning-based emergency voltage control methods face challenges in learning speed and data utilization efficiency. Meanwhile, the accumulated data containing expert experience and domain knowledge has not been fully utilized to improve the performance of the deep reinforcement learning methods. To address the above issues, a novel hybrid emergency voltage control method that combines expert experience and machine intelligence is proposed in this paper. Specifically, the expert experience in the off-line demonstration is extracted through a behavioral cloning model and the deep reinforcement learning method is applied to discover and learn new knowledge autonomously. A special supervised expert loss is designed to utilize the pre-trained behavioral cloning model to assist the self-learning process. The demonstration is dynamically updated during the training process such that the behavioral cloning model and the deep reinforcement learning model can facilitate each other continuously. Experiments are conducted on the open-source RLGC platform to validate the performance and the experimental results show that the proposed method can effectively improve the learning speed and the applicability of the model to different test situations. [ABSTRACT FROM AUTHOR]

Neuroevolution is a machine learning method for evolving neural networks parameters and topology with a high degree of flexibility that makes them applicable to a wide range of architectures. Neuroevolution has been popular in reinforcement learning and has also shown to be promising for deep learning. The major feature of Bayesian optimisation is in reducing computational load by approximating the actual model with an acquisition function (surrogate model) that is computationally cheaper. A major limitation of neuroevolution is the high computational time required for convergence since learning (evolution) typically does not utilize gradient information. Bayesian optimisation, which is also known as surrogate-assisted optimisation, has been popular for expensive engineering optimisation problems and hyper-parameter tuning in machine learning. It has potential for training deep learning models via neuroevolution given large datasets and complex models. Recent advances in parallel and distributed computing have enabled efficient implementation of neuroevolution for complex and computationally expensive neural models. In this paper, we present a Bayesian optimisation framework for deep neuroevolution using a distributed architecture to provide computational efficiency in training. Our results demonstrate promising results for simple to deep neural network models such as convolutional neural networks which motivates further applications. [ABSTRACT FROM AUTHOR]

As one of the core technologies of the automatic control system for unmanned surface vehicles (USVs), autonomous collision avoidance algorithm is the key to ensure the safe navigation of USVs. In this paper, path planning and dynamic collision avoidance (PPDC) algorithm which obeys COLREGs is proposed for USVs. In order to avoid unnecessary collision avoidance actions, the risk assessment model is developed, which is used to determine the switching time of path planning and dynamic collision avoidance. In order to train the algorithm which complies with the COLREGs, the encounter situation is divided quantitatively, which is regarded as the input state of the system, so that the high-dimensional input is successfully avoided. The state space of the USV is defined by relative parameters to improve the generalization ability of the algorithm, meanwhile, a network structure based on DDPG is designed to achieve the continuous output of thrust and rudder angle. Combined with path planning, collision avoidance, compliance with COLREGs and smooth arrival task, four kinds of reward functions are designed. In order to solve the problem of low training efficiency of experience replay mechanism in DDPG, cumulative priority sampling mechanism is proposed. Through the simulation and verification in a variety of scenarios, it is proved that PPDC algorithm has the function of path planning and dynamic collision avoidance in compliance with COLREGs, which has good real-time performance and security. [ABSTRACT FROM AUTHOR]

Reinforcement Learning (RL) technologies are powerful to learn how to interact with environments and have been successfully applied to various important applications. Q-learning is one of the most popular methods in RL, which leverages the Bellman equation to update the Q-function. Considering that data collection in RL is both time and cost consuming and Q-learning converges slowly, different kinds of transfer RL algorithms are designed to improve the sample complexity of the new tasks 1 1 In order to avoid confusion, we use "old/new tasks" instead of "source/target tasks" in this paper.. However, most of the previous transfer RL algorithms are similar to the transfer learning methods in deep learning and are heuristic with no theoretical guarantee of the convergence rate. Therefore, it is important for us to clearly understand how and when will transfer learning help RL method and provide the theoretical guarantee for the improvement of the sample complexity. In this paper, we rethink the transfer Rl problems in the RL perspective and propose to transfer the Q-function learned in the old task to the target Q-function in the Q-learning of the new task. We call this new transfer Q-learning method target transfer Q-Learning (abbrev. TTQL). The transfer process is controlled by the error condition which can help to avoid the harm to the new tasks brought by the transferred target. We design the error condition in TTQL as whether the Bellman error of the transferred target Q-function is less than the current Q-function. We show that TTQL with the error condition will achieve a faster convergence rate than Q-learning. Our experiments are consistent with our theoretical results and verify the effectiveness of our proposed target transfer Q-learning method. [ABSTRACT FROM AUTHOR]

Combinatorial optimization problems have been widely used in various fields. And many types of combinatorial optimization problems can be generalized into the model of unconstrained binary quadratic programming (UBQP). Therefore, designing an effective and efficient algorithm for UBQP problems will also contribute to solving other combinatorial optimization problems. Pointer network is an end-to-end sequential decision structure and combines with deep learning technology. With the utilization of the structural characteristics of combinatorial optimization problems and the ability to extract the rule behind the data by deep learning, pointer network has been successfully applied to solve several classical combinatorial optimization problems. In this paper, a pointer network based algorithm is designed to solve UBQP problems. The network model is trained by supervised learning (SL) and deep reinforcement learning (DRL) respectively. Trained pointer network models are evaluated by self-generated benchmark dataset and ORLIB dataset respectively. Experimental results show that pointer network model trained by SL has strong learning ability to specific distributed dataset. Pointer network model trained by DRL can learn more general distribution data characteristics. In other words, it can quickly solve problems with great generalization ability. As a result, the framework proposed in this paper for UBQP has great potential to solve large scale combinatorial optimization problems. [ABSTRACT FROM AUTHOR]

The performance of object detection is closely related to the quality of input images. However, the current image acquisition is purely guided by human visual perception, and such camera imaging process ignores the subsequent application. In this context, detection performance is impacted by imaging configuration and dynamic camera motion. To address the above problems, an active object detection framework is proposed in this paper, which aims to build the bridge between imaging configuration and object detection task. Within the proposed framework, a dynamic camera configuration learning approach is presented based on deep reinforcement learning, where the camera is actively controlled to maximize the detection performance. Through iterated interactions between imaging, control and object detection, the deep gap between perception and cognition in the object detection system is eliminated, and the transformation from physical imaging to purposeful imaging is realized. The effectiveness and advantages of the proposed framework are demonstrated in three dynamic environments. [ABSTRACT FROM AUTHOR]

Detecting objects of varied sizes in high resolution images is difficult due to the challenges of high memory requirement and huge computation burden. Existing state-of-the-art detectors perform well on low resolution images. However, its performance is greatly limited on high resolution images. In this paper, we propose a selective region enlargement network, called SRENet, which significantly reduces processing time and memory requirement while remaining high detection accuracy. The proposed SRENet does not need to conduct detection on original high resolution images but only needs to conduct detection on down-sampled images and some zoom-in regions selected from high resolution images. SRENet first conducts coarse detection on a low resolution image, and then sequentially selects promising regions that are expected to be analyzed at a higher resolution. Specifically, SRENet is built upon Deep Q-learning Network (DQN) and it outputs an action-reward map. The value of the reward map indicates the possibility that the action can improve detection accuracy. The region selected by the action with the maximum reward value will be analyzed further at a higher resolution. Extensive experiments are conducted to demonstrate SRENet on two challenging datasets obtaining high resolution images. Experimental results show that SRENet achieves state-of-the-art detection performance with high efficiency. [ABSTRACT FROM AUTHOR]

Deep reinforcement learning (DRL) has provided an effective end to end approach for autonomous robot skill learning. However, the task goal in most DRL approaches is always single and fixed which restricts the flexibility of policy. In addition, the data inefficiency also makes it impractical to applied on real-world robots. To address these problems, this paper proposes a data-efficient goal-directed DRL method for robotic skill learning. First, an asymmetric deep deterministic policy gradients algorithm is constructed as the basic framework of the method by taking advantage of the low-dimensional physical state accessible in simulations. Then, a Siamese representation learning network is designed to embed the RGB observation and the human intention at the same feature space to realize human-robot intention transfer. And, an auxiliary similarity evaluation network is added to the DRL algorithm to accelerate representation learning. Finally, a domain randomization method is employed to transfer the learned policies from simulation to reality. In experiments, two typical robotic tasks are set up to evaluate the proposed method. The experimental results validate the effectiveness of the proposed method. The trained robot can switch among multiple goals automatically according to human intentions. And, the Siamese representation learning network and auxiliary similarity evaluation network can improve data efficiency effectively. [ABSTRACT FROM AUTHOR]

In recent years, task-oriented dialogue systems have received extensive attention from academia and industry. Training a dialogue agent through reinforcement learning is often costly because it requires many interactions with real users. Although the Deep Dyna-Q (DDQ) framework uses simulation experience to alleviate the cost of direct reinforcement learning, it still suffers from challenges such as delayed rewards and policy degradation. This paper proposes an Emotion-Sensitive Deep Dyna-Q (ES-DDQ) model which: (1) presents an emotional world model that considers emotion-related cues to improve the ability of the traditional DDQ framework to model and simulate users, and (2) designs two kinds of emotion-related immediate rewards to mitigate the delayed reward problem. Experimental results show that our proposed approach effectively simulates users' behaviors and is superior to the state-of-the-art benchmarks. [ABSTRACT FROM AUTHOR]

A Reinforcement Learning (RL) agent interacts with the environment to learn a policy with high accumulated rewards through attempts and failures. However, RL suffers from its own trial-and-error learning nature, which results in an unstable learning process. In this paper, we investigate a common phenomenon called rewards drop at the late-stage RL training session, where the rewards trajectory oscillates dramatically. In order to solve such a problem, we propose a novel rewards shaping technique named Variance Aware Rewards Smoothing (VAR). We show that the proposed method reduces the variance of rewards and mitigates the rewards drop problem without changing the formulation of the value function. Furthermore, the theoretical analysis of convergence of VAR is provided, which is derived from the γ -contraction operator and the fixed point attribute of the value function. Finally, the theoretical results are illustrated by extensive results on various benchmarks and advanced algorithms across different random seeds to demonstrate the effectiveness and the compatibility of VAR. [ABSTRACT FROM AUTHOR]

[Display omitted] • We summarize the state-of-the-art of convolutional neural architecture search. • We model the convolutional neural network design as a bi-level optimization problem. • We develop BLOP-CNN as a new solution approach to our bi-level model. • We evaluate the performance of our proposal with respect to relevant existing works. During the last decade, deep neural networks have shown a great performance in many machine learning tasks such as classification and clustering. One of the most successful networks is the CNN (Convolutional Neural Network), which has been applied in many application domains such as pattern recognition, medical diagnosis, and signal processing. Despite the very interesting performance of CNNs, their architecture design is still so far a major challenge for researchers and practitioners. Several works have been proposed in the literature with the aim to find optimized architectures such as ResNet and VGGNet. Unfortunately, most of these architectures are either manually defined by experts or automatically designed by greedy induction algorithms. Recent works suggest the use of Evolutionary Algorithms (EAs) thanks to their ability to escape locally-optimal architectures. Despite the fact that EAs have shown interesting performance, researchers in this direction have considered the design task as a single-level optimization problem; which represents the main research gap we tackle in this paper. The main contribution behind our work consists in the fact that CNN architecture design has a hierarchical nature and thus could be seen as a Bi-Level Optimization Problem (BLOP) where: (1) the upper level minimizes the network complexity defined by the number of blocks and the number of nodes per block; and (2) the lower level optimizes the convolution block 'graphs' topologies by maximizing the classification accuracy. Motivated by the originality of our observation with respect to the state of the art, we frame for the first time the CNN architecture design problem as a BLOP and then solve it using an adapted version of an existing efficient bi-level EA; through the definition of the solution encoding, the fitness function, and the variation operators at each level. The adapted EA is named BLOP-CNN and is assessed on the image classification task using the commonly employed CIFAR-10 and CIFAR-100 benchmark data sets. The analysis of our experimental results show the merits of our proposed method in providing the user with optimized architectures that outperform many recent and prominent architectures coming from the three different approaches, namely: manual design, reinforcement learning-based generation, and evolutionary optimization. Moreover, to show the applicability of our approach, we have conducted a case study on the detection of the COVID-19 using a set of benchmark chest X-ray and Computed Tomography (CT) images. [ABSTRACT FROM AUTHOR]

Gold immunochromatographic strip (GICS) is a widely used lateral flow immunoassay technique. A novel image segmentation method is developed in this paper for quantitative analysis of GICS based on the deep reinforcement learning (DRL), which can accurately distinguish the test line and the control line in the GICS images. The deep belief network (DBN) is employed in the deep Q network in our DRL algorithm. Meanwhile, the multi-factor learning curve is introduced in the DRL algorithm to dynamically adjust the capacity of the replay buffer and the sampling size, which leads to enhanced learning efficiency. It is worth mentioning that the states, actions, and rewards in the developed DRL algorithm are determined based on the characteristics of GICS images. Experiment results demonstrate the feasibility and reliability of the proposed DRL-based image segmentation method and show that the proposed new image segmentation method outperforms some existing image segmentation methods for quantitative analysis of GICS images. [ABSTRACT FROM AUTHOR]

Compared with 2D Magnetic Resonance Imaging (MRI), 3D MRI is more powerful for generating high resolution images and visualizing small anatomical structures. However, 3D MRI acquisition is much more time-consuming due to the significantly larger number of phase encoding steps, which is directly proportional to the acquisition time. This paper proposes to select a volume-adaptive small subset of phases to accelerate 3D MRI scans and accurately reconstruct 3D images from the corresponding undersampled 3D k-space data. To avoid the delays caused by computationally expensive yet high-performance volume-adaptive phase selection, we propose a strategy of selecting multiple phases based on sampled slices from the volume during idle time within the repetition time (TR). To enhance the performance of phase selection, we propose a novel three-directional cross-attention phase selection network. Additionally, to improve the reconstruction performance, we introduce a three-directional slice-wise volume reconstruction. To the best of our knowledge, the proposed method, which we called EVOLVE (l e arning vol ume-adapti ve phases), is the first work that learns volume-adaptive phases for fast 3D MRI. The extensive experimental results on a large-scale 3D MRI dataset at various acceleration factors demonstrate the substantial performance improvement in terms of image reconstruction achieved by using the EVOLVE method for phase selection compared to traditional learning free 3D MRI phase selection methods. [ABSTRACT FROM AUTHOR]

Reinforcement learning (RL) depends on carefully engineering environment rewards. However, rewards from environments are extremely sparse for many RL tasks, challenging for the agent to learn skills and interact with the environment. One solution to this problem is to create intrinsic rewards for agents and to make rewards dense and more suitable for learning. Recent algorithms, such as curiosity-driven exploration, usually estimate the novelty of the next state through the prediction error of dynamics models. However, these methods are typically limited by the capacity of their dynamics models. In this paper, a random curiosity-driven model using deep reinforcement learning is proposed, which uses a target network with fixed weights to maintain the stability of dynamics models and create more suitable intrinsic rewards. We integrate the parametric exploration method for further promoting sufficient exploration. Besides, a deeper and more closely connected network is utilized for encoding the pixel images for policy-gradient. By comparing our method against the previous approaches in several environments, the experiments show that our method achieves state-of-the-art performance on most but not all of the Atari games. [ABSTRACT FROM AUTHOR]

For solving the vehicle Re-identification (Re-ID) task, we need to focus our attention on the details with arbitrary size in the image, and it's tough to locate these details accurately. In this paper, we propose a Multi-Attention Deep Reinforcement Learning (MADRL) model to focus on multi-attentional subregions that spreading randomly in the image, and extract the discriminative features for the Re-ID task. First, we obtain multiple attentions from the representative features, then group the feature channels into different parts, then train a deep reinforcement learning model to learn more accurate positions of these fine-grained details with different losses. Unlike existing models with complex strategies to keep the patch-matching constrains, our MADRL model can automatically locate the matching patches (multi-attentional subregions) in different vehicle images with the same identification (ID). Furthermore, based on the fine-grained attention and global features we re-calculate the distance between the inter- and intra- classes, and we get better re-ranking results. Compared with state-of-the-art methods on three large-scale vehicle Re-ID datasets, our algorithm greatly improves the performance of vehicle Re-ID. [ABSTRACT FROM AUTHOR]

Recently, deep-learning based lane detection methods effectively boost the development of Advanced Driver Assistance Systems (ADAS) and Self-Driving Systems. However, these methods only detect lane lines with sketchy bounding boxes while ignore the shape of specific curved lanes. To address the above problems, this paper introduces deep reinforcement learning into cursory lane detection models for accurate lane detection and localization. This model consists of two stages, namely the bounding box detector and landmark point localizer. To be specific, a bounding box level convolution neural network lane detector outputs the preliminary location of lanes in the form of bounding boxes. Then, a reinforcement based Deep Q-Learning Localizer (DQLL) accurately localizes the lanes as a group of landmarks to achieve better representation of curved lanes. Moreover, a pixel-level lane detection dataset named NWPU Lanes Dataset is constructed and released. It contains a variety of real traffic scenes and accurate masks of the lane lines. This approach achieves competitive performance in the released dataset and TuSimple Lane dataset. Furthermore, the codes and dataset will be released on https://github.com/tuzixini/DQLL. [ABSTRACT FROM AUTHOR]

We explored the problem about function approximation error and complex mission adaptability in multi-agent deep reinforcement learning. This paper proposes a new multi-agent deep reinforcement learning algorithm framework named multi-agent time delayed deep deterministic policy gradient. Our work reduces the overestimation error of neural network approximation and variance of estimation result using dual-centered critic, group target network smoothing and delayed policy updating. According to experiment results, it improves the ability to adapt complex missions eventually. Then, we discuss that there is an inevitable overestimation issue about existing multi-agent algorithms about approximating real action-value equations with neural network. We also explain the approximate error of equations in the multi-agent deep deterministic policy gradient algorithm mathematically and experimentally. Finally, the application of our algorithm in the mixed cooperative competition experimental environment further demonstrates the effectiveness and generalization of our algorithm, especially improving the group's ability of adapting complex missions and completing more difficult missions. [ABSTRACT FROM AUTHOR]

In recent years, deep reinforcement learning has achieved impressing accuracies in games compared with traditional methods. Prior schemes utilized Convolutional Neural Networks (CNNs) or Long Short-Term Memory networks (LSTMs) to improve the performances of the agents. In this paper, we consider the issue from a different perspective when the training and inference of deep reinforcement learning are required to be performed with limited computing resources. Mainly, we propose two efficient neural network architectures of deep reinforcement learning: Light-Q-Network (LQN) and Binary-Q-Network (BQN). In LQN, The depth-wise separable CNNs are utilized in memory and computation saving. While, in BQN, the weights of convolutional layers are binary that help in shortening the training time and reduce memory consumption. We evaluate our approach on Atari 2600 domain and StarCraft II mini-games. The results demonstratethe efficiency of the proposed architectures. Though performances of agents in most games are still super-human, the proposed methods advance the agent from sub to super-human performance in particular games. Also, we empirically find that non-standard convolution and non-full-precision networks do not affect agent learning game strategy. [ABSTRACT FROM AUTHOR]

• A framework based on Bidirectional Gated Neural Networks is proposed for fault diagnosis with uncertainty in dynamic environments. • A Sample Sensitive Bidirectional Gated Neural Networks model is developed to tackle imbalanced fault diagnosis challenges. • Cost sensitive active learning is used to explore unlabeled data. • Effective methods are developed to address both binary Fault Diagnosis and multi-class Fault Diagnosis. Most existing fault diagnosis methods may fail in the following three scenarios: (1) serial correlations exist in the process data; (2) fault data are much less than normal data; and (3) it is impractical to obtain enough labeled data. In this paper, a novel form of the bidirectional gated recurrent unit (BGRU) is developed to underpin effective and efficient fault diagnosis using cost sensitive active learning. Specifically, BGRU is devised to consider the dynamic behavior of a complex process. In the training phase of BGRU, the idea of weighting each training example is proposed to reduce the effect of class imbalance. Besides, in order to explore the unlabeled data, cost sensitive active learning is utilized to select the candidate instances. The effectiveness of the proposed method is evaluated on the Tennessee Eastman (TE) dataset and a real plasma etching process dataset. The experiment results show that the proposed cost senstive active learning bidirectional gated recurrent unit (CSALBGRU) method achieves better performance in both binary fault diagnosis and multi-class fault diagnosis. [ABSTRACT FROM AUTHOR]

• Proposed a novel approach for multi-robot coordination. • Our approach can solve the resource competition problem. • Our approach can solve the obstacle avoidance problems in real time. • Results indicate that our method effectively applied to multi-robot coordination. This paper proposes a multi-robot cooperative algorithm based on deep reinforcement learning (MRCDRL). We use end-to-end methods to train directly from each robot-centered, relative perspective-generated image, and each robot's reward as the input. During training, it is not necessary to specify the target position and movement path of each robot. MRCDRL learns the actions of each robot by training the neural network. MRCDRL uses the neural network structure that was modified from the Duel neural network structure. In the Duel network structure, there are two streams that each represents the state value function and the state-dependent action advantage function, and the results of the two streams are merged. The proposed method can solve the resource competition problem on the one hand and can solve the static and dynamic obstacle avoidance problems between multi-robot in real time on the other hand. Our new MRCDRL algorithm has higher accuracy and robustness than DQN and DDQN and can be effectively applied to multi-robot collaboration. [ABSTRACT FROM AUTHOR]

Ensemble models achieve high accuracy by combining a number of base estimators and can increase the reliability of machine learning compared to a single estimator. Additionally, an ensemble model enables a machine learning method to deal with imbalanced data, which is considered to be one of the most challenging problems in machine learning. In this paper, the capability of Adaptive Boosting (AdaBoost) is integrated with a Convolutional Neural Network (CNN) to design a new machine learning method, AdaBoost-CNN, which can deal with large imbalanced datasets with high accuracy. AdaBoost is an ensemble method where a sequence of classifiers is trained. In AdaBoost, each training sample is assigned a weight, and a higher weight is set for a training sample that has not been trained by the previous classifier. The proposed AdaBoost-CNN is designed to reduce the computational cost of the classical AdaBoost when dealing with large sets of training data, through reducing the required number of learning epochs for its ingredient estimator. AdaBoost-CNN applies transfer learning to sequentially transfer the trained knowledge of a CNN estimator to the next CNN estimator, while updating the weights of the samples in the training set to improve accuracy and to reduce training time. Experimental results revealed that the proposed AdaBoost-CNN achieved 16.98% higher accuracy compared to the classical AdaBoost method on a synthetic imbalanced dataset. Additionally, AdaBoost-CNN reached an accuracy of 94.08% on 10,000 testing samples of the synthetic imbalanced dataset, which is higher than the accuracy of the baseline CNN method, i.e. 92.05%. AdaBoost-CNN is computationally efficient, as evidenced by the fact that the training simulation time of the proposed method is 47.33 s, which is lower than the training simulation time required for a similar AdaBoost method without transfer learning, i.e. 225.83 s on the imbalanced dataset. Moreover, when compared to the baseline CNN, AdaBoost-CNN achieved higher accuracy when applied to four other benchmark datasets including CIFAR-10 and Fashion-MNIST. AdaBoost-CNN was also applied to the EMNIST datasets, to determine its impact on large imbalanced classes, and the results demonstrate the superiority of the proposed method compared to CNN. [ABSTRACT FROM AUTHOR]

In recent years, aided by deep neural networks (DNNs), reinforcement learning (RL) algorithms have been achieving great success in more and more tasks. In general, model-free RL algorithms are widely applicable, but sometimes suffer from low sample efficiency. Although the environment dynamics can be incorporated into model-free RL algorithms to enhance sample efficiency, which can be transformed into the joint algorithm of model-free and model-based RL algorithms, the model bias of dynamics model may still hurt the performance. Another attractive study direction for RL algorithms, is using parallel strategy. Meanwhile, parallel RL algorithms can achieve outstanding results under less time cost, but are still with low sample efficiency, because most of such algorithms are vanilla model-free ones. Therefore, aiming to enhance the model performance for RL algorithms with DNNs, in this paper, we propose a novel parallel model-augmented framework, called PMA-DRL, to combine the dynamics model and parallel RL algorithms together. First, we introduce a local dynamics model (LDM) for each local agent model (LAM) in parallel RL algorithms. Next, we propose to build a better LDM by using an ensemble of LDMs, and such ensemble can help to reduce model bias, because the experienced observations under different LAMs are always diverse. Then, the observation diversity can be further increased by employing LDMs to explore. Finally, we implement experiments on classic control tasks (CCTs) and Atari games, which demonstrates the efficiency of our proposed framework. [ABSTRACT FROM AUTHOR]

Deceptive games are games that utilize the reward structure to keep the agent away from the global optimization and have been grown up to become a huge challenge in the field of deep reinforcement learning intelligent exploration. Most of the cutting-edge exploration approaches, such as count-based and curiosity-driven, even with intrinsic motivation, which achieves better performance in the sparse reward game, still easily fall into local optimal traps in the deceptive game. To address this shortfall, we introduce a further exploration approach called Maximum Entropy Explore (MEE). Based on entropy rewards and the off-policy actor-critic reinforcement learning algorithm, we divided the agent exploration policy into two independent parts, namely, the target policy and the explorer policy. The explorer policy, taking the maximum entropy of the target policy as the optimization goal, is used to interact with the environment and generated trajectories for the target policy. The target policy regards the maximization of external reward as the optimization goal to achieve the global solution. To alleviate the catastrophic forgetting problem which leads to the training of the agent not stabilized during the off-policy exploration phrase, the optimal experience replay is applied. An on-policy mode switch trick is used to validly prevent the unstable and diverge which caused by the deadly triad. We conduct experiments comparing our approach with state-of-the-art deep reinforcement learning algorithm and exploration methods in the grid world and StarCraft II environments with deceptive reward. The experiment indicates that the MME approach sets out to be in the present paper effectively avoids the deceptive reward trap and learns the global optimal strategy. [ABSTRACT FROM AUTHOR]

As a hot topic in the financial engineering, the portfolio optimization aims to increase investors' wealth. In this paper, a portfolio management system based on deep-reinforcement learning is proposed. In contrast to inflexible traditional methods, the proposed system achieves a better trading strategy through Reinforcement learning. The reward signal of Reinforcement learning is updated by action weights from Deep learning networks. Low price, high price and close price constitute the inputs, but the importance of these three features is quite different. Traditional methods and the classical CNN can't deal with these three features separately, but in our method, a designed depth convolution is proposed to deal with these three features separately. In a virtual currency market, the price rise only occurs in a flash. Traditional methods and CNN networks can't accurately judge the critical time. In order to solve this problem, a three-dimensional attention gating network is proposed and it gives higher weights on rising moments and assets. Under different market conditions, the proposed system achieves more substantial returns and greatly improves the Sharpe ratios. The short-term risk index of the proposed system is lower than those of the traditional algorithms. Simulation results show that the traditional algorithms (including Best, CRP, PAMR, CWMR and CNN) are unable to perform as well as our approach. [ABSTRACT FROM AUTHOR]

Knowledge Graphs (KGs) have been applied to various application scenarios including Web searching, Q&A, recommendation system, natural language processing and so on. However, the vast majority of Knowledge Bases (KBs) are incomplete, necessitating a demand for KB completion (KBC). Methods of KBC used in the mainstream current knowledge base include the latent factor model, the random walk model and recent popular methods based on reinforcement learning, which performs well in their respective areas of expertise. Recurrent neural network (RNN) and its variants model temporal data by remembering information for long periods, however, whether they also have the ability to use the information they have already remembered to achieve complex reasoning in the knowledge graph. In this paper, we produce a novel framework (ALSTM) based on the Attention mechanism and Long Short-Term Memory (LSTM), which associates structure learning with parameter learning of first-order logical rules in an end-to-end differentiable neural networks model. In this framework, we designed a memory system and employed a multi-head dot product attention (MHDPA) to interact and update the memories embedded in the memory system for reasoning purposes. This is also consistent with the process of human cognition and reasoning, looking for enlightenment for the future in historical memory. In addition, we explored the use of inductive bias in deep learning to facilitate learning of entities, relations, and rules. Experiments establish the efficiency and effectiveness of our model and show that our method achieves better performance in tasks which include fact prediction and link prediction than baseline models on several benchmark datasets such as WN18RR, FB15K-237 and NELL-995. [ABSTRACT FROM AUTHOR]

In this paper we present an approach to training neural network to generate sequences using successor feature learning from reinforcement learning. The model can be thought as two components, an MLE-based token generator and an estimator that predicts the future value of whole sentence. As we know, reinforcement learning has been applied to dealing with the exposure bias problem of generating sequences. Compared with other RL algorithm, successor feature(SF) can learn robust value function provided observations and reward by decomposing the value function into two components - a reward predictor and a successor map. The encoder-decoder framework with SF enables the decoder to generate outputs that receive more future reward, which means that the model pays attention on not only the current word but also the rest words. We demonstrate that the approach improves performance on two translation tasks. [ABSTRACT FROM AUTHOR]

The problem of Multiple Object Tracking (MOT) consists in following the trajectory of different objects in a sequence, usually a video. In recent years, with the rise of Deep Learning, the algorithms that provide a solution to this problem have benefited from the representational power of deep models. This paper provides a comprehensive survey on works that employ Deep Learning models to solve the task of MOT on single-camera videos. Four main steps in MOT algorithms are identified, and an in-depth review of how Deep Learning was employed in each one of these stages is presented. A complete experimental comparison of the presented works on the three MOTChallenge datasets is also provided, identifying a number of similarities among the top-performing methods and presenting some possible future research directions. [ABSTRACT FROM AUTHOR]

Path planning plays a significant role in autonomous navigation for robots in complex environments and hence has been extensively studied for decades. However, the computational time of most existing methods are dependent on the scale and complexity of environment, which leads to the compromise between time efficiency and path quality. To tackle this challenge, deep neural network based (DNN-based) planning methods have been actively explored. However, despite the success of DNN-based 2D planner, 3D path planning, which is a significant primitive for quite a few autonomous robots, is rarely handled by DNNs. In this paper, we propose a novel end-to-end neural network architecture named Three-Dimensional Path Planning Network (TDPP-Net) to realize DNN-based 3D path planning. Embedding the action decomposition and composition concept, our network predicts 3D actions merely through 2D convolutional neural networks (CNNs). Besides, the computational time of TDPP-Net is almost independent of environmental scale and complexity for each action prediction. The experimental results demonstrate that our approach exhibits remarkable performance for planning real-time paths in unseen 3D environments. [ABSTRACT FROM AUTHOR]

Highlights • A reinforcement learning method for cinematography-oriented control is proposed. • A hint-based reward function is used to improve the quality of the obtained models. • Two simulation environments, both for drone and camera control, are provided. Abstract Drones, also known as Unmanned Aerial Vehicles, are capable of capturing spectacular aerial shots and can be used to aid several cinematography-oriented tasks. However, flying drones in a professional setting requires the cooperation of several people, increasing the production cost and possibly reducing the quality of the obtained shots. In this paper, a generic way for formulating cinematography-oriented control objectives, that can be used for training any RL agent to automate the drone and camera control processes, is proposed. To increase the convergence speed and learn more accurate deep RL agents, a hint-based reward function is also employed. Two simulation environments, one for drone control and one for camera control, were developed and used for training and evaluating the proposed methods. The proposed method can be combined both with methods capable of performing discrete control, as well as with continuous control methods. It was experimentally demonstrated that the proposed method improves the control accuracy over both handcrafted control techniques and deep RL models trained with other reward functions. [ABSTRACT FROM AUTHOR]