Search

With the rapid development and integration of biomedicine and information technology,massive amounts of imaging data,patient report data,electronic health records,and omics data have been accumulated rapidly in healthcare.These data are cha-racterized by complexity,heterogeneity and high dimensionality.Deep learning has the ability of complex function simulation and automatic feature learning,which can provide efficient technical support for research in medical diagnosis and drug development.Currently,deep learning has been extremely successful in medical imaging and further more,some medical imaging diagnostic systems based on deep learning have achieved performance that is even comparable to that of relevant experts.Due to the progress of natural language processing technology,deep learning has also made remarkable progress in the use of non-image data tasks.This paper first briefly describes the development of deep learning in healthcare.Subsequently,the application of deep learning model in healthcare is statistically analyzed,and some available datasets are sorted out.In addition,this paper also introduces the research progress of deep learning in medical diagnosis and treatment processes such as disease diagnosis and health monitoring,and its research progress in protein structure prediction and drug discovery.Finally,key challenges of deep learning in healthcare applications such as data quality,interpretability,privacy security and practical application limitations are discussed.It also discusses feasible solutions or approaches to these challenges.

In recent years,artificial intelligence represented by deep learning has made breakthroughs in theories and technologies.With the strong support of data,algorithms and computing power,deep learning has received unprecedented attention and has been widely used in various fields,bringing great improvements to the corresponding fields.With the wide application of deep learning technology in various fields including security critical ones,the security issue of deep learning has attracted more and more attention.Researchers have found many security risks in deep learning systems.In terms of the security of deep learning models,researchers have extensively explored the new attack paradigm of backdoor attack.Backdoor attack can threaten deep learning models throughout their whole life cycle.A large number of researchers have proposed series of attack scheme from different angles.This paper takes the security threats of deep learning system as a starting point,introduces the current attack paradigms.On this basis,it gives the back-ground and principle of backdoor attack,distinguishes the similar attack paradigms such as adversarial attack and data poisoning attack,then continues to elaborate on the attack principle and outstanding features of the classic methods of backdoor attack to date.According to the working principle,the attack schemes are divided into data poisoning based attack and model poisoning based attack and others,the paper systematically summarizes them and clarify the advantages and disadvantages of current research.Then,this paper surveys the state-of-the-art works of backdoor attack against various typical applications and popular deep learning paradigms,which further reveal the threat of backdoor attack towards deep learning models.Finally,this paper summarizes the research work on applying backdoor attack characteristics to positive applications and explores the current challenges of backdoor attack,as well as discusses future research directions worthy of in-depth exploration,aiming to provide guidance for the follow-up researchers to further promote the development of backdoor attack and security of deep learning.

The task of object detection is a basic and highly concerned work in the field of computer vision.Because object detection in remote sensing has important application value in transportation,military,agriculture,etc.,it has also become a major research hotspot.Compared with natural images,remote sensing images are affected by many factors such as complex background interference,weather,irregularities,and small objects.It is extremely challenging to achieve higher accuracy in remote sensing image object detection tasks.This paper proposes a novel object detection network based on Transformer,swin pyramid Transformer(SPT).SPT uses a sliding window Transformer module as the backbone of feature extraction.Among it,the self-attention mechanism of Transformer is very effective for detecting objects in a chaotic background,and the sliding window mode efficiently avoids a large number of square-level complexity calculations.After obtaining the feature map extracted by the backbone network,SPT uses a pyramid architecture to fuse different scale and semantic features,pithily reducing the loss of information between feature layers,and capturing the inherent multi-scale hierarchical relationship.In addition,this paper proposes self-mixed Transformer(SMT) module and cross-layer Transformer(CLT) module.SMT re-renders the highest-level feature map to enhance object feature recognition and expression.According to the feature context interaction,the feature expressions of the pixels of each feature layer are rearranged by CLT,and the CLT module is integrated into the bottom-up and top-down dual paths of the pyramid to make full use of global and local information containing different semantics.Our SPT network model is trained and tested on the UCAS-AOD and RSOD datasets.Experimental results show that SPT is high-performing in remote sensing image object detection tasks,especially suitable for irregular and small target categories,such as overpass and car.

As an important branch in the field of computer vision,object tracking has been widely used in many fields such as intelligent video surveillance,human-computer interaction and autonomous driving.Although object tracking has achieved good development in recent years,tracking in complex environment is still a challenge.Due to problems such as occlusion,object deformation and illumination change,tracking performance will be inaccurate and unstable.In this paper,an effective object tracking method AFTM,is proposed with attention features.Firstly,this paper constructs an adaptively generated attention weight factor group,which implements an efficient adaptive fusion strategy for response map to improve the accuracy of object positioning and bounding box scale calculation in the process of classification and regression.Secondly,aiming at the class imbalance in the data set,the proposed method uses the dynamically scaled cross entropy loss as the loss function of the object positioning network,which can modify the optimization direction of the model and make the tracking performance more stable and reliable.Finally,this paper designs a corresponding learning rate adjustment strategy to stochastically average the weight of a number of models,which can enhance the generalization ability of the model.Experimental results on public data sets show that the proposed method has higher accuracy and more stable tracking performance in complex tracking environment.

As the core task of text mining and information extraction,entity relation extraction intends to identify and determine the specific relation between entity pairs from natural language texts,provides basic support for intelligent retrieval and semantic analysis,and helps to improve search efficiency.It is a research hotspot in the field of natural language processing.Compared with relation extraction from single sentence,documents contain richer entity relation semantics.Therefore,recently many new extraction methods have shifted their research focus from sentence-level to document-level,and achieved rich research results.This paper systematically summarizes the mainstream methods and research progress of document-level entity relation extraction in recent years.Firstly,the paper summarizes the problems and challenges of document-level relation extraction,and then introduces a variety of document-level relation extraction methods from three aspects:sequence based,graph based and pre-trained language model based.Finally,the data sets and experiments used by each method are compared and analyzed,and the possible research directions in the future are discussed and prospected.

Due to the rapid updating of computer knowledge with many ambiguities,it is difficult for students to seek reasonable solutions for independent innovation.As an auxiliary innovation tool,intelligent question-answering system can help students to grasp the frontier of subject development,find out solutions for problems faster and precisely.In this paper,a knowledge graph of scientific research is constructed based on a large-scale database of scientific and technological documents,which realizes an intelligent question answering system for assisting students in innovation.In order toreduce the influence of noisy entities on query questions,this paper proposes an auxiliary task enhanced intent information for question answering in computer domain(ATEI-QA).Compared with the traditional method,this method can extract the question intention information more accurately and further reduce the influence of noisy entity on intention recognition.Additionally,we conduct a series of experimental studies on computer and common datasets,and compare with three mainstream methods.Finally,experimental results demonstrate that our model achieves significant improvements against with three baselines,improving MAP and MRR scores by average of 3.27%,1.72% in the computer dataset and 4.37%,2.81% in the common dataset respectively.

Typically, the strategy of central heating for buildings in winter is climate compensator.However, this strategy heavily relies on manual experience with a relatively simple regulation.Therefore, how to optimize the heating control strategy is very important to keep the indoor temperature stable and comfortable.For this task, this paper proposes a heating strategy optimization method based on deep learning and deep reinforcement learning, which can optimize the original control strategy based on real historical data.The paper first develops a deep MTDN (Multiple Time Difference Network) as the simulator to predict the next time slot's room temperature.By learning the thermodynamic law of indoor temperature change, the network has high accuracy and confirms the physical laws.After that, the SAC (Soft Actor-Critic) algorithm based on maximum entropy reinforcement learning is employed as the strategy optimizer to interact with the simulator.Here, we use the evaluation index of the human body's thermal response as the reward to train and optimize the heating control strategy.Based on the real data of a heat exchange station in Tianjin, we evaluate the predictive ability of the simulator and the control ability of the strategy optimizer, respectively.The results verify that, compared with other types of prediction simulators, this simulator not only has high prediction accuracy but also conforms to physical laws.At the same time, compared with the original strategy, the strategy learned by the strategy optimizer can ensure that the indoor temperature is more stable and comfortable in multiple time periods of random sampling.

Recently the application of location-based services has gradually become popular.It provides convenience in people's daily life,and also brings a great threat to personal privacy.The existing research shows that,with a large amount of historical trajectory data,attackers can identify the user who generates the trajectory from the anonymous trajectory dataset.In these rela-ted studies,both data sparsity and poor data quality are faced.Data sparsity refers to the fact that trajectories are often distributed only in a few local areas,and there is no large corpus contrast to the natural language processing field.The poor data quality refers to the low sampling rate and existing noise of the location points in a trajectory.To address these two problems,this paper proposes a user trajectory identification model based on attention mechanism,including the location embedding module,the attention-based transitional feature encoder and trajectory-user identification module.The location embedding module is used to embed the spatial relation of the trajectory points into the location vector;the attention-based transitional feature encoder is used to extract the sequential dependencies from a single trajectory;and the trajectory-user identification module is used to predict the user identity of the trajectory based on the outputs of the transitional feature encoder.Finally,the experimental verification is carried out on Gowalla and Geolife datasets.The experimental results show that the proposed model in this paper can effectively alleviate the problem of data sparsity and poor data quality,and can achieve better accuracy than existing methods.

With the development of deep learning theory,deep neural network has made a series of breakthrough progress and has been widely applied in various fields.Among them,applications in the image field such as image classification are the most popular.However,research suggests that deep neural network has many security risks,especially the threat from adversarial examples,which seriously hinder the application of image classification.To address this challenge,many research efforts have recently been dedicated to adversarial examples in images,and a large number of research results have come out.This paper first introduces the relative concepts and terms of adversarial examples in images,reviews the adversarial attack methodsand defense me-thods based on the existing research results.In particular,it classifies them according to the attacker's ability and the train of thought in defense methods.This paper also analyzes the characteristics and the connections of different categories.Secondly,it briefly describes the adversarial attacks in the physical world.In the end,it discusses the challenges of adversarial examples in images and the potential future research directions.

Currently,the recognition technology targeted at human action in an interior scene is widely used in video content understanding,home-based care,medical care and other fields,and existing researches pay more heed to the modelling of human action,while ignoring the connection between interior scene and human action in videos.With a view to making full use of the relevance between the scene information and the human motion,this paper studies the recognition approaches for human action in an interior scene based on scene-prior knowledge.Yet,the paper proposes scene-prior knowledge inflated 3D ConvNet(SPI3D).Firstly,the ResNet152 network is adopted to extract scene features for scene classification.Then,based on the results,combined with scene-prior knowledge,this paper introduces quantified scene prior knowledge,optimizes the overall objective function by constraining the weights.Additionally,aiming at the problem that most of the existing data sets focus on the characteristics of human action,whereas the scene information remains complex and plain,an interior scene-action database(SADB) is established.It is shown in experimental results,on the SADB,the recognition accuracy rate of SPI3D reaches 87.9%,6% higher than the recognition accuracy of I3D directly.It can be seen that the modelling for the recognition on human action in interior scene is featured by better performance after introducing the prior knowledge of the scene.

According to the "White Paper on China's Intelligent Internet of Things (AIoT) 2020",with the prompt development of China's 5G network,the rapid popularization of large-capacity with low-price IoT sensor devices and the explosive growth of data,image processing is widely used in various fields of Internet of Things,such as smart city,smart transportation,smart healthcare,and other industry,etc.In these research areas,researchers usually ignore the actual problems in the data collection process,for instance,data degradation caused by time changes:seasonal shifting,diurnal variation,weather changes,and noise problems caused by spatial changes:object superposition,blur,and partial occlusion.Among those problems,the weather pro-blems represented by rainy days are the most challenging and common.Therefore,this paper systematically investigates the actual problems in the data collection process above,classifies and summarizes the image rain-removal algorithms under complex weather conditions.At the same time,regarding the compute-intensive execution of such algorithms,we utilize the Amazon EC2 cloud instance G4 and P3 series to quantitatively evaluate the processing time and effect of various reviewed rain removal algorithms.Finally,we illustrate the characteristics of various rain removal algorithms and the latest trends in Cloud-IoT applications.

Person re-identification aims to retrieve specific pedestrian targets from the target database under the condition of cross camera.It has important application value in the field of video surveillance.At present,the difficulty of the research is that the sample images have large intra class differences and small inter class differences.Therefore,how to design and train the deep neural network to extract a more discriminative feature from pedestrian images is the key.In this paper,we propose a network structure combining global features and local features learning,which can extract global features and local features simultaneously.In view of the different importance of each part of the local features to the description of pedestrian features,this paper proposes a fusion method of local features,which can adaptively generate the weight of each local feature.Finally,the local features and glo-bal features are combined to make the pedestrian features get more comprehensive representation.In addition,in view of the fuzzy optimization objective of the previous triple loss based on hard sample mining,this paper proposes an improved triple loss function based on hard sample mining.The effectiveness of the proposed method is verified on the mainstream person re-identification data sets Market-1501 and DukeMTMC-reID,respectively,and the mAP values are 82.16% and 74.02%,and the Rank-1 values are 92.75% and 86.8%,respectively.

Voice information processing technology is developing rapidly under the impetus of deep learning.The combination of speech synthesis and voice conversion technology can achieve real-time high-fidelity voice output of designated objects and content,and has broad application prospects in man-machine interaction,pan-entertainment and other fields.This paper aims to provide an overview of speech synthesis and voice conversion technology based on deep learning.First,this paper briefly reviews the development of speech synthesis and voice conversion technology.Next,it enumerates the common public datasets in these fields so that it is convenient for researchers to carry out related explorations.Then,it discusses the TTS models,including the classic and cutting-edge models and algorithms in terms of style,rhythm,speed,and compares their effects and development potentials respectively.Then,it reviews voice conversion by summarizing the voice conversion methods and optimization methods.Finally,it summarizes the applications and challenges of speech synthesis and voice conversion,and looks forward to their future development direction in model compression,few-shot learning and forgery detection,based on the problems faced by them in terms of model,application and regulation.

With the development of blockchain technology,smart contracts have been widely used in various fields,and Ethereum has become the largest smart contract platform.At the same time,the frequent smart contract vulnerabilities have caused huge economic losses.The vulnerability detection of smart contract has become the focus of research,while the previous smart contract vulnerability detection tools can not make good use of the syntax information of the contract source code.Aiming at the re-entrancy vulnerability of smart contract,firstly,this paper proposes SCDefender,a vulnerability detection tool based on deep learning.Taking the abstract syntax tree form of the Solidity source code of smart contract as the research object,the tree-based convolutional neural networks is used for vulnerability detection.Secondly,an abstract syntax tree pruning algorithm is proposed to remove the nodes irrelevant to the vulnerability detection task and retain the key information in the abstract syntax tree.The accuracy,recall rate and F1 value of SCDefender vulnerability detection is 81.43%,92.12% and 86.45% respectively,which has a good vulnerability detection effect.Ablation experiments show that the abstract syntax tree pruning algorithm has an important contribution to the vulnerability detection task of SCDefender.

With the rapid development of social media and the popularity of mobile devices,the interaction between users has become more convenient.But at the same time,rumors on social media are more and more rampant,which brings hidden dangers to the public and social safety.In the real world,users often express their own opinions after observing other microblogs that have been posted,especially the context of the microblog to be replied.Although some existing rumor detection methods learn the propagation patterns on propagation trees of rumors to extract clues of user interrogation or factual evidences based on the principle of crowd wisdom,which greatly improves the performance of rumor detection,they only focus on those microblogs that have direct response relationships,and Lack of ability to fully mine the indirect and implicit relationships among microblogs in the process of rumor propagation.Therefore,in this paper,a node and path dual-attention network on propagation tree structures(DAN-Tree) for debunking rumors is proposed.First,the model uses the Transformer structure to fully learn the implicit semantic relationship between posts in the propagation path,and then uses the attention mechanism to perform weighted fusion to obtain the feature vector of the propagation path.Secondly,the path representation is weighted and aggregated by using the attention mechanism to obtain the representation vector of the whole propagation tree.In addition,the structure embedding method is used to learn the spatial location information of the post on the propagation tree,which realizes the effective fusion of the deep structure and semantic information in the rumor propagation structure.The effect of the DAN-Tree model is verified on four classic datasets.Experimental results show that the DAN-Tree model surpasses the best results of the existing literature on the three datasets:the accuracy of the Twitter15 and Twitter16 datasets increases by 1.81% and 2.39%,respectively,and the F1 score of the PHEME dataset increases by 7.51%,which proves the effectiveness of DAN-Tree model.

Rigid object pose estimation aims to obtain 3D translation and 3D rotation information of the rigid object in the camera coordinate system,which plays an important role in rapidly developing fields such as autonomous driving,robotics and augmented reality.The representative papers on rigid object pose estimation based on deep learning from 2017 to 2021 are summarized and analyzed.The rigid object pose estimation methods are divided into coordinate-based,keypoints-based and template-based me-thods.The rigid object pose estimation task is divided into four sub-tasks:image preprocessing,spatial mapping or feature ma-tching,pose recovery,and pose optimization.The subtask realization of each method and its advantages and problems are introduced in detail.The challenges of rigid object pose estimation are analyzed,and the existing solutions and their advantages and disadvantages are summarized.Based on the rigid object pose estimation method,the articulated object and deformable object pose estimation are analyzed.The common datasets and performance evaluation indexes of rigid object pose estimation are introduced,and the performance of existing methods on common datasets is compared and analyzed.Finally,the future research directions of pose tracking and class rigid object pose estimation are prospected.

Accurate flight trajectory predictions can help air traffic management systems make warnings for potential hazards and effectively provide guidance for safe travel.However,the atmospheric situation in which the planes flying is complicated and changeable.The flight track is affected by external factors such as atmospheric disturbance,the air cloud,making prediction difficult.In addition,due to the harsh ground environment where some flight areas are located,it is impossible to deploy enough signal base stations,while the flight signals in some flight areas are collected and combined by multiple signal base stations,resulting in sparse and noisy aircraft track data,which further increases the difficulty of flight track prediction.This paper proposes a technically enhanced self-supervision flight trajectory learning method.This method uses a regularization-based data enhancement mode to extend the sparse track data and process the abnormal values included in the dataset.It provides a self-supervised learning diagram by maximizing mutual information to dig the mobility pattern contained in the flight trajectory.The method employs a multi-head self-attention model with a distillation mechanism as a fundamental model to solve the long-term dependence problem of the recurrent neural network.In addition,the approach uses the distillation mechanism to reduce the complexity of the model and utilizes the generating decoding method to accelerate the speed of its training and prediction.The evaluation results on the flight trajectory dataset show that our method has a significant increase in the results of trajectory prediction compared with the state-of-the-art method that our approach reduces the root mean square error of the prediction results in latitude,longitude,and altitude by 20.8%,26.4%,and 25.6%,respectively.

Few-shot object detection aims to train target detection model through a small amount of sample learning.At present,most of the existing few-shot object detection methods are based on classical target detection algorithms.In the two-stage detection method,due to the small number of new class samples,many irrelevant border boxes are generated,resulting in low accuracy of candidate regions.To solve this problem,this paper proposes a few-shot object detection algorithm FF-FSOD based on feature fusion.It uses the feature fusion method to enhance the data,supplements the new category samples,increases the coverage range of the sample,and introduces the FPN network to extract multi-scale feature.Then,the RPN network is improved,and the support set image branch is introduced.The depth correlation between the support set image feature and the query set image feature is calculated,and the attention feature map is obtained,and the more accurate candidate box is obtained.The effectiveness of the proposed model is verified on MS COCO and FSOD datasets.Experimental results show that the proposed method obtains more accurate candidate boxes and improves the detection accuracy.

In the era of big data,with the explosive growth of multi-source heterogeneous data,multi-modal data fusion has attracted much attention of researchers,and visual question answering(VQA) has become a hot topic in multi-modal data fusion due to its image and text fusion processing characteristics.Visual Q&A task is mainly based on the deep feature fusion association and representation of image and text multi-modal data,and inference learning of the fusion feature results,so as to get the conclusion.Traditional visual question answering models tend to miss key information and mostly focus on the superficial modal feature association representation learning between data,but less on the deep semantic feature fusion.To solve the above pro-blems,this paper proposes a visual question answering model based on cross-modal deep interaction of of graphic features.The proposed method uses convolutional neural network and LSTM network to obtain the data features of image and text modes respectively,and builds a novel deep attention learning network based on combination of meta-attention units,to realize interactive learning of attention features within or between modes of image and text.At last,we represent the learning features so as to output the results.The model is tested and evaluated on VQA-v2.0 dataset.Compared with the traditional baseline model,the expe-rimental results show that the performance of the proposed model is significantly improved.

The tensor compiler compiles the tensor algorithm and schedule of the operator into the code of the target hardware.In order to accelerate tensor operation,the special processor in the field of deep learning is designed as a special architecture with special instructions,which supports multi-core parallel,multi-level special memory architecture and tensor calculation.On top of the hardware,there is a tensor instruction set closely related to the characteristics of the hardware.In such a complex architecture,the use of tensor instructions has many constraints and limitations,and there are the following problems and challenges.Firstly,the conditional branches introduced by loop tiling such as computing task division or data chunking increase the difficulty of pattern matching.Secondly,tensor instructions have hardware constraints such as alignment and data layout.To solve the above problems and research challenges,an optimization algorithm of tensor instruction ge-neration based on loop partitioning is proposed.By dividing the loop interval,the algorithm eliminates the conditional branches introduced by task division or data segmentation.The instruction and hardware constraints are solved by filling zeros,replacing equivalent instructions and adding additional calculations.The tensor instruction is generated by pattern matching method.This paper studies and extends the open source deep learning compiler TVM version 0.7,and implements a compiler prototype system supporting tensor instruction ge-neration of DianNao architecture machine learning accelerator.In order to evaluate the effectiveness of the algorithm,the operator performance and development efficiency of element-wise binary tensor operator,in-place unary tensor operator and convolution operator are tested on the DianNao architecture machine learning accelerator hardware platform.Experimental results show that the average speedup of the three types of operators is 125.00%,the maximum speedup is 194.00%,and the maximum development efficiency increases by 7 times.

In recent years,pretrained language models have developed rapidly,pushing natural language processing into a whole new stage of development.To help researchers understand where and how the powerful pretrained language models can be applied in natural language processing,this paper surveys the state-of-the-art of its application.Specifically,we first briefly review typical pretrained language models,including monolingual,multilingual and Chinese pretrained models.Then,we discuss these pretrained language models' contributions to five different natural language processing tasks:information extraction,sentiment analysis,question answering,text summarization,and machine translation.Finally,we discuss some challenges faced by the applications of pretrained language models.

The task of human pose estimation is to locate and detect the key points of human body in images or videos.It has always been one of the hot research directions in the field of computer vision,and it is also a key step for computers to understand human actions.In recent years,it has wide application for predicting the poses of two-dimensional human body key points in images and videos.Using the powerful image feature extraction capabilities of deep learning,two-dimensional human pose estimation has been improved in robustness,accuracy,and processing time,and the performance effect is far beyond traditional methods.According to the different number of objects in the two-dimensional human body pose,it can be divided into single-person and multi-person pose estimation methods.For single-person pose estimation,according to the different representations of the extracted key points,coordinate regression methods based on the direct prediction of human coordinate points and heat map detection methods based on predicting the Gaussian distribution of human key points can be used.In multi-person pose estimation,it is divided into the top-down method which solves the process from multiple people to a single person,and a bottom-up method that directly deals with the key points of multiple people.Based on the existing estimation methods of human body posture,this paper analyzes the internal mechanism of the network structure,analyzes the commonly used datasets and evaluation indicators,and elaborates the current problems and future development trends.

Visual question answering(VQA) is a multi-modal task that combines computer vision and natural language proces-sing,which is extremely challenging.However,the current VQA model is often misled by the apparent correlation in the data,and the output of the model is directly guided by language bias.Many previous researches focus on solving language bias and assisting the model via counterfactual sample methods.These studies,however,ignore the prediction information and the difference between key features and non-key features in counterfactual samples.The proposed model can distinguish the difference between the original sample,the factual sample and the counterfactual sample.In view of this,this paper proposes a paradigm of contrastive learning based on counterfactual samples.By comparing these three samples in terms of feature gaps and prediction gaps,the VQA model has been significantly improved in its robustness.Compared with CL-VQA method,the overall precision,average precision and Num index of this method improves by 0.19%,0.89% and 2.6% respectively.Compared with the CSSVQA method,the Gap of the proposed method decrease to 0.45 from 0.96.

Recommendation models typically use user’s historical behaviors to obtain user preference representations for recommendations.Most of the methods of learning user representations always entangle different preference factors,while the disentangled learning method can be used to decompose user behavior characteristics.In this paper,a variational autoencoder based framework DSVAECF is proposed to disentangle the static and dynamic factors from user’s historical behaviors.Firstly,two encoders of the model use multi-layer perceptron and recurrent neural network to model the user behavior history respectively,so as to obtain the static and dynamic preference representation of the user.Then,the concatenate static and dynamic preference representations are treated as disentangled representation input decoders to capture user’s decisions and reconstruct user’s behavior.On the one hand,in the model training phase,DSVAECF learns model parameters by maximizes the mutual information between reconstructed user’s behaviors and actual user’s behaviors.On the other hand,DSVAECF minimizes the difference between disentangled representations and their prior distribution to retain the generation ability of the model.Experimental results on Amazon and MovieLens data sets show that,compared with the baselines,DSVAECF significantly improves the normalized discounted cumulative gain,recall,and precision,and has better recommendation performance.

Software bug localization is an important task in bug fixing process.The bug report-oriented localization approaches typically use software bug reports that describe the phenomenon of bugs as queries,and source code as corpus.First,correlations between the bug report and each source code unit are analyzed.Then,a bug localization data set is created by mining the software repository,aiming to construct a bug localization model to localize the source code unit(i.e.bug location) corresponding to the bug report.This paper offers a systematic survey of existing research achievements of the domestic and foreign studies of bug localization in recent years.First,the related concepts in bug report-oriented bug localization are introduced,and the main localization process is summarized,followed by discussing the existing research works that focus on the three key steps in the localization process.Then,the commonly used data sets and evaluation metrics for bug localization are summarized.Finally,future work in this research area is discussed.

Most of the existing medical image segmentation models with excellent performance are manually designed by domain experts.The design process usually requires a lot of professional knowledge and repeated experiments.In addition,the over complex segmentation model not only has high requirements for hardware resources,but also has low segmentation efficiency.An neural architecture search method named Auto-LW-MISN(Automatically Light-weight Medical Image Segmentation Network) is proposed for automatic construction of light-weight medical image segmentation network.In this paper,by constructing a light-weight search space,designing a search super network for medical image segmentation,and designing a differentiable search stra-tegy with complexity constraints,a neural architecture search framework for automatic search of light-weight medical image segmentation network is established.Experimental results on microscope cell images,liver CT images and prostate MR images show that Auto-LW-MISN can automatically construct light-weight segmentation models for different modes of medical images,and its segmentation accuracy is improved compared with U-net,Attention U-net,Unet+＋and NAS-Unet.

Industrial security issues have always been an important and urgent issue globally.Industrial control protocols are widely used in the communication between industrial control system(ICS) components.Their security is related to the safe and stable operation of the entire system,and there is an urgent need to ensure the security of all industrial control protocols.The network protocol fuzzing plays an important role in ensuring the security and reliability of ICS.Traditional fuzzing methods can improve the security testing of industrial control protocols,and many of which have practical applications.However,most traditional fuzzing methods rely heavily on specifications of industrial control protocols,making the test process costly,time-consuming,cumbersome and boring.If the norm does not exist,the task is difficult to carry out.This paper proposes an intelligent and automatic protocol fuzzing method based on pointer-generation networks(PGN),and gives a series of performance indicators.On the basis of this method,an intelligent and automatic fuzzing framework based on PGNFuzz for application is designed,which can be used for various industrial control protocols.Several typical industrial control protocols such as Modbus and EtherCAT are used to test the validity and efficiency of our framework.Experiment results show that our method is superior to other general purpose fuzzers(GPF) and other deep learning based fuzzing methods in terms of convenience,effectiveness and efficiency.

Person re-identification(Re-ID) aims to study the matching of specific persons among multiple disjoint cameras.To the best of our knowledge,it’s the first work that uses the types of challenges that the Re-ID technology needs to overcome in complex scenes as the classification basis,and classifies the Re-ID articles published during 2010-2021 into seven categories:person posture issues,occlusion issues,lighting issues,viewpoint issues,background issues,resolution issues and other open issues.This classification method is convenient for researchers to start from actual needs and find corresponding solutions according to the problems.Firstly,it reviews the research background,significance and research status of Re-ID,summarizes the current mainstream Re-ID framework,counts the papers published in the three top conferences of computer vision,i.e.CVPR,ICCV and ECCV,and counts the Re-ID related projects in the national fund projects since 2013.Secondly,with regard to the seven types of challenges faced in complex scenarios,the existing literatures are classified and analyzed in detail from the two aspects:the cause of the problems and the solutions.The mainstream methods for dealing with various challenges are summarized and listed again.Afterwards,we summarize the Re-ID methods with high generalization and list the difficulties of the current Re-ID research.Finally,the future development trend of Re-ID is discussed.

Question answering over knowledge graph(KG-QA) systems map the natural language question to the 〈subject,predicate,object〉 triple in the knowledge graph(KG) by semantic analysis of the given question,and infer the triple to get the answer of the question.Due to the diversity of natural languages,a question may be expressed in multiple forms but the triples in KGs are structured data in a standard form.It is challenging to map questions to triples in KGs.This paper proposes a novel Key-Value relational memory network,starting from the perspective of KGs,and focusing on the relationship between the candidate answer knowledge and the relationship between the knowledge in KGs and the question representations.In addition,the attention mechanism is applied in the proposed model,so that it has better interpretability than other baseline models.We evaluate the method on WebQuestions benchmark.Experiment results show that,compared with the best methods based on information extraction,the F1 value of the proposed method increases by 5.9% and is slightly higher than that of the optimal methods based on semantic analysis,which verifies the effectiveness of the proposed method.

As a structured form of human knowledge,knowledge graphs have played a great supportive role in supporting the semantic intercommunication of massive,multi-source,heterogeneous data,and effectively support tasks such as data analysis,attracting the attention of academia and industry.At present,most knowledge graphs are constructed based on non-real-time static data,without considering the temporal characteristics of entities and relationships.However,data in application scenarios such as social network communication,financial trade,and epidemic spreading network are highly dynamic and exhibit complex temporal properties.How to use time series data to build knowledge graphs and effectively model them is a challenging problem.Recently,numerous studies use temporal information in time series data to enrich the characteristics of knowledge graphs,endowing know-ledge graphs with dynamic features,expanding fact triples into quadruple representation(head entity,relationship,tail entity,time).The knowledge graph which utilizes time-related quadruples to represent knowledge are collectively referred to as temporal knowledge graph.This paper summarizes the research work of temporal knowledge graph representation learning by sorting out and analyzing the corresponding literature.Specifically,it first briefly introduce the background and definition of temporal know-ledge graph.Next,it summarizes the advantages of the temporal knowledge graph representation learning method compared with the traditional knowledge graph representation learning method.Then it elaborates on the recent method of temporal knowledge graph representation learning from the perspective of the method modeling facts,introduces the dataset used by the above method and summarizes the main challenges of this technology.Finally,the future research direction is prospected.

Federated learning provides a novel solution to collaborative learning among untrusted entities. Through a local-trai-ning-and-central-aggregation pattern,the federated learning algorithm trains a global model while protects local data privacy of each entity. However,recent studies show that local models uploaded by clients and global models produced by the server may still leak users' private information. Secure multi-party computation and differential privacy are two mainstream privacy-preserving techniques,which are used to protect the privacy of computation process and computation outputs respectively. There are few works that exploit the benefits of these two techniques at the same time. This paper proposes a privacy-preserving federated learning scheme for deep learning by combining secure multi-party computation and differential privacy. Clients add noise to local models,and secret share them to multiple servers. Servers aggregate these model shares by secure multi-party computation to obtain a private global model. The proposed scheme not only protects the privacy of local model updates uploaded by clients,but also prevents adversaries from inferring sensitive information from globally shared data such as aggregated models. The scheme also allows dropout of unstable clients and is compatible with complex aggregation functions. In addition,it can be naturally extended to the decentralized setting for real-world applications where no trusted centers exist. We implement our system in Python and Pytorch. Experiments validate that the proposed scheme achieves the same level of efficiency and accuracy as plaintext fede-rated learning.

With the rapid development of social media platforms,public opinion information can be widely disseminated in a very short period of time.If the information of public opinion is not managed and controlled,it will pose a great threat to the network environment and even the social environment.Information extraction technology has become the first and the most significant step in public opinion analysis and management due to its semantization and accuracy.Over the last few years,with the development of deep learning,its ability to automatically learn potential features and combine these features has dramatically improved the accuracy of each sub-task of information extraction.This paper systematically composes and summarizes the methods of extracting information by combining the characteristics of social media public opinion and deep learning technology.Firstly,we sort out the organization of public opinion information in social networks,elaborate the framework and evaluation indexes of public opinion information extraction.Then we conduct a comprehensive review and analysis of existing deep learning-based public opinion information extraction models,discuss the applicability and limitations of existing methods.Finally,the future research trends is prospected.

Cancer is one of the deadliest diseases in the world.Using machine learning to process microarray data plays an important role in assisting the early diagnosis of cancer,but the numbers of genetic features are much more than samples,leading to an imbalance in the sample,and the efficiency and accuracy of classification are affected,so it is important to select the feature of gene array data.Most of the existing feature selection algorithms are single condition feature selection,which seldom consider feature extraction.Most of them use the long-existing neural network and have low classification accuracy.So,a two-stage deep feature selection(TSDFS) algorithm is proposed.The first stage aggregates three feature selection algorithms for comprehensive feature selection,and feature subsets are obtained.In the second stage,unsupervised neural network is used to obtain the best representation of feature subset and improve the final classification accuracy.This paper analyzes the effectiveness of TSDFS by comparing the classification effect before and after feature selection and different feature selection algorithms.Experimental results show that TSDFS algorithm can reduce the number of features while maintaining or improving the accuracy of classification.

In recent years,pre-training models have flourished in the field of natural language processing,aiming at modeling and representing the implicit knowledge of natural language.However,most of the mainstream pre-training models target at the English domain,and the Chinese domain starts relatively late.Given its importance in the natural language processing process,extensive research has been conducted in both academia and industry,and numerous Chinese pre-training models have been proposed.This paper presents a comprehensive review of the research results related to Chinese pre-training models,firstly introducing the basic overview of pre-training models and their development history,then sorting out the two classical models Transformer and BERT that are mainly used in Chinese pre-training models,then proposing a classification method for Chinese pre-training models according to model categories,and summarizes the different evaluation benchmarks in the Chinese domain.Finally,the future development trend of Chinese pre-training models is prospected.It aims to help researchers to gain a more comprehensive understanding of the development of Chinese pre-training models,and then to provide some ideas for the proposal of new models.

Recently,deep learning models have been widely used in various real-world tasks.Improving the robustness of deep neural networks has become an important research direction in machine learning field.Recent works show that training the deep model with noise perturbations can significantly improve the model robustness.However,its training requires a large set of precisely labeled examples,which is often expensive and difficult to collect in real-world scenario.Active learning(AL) is a primary approach for reducing the labeling cost,which progressively selects the most useful samples and queries their labels,with the target of training an effective model with less queries.This paper proposes an active sampling based neural network learning framework,which aims to improve the model robustness with low labeling cost.In this framework,the proposed inconsistency sampling strategy is employed to measure the potential utility for improving the model robustness of each unlabeled example with a series of perturbations.Then,those examples with the largest inconsistency will be selected for training the deep model with noise perturbations.Experimental results on the benchmark image classification task data set show that the inconsistency-based active sampling strategy can effectively improve the robustness of the deep neural network model with lower sample labeling cost.

Traditional radar emitter identification methods can no longer meet the needs of identifying new-system radar emitters in the complicate and changeable electromagnetic environment.Deep learning methods can effectively extract the intra-pulse features of the unsorting radar emitter signal,quickly and accurately identify the radar intra-pulse modulation type,model type and emitter individual under complex environments such as low signal-to-noise ratio.However,in the reality,radar emitter signal is difficult to collect and cannot satisfy the training needs of traditional deep learning models.Therefore,the small sample radar emitter identification is one of hotspot and difficult questions of current research.Firstly,this paper reviews the research progress and application of various deep learning methods based on supervised learning for radar emitter recognition with small samples in recent years.Secondly,the research progress of radar emitter identification by small sample learning is introduced.Last,according to the current radar emitter identification research,the challenges and outlook for future research are put forward.

As a typical industrial Internet of things (IIOT) service,demand response(DR) is becoming a promising enabler for intelligent energy management in 6G-enabled smart grid systems,to achieve quick response for supply-demand mismatches.How-ever,existing literatures try to adjust customers’ load profiles optimally,instead of electricity overhead,energy consumption patterns of residential appliances,customer satisfaction levels,and energy consumption habits.In this paper,a novel DR method is investigated by mixing the aforementioned factors,where the residential customer cluster is proposed to enhance the performance.Clustering approaches are leveraged to study the electricity consumption habits of various customers by extracting their features and characteristics from historical data.Based on the extracted information,the residential appliances can be scheduled effectively and flexibly.Moreover,we propose and study an efficient optimization framework to obtain the optimal scheduling solution by using clustering and deep learning methods.Extensive simulation experiments are conducted with real-world traces.Numerical results show that the proposed DR method and optimization framework outperform other baseline schemes in terms of the system overhead and peak-to-average ratio (PAR).The impact of various factors on the system utility is further analyzed,which provides useful insights on improving the efficiency of the DR strategy.With the achievement of efficient and intelligent energy management,the proposed method also promotes the realization of China’s carbon peaking and carbon neutrality goals.

The key problem of financial asset allocation is asset price.Asset pricing is the core content of modern finance,which indicates that asset pricing law has always been one of the hot topics of financial research.This paper reviews the methods used by machine learning in the field of asset pricing and research progresses,classifies machine learning asset pricing method into machine learning method based on the characteristics processing and deep learning method based on end-to-end processing,compares the differences between different algorithms in principle and application scenarios,points out the applicability and limitations of the two kinds of machine learning methods,prospects the research direction on machine learning asset pricing in the future.

To address the problem that the existing text classification work usually ignores the semantic interaction between words when generating text representation,this paper proposes a novel text classification model based on attention gated graph neural network.It makes effective use of the semantic features of words and improves the accuracy of text classification based on the adequate semantic interaction.Firstly,each input text is converted to a single graph-structured data and the semantic features of word nodes are extracted.Secondly,attention gated graph neural network is used to interact and update the semantic features of word nodes.In addition,the attention-based text pooling module is used to extract the word nodes with discriminative semantic features to construct text graph representation.Finally,effective text classification is implemented based on the text graph representation.Experimental results show that the proposed method achieves an accuracy of 70.83%,98.18%,94.72% and 80.03% on Ohsumed,R8,R52 and MR datasets,respectively,and outperforms existing methods.

Different from the data distribution in general image classification scenarios,in the scenario of multi label image classification,the sample number distribution among different label categories is unbalanced,and a small number of head categories often account for the majority of sample size.However,due to the correlation between multiple labels,and the distribution of diffi-cult samples under multiple labels is also related to the data distribution and category distribution,the re-sampling and other methods for solving the data imbalance in the single label problem cannot be effectively applied in the multi label scenario.This paper proposes a classification method based on the loss of sample distribution in multi label image scene and deep learning.Firs-tly,the unbalanced distribution of multi label data is set with category correlation,and the loss is re-used,and the dynamic lear-ning method is used to prevent the excessive alienation of distribution.Then,the asymmetric sample learning loss is designed,and different learning abilities for positive and negative samples and difficult samples are set.At the same time,the information loss is reduced by softening the sample learning weight.Experiments on related data sets show that the algorithm has achieved good results in solving the sample learning problem in the scene of uneven distribution of multi-label data.

Automatic modulation recognition (AMR) is critical to realize efficient spectrum sensing,spectrum management and spectrum utilization in non-cooperative communication scenarios.It is also an important prerequisite for efficient signal proces-sing.Traditional AMR methods based on pattern recognition need to extract features manually,which faces many problems such as high design complexity,low recognition accuracy and weak generalization ability.Therefore,practitioners turn to deep learning (DL) methods,which are good at extracting hidden features from the data,and propose a number of AMR-oriented deep neural network (ADNN) architectures.Compared with traditional methods,ADNN has achieved higher recognition accuracy,higher generalization ability and wider application range.This paper provides a comprehensive survey of ADNN to help practitioners understand the current research status in this field,and analyzes the future directions after pinpointing several open issues.Firstly,typical deep learning methods involved in ADNN design are introduced.Secondly,a few traditional AMR methods are briefly described.Thirdly,typical ADNNs are introduced in detail.Finally,a series of experiments are conducted on an open dataset to compare typical proposals,and several key research directions in this field are put forward.

The imbalanced problem is common in the real world,and the highly-skewed distribution of imbalanced data seriously affects the performance of the model.In general,the imbalanced data affects the model performance from two aspects.On the one hand,the imbalance in sample size leads to more updates of parameters in majority classes,which leads to the model biased to majority classes.On the other hand,the sample size of minority classes is too small,and the diversity is insufficient,which leads to the insufficient representation ability of the model.To solve these problems,this paper proposes a feature fusion framework combining attention mechanism and geometric information.Specifically,in the first stage,the model learns the semantic information and discriminative information of the data through pre-training,and combines the attention mechanism to discover where the mo-del pays more attention.In the second stage,the model uses geometric information to mine boundary features,and combines the attention weight obtained in the first stage to fuse the boundary features,so as to supplement minority classes.Experimental results on long tail CIFAR10,CIFAR100 and KDD Cup99 datasets show that the proposed feature fusion framework combining attention mechanism and geometric information can effectively improve the classification performance of imbalanced data,and can effectively improve the classification performance for different types of data,including image data and structured data.

Automatic sketch colorization has become an important research topic in computer vision.Previous methods intent to improve the colorization quality with advanced network architecture or innovative pipeline.However, they usually generate results with concentrated hue, unreasonable saturation and gray distribution.To alleviate these problems, this paper proposes a sketch colorization method with drawing priors.Inspired by the actual coloring process, this method learns the widely used drawing priors (such as hue variation, saturation contrast, and gray contrast) to improve the quality of automatic sketch colorization.Speci-fically, it incorporates pixel-level loss in the HSV color space to gain more natural results with less artifacts.Meanwhile, three heuristic loss functions that introduce the drawing priors such as hue variation, saturation and gray contrast are used to train our method to generate results with harmonious color composition.We compare our method with current state-of-the-art methods on test dataset constructed by real sketch images.Fréchet inception distance (FID) and mean opinion score (MOS) are adopted to measure the similarity between the distribution of real and generated images and the visual quality, respectively.Compared to the second-best method, the experimental results show that the FID of our method decreases by 21.00 and the MOS increases by 0.96, respectively.All the experimental results prove that the proposed method effectively improves the visual quality of automa-tic sketch colorization.

The Text-to-SQL task in the field of semantic parsing is of great significance for realizing database-based automatic question and answer.At present, deep learning models, such as sequence generation model Seq2Seq, has achieved significant effects in single-table SQL queries.However, the problem of multi-table SQL queries remains to be solved.Graph neural network can effectively extract the associated information between databases, tables and questions, enrich the semantic information in the parsing process, and improve the accuracy of multi-table SQL queries.This paper proposes an adaptive graph construction method and graph encoding method.Question information is introduced into the existing Text-to-SQL model, and the graph network initialized weights are generated by convolution operation on the splicing word vector of the question sentence and the database.General training can be achieved for different databases of the same type.The IRNet framework and relational expansion are used to design the overall model, and it is verified on the open Text-to-SQL data set——Spider.Results show that the technology can effectively improve the matching accuracy of multi-table SQL statement generation, and the algorithm has an important reference value for the research of graph neural network in the text-to-SQL field.

Socioeconomic Status (SES), an overall measure of a person's economic and social status relative to others combining factors such as economics and sociology, has received a lot of attention from researchers, as its assessment can help relevant orga-nizations to make various policies and decisions (governmental formulation of social policies, advertising personalized services, etc).In addition, with the development of big data technology and machine learning in recent years, assessing people's socioeconomic attributes (SEAs) and further obtaining the corresponding socioeconomic status with a data-driven approach can address the issue of extremely high cost of traditional methods.Therefore, this paper summarizes the research progresses of applying big data techniques to socioeconomic status analysis in recent years.It first introduces the basic concept of socioeconomic status and discusses the challenges posed by big data methods compared to traditional methods.After that, it systematically summarizes and classifies the state-of-the-art related methods based on the information in the learning process, and present them in detail, discusses the pros and cons of each type of method.Finally, it discusses the challenges and problems of inferring people's socioeconomic status and provides an outlook on future research directions.

With the rapid development of human-computer interaction in computer aided field, EEG has become the main means of emotion recognition.Meanwhile, graph network has attracted wide attention due to its excellent ability to represent topological data.To further improve the representation performance of graph network on multi-channel EEG signals, in this paper, conside-ring the sparsity and infrequency of EEG signals, a self-adaptive brain graph convolutional network with spatiotemporal attention (SABGCN-ST) is proposed.The method solves the sparsity of emotion via the spatiotemporal attention mechanism and explores the functional connections between different electrode channels via the self-adaptive brain network topological adjacent matrix.Finally, the feature learning of graph structure is operated via graph convolution, and the emotion is predicted.Extensive experiments conduct on two benchmark datasets DEAP and SEED prove that SABGCN-ST has a significant advantage in accuracy compared with baseline models, and the average accuracy of SABGCN-ST reaches 84.91%.

Reviews contain rich user and item information,which helps to alleviate the problem of data sparsity.However,the existing recommendation model based on reviews is not sufficient and effective enough to mine the review texts,and most of them ignore the migration of user interest over time and the item description documents containing the item attribute,which makes the recommendation result not accurate enough.In this paper,a deep semantic mining based recommendation model (DSMR) is proposed.By mining the semantic information of review texts and item description documents in depth,user characteristics and item attributes can be extracted more accurately,so as to realize more accurate recommendation.Firstly,the BERT pre-training model is used to process the comment text and item description document,and the user characteristics and item attributes are excavated deeply,which effectively alleviated the problems of data sparse and item cold start.Then,the forward LSTM is used to pay attention to the change of user preferences over time,and more accurate recommendations are obtained.Finally,in the model training stage,the experimental data are randomly selected from 1 to 5 points at 1:1:1:1:1 to ensure the same amount of data for each score value,so as to make the results more accurate and the model more robust.Experiments on four commonly used Amazon open datasets show that the root mean square error (RMSE) of DSMR is at least 11.95% lower than the two classical recommendation models based only on rating data,and it is better than the three new recommendation models based only on review text,and 5.1% lower than the optimal model.

With the rapid development of mobile internet,social network platform is full of a large number of text data with emotional color.Mining the emotion in such text not only helps to understand the attitude and emotion of internet users,but also plays an important role in scientific research institutions and the government to grasp the emotional changes and trends of society.Traditional sentiment analysis mainly focuses on the analysis of sentiment tendency,which can not accurately and multi-dimensionally describe the emotion of the text.In order to solve this problem,this paper studies the emotion analysis of the text.Firstly,aiming at the lack of fine-grained sentiment tags in text data sets of different fields,a deep learning based emotion analysis model,FMRo-BLA,is proposed.The model pre-trains the general domain text,and then applies the pre-trained model to the downstream situation of specific domain through parameter based migration learning,feature fusion and FGM Adversarial training.Compared with the best performance of RoBERTa pre-trained language model,the F1 value of the proposed method is improved by 5.93% on the target domain dataset,and it achieves 12.38% by further adding transfer learning.

In the field of computer vision,3D face reconstruction is a valuable research direction.High quality reconstruction of 3D faces can find applications in face recognition,anti-proofing,animation and medical cosmetology.In the last two decades,although great progress has been made 3D face reconstruction based on a single image,the results of reconstruction using traditionalalgorithms are still facing the challenge of facial expression,occlusion and ambient light,and there will be problems such as poor reconstruction accuracy and robustness.With the rapid development of deep learning in 3D face reconstruction,various methods which are superior to traditional reconstruction algorithms have emerged.Firstly,this paper focuses on deep-learning-based reconstruction algorithms.The algorithms are divided into four categories according to different network architecture,and the most popular methods are described in detail.Then commonly used 3D face data sets are introduced,and performance of representative methods are evaluated.Finally,conclusions and prospects of the single-image-based 3D face reconstruction are given.

Micro-expression refers to an uncontrollable muscle movement on the face when people try to hide or suppress their true emotions.Due to the short duration,small motion range,and difficulty in concealing and restraining,the recognition accuracy of such emotional facial expressions is restricted.In order to cope with these challenges,this paper proposes a novel micro-expression recognition method combining feature fusion and attention mechanism,considering optical flow features and face features,and further adding attention mechanism to improve the recognition performance.The processing steps of this method are as follows:1)Extract the optical flow and optical strain from Onset to Apex in each micro-expression segment,input the vertical optical flow,horizontal optical flow and optical strain into a shallow 3DCNN,and extract the optical flow features.2)Taking the deep convolution neural network ResNet-10 as the backbone network,the convolution attention module is added to extract face features.3)Combine the two feature vectors for classification.The experimental results reveal that the proposed method is superior to the traditional methods and existing deep learning methods in micro-expression recognition.