Search

How to preserve the complementary information in multiple images to represent the scene in one image is a challenging topic. Based on this topic, various image fusion methods have been proposed. As an important branch of image fusion, infrared and visible image fusion (IVIF) has a wide range of applications in segmentation, target detection and military reconnaissance fields. In recent years, deep learning has led the development direction of image fusion. Researchers have explored the field of IVIF using deep learning. Relevant experimental work has proven that applying deep learning to achieving IVIF has significant advantages compared with traditional methods. This paper provides a detailed analysis on the advanced algorithms for IVIF based on deep learning. Firstly, this paper reports on the current research status from the aspects of network architecture, method innovation, and limitations. Secondly, this paper introduces the commonly used datasets in IVIF methods and provides the definition of commonly used evaluation metrics in quantitative experiments. Qualitative and quantitative evaluation experiments of fusion and segmentation and fusion efficiency analysis experiments are conducted on some representative methods mentioned in the paper to comprehensively evaluate the performance of the methods. Finally, this paper provides conclusions and prospects for possible future research directions in the field.

Recent image restoration and image enhancement methods are difficult to balance the robustness of multiple subtasks with the small number of parameters and computational costs. To solve this problem, this paper proposes a lightweight cross-gating transformer (CGT) for efficient image restoration task. On the one hand, this paper summarizes the limitations of traditional global self-attention mechanism to capture global dependencies, and improves the global self-attention mechanism to a cross-level cross-gating self-attention mechanism. Meanwhile, a lightweight feed-forward neural network is proposed to learn cross-level local dependencies at a very small computational cost and reconstruct clear features in the adjacent locality. On the other hand, in view of the defect that the traditional method of adding or concatenating encoder and decoder equally leads to information interference, a long-distance reset update module is proposed to suppress and update useless information and clear features respectively. This paper conducts extensive quantitative experiments and is compared with 25 state-of-the-art methods on 9 datasets for image denoising, image deraining and low-light image enhancement, respectively. Experimental results prove that the proposed lightweight cross-gating transformer achieves high peak signal-to-noise ratio and structural similarity in image restoration and image enhancement tasks with a small number of parameters and computation, and reconstructs clear images close to real-world scenes, achieving state-of-the-art image restoration performance.

Traditional 2D animation is a distinct visual style with a production process and image characteristics that differ significantly from real-life scenes. It usually requires drawing pictures frame by frame and saving them as bitmaps. During the storage, transmission, and playback process, 2D animation may encounter problems such as picture quality degradation, insufficient resolution, and discontinuous timing. With the development of deep learning technology, it has been widely used in the field of animation restoration. This paper provides a comprehensive summary of 2D animation restoration based on deep learning. Firstly, exploring existing animation datasets can help identify the available data support and the bottleneck in establishing animation datasets. Secondly, investigating and testing deep learning-based algorithms for animation image quality restoration and animation interpolation can help identify key points and challenges in animation restoration. Additionally, introducing methods designed to ensure consistency between animation frames can provide insights for future animation video restoration. Analyzing the effectiveness of existing image quality assessment (IQA) methods for animation images can help identify practical IQA methods to guide restoration results. Finally, based on the above analysis, this paper clarifies the challenges in animation restoration tasks and presents future development directions of deep learning in animation restoration field.

Finger vein recognition technology has become a research hotspot in the new generation of biometrics because of its advantages of non-contact, high security and living body detection. With the development of deep learning, finger vein recognition technology based on deep neural network has made remarkable achievements. This paper firstly introduces the common public datasets in the field of finger vein recognition, and then classifies the applications of deep learning methods in finger vein recognition in recent years according to different neural network learning tasks, and analyzes the technical characteristics and application scenarios of each type. This paper also introduces the design techniques of deep learning in finger vein recognition from the aspects of lightweight network, data augmentation, attention mechanism and so on, and then expounds the common loss function in the model from two aspects of classifying loss and measuring learning loss. Finally, the evaluation indices of finger vein recognition system are introduced and the results of some researches on accuracy and equal error rate are summarized. In addition, the challenges and potential development directions of finger vein recognition are also presented.

Knowledge tracking, which aims to model students’ changing knowledge states over learning time based on their historical answer records and then predict students’ answer performance, is a core module supporting smart education systems and has received increasing attention from researchers. This paper comprehensively compares the research progress in this field, analyzes the basic theoretical research related to knowledge tracking, and analyzes the knowledge tracking models from probabilistic models, logical models, and deep learning-based models according to different research methods. Probabilistic models assume that learning follows Markov processes, logical models are a class of logic function-based models, and deep learning-based knowledge tracking models relying on the powerful feature extraction ability of deep learning have become a hot research topic in recent years. The improvement methods proposed for the problems faced by deep learning-based knowledge tracking models such as interpretability and lack of learning features are presented. The public datasets currently available to researchers are given as well as a comparison of the performance of different models. Finally, this paper concludes with a summary of this rapidly growing field on knowledge tracking, suggesting some possible future research directions for the problems of research in this area.

Music is one of the most popular forms of art and entertainment, and it is an artistic language to express or entrust people??s feelings. However, with the rapid increase of digital music, it is very difficult to manage and filter music through shallow information. As an effective means to organize massive music and enrich the music information, automatic music annotation can overcome the semantic gap in music information retrieval, improve music information, make music more intuitive in expression, and promote the in-depth research of music information retrieval tasks such as music classification, music recommendation, and instrument identification. The current automatic music annotation mainly focuses on solving two problems: feature extraction and model selection. Combined with the current research focus, this paper expounds the relevant knowledge of automatic music annotation. This paper systematically sorts out various audio feature representation and feature extraction methods in the field of music automatic annotation, and conducts quantitative and qualitative analysis of each extraction method. This paper summarizes the related research results in this field, and focuses on the differences between different model methods from the perspectives of machine learning and deep learning. The commonly used datasets and performance evaluation indicators are introduced, the characteristics of different datasets are summarized, and the evaluation indicators are classified and analyzed. Finally, the difficulties and challenges faced by the research in the field of music automatic annotation are pointed out and the future is prospected.

Convolutional neural networks (CNN) in deep learning can make full use of the computing power of computers to efficiently extract the features of remote sensing images. This has achieved good results, especially in the classification of hyperspectral images. In order to fully extract spectral and spatial features from limited hyper spectral samples and improve the accuracy of hyperspectral image classification, a hybrid deep CNN-Attention (HDC-Attention) model is proposed. Firstly, this paper uses kernel principal component analysis (KPCA) and minibatch K-means (MBK-means) to reduce the dimensionality of hyperspectral images. This effectively eliminates data redundancy and retains the main amount of information. The result is that the dimensionality-reduced data have the best discrimination. Then, this paper uses the HDC network to extract the spectral-spatial features from the dimensionality-reduced data. Finally, the spectral-spatial attention is used to redistribute the weights of spectral-spatial features. It enhances useful spatial-spectral features and suppresses useless features. The proposed model has been tested on three public datasets for many times. With a limited number of labeled samples, the OA, AA, and Kappa classification indicators of the three datasets all exceed 99%.

Epilepsy is a life-threatening and challenging nervous system disease. There are still many challenges in the detection of epilepsy based on electroencephalogram (EEG). Because the EEG signal is unstable, different patients show different seizure patterns. In addition, EEG detection is time-consuming and laborious, which will not only bring heavy burden to medical staff, but also easily lead to false detection. Therefore, it is necessary to study an efficient automatic epilepsy detection technology across multiple patients. In this paper, an epileptic EEG detection method (convolutional attention bidirectional long short-term memory network, CABLNet) based on the multi-head self-attention mechanism neural network is proposed. Firstly, the convolution layer is used to capture short-term temporal patterns of EEG time series and local dependence among channels. Secondly, this paper uses the multi-head self-attention mechanism to capture the long-distance dependence and time dynamic correlation of the short-term time pattern feature vectors with temporal relationship. Thirdly, the context representation is sent into a bidirectional long short-term memory (BiLSTM) to extract the information in the front and back directions. Finally, logsoftmax function is used for training and classification. Using CHB-MIT scalp EEG database data, the sensitivity, specificity, accuracy and F1-score are 96.18%, 97.04%, 96.61% and 96.59% respectively. The results show that the proposed method is superior to the existing methods and significantly improved in epilepsy detection performance, which is of great significance to the auxiliary diagnosis of epilepsy.

Video multi-object tracking is a key task in the field of computer vision and has a wide application prospect in industry, commerce and military fields. At present, the rapid development of deep learning provides many solutions to solve the problem of multi-object tracking. However, the challenging problems such as mutation of target appearance, serious occlusion of target area, disappearance and appearance of target have not been completely solved. This paper focuses on online multi-object tracking algorithm based on deep learning, and summarizes the latest progress in this field. According to the three important modules of feature prediction, apparent feature extraction and data association, as will as the two frameworks of detection-based-tracking (DBT) and joint-detection-tracking (JDT), this paper divides deep online multi-object tracking algorithms into six sub-classes, and discusses the principles, advantages and disadvantages of different types of algorithms. Among them, the multi-stage design of the DBT algorithm has a clear structure and is easy to optimize, but multi-stage training may lead to sub-optimal solutions; the sub-modules of the JDT algorithm that integrates detection and tracking achieve faster inference speed, but there is a problem of collaborative training of each module. Currently, multi-target tracking begins to focus on long-term feature extraction of targets, occlusion target processing, association strategy improvement, and end-to-end framework design. Finally, combined with the existing algorithms, this paper summarizes urgent problems to be solved in deep online multi-object tracking and looks forward to possible research directions in the future.

In order to further improve the performance of the SSD (single shot multibox detector) algorithm, and solve the problems of unbalanced feature map information and difficulty in small target recognition during multi-scale prediction of the SSD algorithm, in this paper, plug-and-play modules are designed to fully integrate the information contained in feature maps of different scales and model the relationships within feature maps to enhance the representation ability of feature maps. Firstly, a novel feature fusion method is designed to solve the problem of information disparity in cross-scale feature fusion. Secondly, according to the idea of pooling pyramid, a depth feature extraction module is designed to extract the information of different receptive fields, so as to improve the detection ability of the model to object of different sizes. Finally, in order to further optimize the feature map, highlight the effective information of the feature map for the current task, and establish the global long-distance relationship between pixels and the importance relationship between each channel, a lightweight attention module is proposed. Through the above mechanism, the structure of SSD model is modified in this paper, which effectively improves the detection accuracy and robustness of SSD algorithm. Extensive experiments have been conducted on PASCAL VOC datasets to verify the efficiency of the proposed method. On PASCAL VOC2007 test datasets, the proposed method improves 2.9 percentage points mean average precision (mAP) over SSD algorithm, while maintaining the ability of real-time detection.

Community question answering (CQA) has become the most important knowledge sharing and exchange platform on the Internet, effectively recommending massive questions raised by users to users who may answer them, and mining the questions that users are interested in is the core function of such platforms. Some expert recommendation algorithms for CQA have been proposed to improve the efficiency of platform answering, but most of the existing work focuses on matching user interests with question information, ignoring the dynamic change of user interests, which may seriously affect the quality of recommendation. This paper proposes an expert recommen-dation algorithm combining attention and recurrent neural network (RNN), which not only realizes deep feature coding of the question content, but also captures the dynamically changing user interest. First, the question encoder combines convolutional neural network (CNN) and Attention mechanism on the basis of pre-trained word embeddings to realize joint representation of deep feature of question title and bound topics. Then, the user encoder captures the dynamic interest using Bi-GRU model on the time series of user's historical answers to questions, and combines the user's fixed label information to represent the long-term interest. Finally, a recommendation result combining the user's dynamic interest and long-term interest is generated according to the similarity calculation of output vectors of two encoders. This paper conducts comparative experiments on different parameter configurations and different algorithms on real data from Zhihu. Experimental results show that the performance of the algorithm is significantly better than the current popular deep learning expert recommendation methods.

The analysis and inference of program play an important role in software development, maintenance and migration. How to efficiently obtain high quality information from program code has become a hot research topic. In recent years, a large number of researchers have introduced the deep learning-based representation technology into the code analysis tasks. The deep learning model can automatically extract the implicit and useful features implicit in the source code, which can alleviate the dependence on the manual construct feature. This paper first introduces the background and basic concepts of code representation, and summarizes the recent research work on deep learning-based code representation learning from the perspective of code static information analysis. Furthermore, this paper introduces the application of code representation on three tasks, code clone detection, code search and code completion. Finally, it discusses the challenges of deep learning-based code representation and the possible research directions in this field.

In the field of natural language processing, named entity recognition is the first key step of information extraction. Named entity recognition task aims to recognize named entities from a large number of unstructured texts and classify them into predefined types. Named entity recognition provides basic support for many natural language processing tasks such as relationship extraction, text summarization, machine translation, etc. This paper first introduces the definition of named entity recognition, research difficulties, particularity of Chinese named entity recognition, and summarizes the common Chinese and English public datasets and evaluation criteria in named entity recognition tasks. Then, according to the development history of named entity recognition, the existing named entity recognition methods are investigated, which are the early named entity recognition methods based on rules and dictionaries, the named entity recognition methods based on statistic and machine learning, and the named entity recognition methods based on deep learning. This paper summarizes the key ideas, advantages and disadvan-tages and representative models of each named entity recognition method, and summarizes the Chinese named entity recognition methods in each stage. In particular, the latest named entity recognition based on Transformer and based on prompt learning are reviewed, which are state-of-the-art in deep learning-based named entity recognition methods. Finally, the challenges and future research trends of named entity recognition are discussed.

The current scheme of manually extracting features for object detection has been replaced by deep learning. Deep learning technology has greatly promoted the development of object detection technology. Object detection has also become one of the most important application fields of deep learning. Object detection is to simultaneously predict the category and position of object instances in a given image. This technology has been widely used in medical imaging, remote sensing technology, monitoring and security, automatic driving and other fields. However, with the diversification of object detection application fields, the imbalance problem in the application of deep learning to object detection has become a new entry point to optimize the object detection training model. This paper mainly analyzes the use of machine learning technology to solve the object detection problem. There are four kinds of imbalance problems in each training stage of the model: data imbalance, scale imbalance, relative space imbalance and classification and regression imbalance. This paper analyzes the main reasons for the problem, studies representative classical solutions, and expounds the problems existing in object detection in various fields. By analyzing and summarizing the object detection imbalance problems, this paper discusses the directions of the imbalance of object detection in the future.

Deep learning has the following problems in medical image classification application: first, it is impossible to construct a deep learning model hierarchy for medical image properties; second, the network initialization weight of the deep learning model has not been optimized. To this end, this paper starts from the perspective of network optimization, and then improves the nonlinear modeling ability of the network through optimization methods. Then, this paper proposes a new network weight initialization method, which alleviates the problem that the initialization theory of existing deep learning is limited by the nonlinear unit type, and increases the potential of neural network to deal with different visual tasks. At the same time, in order to make full use of the characteristics of medical images, this paper deeply studies the multi-column convolutional neural network framework and finds that through changing the number of features and the convolution kernel size of different levels of convolutional neural networks, it can construct different convolutional neural network models to better adapt to the medical characteristics of the medical images to be processed and train the obtained heterogeneous multi-column convolutional neural networks. Finally, the classification task of medical images is completed by the method proposed in this paper. Based on the above ideas, this paper proposes a medical classification algorithm based on weight initialization-sliding window fusion of multi-layer convolutional neural networks. The methods of this paper are used to classify breast mass classification, brain tumor tissue classification experiment and medical image database classification. The experimental results show that the proposed method not only has higher average accuracy than traditional machine learning and other deep learning methods, but also has better stability and robustness.

Video object tracking is an important research content in the field of computer vision, mainly studying the tracking of objects with interest in video streams or image sequences. Video object tracking has been widely used in cameras and surveillance, driverless, precision guidance and other fields. Therefore, a comprehensive review on video object tracking algorithms is of great significance. Firstly, according to different sources of challenges, the challenges faced by video object tracking are classified into two aspects, the objects’ factors and the backgrounds’ factors, and summed up respectively. Secondly, the typical video object tracking algorithms in recent years are classified into correlation filtering video object tracking algorithms and deep learning video object tracking algorithms. And further the correlation filtering video object tracking algorithms are classified into three categories: kernel correlation filtering algorithms, scale adaptive correlation filtering algorithms and multi-feature fusion corre-lation filtering algorithms. The deep learning video object tracking algorithms are classified into two categories: video object tracking algorithms based on siamese network and based on convolutional neural network. This paper analyzes various algorithms from the aspects of research motivation, algorithm ideas, advantages and disadvantages. Then, the widely used datasets and evaluation indicators are introduced. Finally, this paper sums up the research and looks forward to the development trends of video object tracking in the future.

Multimodal emotion recognition aims to recognize human emotional states through different modalities related to human emotion expression such as audio, vision, text, etc. This topic is of great importance in the fields of human-computer interaction, a.pngicial intelligence, affective computing, etc., and has attracted much attention. In view of the great success of deep learning methods developed in recent years in various tasks, a variety of deep neural networks have been used to learn high-level emotional feature representations for multimodal emotion recog-nition. In order to systematically summarize the research advance of deep learning methods in the field of multi-modal emotion recognition, this paper aims to present comprehensive analysis and summarization on recent multi-modal emotion recognition literatures based on deep learning. First, the general framework of multimodal emotion recognition is given, and the commonly used multimodal emotional dataset is introduced. Then, the principle of representative deep learning techniques and its advance in recent years are briefly reviewed. Subsequently, this paper focuses on the advance of two key steps in multimodal emotion recognition: emotional feature extraction methods related to audio, vision, text, etc., including hand-crafted feature extraction and deep feature extraction; multi-modal information fusion strategies integrating different modalities. Finally, the challenges and opportunities in this field are analyzed, and the future development direction is pointed out.

Occluded object detection has long been a difficulty and hot topic in the field of computer vision. Based on convolutional neural network, the deep learning takes the object detection task as a classification and regression task to handle, and obtains remarkable achievements. The mask confuses the features of object when the object is occluded, making the deep convolutional neural network cannot handle it well and reducing the performance of detector in ideal scenes. Considering the universality of occlusion in reality, the effective detection of occluded object has important research value. In order to further promote the development of occluded object detection, this paper makes a comprehensive summary of occluded object detection algorithms, and makes a reasonable classification and analysis. First of all, based on a simple overview of object detection, this paper introduces the relevant theoretic background, research difficulties and datasets about occluded object detection. After, this paper focuses on the algo-rithms to improve the performance of occluded object detection from the aspects of object structure, loss function, non-maximum suppression and semantic partial. This paper compares the performance of different detection algo-rithms after summarizing the relationship and development of various algorithms. Finally, this paper points out the difficulties of occluded object detection and looks forward to its future development directions.

Video anomaly detection refers to the detection and identification of events that deviate from normal behavior, which has a wide range of applications in surveillance video. In this paper, the video anomaly detection algorithm based on deep learning is investigated in depth and summarized comprehensively. Firstly, this paper analyzes the related content of video anomaly detection and the challenges faced by anomaly detection, then introduces and analyzes the related algorithms of video anomaly detection from three aspects: supervised, semi-supervised and unsupervised. The algorithms in three different scenarios are further refined and classified. The algorithms in the supervised scenario are divided into two types: binary classification and multi-classification. The algorithms in the semi-supervised scenario are divided into two types: calculating anomaly scores and clustering discrimination. The algorithms in the unsupervised scenario are divided into two types: reconstruction discrimination and prediction discrimination. The characteristics, advantages and disadvantages of different technologies are analyzed. The commonly used datasets in the field of video anomaly detection and the evaluation criteria of detection performance are introduced, and the performance of current mainstream video anomaly detection algorithms is compared and analyzed. Finally, the future research direction of video anomaly detection algorithm is discussed and prospected.

As the rapid development of deep neural networks, multi-modal learning techniques are widely concerned. Cross-modal retrieval is an important branch of multimodal learning. Its fundamental purpose is to reveal the relation between different modal samples by retrieving modal samples with identical semantics. In recent years, cross-modal retrieval has gradually become the forefront and hot spot of academic research. It’s an important direction in the future development of information retrieval. This paper focuses on the latest development of cross-modal retrieval based on deep learning, reviews the development trends of real value representation-based and binary representation-based learning methods systematically. Among them, the real value representation-based method is adopted to improve the semantic relevance, and improve the accuracy, and the binary representation-based learning method is used to improve the efficiency of image-text cross-modal retrieval and reduce storage space. In addition, the common open datasets in the field of image-text cross-modal retrieval are summarized, and the performance of various algorithms on different datasets is compared. Especially, this paper summarizes and analyzes the specified implementations of cross-modal retrieval techniques in the fields of public security, media and medicine. Finally, combined with the state-of-the-art technologies, development trends and future research directions are discussed.

The continuous development of deep learning technology has provided new ideas for image repair research over the years, and the image repair methods can understand the semantic information of image through the study of massive image data. Although the existing image repair methods have been able to generate desirable repair results, it is insufficient to deal with the details of missing part from the image when facing the image with more complex missing part, thus the restoration results are excessively smooth or blurry, and the complex structural information that misses from the image cannot be repaired well. In order to solve the issues above, an edge-guided image repair method based on generative adversarial networks technology and the corresponding algorithm are proposed in this paper, and the repair process is divided into two stages. First, the edge repair model is trained to generate more realistic edge information of the missing area. Then, the content generation model is trained to fill in the missing content information based on the edge information that has been repaired. Lastly, the experimental verification is conducted on the CelebA dataset and ParisStreet-View dataset to compare with the Shift-Net model,deep image prior (DIP) model and field factorization machine (FFM) model, and the visual qualitative analysis and quantitative index analysis are carried out on the experimental repair results. The experimental results prove that the repair method proposed in this paper for the missing complex structure information in the image is superior to the existing methods, and also reflect the edge information plays a crucial role in image repair.

Traditional neural networks have the disadvantages of over-reliance on hardware resources and high requirements for application equipment performance. Therefore, they cannot be deployed on edge devices and mobile terminals with limited computing power. The application development of artificial intelligence technology is limited to a certain extent. However, with the advent of the technological age, artificial intelligence, which is affected by user requirements, urgently needs to be able to successfully perform operations such as computer vision applications on portable devices. For this reason, this paper takes the convolution of popular lightweight neural networks in recent years as the research object. Firstly, by introducing the concept of lightweight neural network, the development status of lightweight neural networks and the problems faced by convolution in the network are introduced. Secondly, the convolution is divided into three aspects: lightweight of convolution structure, lightweight of convolution module and lightweight of convolution operation, specifically through the study of the convolution design in various lightweight neural network models, the lightweight effects of different convolutions are demonstrated, and the advantages and disadvantages of the optimization methods are explained. Finally, the main ideas and usage methods of all lightweight model convolutional design in this paper are summarized and analyzed, and their possible future development is prospected.

Behavior recognition is a hot topic in the field of computer vision. It has experienced the development process from manual design feature representation to deep learning feature expression. This paper classifies the mainstream algorithms in the development of behavior recognition from two aspects of traditional behavior recognition models and deep learning models. The traditional behavior recognition models mainly include feature description methods based on silhouette, space-time interest points, human joint point and trajectories. Among them, the improved dense trajectory method has good robustness and reliability. Deep learning network architecture mainly includes two-stream network, 3D convolution network and hybrid network. Firstly, this paper focuses on the main research ideas and innovations of each behavior recognition algorithm, and introducees the model architecture, algorithm features, application scenarios of each kind of algorithm. Then, the widely used public behavior databases are classified, and the HMDB51 and UCF101 datasets are introduced in detail. The recognition effects of traditional methods and deep learning algorithms on each dataset are compared and analyzed. Through comparative analysis, the traditional methods are not suitable for high-precision behavior recognition, and it is not easy to achieve cross database or cross scene promotion. In depth architecture, two-stream network and 3D convolution network have achieved good behavior recognition effect and are widely used. Finally, the future development of behavior recognition is prospected, and some feasible research directions in the future are pointed out.

Human-machine dialogue, an important research component of artificial intelligence, has received widespread attention from academia and industry. Inspired by the successful application of deep learning in natural language processing, a growing number of neural network models are being focused on by researchers. Among them, end-to-end based neural network models are able to learn valuable patterns and features from large-scale corpus to generate meaningful and diverse responses, and are widely used in research on emotional dialogue generation. This paper presents a review of the research on end-to-end models for emotional dialogue generation. Firstly, the tasks and main problems addressed by current research on emotional dialogue generation are outlined and defined in detail in the light of existing research results. The datasets required for modeling emotional dialogue generation models are organized and presented. Secondly, a brief overview of the principles of end-to-end neural network models is given, and the improvements in each of the underlying models, the current state of research, the evaluation metrics involved in the models, and the performance of the models are analyzed and summarized. Thirdly, the evaluation methods involved in the current stage of model evaluation are summarized in terms of automatic evaluation as well as manual evaluation. Finally, this paper prospects the development direction of the research on the generation of emotional dialogue in the future.

Current research on lip recognition focuses on improving recognition accuracy and studying features of multimodal inputs. However, little attention has been paid to improving the effectiveness of lip visual features. Lip visual information plays a key role in visual speech recognition and lip recognition. It is important when audio is destroyed or has no information. How to obtain accurate and effective lip visual features is one of the most difficult tasks in lip recognition. This paper reviews the latest research work on lip recognition in recent years from three aspects: lip dataset, traditional visual feature extraction methods, and in-depth learning methods for visual feature extraction. Firstly, this paper summarizes the dataset for lip recognition. The lip dataset is divided into two types: front view and multi-view. Further two types of datasets are summarized from their characteristics, limitations, and download addresses. Secondly, this paper introduces the traditional methods of lip visual feature extraction from the perspective of pixel point, shape and mixed features. The basic idea, network structure and features of each method are mainly introduced. In the deep learning method of lip visual feature extraction, the network structure, advantages and disadvantages of four deep learning methods are mainly introduced, such as 2D CNN, 3D CNN, 2D CNN combined with 3D CNN, and other neural networks. The performance of these methods on open datasets is compared. Finally, the challenges faced by lip visual feature extraction methods and future research trends are prospected.

Face synthesis is one of the hot topics in the field of computer vision because of its application and technical value. In recent years, the breakthrough of deep learning has attracted much attention in this field. This paper divides the research in this field into four subcategories: face identity synthesis, face movements synthesis, face attributes synthesis and face generation, and systematically summarizes the development process, status quo, and existing problems of these subcategories. First of all, for face identity synthesis, three approaches are summarized, including computer graphics, digital image processing and deep learning. This paper summarizes their respective routine processes, and analyzes the technical principles of milestone work in detail. Secondly, face movements synthesis is further divided into label driven expression editing and real face driven face reenactment, where the shortcomings and problems in each field are pointed out. Then, the development of face attribute synthesis based on generative model is introduced, especially generative adversarial network. Finally, this paper briefly describes all kinds of researches on face generation. In addition, this paper also introduces the practical application and related problems of face synthesis field and provides the possible research direction in this field.

Fine-grained image categorization aims to distinguish the sub-categories from a certain category of images. Generally, fine-grained data sets have the characteristics of the intra-class similarity and inter-class variation, which makes the task of fine-grained image categorization more challenging. With the increasing development of deep learning, the methods of fine-grained image categorization based on deep learning exhibit more powerful feature representation and generalization capabilities, and can obtain more accurate and stable classification results. Therefore, deep learning has been attracting more and more attentions and research from the researchers in the fine-grained image categorization. In this paper, starting from the background of fine-grained image categorization, the difficulties and the meaning of fine-grained image categorization are introduced. Then, from the perspectives of strong supervision and weak supervision, this paper reviews the research progress of fine-grained image classification algorithms based on deep learning, and a variety of typical classification algorithms with excellent performance are introduced. In addition, the YOLO (you only look once), multi-scale CNN (convolutional neural network), and GAN (generative adversarial networks) model are further discussed in the application of fine-grained image categorization, the perfor-mance of the latest relevant fine-grained data augmentation methods is compared and an analysis of different types of fine-grained categorization methods is made under complex scenarios. Finally, by comparing and summarizing the categorization algorithms, the future improvement directions and challenges are discussed.

Generative adversarial networks (GAN) has become a popular research direction in the field of deep learning. The unique adversarial idea of GAN comes from the two-player zero-sum game in game theory. How to solve the problems of unstable GAN training, poor quality of generated samples, inadequate evaluation system, poor interpretability and other issues is critical and difficult in current GAN research. This paper investigates the research background and development trend of GAN. First, the basic idea and algorithm implementation of GAN are described, the advantages and disadvantages of GAN are analyzed, a more systematic classification of existing improved methods is made, and some typical optimization methods and derivative models of GAN are sorted out from two categories based on structure change and loss function variants respectively. Next, the similarities and differences between GAN and other generative models are compared, and their respective advantages and disadvantages are introduced. Then, this paper compares the performance of GAN and its derivative models, and summarizes their operation mechanism, advantages, limitations and applicable scenarios. The applications of GAN in the field of image generation are introduced. Finally, the mainstream evaluation indicators of GAN are listed, the main problems still faced in GAN research are analyzed and corresponding solutions are given. The listed mainstream solutions are compared and analyzed in terms of solution effects and applicability, and the future research direction is prospected.

Knowledge graph question answering (KGQA) is the procedure of processing natural language questions posed by users to obtain relevant answers from knowledge graph (KG) based on some form of KG. Due to the limitation of knowledge scale, computing power and natural language processing capability, the early knowledge base question answering systems were limited to closed-domain questions. In recent years, with the development of KG and the construction of open-domain question answering (QA) datasets, KG has been used for open-domain QA research and practice. In this paper, in accordance with the development of technology, the open-domain KGQA is summarized. Firstly, five rule and template based KGQA methods are reviewed, including traditional semantic parsing, traditional information retrieval, triplet matching, utterance template, and query template. This type of methods mainly relies on manually defined rules and templates to complete QA task. Secondly, five deep learning based KGQA methods are introduced, which use neural network models to complete the subtasks of QA process, including knowledge graph embedding, memory network, neural network-based semantic parsing, neural network-based query graph, and neural network-based information retrieval method. Thirdly, four general domain KG and eleven open-domain QA datasets, which KGQA commonly used are described. Fourthly, three classic KGQA datasets are selected according to the difficulty of questions to compare and analyze the performance metric of each KGQA system, and the effect between above methods. Finally, this paper looks forward to the future research directions on this topic.

Video object detection is to solve the problem of object localization and recognition in every video frame. Compared with image object detection, video is featured by high redundancy, which contains a lot of local spatio-temporal information. With the rapid popularity of deep convolutional neural network in the field of static image object detection, it shows a great advantage over traditional methods in performance. Besides, it plays a due role in video-based object detection task. However, the current video object detection algorithms still face many challenges, such as improving and optimizing the performance of mainstream object detection algorithms, maintaining the spatiotemporal consistency of video sequences, and making detection of model lightweight. In view of the above problems and challenges, on the basis of investigating a large number of literature, this paper systematically sum-marizes the video object detection algorithm based on deep learning. Based on the basic methods like optical flow and detection, these algorithms are classified. In addition, in the angles of backbone network, algorithm structure and data sets etc., these methods are explored. Combined with the experimental results in the ImageNet VID data set, this paper analyzes the performance advantages and disadvantages of typical algorithms of this field, and the relations between these algorithms. As for video object detection, the problems to be solved as well as the future research direction are expounded and prospected. Video object detection has become a hot spot pursued by many computer vision scholars. More efficient and accurate algorithms will be proposed, and its development direction will be better and better.

The objects in remote sensing images have the characteristics of arbitrary direction and dense arrangement, and thus objects can be located and separated more precisely by using inclined bounding boxes in object detection task. Nowadays, oriented object detection in remote sensing images has been widely applied in both civil and military defense fields, which shows great significance in the research and application, and it has gradually become a research hotspot. This paper provides a systematic summary of oriented object detection methods in remote sensing images. Firstly, three widely-used representations of inclined bounding boxes are summarized. Then, the main challenges faced in supervised learning are elaborated from four aspects: feature misalignment, boundary discontinuity, inconsistency between metric and loss and oriented object location. Next, according to the motivations and improved strategies of different methods, the main ideas and advantages and disadvantages of each algorithm are analyzed in detail, and the overall framework of oriented object detection in remote sensing images is summarized. Furthermore, the commonly used oriented object detection datasets in remote sensing field are introduced. Experimental results of classical methods on different datasets are given, and the performance of different methods is evaluated. Finally, according to the challenges of deep learning applied to oriented object detection in remote sensing images tasks, the future research trend in this direction is prospected.

Cryptomining malware can compromise system security, reduce hardware lifetime, and cause significant power consumption. Therefore, implementing cryptomining malware early detection to stop its damage in time is critical to system security. The existing dynamic analysis-based cryptomining malware early detection methods are hard to balance the timeliness and accuracy of detection. To detect cryptomining malware accurately and timely, this paper integrates a certain length of API (application programming interface) names, API operation categories and DLLs (dynamic link libraries) called by cryptomining malware in the early stage of operation to more fully describe its behavioral information in this stage, and proposes the AECD (API embedding based on category and DLL) embedding and further proposes a cryptomining malware early detection method based on AECD embedding (CEDMA). CEDMA uses the API sequence called by software in the early stage of operation as the object of detection and uses AECD embedding and TextCNN (text convolutional neural network) to build a detection model to implement cryptomining malware early detection. Experimental results show that when CEDMA takes the 3000 API sequence called for the first time after the software runs as input, it can detect the known and unknown cryptomining malware samples in the experiment with 98.21% and 96.76% accuracy values, respectively.

As static scene 3D reconstruction algorithms become more mature, dynamic scene 3D reconstruction has become a hot and challenging research topic in recent years. Existing static scene 3D reconstruction algorithms have good reconstruction results for stationary objects. However, when objects in the scene undergo deformation or relative motion, their reconstruction results are not ideal. Therefore, developing research on 3D reconstruction of dynamic scenes is essential. This paper first introduces the related concepts and basic knowledge of 3D reconstruction, as well as the research classification and current status of static and dynamic scene 3D reconstruction. Then, the latest research progress on dynamic scene 3D reconstruction is comprehensively summarized, and the reconstruction algorithms are classified into dynamic 3D reconstruction based on RGB data sources and dynamic 3D reconstruction based on RGB-D data sources. RGB data sources can be further divided into template based dynamic 3D reconstruction, non rigid motion recovery structure based dynamic 3D reconstruction, and learning based dynamic 3D reconstruction under RGB data sources. The RGB-D data source mainly summarizes dynamic 3D reconstruction based on learning, with typical examples introduced and compared. The applications of dynamic scene 3D reconstruction in medical, intelligent manufacturing, virtual reality and augmented reality, and transportation fields are also discussed. Finally, future research directions for dynamic scene 3D reconstruction are proposed, and an outlook on the research progress in this rapidly developing field is presented.

In recent years, animal behavior analysis has become one of the most important methods in the fields of neuroscience and artificial intelligence. Taking advantage of the powerful deep-learning-based image analysis technology, researchers have developed state-of-the-art automatic animal behavior analysis methods with complex functions. Compared with traditional methods of animal behavior analysis, special labeling is not required in these methods, animal pose can be efficiently estimated and tracked. These methods like in a natural environment, which hold the potential for complex animal behavior experiments. Therefore, the application of deep learning in animal behavior analysis is reviewed. Firstly, this paper analyzes the tasks and current status of animal behavior analysis. Then, it highlights and compares existing deep learning-based animal behavior analysis tools. According to the dimension of experimental analysis, the deep learning-based animal behavior analysis tools are divided into two-dimensional animal behavior analysis tools and three-dimensional animal behavior analysis tools, and the functions, performance and scope of application of tools are discussed. Furthermore, the existing animal datasets and evaluation metrics are introduced, and the algorithm mechanism used in the existing animal behavior analysis tool is summarized from the advantages, limitations and applicable scenarios. Finally, the deep learning-based animal behavior analysis tools are prospected from the aspects of dataset, experimental paradigm and low latency.

In recent years, with the popularization of Internet and technologies like big data analysis, the demand for mobile medical services has become more and more urgent, which mainly focuses on ascertaining their diseases based on symptoms and further choosing hospitals and doctors with good service quality based on diseases. In order to tackle problems above, this paper designs and implements a recommendation system for medical consultation based on knowledge graph and deep learning. Using the open data on Internet, a “disease-symptom” knowledge graph is constructed. Once given symptom description, a disease candidate set is built to help user self-diagnose. To improve the accuracy of disease diagnosis, a vector representation of entities in the knowledge graph is trained by a knowledge graph embedding model. Then the disease candidate set is expanded by selecting disease entity with the shortest Euclidean distance with diseases in the set. Combining the two above, disease diagnosis service is provided. To recommend hospitals and doctors, given open media data, this paper uses a deep learning model and combines it with existing quality evaluation indicators for medical service to achieve scoring for doctors multi-dimensional service quality automatically. Finally, this paper verifies the accuracy of the disease diagnosis service and the doctor recommendation service by constructing test sets and designing questionnaires, which reach 74.00% and 90.91%, respectively.

Sleep disorders seriously affect human health and life. Accurate classification of sleep stages is the key to detecting and treating sleep disorders. In recent years, methods based on deep learning have surpassed traditional machine learning methods and human experts. However, the internal structure of deep learning is complex and requires to be designed by expert who is familiar with the computer and medical fields. This paper aims to analyze the relational induction biases in the existing sleep staging model based on deep learning, and explores the basic principles of sleep staging model design. This paper analyzes relational induction biases such as translation invariance, time invariance and hierarchical processing. Firstly, the model is divided into three categories according to whether it contains convolution layer with translation invariance and recurrent layer with time invariance: convolutional neural network framework, recurrent neural network framework and hybrid neural network framework. Secondly, according to the hierarchical processing method of the frame, segment and sequence in the model, a more detailed classification is carried out. Thirdly, it analyzes the impact of different relational induction biases in the model on the performance of sleep staging. It is proposed that a relational inductive bias matched the task should be introduced when the automatic sleep staging model is designed. Finally, the advantages and limitations of sleep staging based on deep learning are discussed, and it may be necessary to use more advanced relationship induction bias to express knowledge more abstractly and combine it with other artificial intelligence technologies in the future.

Deep learning is a promising solution for computer vision at present. To solve the computer vision problem, it requires massive and high-quality image training datasets. Collecting and accurately labeling image datasets is a very time-consuming and expensive process. As computer vision applications become more widespread, it makes this problem even more pronounced. Image augmentation technologies are technical methods to effectively solve the problem of deep learning training under the condition of small-scale or low-quality training data. These technologies are continually accompanied with the development of deep learning and computer vision. This paper first reviews these image augmentation researches from the perspective of augmentation objects, operation spaces, label processing methods, and augmentation strategies and then concludes corresponding paradigms of current image data augmentation methods. After that, this paper proposes a taxonomy for current image data augmentation guided by the above paradigms, and reviews corresponding representative methods of each image data augmentation category. Finally, this paper makes conclusions on existing image data augmentation, points out the problems existing in the current image augmentation research and presents promising directions for future research.

Video quality assessment (VQA) is based on the subjective quality assessment results of the human eye, using models to evaluate distorted videos. It is difficult for traditional assessment methods to make subjective assessment results consistent with objective assessment results. The VQA methods based on deep learning can be evaluated through independent learning of the model without adding manual features. It is of great significance to video quality monitor and assessment, and has become one of the research hotspots in computer vision. First, this paper introduces the research background and main research methods of VQA. Second, it introduces the VQA methods based on deep learning from both the full-reference and no-reference ones, and uses the convolutional neural network model to classify and compare the no-reference evaluation methods. Then it introduces the related databases and performance evaluation indices of the VQA models and compares the algorithm performance. Finally, this paper summarizes the existing problems in the current VQA researches and looks forward to the challenges and development directions in this field.

In recent years, the popularity of depth sensors and 3D laserscanners has led to a rapid development of 3D point clouds processing methods. Semantic segmentation of point cloud, as a key step in understanding 3D scenes, has attracted extensive attention of researchers. With the rapid development of deep learning and its widespread applications in 3D semantic segmentation, the quality of point cloud semantic segmentation has been significantly improved. This paper mainly reviews the deep learning-based point cloud semantic segmentation methods and their research status. This paper categories these deep learning-based methods for point clouds into two groups: indirect and direct semantic segmentation methods. In terms of the characteristics of the algorithm, each of groups is further subdivided. The representative algorithms are analyzed and introduced. This paper summarizes the theories, principles, advantages and disadvantages of each type of method, and systematically explains the contribution of deep learning to the field of semantic segmentation. Moreover, the current mainstream datasets and remote sensing datasets are summarized and the experimental results of some algorithms are compared. Finally, the future development direction of semantic segmentation technology is prospected.

Transformer is a deep learning model based on the self-attention mechanism, showing tremendous potential in computer vision. In image classification tasks, the key challenge lies in efficiently and accurately capturing both local and global features of input images. Traditional approaches rely on convolutional neural networks to extract local features at the lower layers, expanding the receptive field through stacked convolutional layers to obtain global features. However, this strategy aggregates information over relatively short distances, making it difficult to model long-term dependencies. In contrast, the self-attention mechanism of Transformer directly compares features across all spatial positions, capturing long-range dependencies at both local and global levels and exhibiting stronger global modeling capabilities. Therefore, a thorough exploration of the challenges faced by Transformer in image classification tasks is crucial. Taking Vision Transformer as an example, this paper provides a detailed overview of the core principles and architecture of Transformer. It then focuses on image classification tasks, summarizing key issues and recent advancements in visual Transformer research related to performance enhancement, computational costs, and training optimization. Furthermore, applications of Transformer in specific domains such as medical imagery, remote sensing, and agricultural images are summarized, highlighting its versatility and generality. Finally, a comprehensive analysis of the research progress in visual Transformer for image classification is presented, offering insights into future directions for the development of visual Transformer.

Personalized news recommendation is an important technology to help users obtain the news information they are interested in and alleviate information overload. In recent years, with the development of information technology and society, personalized news recommendation has been increasingly widely studied, and has achieved remarkable success in improving the news reading experience of users. This paper aims to systematically summarize personalized news recommendation methods based on deep learning. Firstly, it introduces personalized news recommendation methods and analyzes their characteristics and influencing factors. Then, the overall framework of personalized news recommendation is given, and the personalized news recommendation methods based on deep learning are analyzed and summarized. On this basis, it focuses on personalized news recommendation methods based on graph structure learning, including user-news interaction graph, knowledge graph and social relationship graph. Finally, it analyzes the challenges of the current personalized news recommendation, discusses how to solve the problems of data sparsity, model interpretability, diversity of recommendation results and news privacy protection in personalized news recommendation system, and puts forward more specific and operable research ideas in the future research direction.

Epilepsy is a common neurological illness. By analyzing the electroencephalo-graph (EEG) signals of patients with epilepsy, their conditions can be monitored so that seizure can be detected and intervened in time. In epilepsy research, how to obtain effective features and construct an effective classifier is the key to epilepsy detection and recognition. In order to obtain better epilepsy detection effect, this paper proposes a multi-view seizure detection algorithm that combines deep and shallow features. The method first uses FFT (fast Fourier transform) and WPD (wavelet packet decomposition) to obtain the shallow features in the frequency domain and the time-frequency domain. Then, CNN (convolutional neural network) is used to obtain the deep features in the frequency domain and the time-frequency domain. Further the multi-view TSK fuzzy system is used to construct the classification model of shallow and deep features. Experimental studies show that in the detection of EEG signal epilepsy, the effects of shallow features and deep features proposed in this paper are more than 1% higher than those of commonly used feature extraction methods such as PCA (principal component analysis), LDA (linear discriminant analysis) and etc. The multi-view epileptic detection algorithm which combines deep features and shallow features has a classification effect higher than that of the single-view algorithm by more than 1%, and the average detection effect is more than 5% higher than that of the single-view algorithm.

Spoken language understanding (SLU) is an important part of human-machine dialogue system. Intent detection is an important sub-task in SLU, because it can expand the fields of dialogue in a limited field. Due to the increase in the demand for dialogue systems in practical application fields, and the new fields that need to be developed cannot obtain a large amount of data in a short time, it poses challenges for building deep learning models in new fields. Transfer learning is a special application of deep learning. In transfer learning, the source and target domains can be used to complete the construction of a target domain model with only a small amount of labeled data. The learning process is completed by transferring knowledge between the source and target domains. Based on labeled data and models in existing fields, building a new dialogue system with only a small amount of labeled data is a current research focus. This paper summarizes the task of intent detection, classifies and elaborates transfer learning methods, and summarizes their problems and solutions. This paper further thinks about how to apply transfer learning to the intent detection task, so as to promote a new field of intent detection research with a small amount of data.

Online singing platforms as a new type of entertainment attract huge number of users in recent years. On online singing platform, writing comments for published music is a way for users to share their music pieces and communicate with each other, which has an incentive effect on users to publish music. However, there is usually no comment for a new user??s music or a newly released music. Automatic generating comments for these music pieces can solve this problem to some extent. There is a certain relationship between the comments and the performance grade of the music on the online singing platform. Therefore, this paper studies how to automatically generate comments for published music considering its performance level information. To this end, this paper proposes a deep text generation model based on generative adversarial network, called GradeGAN. Major components of this model include generators, text discriminator and grade discriminators, where both text discriminator and grade discriminator are used to guide the generator to generate accurate text and make the generated text match their grades. Experiment conducted on real data sets shows that compared with existing models, the proposed model not only has higher accuracy, but also has high diversity in generating comments.

As one of the most challenging problems in data management, query processing and optimization are always widely concerned by researchers. However, it is very difficult for traditional techniques to meet the diverse requirements of modern database system due to the needs of hand-tuning for specific workloads and datasets. Inspired by advances in applying artificial intelligence (AI) to multi-field researches, recently, the AI powered new techniques for query processing and optimization have been proposed and made significant success. In view of these researches, this paper first presents the main tasks of the AI powered new techniques of query processing and optimization, and analyzes the differences between the new tasks and traditional AI tasks. Second, the recent research progress is reviewed, and the main advantages and application bottlenecks are summarized. Third, this paper discusses the main challenges of the AI powered new techniques of query processing and optimization. Finally, the future research directions are prospected.

SQL (structured query language) query generation from natural language is not only one of the most important parts of constructing intelligent database query system, but also one of the difficulties in the individualized operation and maintenance of hybrid temporal big data in the new power supply rail transit system. At present, the deep learning models almost focus on SQL query generation in a single table, but cannot solve multi-table SQL query generation in database. In order to solve this problem, this paper adopts a method named SQL sketch filling to transform the sequence generation problem into multiple classification problems. In the process of training the deep learning models, this paper makes full use of the dependencies of components in SQL clauses. In the generation of multi-table JOIN path of FROM clause, it is modeled as Steiner Tree problem and solved by a globally optimal algorithm. This method is validated on an open text-to-SQL dataset named Spider. The experimental results show that the model can improve the query-match accuracy of multi-table SQL query generation.

Aiming at the problem of detecting different sizes and types of ships in complex sea conditions, a ship detection method based on deep learning is proposed, which is mainly for the improvement of regional fully convolutional networks (R-FCN). Firstly, the ResNet50 network is selected for automatic extraction of features, and the feature map is automatically provided for the improved R-FCN. Secondly, the R-FCN is improved according to the characteristics of the ship identification, which allows the R-FCN to fully perform its performance on ship detection. Finally, according to the problem that the recognition rate of small ships in some categories is small, on the first step, the method of Maxpooling increases the recognition rate of small ships by 4.08 percentage points; on the second step, the improvement of ROIAlign makes the improved R-FCN in this paper perform much better on small target ship identification than original R-FCN, and the recognition rate is increased by 13 percentage points totally. This paper is also compared with the current mainstream target detection algorithms such as Faster-RCNN. Experimental results show that the method has higher recognition accuracy and the rate is basically the same as other methods.

With the continuous development of deep learning and image segmentation, image panoptic segmentation has become a research hotspot in the field of computer vision, and many image panoptic segmentation methods have been proposed. This paper summarizes the research methods of image panoptic segmentation based on deep learning. Firstly, the research status of image panoptic segmentation at home and abroad is introduced, and the existing image panoptic segmentation methods are classified according to different optimization tasks in the network architecture, mainly including image panoptic segmentation optimized by feature extraction, image panoptic segmentation optimized by sub-task segmentation, image panoptic segmentation optimized by sub-task fusion, and other image panoptic segmentation. Secondly, 5 commonly used datasets, i.e. MS COCO, PASCAL VOC, Cityscapes, ADE20K and Mapillary Vistas, and 2 evaluation criteria, i.e. panoptic quality (PQ) and parsing covering (PC) in image panoptic segmentation are briefly introduced. And then, performance comparison of typical image panoptic segmentation methods has been conducted on different datasets. Thirdly, the application of image panoptic segmentation in medicine, autonomous driving, drones, agriculture, animal husbandry, military and other fields are listed. Finally, the deficiencies and challenges of existing methods in complex scene applications, real-time performance, and conflicts are pointed out, and the potential directions of image panoptic segmentation are discussed, including image panoptic segmentation based on a simple unified framework, real-time high-quality image panoptic segmentation, and image panoptic segmentation in complex application scenarios.

Multi-label classification refers to the classification problem where multiple labels may coexist in a single sample. It has been widely applied in fields such as text classification, image classification, music and video classification. Unlike traditional single-label classification problems, multi-label classification problems become more complex due to the possible correlation or dependence among labels. In recent years, with the rapid development of deep learning technology, many multi-label classification methods combined with deep learning have gradually become a research hotspot. Therefore, this paper summarizes the multi-label classification methods from the traditional and deep learning-based perspectives, and analyzes the key ideas, representative models, and advantages and disadvantages of each method. In traditional multi-label classification methods, problem transformation methods and algorithm adaptation methods are introduced. In deep learning-based multi-label classification methods, the latest multi-label classification methods based on Transformer are reviewed particularly, which have become one of the mainstream methods to solve multi-label classification problems. Additionally, various multi-label classification datasets from different domains are introduced, and 15 evaluation metrics for multi-label classification are briefly analyzed. Finally, future work is discussed from the perspectives of multi-modal data multi-label classification, prompt learning-based multi-label classification, and imbalanced data multi-label classification, in order to further promote the development and application of multi-label classification.

As one of the most important semantic segmentation frameworks in convolutional neural networks (CNN), UNet is widely used in image processing tasks such as classification, segmentation, and target detection of medical images. In this paper, the structural principles of UNet are described, and a comprehensive review of UNet-based networks and variant models is presented. The model algorithms are fully investigated from several perspectives, and an attempt is made to establish an evolutionary pattern among the models. Firstly, the UNet variant models are categorized according to the seven medical imaging systems they are applied to, and the algorithms with similar core composition are compared and described. Secondly, the principles, strengths and weaknesses, and applicable scenarios of each model are analyzed. Thirdly, the main UNet variant networks are summarized in terms of structural principles, core composition, datasets, and evaluation metrics. Finally, the inherent shortcomings and solutions of the UNet network structure are objectively described in light of the latest advances in deep learning, providing directions for continued improvement in the future. At the same time, other technological evolutions and application scenarios that can be combined with UNet are detailed, and the future development trend of UNet-based variant networks is further envisaged.